ROBLOX’s Encounter with a One-in-Four-Billion .NET Bug

By -

ROBLOX Home Page

On July 28, 2012, user Yaman100 did something thousands of users do without a second thought every day: put a new t-shirt on his ROBLOX character. Only, this time it was special. It set off a bug, with approximately one-in-four-billion chances of occurring, that would end up breaking any ROBLOX page on which Yaman100′s character thumbnail appeared.

An innocent action, with such far-reaching effects.

The bug came to light via Spetsnaz, a group whose page on ROBLOX simply wouldn’t load. It was an isolated incident – there were no problems with other ROBLOX groups – so we dug further to find the bug was related to a particular group rank. A little more digging, and we found a single user, Yaman100, was unknowingly the cause of the bug; the group page wouldn’t load due to a post by Yaman100 on its wall.

What happened?

Understanding the bug requires a look at how ROBLOX stores and fetches user data.

Every ROBLOX user has a unique record. Each article of clothing he or she is wearing – in Yaman100’s case, a t-shirt – has a unique record. There’s also a unique record for a specific article of clothing on a specific user. We store all of these records as rows in a table in our database.

If this was all we did, fetching data would be painstakingly slow. To improve performance, we add recent requests to the database to our cache. We look up the records in the cache by assigning each one a unique string key, like “Wearing_67583028″, where the number indicates the unique record. This makes the process of fetching and displaying data on Roblox.com much faster.

Roblox.com runs on Microsoft’s ASP.NET framework. The ASP.NET Web Cache takes those string keys and uses a hash function to turn them into 32-bit numbers. Looking up the number is even faster than searching for the string.

32-bit numbers range from -2,147,483,648 to 2,147,483,647. Do the math and that’s a range of about 4.29 billion numbers.32-bit numbers range from -2,147,483,648 to 2,147,483,647. Do the math and that’s a range of about 4.29 billion numbers. As you can see, the smallest 32-bit number you can have is the opposite of the largest and one smaller.

When storing data in the cache, ASP.NET uses the absolute value of the hash. In the rare instance that a record’s string key is hashed down to -2,147,483,648, ASP.NET chokes because it pulls the absolute (positive) value, which is one over the largest 32-bit number — or 2,147,483,648 (231 + 1).

The string key for Yaman100’s character wearing a specific t-shirt hashed to the infamous, smallest-possible 32-bit number. Wherever his auto-generated character thumbnail appeared on Roblox.com, the pages threw an error.

How did we fix it?

The fix was actually less interesting than the circumstances of the bug. We left Yaman100’s account as it was for a short while so we had a test case. We implemented a quick check to determine, going forward, whether any hashed numbers are set to the smallest-possible 32-bit number. If so, we change them to something that’s… Well, not the smallest-possible 32-bit number.

Lightning BoltYaman100 is the only ROBLOX user who has caused – and ever will cause – this bug to rear its face on ROBLOX. He inadvertently helped us squash an ASP.NET bug with slim chances of occurring and gave us an interesting story. For that, he goes down as a small part of ROBLOX history.

Likelihood

To put this all in perspective, before our fix, you were more likely to get struck by lightning (one in a million in a given year) or experience a plane crash (one in 11 million) than cause this bug yourself. But ROBLOX deals with such a wealth of data that the bug, despite its rarity, was statistically certain to show up one day.

134 thoughts on “ROBLOX’s Encounter with a One-in-Four-Billion .NET Bug”

  2. Yaman100 shall be famous now! Well, thanks to him the ROBLOX team killed the bugs!

  7. The same thing happened to me today when the roblox website was acting up. Thank you for fixing the problem instaly roblox.

  8. Wow. This has shown ROBLOX is really big to cause a very rare bug. I’m in awe right now.

    • Nothing will happen at all.
      It’s not a bad glitch either. You must not have read.

      It was the web page that was buggy because of the bit value in the 32-bit number. NO computer handles the number, at least not a Microsoft computer. The pages weren’t loading because the web was trying to fetch the number from the cache and it was causing it not to load right. It’s like not being able to play a Flash game because you don’t have the right version of Flash.
      Anyway, they manually changed that number and everything went back to normal.

  11. I was dressing my character once and I put a item on and my character appeared as some random person’s character.

    • That EXACT same thing happend to me a month ago!!!!! [or two]. I even reloaded the page, redrew my character, and it didnt stop… I even looked like them in-game.!!! :3

  15. AWESOME. Read the “Likelihood” Section. It’s freaky. Yaman’s still alive. (I think)

  16. I was a part of spetsnaz when this happened. right when i joined this bug had only started recently.

  17. Wow, that is some odds. One in four BILLION, when you think about that it’s nearly impossible. Just think for a long time, you’ll see how impossible that was.

  18. This is interesting.
    Something else interesting: I have a bug to report. Whenever there’s a pie, I go in the kitchen to get a piece, and it disappears.
    XD

  20. Lol, interesting. Wonder what other bugs will come up in the future. I also wonder when the “ERROR: Could not read CAPTCHA token file.” error will be fixed. It is getting in the way two out of three times when I try to post something on this site.

  23. Hey, he just won a lottery.
    You should give him a million ROBUX, because statistics and stuff

  24. So, all this is what happens when we put on a simple T-Shirt. Also to think that this is a small portion of Modern Technology today… Mind-Blowing.

  28. Whoa, this is something people can look up to in ROBLOX history as a great discovery.
    Congrats Yaman100

  30. Wow, that is cool. I wonder how long the bug existed. Did it exist since ROBLOX first launched, and it was only a matter of time before this happened?

  32. i once had this shirt that when i wore it my skin turned white and my gear while playing a game bugged with dropping and using it, and caused MAJOR lag for me in my game and on the site.

  34. What are the chances? Oh yeah, one in 4,290,000,000.

    How many Roblox Players are there?

    How many items on Roblox are there?

    Both of these could change the odds.

  35. I have a glitch i want to report. If you click “catalog” a random search was put in. Once it was Green Bay Packers Jersy. Has anyone else experienced this?

  36. I can’t play games on ROBLOX anymore. I’m wondering if this is a bug?
    Its pretty awesome how one Robloxian caused a bug!
    But its fixed now.
    Anways, could you please tell me whats wrong with my character or if its just some weird bug that stops me playing games?
    Thanks!

    • What system does your computer have? I know I couldn’t play games with linux(I think that was it’s name). And you may want to try reloading the page after you hit play and can’t play.

  37. Yeah and you also know what the insane thing is?
    I didnt even know that i caused it O_O
    Until this Morning