Speeding ROBLOX Development with Continuous Testing

By -

No bugsWe’ve been talking a lot lately about the new features and engine improvements coming to ROBLOX. But behind the development is a relatively unseen – and extensive – process that ensures the changes don’t break the game for players: Testing.

ROBLOX undergoes constant, rigorous, automated tests. Every time a developer “checks in” a new line of code, our testing system creates a trial version of ROBLOX and runs it through a battery of automated tests. This not only ensures the new code works, but also that it didn’t inadvertently introduce new bugs somewhere else.

Testing: All day, all night

We run full test suites on ROBLOX dozens of times each day. Each test suite consists of hundreds of tests – and this number is growing daily – for multiple platforms. Do the math and that means we run tens of thousands of tests per day, with each test covering hundreds of thousands of lines of code.

ROBLOX testing activity chart
Each colored block represents activity by ROBLOX’s various testing systems.
As you can see, our PC and Mac testing machines are active throughout the day. 

Testing: Any type, anywhere

Our testing framework consists of three varieties of tests: manual, unit and automated. Considering the size of ROBLOX, each one is necessary for us to release regular, quality updates.

Manual tests
Manual testing, in the most basic sense, consists of jumping into a ROBLOX game and seeing whether a specific behavior works. ROBLOX is too complex for us to test it in this Manual test descriptionway, so we’ve built up a library of testing scripts, written with very specific steps, that we send to a dedicated testing house. Using their extensive resources, they run the tests – usually overnight – and return detailed diagnostics the next day.

For example, we wrote a script that tests whether rockets function as they should. If a code change causes them to, say, fall to the ground, the test fails and we know there’s a regression that needs fixing. We have a lot of these scripts, covering a spectrum of ROBLOX behaviors.

Unit tests
Unit testing refers to ROBLOX running tests on itself. If ROBLOX were a car, its unit tests would be checking its own oil level, transmission temperature, etc., and reporting problems – like being low on oil. We have a bank of machines that almost always have ROBLOX running tests on itself because the tests fire up each time a developer checks in a line of code.

To write unit tests, we use the boost::test framework. Here’s an example of a test that makes sure our raycasting code works:

Test code sample

Automated tests
Automated testing determines whether a specific behavior happens as it should. The tests actually open instances of ROBLOX and run Lua scripts – the same language ROBLOX players use to create games – to determine whether the behavior happened as desired. For instance, an automated test might open ROBLOX and run for 30 seconds to ensure a ball continues bouncing until 30 seconds have passed (as opposed to bouncing away, falling through the floor or some other buggy behavior).

Testing: You can, too

To make writing automated tests easy, we extended the boost::test framework into our Lua environment. If you’re an advanced ROBLOX Studio user, you might have noticed a new TestService feature. This lets you write sophisticated tests right inside a game. All scripts inside the TestService are executed when a test is run, but not during normal gameplay.

Here’s a script we run to make sure the BindableEvent feature is working:

Test code sample

Let’s say there was a bug in BindableEvent. In that case, when we run the test we might see the following output:

Test code sample

What good does all this testing do you? Continuously testing ROBLOX is allowing us develop new functionality more quickly, and with less risk of causing regressions in the process. This ultimately means a better gaming experience and more consistent releases for our users.


104 thoughts on “Speeding ROBLOX Development with Continuous Testing”

  4. Testing? Oh, yes. I love testing! Let’s just hope that you don’t die like last time. Maybe I’ll even let you keep the Portal Gun if you succeed.

  5. ive had problems with the connenction too. the internet status on my pc taskbar is 5 bars and i just got a new router and its in the same room as my computer. i can connect to a game, but after a while of playing, the game freezes and it says the connection was lost. please fix it as soon as possible.

    • Same. roblox been having toubles loading but other gaming sites load fast no lag. idk if it my computer router. or if roblox have a bug

    • I have had this issue to and all I can say so far about ideas to resolve it is to try disconnecting and then reconnecting to the router. Doing this once I went from a download rate of about 60 KBs/sec to over 1 MB/sec.

  6. Yep, that’s Portal all right. Are you sure Shedletsky/Telamon isn’t just a fake account run by GlaDOS?

  7. I know having the same trouble here ever since the latest update I’m losing connection all the time!

  10. Ermm… I can’t log in to my account and my sister can’t eather… Whats wrong?

  11. I just scrolled through the page and I kept seeing the word “test,” so I immediately thought of Portal.

  12. Hah, I bet if you made BC packages TBC, BC would be louder than NBC. And, there would be less ODing.

  13. Mannnnn i hate the new updates.
    I haven’t been able to get on a game ever since.
    ALways losing connection while Loading the game up.
    I even missed the egg hunt :(

  14. Hello there,

    I’ve lost all access to my older accounts created in 08 today. At the time when these were created, roblox passwords only required 2 number character, as opposed to 2 for current accounts.

    It appears that all of these accounts containing 1 number passwords have been suspended from access, as I can no longer use even one of my 3 old accounts. Please help!

  16. I don’t know WHAT you did, but since the latest roblox update, ive lost connection 10x as much

  17. glad again that yur working on game play. i noticed intervals of my screen freezing when i was building on a place with only a plate and a basic humanoid i was working on. why?

  18. Epic! :3. I agree. Above, Well.. Sorta.. But Yeah, We need less Od-ers, Less Hack, Less Ban.

  20. Anyone notice the video ads and game pics are realeased? :D. The bad part is they are 1k robux. D:.

  22. Great work on tsting and fixing new stuff, but maybe you should…. Idk, MAKE ROBLOX MORE SECURE FROM HACKERS ( Also known as exploiters but people apparently use hackers as the correct word )

  24. This is cool, haven’t noticed it yet… but won’t this bring up even more exploits?

  25. Awesome job I can’t wait to not have all those bugs and while your at it could you make a *Cough* Trading *Cough* System? Troll =3

  28. Man I Wish I Could Build My Place With Free Models Without Computer Freeze. Btw Thats Pretty Cool Roblox.

    • I totally agree. Whenever I try using a free model that is big it always freezes my computer.

  32. Well, we’ve seen Universes, high scalability water, performance updates, video clips, ropes and new physics, but when is it going to be added?

  35. Im really excited for the upcoming features & adjustments that are being made for ROBLOX. I honestly have to say this, ROBLOX has surely gotten better over the years mostly this year [2012] without a doubt! Keep up the AMAZING work ROBLOX Staff…ROBLOX just keeps getting better & better :)

  36. So that’s what the TestService is for? I’ve been wondering for some time…

  40. Just type smaller code, seriously your putting code in that wrecks your project.

    Simply make it simpler.

    I know this sounds dumb but the more deeper into code you get the worse it gets.

    • actually kind of true. the more scripts needed to make the creation work means some scripts cant respond properly with all the other scripts

  42. I still believe that Roblox should realease their updates sooner and at least not tell us when its in development. Cause what we dont know wont hurt us. Like if you are making a trade system and it is just being thought of dont get us all excited over lost hopes D:* , and i still belive you sould hire me :D Tweetytime :D

  43. Hey GLaDoS Glad to see you having your minions test vigorously day and night for years to come.

  45. I don’t entirely see the truth in this post. I do appreciate the effort that you guys put into testing everything, however with each update, there’s always something, mostly scripting wise that breaks. That is truly frustrating.

  46. Wish you guys did this on the last update. You guys broke startergui so it wouldn’t render gui’s.

  47. There is a major problem, the guis do not show up in edit mode but show up in the Test/PlaySolo and in regular games. Fix this please!

    • That’s been doing that since Studio Mode (What you now call Edit Mode) was created. GUI’s are for the player. On Studio Mode, you are not a player (Technically, there is no player you control) during that time. If you want to work with GUI, I suggest opening ROBLOX Studio and use Solo Mode (Build Mode today) and use the tools as a player.

    • There is no “bug” with edit mode. Obviously what is happening in edit mode is it’s showing the GUI before any scripts are run. So if your GUI is off the screen before a script changes its position you wont see it until you hit the “play” button in tools.

  49. Sounds sweet, I can’t wait to see what’s in-store for ROBLOX. *cough* tradingsystem *cough*

  51. That’s pretty cool. You guys amaze me. Updates have been getting better. No more ones like bodies.

  54. Does that mean I can now remove my DEBUG.Lua file from my Rbx games? ( I hope so.)
    I would be nice to have some more debugging and loging tools those…

  57. Keep “Killing those bugs”

    Hope to see some awesome new featuers soon

    ;)