No bugs

Speeding ROBLOX Development with Continuous Testing

No bugsWe’ve been talking a lot lately about the new features and engine improvements coming to ROBLOX. But behind the development is a relatively unseen – and extensive – process that ensures the changes don’t break the game for players: Testing.

ROBLOX undergoes constant, rigorous, automated tests. Every time a developer “checks in” a new line of code, our testing system creates a trial version of ROBLOX and runs it through a battery of automated tests. This not only ensures the new code works, but also that it didn’t inadvertently introduce new bugs somewhere else.

Testing: All day, all night

We run full test suites on ROBLOX dozens of times each day. Each test suite consists of hundreds of tests – and this number is growing daily – for multiple platforms. Do the math and that means we run tens of thousands of tests per day, with each test covering hundreds of thousands of lines of code.

ROBLOX testing activity chart
Each colored block represents activity by ROBLOX’s various testing systems.
As you can see, our PC and Mac testing machines are active throughout the day. 

Testing: Any type, anywhere

Our testing framework consists of three varieties of tests: manual, unit and automated. Considering the size of ROBLOX, each one is necessary for us to release regular, quality updates.

Manual tests
Manual testing, in the most basic sense, consists of jumping into a ROBLOX game and seeing whether a specific behavior works. ROBLOX is too complex for us to test it in this Manual test descriptionway, so we’ve built up a library of testing scripts, written with very specific steps, that we send to a dedicated testing house. Using their extensive resources, they run the tests – usually overnight – and return detailed diagnostics the next day.

For example, we wrote a script that tests whether rockets function as they should. If a code change causes them to, say, fall to the ground, the test fails and we know there’s a regression that needs fixing. We have a lot of these scripts, covering a spectrum of ROBLOX behaviors.

Unit tests
Unit testing refers to ROBLOX running tests on itself. If ROBLOX were a car, its unit tests would be checking its own oil level, transmission temperature, etc., and reporting problems – like being low on oil. We have a bank of machines that almost always have ROBLOX running tests on itself because the tests fire up each time a developer checks in a line of code.

To write unit tests, we use the boost::test framework. Here’s an example of a test that makes sure our raycasting code works:

Test code sample

Automated tests
Automated testing determines whether a specific behavior happens as it should. The tests actually open instances of ROBLOX and run Lua scripts – the same language ROBLOX players use to create games – to determine whether the behavior happened as desired. For instance, an automated test might open ROBLOX and run for 30 seconds to ensure a ball continues bouncing until 30 seconds have passed (as opposed to bouncing away, falling through the floor or some other buggy behavior).

Testing: You can, too

To make writing automated tests easy, we extended the boost::test framework into our Lua environment. If you’re an advanced ROBLOX Studio user, you might have noticed a new TestService feature. This lets you write sophisticated tests right inside a game. All scripts inside the TestService are executed when a test is run, but not during normal gameplay.

Here’s a script we run to make sure the BindableEvent feature is working:

Test code sample

Let’s say there was a bug in BindableEvent. In that case, when we run the test we might see the following output:

Test code sample

What good does all this testing do you? Continuously testing ROBLOX is allowing us develop new functionality more quickly, and with less risk of causing regressions in the process. This ultimately means a better gaming experience and more consistent releases for our users.

About Erik Cassel

Co-Founder & Chief Scientist of ROBLOX. @CasselErik on Twitter

104 thoughts on “Speeding ROBLOX Development with Continuous Testing

  1. GLaDOS

    Testing? Oh, yes. I love testing! Let’s just hope that you don’t die like last time. Maybe I’ll even let you keep the Portal Gun if you succeed.

  2. cadelaser77

    ive had problems with the connenction too. the internet status on my pc taskbar is 5 bars and i just got a new router and its in the same room as my computer. i can connect to a game, but after a while of playing, the game freezes and it says the connection was lost. please fix it as soon as possible.

    1. materking

      Same. roblox been having toubles loading but other gaming sites load fast no lag. idk if it my computer router. or if roblox have a bug

    2. Cobra Ball

      I have had this issue to and all I can say so far about ideas to resolve it is to try disconnecting and then reconnecting to the router. Doing this once I went from a download rate of about 60 KBs/sec to over 1 MB/sec.

  3. Zxczxczbfgman

    Yep, that’s Portal all right. Are you sure Shedletsky/Telamon isn’t just a fake account run by GlaDOS?

  4. rockyo11

    I know having the same trouble here ever since the latest update I’m losing connection all the time!

  5. HintCore

    I just scrolled through the page and I kept seeing the word “test,” so I immediately thought of Portal.

  6. LikeABAUS

    Hah, I bet if you made BC packages TBC, BC would be louder than NBC. And, there would be less ODing.

  7. Memzi

    Mannnnn i hate the new updates.
    I haven’t been able to get on a game ever since.
    ALways losing connection while Loading the game up.
    I even missed the egg hunt :(

  8. GenericBloxxer's Urgent Question!

    Hello there,

    I’ve lost all access to my older accounts created in 08 today. At the time when these were created, roblox passwords only required 2 number character, as opposed to 2 for current accounts.

    It appears that all of these accounts containing 1 number passwords have been suspended from access, as I can no longer use even one of my 3 old accounts. Please help!

  9. n00b

    I don’t know WHAT you did, but since the latest roblox update, ive lost connection 10x as much

    1. coulson8880

      nothing is going wrong with roblox every thing is fine I havent had any problems this update that roblox is doigng is going to be great I hope it works

  10. kozumi

    glad again that yur working on game play. i noticed intervals of my screen freezing when i was building on a place with only a plate and a basic humanoid i was working on. why?

  11. Avery Walker

    Epic! :3. I agree. Above, Well.. Sorta.. But Yeah, We need less Od-ers, Less Hack, Less Ban.

  12. jimmysnow02

    Anyone notice the video ads and game pics are realeased? :D. The bad part is they are 1k robux. D:.

    1. peacelove2002

      I know right! Cant they be 500! Oh and TheAmazeMan did the vid on his titanic game.

  13. gigabicgigabic

    Great work on tsting and fixing new stuff, but maybe you should…. Idk, MAKE ROBLOX MORE SECURE FROM HACKERS ( Also known as exploiters but people apparently use hackers as the correct word )

  14. warninja97

    Awesome job I can’t wait to not have all those bugs and while your at it could you make a *Cough* Trading *Cough* System? Troll =3

  15. rsfretgas

    Man I Wish I Could Build My Place With Free Models Without Computer Freeze. Btw Thats Pretty Cool Roblox.

    1. Meh is cooleh

      I totally agree. Whenever I try using a free model that is big it always freezes my computer.

  16. Nooby

    Well, we’ve seen Universes, high scalability water, performance updates, video clips, ropes and new physics, but when is it going to be added?

  17. Eminem117796

    Im really excited for the upcoming features & adjustments that are being made for ROBLOX. I honestly have to say this, ROBLOX has surely gotten better over the years mostly this year [2012] without a doubt! Keep up the AMAZING work ROBLOX Staff…ROBLOX just keeps getting better & better :)

  18. VIPdude007

    Just type smaller code, seriously your putting code in that wrecks your project.

    Simply make it simpler.

    I know this sounds dumb but the more deeper into code you get the worse it gets.

    1. patrizl001

      actually kind of true. the more scripts needed to make the creation work means some scripts cant respond properly with all the other scripts

  19. tweetytime

    I still believe that Roblox should realease their updates sooner and at least not tell us when its in development. Cause what we dont know wont hurt us. Like if you are making a trade system and it is just being thought of dont get us all excited over lost hopes D:* , and i still belive you sould hire me :D Tweetytime :D

  20. RRuS

    Hey GLaDoS Glad to see you having your minions test vigorously day and night for years to come.

  21. Nameless

    I don’t entirely see the truth in this post. I do appreciate the effort that you guys put into testing everything, however with each update, there’s always something, mostly scripting wise that breaks. That is truly frustrating.

  22. Abaw7

    Wish you guys did this on the last update. You guys broke startergui so it wouldn’t render gui’s.

  23. thedestroyer115

    There is a major problem, the guis do not show up in edit mode but show up in the Test/PlaySolo and in regular games. Fix this please!

    1. GreenDayDude75

      That’s been doing that since Studio Mode (What you now call Edit Mode) was created. GUI’s are for the player. On Studio Mode, you are not a player (Technically, there is no player you control) during that time. If you want to work with GUI, I suggest opening ROBLOX Studio and use Solo Mode (Build Mode today) and use the tools as a player.

    2. coolboy10000

      There is no “bug” with edit mode. Obviously what is happening in edit mode is it’s showing the GUI before any scripts are run. So if your GUI is off the screen before a script changes its position you wont see it until you hit the “play” button in tools.

  24. LordDefiant

    Sounds sweet, I can’t wait to see what’s in-store for ROBLOX. *cough* tradingsystem *cough*

  25. Pieguardian

    That’s pretty cool. You guys amaze me. Updates have been getting better. No more ones like bodies.

  26. m2mm4m

    Does that mean I can now remove my DEBUG.Lua file from my Rbx games? ( I hope so.)
    I would be nice to have some more debugging and loging tools those…

Comments are closed.