So You Want to Run Tests in Parallel... Now What??

This is a guest blog post by Sarah Goff-Dupont, Product Marketing Manager for Atlassian Bamboo. 

Fast feedback on code changes has been a bit of an obsession ever since the Agile2012 conference got me thinking about how important this is for agile teams in particular--partly because turbo-charged feedback is a natural work-in-progress limiter. How so? Getting results back to devs so quickly that they don't have time to start in on another issue keeps everyone focused on the task at hand. That, in turn, fosters an efficient development process. And efficiency, in my never-humble opinion, is what agile development is all about.

Running automated tests in parallel is a great way to reduce build times and get results to developers and QA engineers faster. Today I'm going to dig back into my long-lost career as a test automation engineer and share with you the technique we used for running tests in parallel. 'Cuz it was super-slick and I loved it. I'll use Atlassian Bamboo (which integrates nicely with Sauce Labs, btw) as my example CI tool as we walk through it.

If you've never used TestNG before, you might think that's a typo above and that I meant "TestING". (Google's autocomplete certainly thinks it's a typo, so hey: you're in good company.) But no. TestNG is an automated testing utility, much like JUnit. In fact, you can use it in combination with JUnit, WebDriver, Selenium, etc. And the reason you'd want to is Groups. By sorting your test suite into groups, then creating separate Jobs in Bamboo to run each group, you can speed up your testing stages by 100% --and likely, better. TestNG is designed for Java tests, but can also be used for Ruby projects. And it's not the only framework to offer groups. Proboscis for Python and PHPUnit also have them. JUnit has "categories", which largely serve the same purpose, but are rather clunky and make TestNG's groups look like a bacon-wrapped Ferrari by comparison (sleek, fast, and yummy).

Like a handful of other testing frameworks, TestNG takes it's cues from an @Test annotation. Inside that, you can place a number of attributes, one of which is Groups. Group names are both dynamic and arbitrary --small sentence, big implications. First, you can create a new group on the fly just by assigning a test to it. No need to declare and define the group in a separate file. The act of assigning a test to a group will also create that group, if it doesn't already exist. (Now, that's a double-edged sword because it also means you can end up with a group called "smoke_test" and another called "smoke_tets" if you're careless.) Second, you can devise whatever system (or systems) of meaning you like for your groups. Slice them along functional area of the application, by level of the code stack, by whether you currently expect the test to pass, etc. And tests can belong to multiple groups. So a single test might conceivably belong to the "user_authentication", "API", "smoke_test" and "in_development" groups. TestNG even supports groups of groups. I haven't played with that personally, but it's pretty bad-ass and their documentation on it is helpful.

Of course, if your application's super-structure doesn't know about TestNG, you'll get all sorts of errors and your builder won't know what do with all those groups at runtime. So be sure to declare TestNG as a dependency for your project or module.

Back in Bamboo, set up Jobs based on the groups you've defined in your test code. If you're executing tests via Maven, Gradle or Ant, this is pretty trivial. In the example shown here, I pass a group name into Maven as an argument. When it's time to run the tests, TestNG will scour my project, pull out all the tests belonging to that group, and run only those. Doesn't matter if the tests for that group are located in different classes or packages. As long as they're in the same project or module, TestNG will find them.

Note that because each Job has it's own working build directory, you'll need to pull the test code into each Job. A source code checkout Task is generally the simplest way to do that. If your code base is large, you can take advantage of the option to check out from a sub-directory in your repo and only pull down that part of the tree. Another option is pulling down all the test code in the very first Stage of your Plan, then zipping it up and passing that as an artifact to downstream Jobs.

It's probably obvious, but splitting your test suite into 10 batches isn't going to help you much if you only have 1 or 2 build agents. Of course we'd love if everyone bought a Bamboo license that supports lots of agents so they can really take advantage of parallelization. Maybe that's feasible for you, and maybe not. Either way, be mindful of your agent count when considering how to split up your test suite so you're not just spinning your wheels. You can take this idea of parallelized tests about as far as you want. There's no restriction as far as TestNG or Bamboo is concerned. Many of our own CI builds have over 20 test groups running simultaneously. Again: you're limited only by the number of build agents you're running.

As with most things in build engineering, there are several ways to accomplish parallelized testing. Tinkering with custom scripts or your suite.xml are two other common approaches. For big multi-module projects, it might make sense to split the suites/groups into separate repositories and run a Job for each repo. Regardless of which method you chose, you'll get the most bang for your buck by focusing your parallelization efforts on integration-, API- and UI-level tests. Unit-level tests already run quite fast, so they're unlikely to be the bottleneck in your builds. (Extra credit for the over-achievers: many testing frameworks allow you to run unit tests multi-threaded!) If you're new to test parallelization, congratulations on making it all the way through this blog! Check out our new automated testing resources page  for more helpful tips & tricks.

----- Sarah Goff-Dupont is a former test automation engineer, and a fan of anything that makes life easier for the nerds. Like Continuous Integration and automation, for example. Because manual testing gets boring. Her heros include Margret Thatcher, MacGyver, and her mom.