In the first blog in this series, I talked about test speed, and why it shouldn’t be your first priority. Rather, you should focus on getting accurate test results back consistently. This week, I’ll discuss the power of parallelization in testing.
By far, the best way to reduce overall execution time for a test suite is to do a better job running tests in parallel. That means running more tests of approximately equal length at the same time. Even if your tests run five times slower on a remote platform, you’ll get your results four times faster if you can run 20 tests at a time instead of everything in one session like you would locally. None of the other performance-enhancing options will provide nearly as much benefit as improving parallel testing.
There are three test attributes that will allow you to improve parallel testing:
Atomic, meaning that each test will test only one thing
Autonomous, so that any test can be run in isolation, at the same time or in any order, with another test
Short, where each test should be of similar length (ideally under two minutes)
Sauce Labs did an evaluation of all the tests run on our platform and determined that tests that run in under two minutes are twice as likely to pass as tests that run in more than seven minutes. Based on this, we believe that teams adopting an approach that facilitates shorter tests are more likely to get accurate results.
Also, you get the most efficiency by keeping all of your tests the same length. Typically our users run multiple sequential builds on the Sauce Labs platform. If a particular build has even one long-running test, it can tie up all of the parallel sessions for the length of time it takes to finish executing. This would increase the overall time it takes for your test suite to finish.
To get results back faster, run more tests in parallel by writing them to be atomic, autonomous and short. In this short video, I review these test attributes and show you an application that I created to illustrate them.
Next, I will show you how to make your tests sufficiently independent with effective test data management.
4 Data Management Approaches for Parallel Testing
There are four different approaches to data management that you can use for parallel testing. Let’s do a quick overview of each one.
1. Grab & hope
The first is what I like to refer to as Grab and Hope. This is a frequently deployed approach where a test needs an object where the specifics don’t matter (perhaps a product to add to the cart), so it locates the first one it finds and uses it and you have to just hope that it’s in the state you need it to be in. This works great until you run enough tests that the back end runs out of stock and your tests suddenly start failing. Also if one test grabs something to edit and another test grabs the same thing to delete, only one of those tests is going to pass.
2. Static fixture
This is the traditional QA staging or testing environment that has a bunch of pre-populated data and your tests have identified specific users and specific objects and have associated them with specific tests. For example: having users with different attributes assigned to specific tests, and some tests always use this object, and other tests always use this other user with that other object. The defining characteristic here is that each piece of data is hard coded in your testing code, often (but not always) in spreadsheets. Obviously this is difficult to maintain, and something as small as “run all these tests but with a user that has this other configuration” becomes difficult to manage. It is a very bad day when you come into work and find out that someone deleted the staging environment database and you have to recreate all of the objects from scratch!
3. Dynamic fixture
The third option is what developers are used to doing for unit tests with SQL code that populates a testing database immediately before the test suite is run. This prevents the problem of someone deleting your test data, but it has the disadvantage of still needing to maintain a one to one relationship between tests and the data they are going to use. Also this is often a type of “grey-box” solution, because, depending on how the data injection is managed, the data ends up in your system in a state that does not accurately represent what would actually be in the database if a user entered that data via the DOM.
4. Just in Time
The fourth and best option is what I call Just In Time. The idea behind Just In Time is that the only way to truly ensure autonomous usage of test data is for every test to have complete control over all of the data it uses. Each test creates and authenticates a brand new user, and each test creates any data it needs to manipulate to evaluate the specific functionality it is testing for. You might be surprised how often you see flakiness in your tests because two tests happen to be running at the same time that are inadvertently using the same data object in different ways, or one test manipulates data required for another test in an unexpected manner. Often you’ll see this when a test fails and leaves the data in an incorrect state for a subsequent test.
I can hear you wondering, “But Titus, won’t it increase overall test time if each test has to also set up its own data?” Remember, though, that you can’t have a successful strategy of atomic and short tests without those tests also being autonomous. This is the only way to reliably increase parallelization, which will more than make up for any increased overhead of setting up data at the start of each test.
I hope you found this helpful. Parallelization and effective data management will go far in helping you increase test speed.
