The chief dilemmas of native mobile app testing include the lack of parallelism, the cost of concurrency with real device hardware, and the challenge of making tests efficient.
In this article, I’d like to discuss a solution that can help overcome each of these problems: Running your Google Espresso tests on every pull request, with faster feedback using emulators. While this technique should not be the be-all, end-all of your testing strategy, it can help to meet your speed goals while still providing broad coverage and accurate feedback.
The Problem: Slow Test Results
For the purposes of this article, I’ll avoid the emulator-vs.-real devices debate. I want instead to focus on when to choose emulators over real devices. For this article, we’ll assume that we are practicing shift-left testing by running Google Espresso tests on every pull request, but those tests are running sequentially on a real device.
Let's break down the problem and discuss the possible solution. Your Espresso smoke test suite is triggered on every pull request, which contains 25 tests. Those tests run sequentially on a real device.
Why doesn't this continuous testing strategy for native app testing cut it? Because it will typically take 20-25 minutes to complete the smoke tests, and your team has strict service-level agreements (SLAs) around pull-request execution timing, so the timing is not acceptable.
We therefore need a solution to run each test in parallel on separate devices in order to reduce the execution time to less than 3-5 minutes—which is a significant amount of time saved, and translates directly into cost savings by finding bugs early in the lifecycle.
In a perfect world, we would achieve this by having unlimited access to real devices at the same cost as virtual machines. But let's be realistic—access to a real-device farm supporting 25-50 parallel tests would be quite expensive.
So, let’s talk about a more realistic, cost-effective solution: using Sauce Labs’ virtual devices (emulators) to execute Espresso tests in parallel, at a fraction of the cost of real devices.
The Solution: Parallel Espresso Testing
The emulator is an application that emulates real mobile device software, hardware, networking, gestures, and operating systems, allowing us to test and debug our native apps on every pull request with maximum speed. Sauce Labs’ virtual device emulator infrastructure will help you because it supports the Google Espresso native testing framework. It enables Android developers and quality engineers to run tests in parallel to deliver fast feedback from local development or a continuous integration pipeline.
That’s a big deal because most mobile native testing frameworks lack parallelism techniques like Selenium Grid to run multiple tests across different browsers, operating systems, and machines in parallel. This means that doing parallel testing for mobile is harder than for other apps.
To be sure, it may take some doing to get your Espresso tests set up. You may need to build the solution or use open source tools like Spoon or Flank to distribute all your Google Espresso tests across several emulators in parallel. Flank supports parallelism distribution, but I am not 100% sure that Spoon can aggregate all the tests, and then execute each test on separate emulators. A solution is needed that distributes all Espresso tests and integrates with Sauce Labs’ emulator infrastructure. Then, you’ll be capable of reducing the execution time on pull requests—resulting in cost savings that allow you to stay within your budget.
In closing, I’ll note that the test strategy described above is no replacement for real-device testing. Real-device tests still belong in your continuous testing strategy for native apps. Use analytics to drive your real-device strategy to execute tests on the top three devices on a nightly scheduled job.
But Espresso testing in parallel on emulators can greatly speed up your test results overall, without compromising test quality. It’s a useful technique for improving the performance of your tests.
Greg Sypolt (@gregsypolt) is Director of Quality Engineering at Gannett | USA Today Network, a Fixate IO Contributor, and co-founder of Quality Element. He is responsible for test automation solutions, test coverage (from unit to end-to-end), and continuous integration across all Gannett | USA Today Network products, and has helped change the testing approach from manual to automated testing across several products at Gannett | USA Today Network. To determine improvements and testing gaps, he conducted a face-to-face interview survey process to understand all product development and deployment processes, testing strategies, tooling, and interactive in-house training programs.
