Best Practices for Effective Mobile Testing: The Modern Mobile Automated Testing Pyramid | Sauce Labs

With the mobile market growing explosively and customer expectations increasing just as rapidly, today’s mobile app development and QA teams are under tremendous pressure to deliver constant updates for their apps while providing a continuous and seamless user experience. Knowing exactly what to test at each stage of the software development lifecycle (SDLC) can help your teams find and fix bugs faster and earlier and keep your customers happy.

The original automated testing pyramid doesn’t account for different use cases, personas, or even beta testing, which puts mobile app development and testing teams at a serious disadvantage. This is why we at Sauce Labs have reimagined Kwo Ding's original mobile test automation pyramid for today's DevOps teams.

Before we dive into the mobile test automation pyramid 2.0, let’s review the original test automation pyramid. 

The test automation pyramid 1.0 is a testing strategy for mobile applications and websites where tests are automated across a mix of user interface (UI), API or integration (service), and unit testing. While every project, system, and delivery team is unique, that mix should generally appear as a pyramid.

The “Test Pyramid” concept was created by Mike Cohn and has been written about by thought leaders across the industry. The automation test pyramid 1.0 (illustrated below) flips the original test pyramid by placing unit testing at the base followed by API or integration (service) and then UI testing. By volume of tests completed, the bottom of the pyramid contains the most tests and the top contains the least.

Our reimagined mobile test automation pyramid is designed to guide mobile app development and QA teams on what to test for each type of mobile app at each stage of the SDLC. This new approach considers not only the specific testing needs of different personas, but also scaling mobile test automation and the mobile use cases developers are actually testing for.

The mobile test automation pyramid 2.0 can be applied to the following use cases:

By volume of tests completed, the bottom of the pyramid contains the most tests and the top contains the least.

Behold, the Sauce Labs mobile test automation pyramid 2.0:

Releasing to production with confidence requires the right quality and debugging tools. Our mobile test automation pyramid 2.0 delivers the insights that set us up for success. Let’s dive into each layer of the pyramid and the best practices for each. 

For web mobile apps and native apps, the bottom of the test pyramid starts with desktop browser testing.

Running your mobile website on emulated browser windows is a fast and scalable way to test compatibility on different browsers. At this level you should focus on system testing, responsive design, and the overall visual feel. This is also a great time to do cross-platform testing, however desktop browser testing will be limited regarding visual testing, native integration, and the fact that you are not using a true mobile device. Remember, any visual testing here will be futile because visuals will render differently on a mobile device.

While desktop browser testing should not replace mobile testing, it helps to see whether the website behaves as it should. But for more information on if our mobile website or hybrid app is behaving properly we will climb to the next level of the pyramid.

The next level of the mobile test automation pyramid 2.0 includes testing your mobile web, hybrid, and native mobile apps on emulators and simulators.

Since native apps don’t truly exist in mobile web, this is the first level of the pyramid where testing applies to them. Let’s dive into how and when you should test on emulators and simulators.

Emulators and simulators give you the option of running your mobile app on either an Android emulator with Chrome or an iOS simulator with Safari. This is another fast and scalable solution that introduces native integrations such as a keyboard, dropdown box, pickerwheels, and so on. Emulators and simulators also provide a better idea of the actual user experience.

At this level, your test focus should be on functional flow, such as the click paths through the website and app. You can also focus on the specific interactions with the native integrations, engagement with a keyboard for example. This is also a good level to mimic actual user behavior by testing gestures such as pinch/zoom/swipe. Visuals are more accurate here than the previous level, but emulators only provide vanilla or stock Android, not with the skins that we will see on actual different user devices.

At this level, you also lack real insight into device performance and real interoperability. To find that, we climb to the apex of our pyramid: real devices.

Real device testing allows you to test the usability of your app under real conditions so you can ensure all visual (UI) components, interactive features (gestures), and integrations (camera, keyboard, drop-down menus) work as designed.

At this level, you test your app on real Android or iOS devices. We recommend choosing a set of real devices that best represent your end users. See our blog on how to choose mobile devices for real device testing for more information.

Real device testing provides additional insights, particularly for hybrid and native apps, in regard to visual testing, since you are running tests in a real environment on an actual device. The OS on real devices, especially Android, has more impact on the visuals than you can see on an emulator or simulator. This is also a good time to focus on the native functionality of integrations such as biometrics and image injection.

Mobile error monitoring is helpful at this level. When embedded in the hybrid app and used during manual and automated testing (Appium/native frameworks), you will get the most accurate insights in where the app uses too much memory/CPU or too many errors are being generated.

At the top of the pyramid with hybrid apps and native apps, you can run mobile beta tests on the devices of real users. This provides the real-time user feedback and meaningful bug reports you need before releasing the app to the app stores.

Focus at this level should be technical, where bugs have the highest priority, and conceptual, where the purpose of the app will be tested by real users.

You can also use beta tests to collect error information post-release. This information can help you understand the type of issues users will encounter in production (errors, hangs, crashes), as well as help developers prioritize the errors to fix. 

The mobile test automation pyramid 2.0 rethinks mobile app testing as a holistic approach that focuses on which tests to run, where in the SDLC they should be run, and how many of certain tests should be run. This new approach is designed to remove bottlenecks and inject quality at every stage of the SDLC.

By following these best practices you are taking a step toward providing your customers with the customer experience they are expecting. As we know, every experience matters, so there is no better time to get started than today.