Getting Started with Mobile App Testing

Mobile testing continues to grow as mobile devices become more important in our every day lives. According to Statista, 68% of the global population had a smartphone as of 2022.

The increasing demand for mobile applications is keenly felt on the development side, an Enterprise Strategy Group (ESG) survey found: in the past three years, 50% of organizations doubled the number of mobile apps they develop, while 76% of organizations plan to increase their testing capabilities in the next 12-24 months.

Mobile application testing (MAT) is the process of testing mobile apps to ensure that their functionality and user interface operate as designed on different types of devices, browsers, and operating systems. Mobile testing can be done manually on real devices and emulators/simulators or with automation.

Mobile users expect rapid innovation – if your app isn’t updated frequently, you risk bleeding users and losing your competitive edge.

Mobile users also have high expectations regarding the personalization and user experience of their apps. Users want information to be presented in context at the right time, in the right place, and in the right way. These demands and the tremendous opportunities they present bring mobile app development to the forefront as businesses compete for supremacy in today’s mobile-first world.

Further, there are fundamental differences between the release cycles for mobile native applications versus web applications. It’s much easier to develop and release updates for web applications that can be deployed at the same time for all users, multiple times per day. New versions of web applications can be accessed automatically by users and rolled back if issues are uncovered.

For mobile applications, release cycles are longer and more complex, which means fixing a bug for an app that’s in production can be both costly and time-consuming. For example, an app installed on a user’s device cannot be rolled back to a previous version—developers are forced to “fix forward” issues in future updates, potentially leading to multiple versions of the same app in production. Thus, testing earlier in the app release cycle becomes even more critical for native apps. In addition, since every app is installed separately, it’s likely that different users will have different versions at any given time.

Mobile testing helps improve the customer experience of software, apps, and more.

In the past, simple device hardware checks were considered “mobile testing.” However, mobile application testing today broadens the scope of testing to include the software of a device (the app) and the end user experience.

Mobile application types include native, hybrid, web, and progressive web apps.

Native apps run on iOS or Android. iOS apps present as an IPA binary file. The file may be tested with open-sourced Appium and/or Apple’s proprietary XCUITest framework.

Android apps are built into an AAB/APK package. Appium and/or Espresso frameworks can be used for this popular mobile OS.

Hybrid apps are essentially web apps that have an application wrapper which is designed to be independent of iOS or Android. This gives the hybrid application the ability to access all of an operating system’s specific capabilities.

Hybrid apps can be tested using the Appium test automation framework, native frameworks (XCUITest, Espresso), as well as Selenium, when the webpages are isolated.

Web apps work with mobile native browsers, like Chrome, Safari, Mozilla’s Firefox, and other “vendor” browsers like Samsung Internet. Because they are pure web applications one can test them by using the Selenium and Appium test automation framework. Though Selenium is a pure web tool, it can be very useful in aspects of MAT.

Progressive web apps are traditional web applications that are enhanced with modern web technologies, allowing them to provide a more app-like experience.

These are web apps that use emerging web browser APIs and features along with traditional progressive enhancement strategy to bring a native app-like user experience to cross-platform web applications. Progressive Web Apps are a useful design pattern, though they aren't a formalized standard. 

Appium, Selenium, or JS based Native frameworks like Cypress, Playwright, and others can be used to test PWAs. The JS based Native frameworks allow users to test PWAs on a desktop web browser.

A critical decision for organizations that are defining their automated testing strategy is choosing the automated testing framework and tools that will help their development and QA teams efficiently write and execute automated tests. The market offers a plethora of frameworks to choose from, and the decision isn’t always easy. 

Read our comprehensive guide to learn: 

Here's how Appium, XCUITest, and Espresso compare.

We also have comprehensive guides about choosing iOS testing frameworks and Android testing frameworks.

Mobile app testing has three major points of focus, each complementary to the others.

App functionality tests include:

This is testing specific to the device itself, and usually requires physical devices rather than simulators. This includes, but is not limited to:

Sometimes called "non-functional" testing, attribute testing explores qualities of the software outside of pure functionality.

Attribute testing may be overlooked by some quality efforts, but it can make or break an app. An app that has an easily exploitable security hole, for example, will be shunned by users, and can force liability onto the developer.

Mobile beta testing allows mobile development teams to validate the applications with “real users” in real-world environments. It helps teams to get feedback from end users who represent an app’s actual user base. By testing application performance and functionality with a beta group prior to general release, teams gain a deeper level of validation that the software meets end-user requirements. They also significantly reduce the risk that an unforeseen application problem post-deployment will necessitate the application release to be rolled back or require the team to push out an update quickly in order to fix a problem with the production release.

Read more about the best practices of mobile app beta testing to learn how beta testing works, why it’s important, and how to get the most out of beta testing processes.

Testing how an app gets used in real-life is important to the decision about testing on production code. Testing in production removes the layer of abstraction introduced by the QA-designed testing environment and brings tests closer to the actual use cases. If it is possible to do so, performing production testing on real devices will reduce unwanted surprises at release time.

With a good mobile application testing strategy, you can improve QA to catch bugs on the front end and backend, test APIs, and simulate different operating systems.

There are two core approaches to testing a mobile application. You can have humans use the application in different situations to see how it responds (manual testing), or you could have coded instructions drive the application into various scenarios and look for expected results (automated testing)..." Both of these mobile testing approaches can be valid in different circumstances, and that validity can change over time in other moments of an application’s lifecycle. In practice, most organizations use a combination of manual and automated testing in their mobile testing strategy. 

Let’s have a closer look at manual vs. automated mobile testing.

Manual testing is a human input, analysis or evaluation. This approach is user-centric, focusing on explorative ways of monitoring, whether a mobile application meets user requirements and expectations. You should test your app for look & feel and for usability, making sure that it is user friendly. You should not be using manual testing for all your testing, but just for about 20% of them, for the rest you can use automated testing

Automated testing is another mobile application testing approach. While the aim of manual testing is to test the “user experience,” automated mobile testing aims to test all the functionality that characterizes an app, and ensures that all features and functionalities work as expected (for example, regression testing) after code changes and app updates are implemented.

Given the benefits and capabilities of each testing method, an integrated testing approach that incorporates both manual and automated tests is the way to go. And with this approach, a best practice is to automate 80% of your test cases while performing 20% manually.

Here are the specific mobile test cases that should be automated:

The best approach for mobile application testing blends both manual testing and automated testing. This approach maximizes efficiency, time, and cost savings. Skipping automated testing results in slower feedback and a reluctance to cut and ship new versions because of the cost; skipping manual testing leaves entire feedback categories ignored.

Sauce Labs recommends 80% automated testing and 20% manual testing for a comprehensive mobile testing strategy. Based on research, Sauce expects combining the two approaches in that ratio can help you save up to 70% of your money and time spent testing. These percentages can change based on your app's complexity and concept.

The best mobile app testing strategy integrates both manual testing and automated testing across the software development lifecycle. Both approaches offer many benefits that can help you deliver high-quality mobile apps at speed. The diagram below shows what tests you should perform through manual testing and which ones through test automation. It also advises an agile approach when developing mobile applications. It is important to note that a tester briefly explores the application while the automated tests are written. The development and testing team will perceive more value if this exploration is done intentionally.

Learn more about manual vs. automated mobile testing.

Real device testing is the practice of installing the latest build of a mobile app on a real mobile device to test the app’s functionality, interactions, and integrations in real-world conditions. Real device testing is a recommended component of a comprehensive mobile app testing strategy, especially when used in combination with virtual devices (Android emulators and iOS simulators).

Organizations committed to their mobile application quality efforts are faced with making an important choice regarding where to run their mobile tests. The default for development teams in large enterprises is to use real mobile devices. While this gives them more accurate test results, it is not ideal for scaling and automation of testing.

In an attempt to move away from testing on physical devices, some organizations have switched to using emulators and simulators for their mobile app testing. However, it’s a best practice to incorporate a combination of real devices and virtual devices (mobile emulators and simulators) for the most comprehensive mobile app testing strategy. Let’s dive into what emulators and simulators are and when they should be used for mobile testing.

A mobile emulator, as the term suggests, emulates the device software and hardware on a desktop PC, or as part of a cloud testing platform. It is a complete re-implementation of the mobile software written in a machine level assembly language. The Android (SDK) emulator is one such example.

A simulator delivers a replica of a phone’s user interface and does not represent its hardware. A simulator is a partial re-implementation of the operating system written in a high-level language. The iOS simulator for Apple devices is one such example.

A real device is the actual hardware (plus OS and built-in support resources) on which your software will run in production. For mobile software, it's the mobile phone or tablet. For specialized industrial, scientific, or medical monitoring software, it's the actual monitoring device.

Here’s a summary of when to use real devices and emulators and simulators in your testing:

Read our white paper to know why automated mobile testing requires both emulators/ simulators and real devices for a best-of-breed mobile QA environment and comprehensive coverage.

A real device cloud is a mobile app testing environment that provides on-demand access to real iOS, Android, and other mobile devices (smartphones and tablets). Mobile app development and QA teams can test their apps on different device/OS combinations to get real-world feedback and ensure optimal coverage for their organization’s target customers. A real device cloud can integrate with popular mobile testing frameworks like Appium, Espresso, and XCUITest.

With so many mobile device models and manufacturers in the market, it’s no longer a safe bet to test only on Apple and Samsung devices. While iPhones are still the most popular mobile phones in every region, manufacturers like Xiaomi, Huawei, and OPPO are disrupting markets in densely populated regions like China and India. Given the magnitude of possibilities, deciding which devices to test on can be overwhelming.

Mobile testing is complex, and therefore you need a testing platform that can serve your needs and help you address the key challenges throughout the mobile app development lifecycle.

Sauce Labs end-to-end mobile quality solutions can help you achieve quality at speed throughout all stages of the mobile app development lifecycle (from development to production), with a secure and scalable test cloud, extensive device coverage, optimized mobile beta testing, and best-in-class error monitoring and reporting.