Continuous Testing for Mobile Devices

Mobile applications have traditionally lagged behind their web counterparts in terms of speed of releases, maturity of testing and available tooling. Yet mobile is driving much of the innovation in today's connected world. Mobile devices and software have become an indispensable part of our day-to-day living.

Since mobile software has assumed so much importance as a driver of innovation, testing mobile software effectively is more important than ever. This is why continuous testing for mobile apps is now essential. This article explains what continuous testing means and what it looks like in the context of mobile development.

Enterprises are big on driving the digital transformation across their organizations and moving their entire workforces to a modern, agile, and mobile-first culture. A key enabler of this is the mobile applications that their employees and customers interact with. Employees bring their own devices and perform their work in applications that are distributed across loosely coupled architecture.

Customers are demanding and won't hesitate to move to a competing solution if they find performance lags and downtime in a mobile app. In today's mobile-first world, quality is table stakes for even showing up with a mobile app that represents your organization and brand.

The DevOps approach to delivering software has brought sweeping changes across the development pipeline. Speed is a key focus and it is enabled by automation. The quicker an organization brings features to market, the better its chances of succeeding. However, with speed comes the challenge of keeping quality on par. As new features are released, if they are not functional, and if they deliver a compromised user experience, all the investment in speed is of no value. To truly operate at speed sustainably, you need to focus on quality. This focus on quality should cover the entire pipeline from start to end. It's just another way of stating the need to adopt continuous testing for mobile. You can't choose between speed or quality. You need both, and continuous testing is the way to achieve this balance.

Mobile is different from web apps, and the challenges associated with continuous testing for mobile are different from web apps.

Gatekeepers: While web apps can be deployed thousands of times a day, mobile apps need to be deployed in timed cycles that take into account app stores and their policies. This leaves less room for failure. Bugs can't be patched as quickly since releases are less frequent. There isn't as much visibility into issues as with web apps, and users are much more impatient when it comes to mobile.

Fragmentation: Mobile is complex because of the number of OS versions, device types, form factors, device specifications, new hardware features, types of mobile apps (native, web, hybrid), different mobile frameworks for building applications, and carrier networks. All these factors act as variables and bring in numerous permutations and combinations to consider when testing mobile apps. Fragmentation is a serious challenge with mobile testing.

Visibility: Error reporting is harder with mobile, as apps crash and sometimes leave no trace of the cause unless appropriately preconfigured. Additionally, when testing, it's harder to spot issues that are specific to a geographical region completely different from the region the tester is in. Mobile is typically used in a more geographically distributed way than web apps, and this needs to be kept in mind when triaging issues and analyzing test results.

Shift left & right: While DevOps has stressed “shift left” and testing that occurs earlier and more frequently, it doesn't mean that testing shouldn't be done post-deployment. Continuous testing requires early testing, but it also focuses on post-deployment testing to relay faster and real-world feedback to the initial stages of development. Error reporting should be configured to catch all errors in production. Additionally, leveraging features like beta testing (which both Android and iOS offer) is necessary to catch bugs while the app is in production, but not yet released to all users. While you can't release as frequently as with web apps, you can leverage these tactics to make every release count.

Piece it together: Mobile is fragmented, but there are testing methodologies and tools to address this issue. Rather than build a device lab in-house, leveraging real devices in the cloud is a smarter choice. It's faster to get started with, easier to maintain, enables agility, and costs much less. Cloud testing solutions like Sauce Labs offer a combination of emulators and real devices so you can make use of both for different types of testing. Unit tests, early-stage tests, and tests that don't rely heavily on hardware specifications can be run on emulators. More complex, late-stage testing can be done on real devices, but in the cloud. This is a realistic way to fight fragmentation with the mobile ecosystem and deliver outstanding apps, no matter which device or platform they run on.

Drive efficiency: As you move faster with mobile app development, your testing should keep pace. The way to do this is to integrate testing with your CI tools, like Jenkins, and make test automation go hand-in-hand with build automation. As the number of builds keeps increasing, you need parallel testing to handle the inflow of new code and features. Continuous integration is not possible without continuous testing.

Analytics brings visibility: As you look to surface hidden issues and fight obvious ones, you need visibility that's end-to-end and deep enough to handle the complexity of mobile. To empower this, Sauce Labs' Continuous Testing Cloud has just released test analytics and extended debugging. Test analytics shows you the most urgent issues to be fixed and proactively guides you. Extended debugging focuses on errors with networking and browsers. Both these features are essential when testing mobile apps.

As your testing matures to match the needs of mobile, you need continuous testing. It brings the right balance between speed and quality, takes into account fragmentation, and brings deep visibility into application performance. If you ship mobile applications, you can no longer do without continuous testing for mobile.

Twain Taylor is a Fixate IO Contributor and began his career at Google, where, among other things, he was involved in technical support for the AdWords team. His work involved reviewing stack traces, and resolving issues affecting both customers and the Support team, and handling escalations. Later, he built branded social media applications, and automation scripts to help startups better manage their marketing operations. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications.