Building a Strategy for Continuous Testing in Mobile

Writing code for both mobile and web apps is pretty much a similar process in terms of the logic, programming language, and architecture used to host them. That’s probably where the similarities end. When it comes to deployment, deploying web apps is like writing on a whiteboard with dry-erase markers, and deploying mobile apps is like doing it in permanent ink. Web apps give you a lot of control over your environment and allow you to make changes to live code as you see fit. When you’re deploying mobile apps, however, it’s a lot like standing in a long queue with your mobile app in one hand and your license and registration in the other. Additionally, in case you don’t get your deployment perfect the first time around, you go back to the end of the line with your updates in one hand and your patches in the other. 

While the downside does involve having to stick to guidelines, publishing in “recommended” formats and making do with fewer and less frequent updates, the upside is that if you get it right, there are literally over a billion potential customers in each store. In today’s super competitive mobile world, the trick is to pick your battles wisely and invest time and effort where it counts.

With so little control over an application once it’s deployed, what do you do to avoid malfunctions that cause applications to be uninstalled, forgotten and replaced? You shift all your energy and effort to the pre-deployment stage and get a bunch of users to test your applications before they go live. Sort of like a virtual wedding rehearsal for apps, beta-testing is a great way to put your application through the ropes, pre-deployment.

Enter continuous testing, the one way to make sure that you’re doing absolutely everything you can pre-deployment, so you can afford the luxury of giving up all control to an app store once it’s showtime. People talk a lot about CI and CD, but there’s really no point of either of those if you can’t come up with a mobile app deployment that’s reliable and relatively bug-free right off the bat. 

Continuous testing, when practiced, involves multiple testing tactics such as beta testing, test automation, and testing in the cloud. Let’s look at these tactics in some detail, and see how they come together to form a continuous testing strategy for mobile. 

In an ideal world, every possible scenario should be run and re-run before you go live. Beta testing not only lets you bypass the app store gate and release your app privately to a large number of testers, but it also helps avoid expenses related to configuring production-like environments for testing. 

While both Apple and Android have their own beta-testing platforms, called TestFlight and Play Console respectively, they’re both exclusive to their own operating systems and still require you to go through a review process.

There are, however, quite a few beta testing platforms that support both iOS and Android and also allow you to share your build with large groups of testers, independent of approval from the public app stores. 

HockeyApp is a good example of a beta testing platform that skips the app store review process and can be used to deploy on iOS as well as on Android. While it’s currently being integrated into App Center by Microsoft, it still retains all its popular features like automated collection of user feedback, team management, and segmented notifications.

Once you’ve picked a beta testing platform and are ready to deploy to testers, the next step is to automate your beta-tests as well as the management of users and test groups. This is because manually collecting feedback on bugs and crashes is both impractical and unsustainable at scale. 

Apple TestFlight users can either use the new App Store Connect API or third-party tools, like FastLane, to automate beta tests. FastLane can be used for Android as well as iOS deployments, and automates the time-consuming aspects of beta-testing like code signing and incrementing versions. FastLane supports over 15 different beta-testing platforms and allows you to switch between platforms freely, without reconfiguration. 

This isn’t to say that automation and beta testing necessarily need separate tools. A lot of times these services have functions that overlap, yet complement each other. A good example is the fact that platforms like HockeyApp are perfectly capable of automation on their own, but are often integrated with automated services like FastLane or Sauce Labs Real Device Cloud for better productivity.

Mobile app testing, in general, can be quite complicated and “resource-hungry” considering the limitless combinations of OS versions and device specifications that make up the mobile web. As opposed to building and maintaining a lab comprised of all the different devices and their drivers, cloud-testing platforms give users access to thousands of real devices, along with their platforms, versions, and network carriers. 

A cloud-based mobile device lab would deliver thousands of actual devices on demand. Testers can log in and instantly access a real device cloud comprising thousands of devices, browsers, operating systems, screen sizes, and resolutions. An added advantage to using cloud-testing platforms like the ones mentioned above is that they integrate well with most popular CI and DevOps tools like Jenkins.  

In the past, QA teams made do with emulators or simulators and manual testing on real devices. Today, however, mobile testing can be automated as powerfully on real mobile devices as on emulators. This is a powerful capability that needs to be leveraged as part of a continuous testing strategy for mobile.

In conclusion, continuously testing a mobile application in order to remain competitive and relevant involves a lot more than the regular, unit, integration and UI tests that most testers are familiar with. With the app store gates being the way they are, it’s important to release bug-free applications right off the bat and make every release count. To that effect, it is critical to test on large groups of users, pre-deployment, and leverage cloud-based testing solutions that enable access to all the different device and OS combinations available today. 

When you’re ready to jumpstart your mobile testing strategy, check out Sauce Labs free trial for instant access to thousands of browsers, OSs, and real devices.

Twain Taylor is a Fixate IO Contributor and began his career at Google, where, among other things, he was involved in technical support for the AdWords team. His work involved reviewing stack traces, and resolving issues affecting both customers and the Support team, and handling escalations. Later, he built branded social media applications, and automation scripts to help startups better manage their marketing operations. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications.