Sauce Labs is part of the Mobile Native Foundation, a place where several companies collaborate on open source projects and work together to enhance processes and technologies for large-scale Android and iOS applications. The foundation provides a platform for discussions and sharing of valuable insights, such as the recent Mobile Ecosystem Survey. In this blog post, I share some of the key highlights of the survey that focus on mobile application testing.
Mobile Application Testing Insights
Testing is, without a doubt, one of the critical elements of any mobile application development process. With that in mind, the Mobile Native Foundation's survey revealed some interesting insights regarding testing practices and approaches.
First, there is a low investment in testing older OS versions, mainly because tests are too complex to maintain, and the user breakdown does not justify the effort needed. Complementing that, there is a trend towards using unit tests, followed by snapshot/screenshot tests, plus a reduction in the number of UI tests. Snapshot/screenshot tests are preferred due to their maintainability and reliability in testing different rendering modes. However, it has proven challenging to manage the images used as baselines for those tests. All this is making UI tests to be non-release blocking, with only a few critical smoke tests being mandatory before a release.
Appium is still the preferred tool by QA teams, given its ability to cover testing in both iOS and Android with a single code base, whether developers tend to author tests through native frameworks like XCUITest and Espresso. There is, however, a shift towards using native frameworks due to the fact that it is common to have different user flows for the same feature in different operating systems, which increases test maintenance when a single code base is used. The effort to onboard new hires is smaller as well when the same role manages both development and testing.
Moreover, simulators and emulators are preferred due to their scalability for testing in parallel, and using real devices for every pull request or commit is difficult because they are a limited resource. And for testing in parallel with native frameworks, tools like Flank, Bluepill, Emcee, and Bazel are becoming popular.
Continuous Integration is Vital
Together with testing, continuous integration (CI) is vital for mobile app development and testing. Adopting CI early in the development process helps to ensure changes made to the app are validated continuously.
Bigger companies, such as Spotify and Lyft, invest heavily in their own infrastructure, using Mac Minis and Kubernetes clusters for Android to reduce build time. Those companies also tend to avoid using cloud services due to the prohibitive cost at that scale. Other companies, however, rely on cloud-based services to manage their build and test environments. Several companies prefer bare metal machines in their environments, for example, AirBnb uses MacStadium and Spotify uses Flow Swiss.
A frequent practice of these companies is to have a dedicated team for CI, which lets the rest of the teams focus on development and testing. A typical task the CI team does is to implement and maintain scripts to manage the setup and cleanup of their environments. Finally, GitHub is the most commonly used system to provide a centralized repository for source codes and team collaboration hub.
Conclusion
The findings from the Mobile Native Foundation’s Mobile Ecosystem Survey serve as a guide for companies looking to improve their mobile testing and development strategy, highlighting the importance of investing in the right infrastructure, using the right tools, and prioritizing the development of a robust testing environment.
