A critical decision for software organizations that are defining their automated testing strategy is choosing the right automated testing framework that will help their development and QA teams efficiently write and execute automated tests.
The market offers a plethora of frameworks to choose from, and the decision isn’t always easy.
To help you in your evaluation, we have created a comprehensive guide that will provide you insight into:
Criteria to consider as you select a mobile test automation framework
Popular frameworks for automated functional testing
The unique benefits and limitations of each framework with a comparison view
What Not to Do When Selecting a Mobile Test Automation Framework
Before we dive into this further, let’s first talk about how not to select a mobile testing framework. Wim Selles, Staff Product Manager, Mobile at Sauce Labs, frequently speaks on this topic and has identified some common mistakes that teams tend to make:
Getting a framework and then hoping it will work for your team
Choosing it because you have used it before and it's familiar
Going after a popular brand (If x built it, it must be good)
Selecting one because it is currently hot in the market
Setting out to do it all on your own and building a new framework
12 Criteria for Selecting a Mobile Test Automation Framework
Now that we know how not to select a framework, let’s talk about what you should do.
As you begin evaluating mobile testing frameworks to determine what's suitable for your organization, here are some key questions you should ask to help uncover your needs and understand the impact of a new tool on your organizational processes, tech stack, and teams:
Organizational process: What does the process for buying/vetting a tool in your organization entail? Is there a separate team that can guide your research process and provide assistance, or do you have the freedom to choose?
Future needs: Will the new framework address your evolving business and testing needs in the future?
Application type: What is the type of application you are developing and need to test: native, hybrid, web?
Requirements: Does the framework meet your usability, speed, parallel testing, and hosting requirements?
Integration: Does the framework integrate well in your CI/CD pipeline?
Environment and tech stack: How does your existing tech stack look? Does the new framework support the technology and environment you are using?
Teams: The structure of your team and how it functions are important considerations when choosing a new tool. Does the team consist of:
Mainly developers embracing the shift-left motion?
Developers and technical QA teams following agile development?
Developers and manual QA currently in a transition to agile?
Siloed QA and dev teams following the more traditional waterfall approach?
Testing responsibilities: Who writes the tests? What is the testing process followed by your team? Will the framework support these specific needs?
Resource skill sets: Does your team have the required skill sets to work with the testing framework? Will they require special training and onboarding? How will the learning curve look?
Budget: What are the costs you need to consider with each framework?
Getting started: Can you evaluate the testing framework with a free trial? How complex is the framework’s workflow?
Community and support: Does the vendor provide good online support and resources? Does it have an active community to help troubleshoot your issues?
Popular Automated Mobile Testing Frameworks to Evaluate
Appium
Appium is a leading open-source test framework that allows for cross-platform (mobile) native test automation, as it supports both iOS and Android platforms. It was derived from Selenium in an effort to extend automated testing functionality to mobile apps.
Benefits: Since it's built on the same JSON wire protocol, QA and development professionals who are familiar with Selenium will find the transition to Appium fairly easy. Its extra functionality allows for the testing of native, mobile web and hybrid apps, and can be run across both the iOS and Android operating systems. Appium tests are language-agnostic and users do not need to install any extra software on their mobile device to support Appium. Backed by a large and thriving community, Appium users get consistent support for troubleshooting their issues.
Some limitations to be aware of: Users who are new to Appium need to familiarize themselves with a new scripting language and learn the rules on how to best interrogate your application to see the benefits of automated testing. Also, to optimally utilize the power of Appium for native app testing, users need to learn the concepts of native app/ selectors and have a reasonable understanding of the Appium architecture, which adds to the learning curve.
XCUITest
XCUITest is the testing framework that ships with Apple’s XCode development system, and as such, is the most popular framework for developers who want to test their iOS apps.
Benefits: XCUITest supports a broad range of tests for macOS environments. It offers developers comprehensive functionality to do basic unit level testing of iOS apps. Due to the framework’s architecture, test execution against iOS devices is faster, with less flaky tests and more reliable results. XCUITest simplifies the testing process by allowing teams to work on the app’s source code and test code in one place. Overall, XCUITest is simpler to work with, intuitive to set up, and easy to maintain when compared to a framework like Appium.
Some limitations to be aware of: As XCUITest is built for testing iOS apps, there are limitations in the programming languages you can use with it. It is dedicated to Objective-C and Swift code, which normally runs exclusively under iOS. So while we find that this native test framework is very popular with iOS developers as it gives them an easy method to run tests on the quality of their code, QA tends to move away from the iOS lock-in of XCUITest, and instead opts for more OS-agnostic testing frameworks. The scope of testing with XCUITest is also limited to the application, while with Appium, users can test the complete flow and interaction between apps, browsers, and OS.
Espresso
Espresso is the testing framework that comes built into Android Studio, and is designed specifically for functional testing of Android applications.
Benefits: Like XCUITest, Espresso is popular among developers, as it is easy to set up, more stable than Appium, and gives them the ability to quickly test code components. With its automatic synchronization of UI, Espresso allows for faster test execution. It also brings faster feedback for developers, as it does not require a server for communication. The simplicity and convenience of this framework also stems from the fact that Android UI tests can be compiled into a separate APK, which allows for the test suites to run next to the app on the device.
Some limitations to be aware of: Espresso also has the same lack of flexibility that we see with XCUITest. Firstly, Espresso is only compatible with Java and Kotlin, meaning that your tests can only be written in these languages. Additionally, Espresso can only be used to test Android apps, and so if your team is developing an app that will be listed on both iOS and Android, you will need to find another framework to help ensure compatibility across these different operating systems.
Comparing Appium vs. XCUITest vs. Espresso
Here's how Appium, XCUITest, and Espresso compare.
Appium | XCUITest | Espresso | |
Application Type | Mobile web, native/hybrid mobile apps | Native/hybrid mobile apps | Native/hybrid mobile apps |
Operating Systems | Cross-platform | iOS | Android |
Languages | Language agnostic | Objective-C/Swift | Java/Kotlin |
Key Users | QA, Test engineers | Developers (iOS) | Developers (Android) |
Enables | Shift-right testing | Shift-left testing | Shift-left testing |
Security Testing Type | Black box | Gray box | Gray box |
Source Code Required | No | Yes | Yes |
Flakiness of Tests | High | Low | Low |
Speed | Slower | Fast | Fast |
Updates | Trails iOS/xCode, Android/UI automation changes | Current/up to date | Current/up to date |
Mobile Testing Frameworks That Aren't Appium, XCUITest, or Espresso
In addition to the "big three" three popular frameworks, there are other tools in the market, including:
Detox is an open-source JavaScript-based test framework for React Native applications. It works cross-platform, accepts tests written in JavaScript, and works well with emulators and simulators. Detox is designed for gray box testing, allowing monitoring of the app from the inside, and is known to reduce test flakiness as it is well synchronized with the app’s activity.
EarlGrey is an open-source iOS UI automation framework developed by Google. It supports only iOS and hence requires tests to be written in Objective-C or Swift. Some benefits of EarlGrey include easy integration into iOS, simplicity in writing tests, and enhanced built-in synchronization features which helps to increase test stability.
