Choosing an Automated Testing Framework

There are a number of things to consider when beginning an automated testing project, to ensure the project is set up for success. However, none might be more critical than choosing the correct automated testing framework. There is no shortage of choice for frameworks for writing and executing automated tests, so this choice can be intimidating for some organizations when defining their strategy.

To help in your evaluation, we have listed some of the most popular frameworks for automated functional testing, along with some of their strengths and weaknesses:

Selenium is the industry standard in open source automated testing frameworks for web applications. An official W3C recommendation has helped elevate the status of this framework, as it has been the choice for QA professionals since its inception in 2004. One of the reasons for its popularity is its test domain-specific language, which allows you to write tests in the programming language of your choice (Java, C#, Python, Ruby and more).

Another big value add is that Selenium allows you to run tests on Mac, Windows or Linux without any additional configuration - you only have to write one test, and it will work in any OS/browser combination. Selenium was intended for desktop web applications, and so if your team is also developing a mobile app or website, it may not fulfill all of your testing requirements.

Appium was derived from Selenium in an effort to extend automated testing functionality to mobile apps. Built on the same JSON wire protocol, QA and development professionals who are familiar with Selenium will find the transition to Appium fairly easy. Its extra functionality allows for testing of native, mobile web and hybrid apps, and can be run across both the iOS and Android operating systems. However, like Selenium, users who are new to Appium need to familiarize themselves with a new scripting language and learn the rules on how to best interrogate your application to see the benefits of automated testing with Appium.

XCUITest is the testing framework that ships with Apple’s XCode development system, and is the most popular framework for developers who want to test their iOS apps. It supports a broad range of tests for macOS environments, but that flexibility does not extend to the programming languages you can use. XCUITest is dedicated to Objective-C and Swift code, which normally runs under iOS exclusively. So while we find that this native test framework is very popular with iOS developers as it gives them an easy method to run tests on the quality of their code, QA tends to move away from the iOS lock-in of XCUITest, and instead opts for more OS-agnostic testing frameworks.

Espresso is the testing framework that comes built into Android Studio, and is designed specifically for functional testing of Android applications. Like XCUITest, Espresso is popular among developers, as it gives them the ability to quickly test code components. However, it also has the same lack of flexibility that we see with XCUITest. Firstly, Espresso is only compatible with Java, meaning that your tests can only be written in that language. Additionally, Espresso can only be used to test Android apps, and so if your team is developing an app that will be listed on both the iOS and Android, you will need to find another framework to help ensure compatibility across these different operating systems.