In the world of software testing, the terms device farm and device cloud may sound interchangeable. In fact, they are often used that way, with some folks assuming that device farms and device clouds are basically the same thing.
That's understandable because device farms and device clouds are similar in key ways, and they both offer similar benefits for mobile testing. However, there are major differences between device farms and device clouds, such as how much each type of solution costs and how much flexibility it offers. Understanding those differences is essential for knowing which type of solution to use when.
Keep reading for a comparison of device farms vs. device clouds, along with guidance on the pros and cons of each.
What is a Device Farm?
A device farm is a collection of devices designed to enable developers and QA teams to test software remotely. By providing access to different types of devices, device farms make it easy to evaluate how applications perform on various device types and configurations.
This is important, of course, because software end users may operate on a wide range of devices, and applications can behave differently under different devices or configurations. By running tests across multiple device types and settings, teams can detect unexpected problems – such as an application interface that fails to render properly on a certain device due to its screen dimensions, or a biometric authentication feature that doesn't work on a specific device because of issues unique to the device's biometric interface.
Developers and QA teams could, of course, run tests across multiple device types by acquiring and deploying each device they want to test themselves. But that would be a costly and time-consuming endeavor. There are tens of thousands of different types of smartphones, tablets, and other devices in the world, and setting up each one locally so you can run tests is simply not practical. Indeed, even setting up just a few dozen devices can be a real challenge because you'd have to purchase each one, set it up, keep it up-to-date, and deploy your applications on it for testing purposes.
Device farms resolve these issues by allowing teams to connect their applications to a wide selection of devices over the network, eliminating the need to purchase and manage devices themselves.
What is a Device Cloud?
A device cloud is a collection of physical and/or simulated or emulated devices that developers and QA teams can access over the Internet for testing purposes. In addition, device clouds typically provide APIs that engineers can use to connect to devices and automate software tests.
Thus, like a device farm, device clouds eliminate the need for teams to set up and manage their own devices for testing purposes. Device clouds provide a flexible, scalable, and convenient way to run tests across a wide range of devices.
To use a device cloud, engineers typically choose which device types and configurations they want to test applications on. Then, they use the device cloud's API, in conjunction with a test automation framework like Selenium, to determine which specific tests they want to run. From there, the tests can be triggered automatically and on demand.
What Is a Real Device Cloud?
A real device cloud is a device cloud that consists of physical devices, as opposed to virtual devices (simulators and emulators). In other words, rather than using software to create a virtual device environment and run tests in it, a real device cloud gives teams access to actual devices, which they can connect to over the Internet and manage using APIs.
The Sauce Labs real device cloud
Benefits of a Real Device Cloud
Real device clouds offer the benefit of more reliable testing because tests are performed on physical devices. With simulators and emulators, there is a risk that the virtual environment where tests occur won't fully replicate the environment of a real device that end-users would operate on in the real world, and so your tests may not surface certain software bugs. In addition, certain types of tests, such as tests that evaluate how application behavior impacts a device's battery life, require access to physical devices; you can't reliably simulate hardware components like batteries.
On balance, it's worth noting that simulators and emulators have benefits, too, such as more flexibility (because virtual devices can be launched on demand, without a physical device being available) and lower costs (because the cost of running virtual devices is based primarily on the CPU and memory they consume on servers, and is not tied to the cost of physical devices). But if your priority is ensuring that tests are as reliable as possible, real device clouds are a better solution than ones based on simulators and emulators.
In many cases, it makes sense to use virtual device clouds and physical device clouds together. You can run the bulk of your tests on emulated or simulated devices, then switch to a real device cloud to support tests that won't work well elsewhere. This approach makes it possible to take advantage of the lower cost of virtual device clouds for most testing needs while still benefiting from the unique features that only real device clouds offer.
How Does a Real Device Cloud Work?
A real device cloud works very similarly to a virtual device cloud: engineers choose which devices they want to use for testing, then configure and execute tests via an API. The only difference is that behind the scenes, the tests run on real, physical devices, as opposed to on virtual devices hosted on a shared server.
A real device cloud mobile app test
4 Key Differences Between Device Farms and Device Clouds
Now that we've discussed the basics of device farms, device clouds, and virtual device vs. real device clouds, let's talk about how device farms compare to device clouds.
To a large extent, device farm and device cloud mean essentially the same thing, since both terms refer to collections of devices that can be used for testing purposes. However, there are four key differences between device farms and device clouds:
Physical vs. virtual devices: In most cases when people talk about device farms, they are referring to collections of physical devices. In contrast, a device cloud could include physical devices (if it's a real device cloud) or simulated or emulated devices.
API-based controls and automation: The term device cloud usually implies an API-based device testing infrastructure where tests are easy to automate and scale. Some solutions billed as device farms (such as AWS Device Farm) also offer APIs, but in general, device farms provide fewer automation options than device clouds.
Scalability: Device clouds are typically designed to be more scalable by providing virtually unlimited infrastructure resources. Because a device farm might not include as many device instances, engineers could end up having to wait for another customer's tests to complete before they can begin their tests. But with device clouds – especially those that use emulation and simulation – there is more flexibility and scalability.
Cost: On the whole, device farms have more rigid costs because they are based on physical devices only. Device clouds typically offer a wider range of price points because customers can choose between different types of devices (virtual vs. real) and different automation features. This makes device clouds advantageous if you want to tailor testing to fit a particular budget.
Thus, in general, device farms are better suited to testing routines that are more manual rather than automated and that require less flexibility or scalability. If you want a broader set of options about where and how you run tests, as well as how you orchestrate your tests, choose a device cloud.
Comparing Device Farms vs. Device Clouds
| Type of infrastructure | Automation level | Scalability | Cost |
Device farm | Physical | Low or medium: may provide some APIs or orchestration tools, but automation features are usually limited | May be limited by physical device availability | Fixed and tied to physical device cost |
Device cloud | Virtual or physical | High, thanks to API-based controls | High, thanks to larger-scale infrastructure and the ability to leverage emulation and simulation when physical devices are not available | Flexible, due to the wide range of device options and platform features |
Device Farm and Device Cloud Tools
If your testing needs include validating applications on real devices only, and you have the budget necessary to run all tests on physical devices, a device farm offering such as AWS Device Farm may be a good tool for you.
On the other hand, if you're looking for a flexible testing infrastructure that lets you pick and choose from among many types of physical and virtual devices, consider Sauce Labs. With the Sauce Labs real device cloud, you can run each test on the type of device best suited for it – which allows you to save money by using emulation or simulation in some cases while still taking advantage of a real device cloud for tests that require physical devices.
Plus, Sauce Labs supports virtually any mobile test automation framework, including Appium, Espresso, and XCUITest.
