A simple mobile development project might be one piece of software developed for iOS. But, things are never so simple are they? There are at least two versions of iOS being used at any given time. There are a handful of devices to consider such as iPad, iPad mini, and a few each of the iPod and iPhone. And then on top of that you have to consider what browsers people are using.
That is the simple version of the story.
The complex version includes a few hundred screen resolutions, different device makers, and too many Android versions to count.
Experienced testers are familiar with the combinatorial problem of emulator vs simulator vs real device. But for mobile devices, there are a few new twists -- and ways to adjust. Here are a few.
Simulate
Ask a dozen testers about the pros and cons of using simulators to test with, and you are likely to get two or three dozen sets of cons. "There's nothing like the real thing" they'll say, or they'll list their favorite bugs that did not appear on the simulator. Yet here is what simulators do: They radically decrease the size of the feedback loop from programming to first exploration.
That feedback loop is critical.
In the old days of software, programmers might work for a day between compiles, and a week between coding and any real testing. Testers familiar with those days tell stories of builds where nothing worked, of crashes, of failure to log in. The first week of the "test cycle" was often spent on blocker issues. Today those things are mostly a memory. Thanks to Test Driven Development (TDD), programmers go from coding to unit testing and back every ten minutes or less.
Yet exploration is not so easy with "the real thing" on mobile devices. For native applications, the programmer needs to make a new build, "sling load" it onto the device, the run the application, not to mention deploying any new server-side code that is needed on the test server, and possibly re-configuring the application to hit the test server, not production. All that takes time. At best it slows down development. Most of the time, the programmer or technical tester will defer the pain, "pushing" to the test device less often. This slows down the feedback cycle while ensuring there is more new code, hence more new hidden errors, between builds.
The main benefit of the simulators, which runs on the same machine as the developer making the build, is saving time and tightening the feedback loop, not more accurate testing. Instead of creating new builds and transferring files, a developer could click play in their development environment to see the latest version of the product they are working on. The simulator is real enough to allow someone to enter data, click buttons, submit forms, and navigate around. But, that is about it. For the most part, simulators are tools of convenience that developers use to get a feel for their latest code change.
Emulate
Emulators are a step in-between the simulator and physical in-your-hand device. Emulators are essentially a real piece of hardware, just somewhere on the internet, running perhaps more than one version of itself as a virtual machine. That server could have several different instances of mobile operating system installed and accessible all at the same time.
Returning to a real software development scenario, the development team is now outside of the TDD (programming) loop. They have worked their way through a few cards on the kanban board, done some cursory testing in their simulator, and are now ready to do some deeper testing. This product is intended for the consumer market. The development team has some statistics on previous usage to help guide decisions about what platforms and operating systems they should start testing on. But, that list is 15 devices deep and can change suddenly. Emulators are probably the right place to start.
Most emulator services have a wide variety of device platforms and operating systems available to their users. A tester could log into this service, select a device, install their new build and be testing in a matter of minutes. Instead of just giving you just a running app, an emulator will give you a simulation of an entire device. That includes hardware resources like memory and CPU cycles, battery levels, wifi availability, and the ability to emulate all of the touch and rotate actions that people do with a real device.
Emulators can be a good deal for companies that need a wide variety of devices and don't want to maintain their own test lab. Emulators are also handy for running test automation in isolated environments that won't disturb any other testing activities.
The Real Deal
The man benefit of a real device is that it is, well, real. A tester could take the latest build of a native app, install it on a real device, and then test in ways that might be relevant to a real customer. That might include walking around in and out of access to wifi or internet connectivity, seeing what happens when hardware resources such as power and memory shrink. These things can not be done in a simulator, and can only be mimicked in an emulator. Emulators do a pretty good job of recreating real mobile environments, but occasionally fall short. Every once in awhile, you will discover a bug in a real mobile device that can not be reproduced in an emulator.
Real mobile devices are perfect for companies that only need to support a handful of devices, and are willing to maintain their own test lab. They are also great tools for augmenting a strategy built around automation and testing in emulators and simulators.
Each of these options is a tool, and like all tools they each serve a specific purpose. Often, the right choice here is some of each. For now, take a look at the devices your customers use, compare that to your mobile test strategy, and find the gaps.
