Why Mobile Game Testing is More Important Than You Think

Yes, July 6, 2016, was a landmark in mobile gaming because it was when Niantic released the global phenomenon Pokémon GO, a game that mixed AR and the real world to force people outside to stalk and catch virtual creatures. 

In 2016, 232 million people were playing Pokémon GO. It was everywhere. This author was working on a Hulu television show on the Universal lot at the time, and the cast and crew would frequently go on lunch hour ‘hunts’ or gym raids. At home, scores of neighborhood folks would wander the streets of Venice Beach, trying to earn enough experience to evolve their beloved Charmander to the menacing dragon Charizard. 

The success not only completely revitalized the franchise, but even spurred a major motion picture starring Ryan Reynolds. It was the piece of pop culture that would go on to define the summer. Pokémon GO made $100 million in its first three weeks, but by 2017 the craze was over. GO dropped to 65 million users by 2017, a drop of 65%.

For most games, this would have been the end. But just a couple of months ago, Pokémon GO crossed $6 billion in revenue. Despite a lower player count than its peak in 2016, (there were 71 million players in 2021) revenues have been consistently growing. After a “low point” in 2017 of $580 million, in the past two years, Pokémon GO has cranked out around $1.2 billion annually. And if you are a business executive wondering what that did to their valuation, I present the following chart.

Year     Valuation ($bn)

2015    0.15

2016    3.65

2019    4

2021    9

So, what did Pokémon GO do to buck the odds? 

Traditionally, a game would launch huge (hopefully) and then slowly tail off until a small community of only the most hardcore fans remain. Titles certainly now have a longer lifetime on both mobile and console alike. Contributing factors to this could be chalked up to new titles in the franchise, HD remakes appearing on newer platforms, nostalgia, or just the cyclical nature of consumer tastes. 

Pokémon GO benefitted from all of that, but they also fixed a quality issue and continued to expand their game with new features and a better overall customer experience.

Put yourself in the shoes of a (very cynical) executive that sells console games. Of course, this person wants to deliver a great product to protect the brand's reputation and generate positive word of mouth to accelerate sales. But at the end of the day, once a customer has given that company their $60 (or $70 or $80) it’s over.*

*Not including microtransactions or DLC, of course :) 

The lifetime value of that customer could be expanded to include their likelihood to buy future titles, but as far as that single product is concerned, the transaction is completed. The tangible benefit to improving the game is a future bet on goodwill and grassroots marketing.

Conversely, with mobile and specifically the freemium game market, the customer relationship is dynamic. Improvements to the game can have a very real bottom-line impact. By reducing churn and keeping the fish on the hook longer, so to speak, one would see immediate results. 

Look at the properties that have moved from computer/console to mobile: Fortnite, PUBG, League of Legends, COD: Warzone. Perhaps it’s the free-to-play Battle Royales that are most well known, but even franchises that were specifically born on console or PC like Final Fantasy and Madden have popular mobile freemium spin-off series now.

If we accept the above hypothesis, it’s clear to see that testing in the mobile game market is even more important than in the legacy console and PC market because mobile games have the opportunity to be constantly improving, delivering more value to gamers, and driving practical revenue. Now, this is not necessarily an endorsement of microtransactions, nor does it dismiss the fact that plenty of console games feature them as well. It merely states the obvious that a high-performing mobile game should have fewer issues with customer retention, which in the long run is good for everyone. 

So how do you deliver quality at speed to those present users? How do you embrace the fact that every experience matters? One solution is to test the product and ensure quality before a user-generated compilation of bugs ends up on the front page of Reddit. (Oh yeah, they exist.)

Backtrace and Sauce Labs give mobile game developers the solutions in a convenient tool chain that enables them to deliver value to users quickly, by enabling quality releases faster than they are currently able. In doing so, game publishers are telling their users they aren’t taking them for granted. They are also spending smart money because 20% of users will abandon a product the first time they encounter an error on a mobile app. The revenue loss of 20% customer abandonment far outweighs anything a company could spend on quality control.

Those that develop for the mobile market know they aren’t just competing against other games. They are competing over a user’s time. Our cell phones are a portal to the world. We can read, watch a video, play games, or communicate with friends. With that much competition, the best way to dazzle and delight your users is to constantly deliver a quality experience. This will keep them engaging with your content and should deliver a strong customer lifetime value. Test early and often, and you won’t have to be scanning the Reddit boards to see where you came up short.