Test Faster and Smarter by Testing in Production

You may dread the term testing in production (TiP). The thought of potential loss of data, downtime, and a damaged reputation to organizations can be daunting. But things need not be that way. In fact, today, testing in production is used by some of the biggest organizations with much success. But can it become a reality for your team?

Testing in production is not a completely new concept. In fact, you’ve probably seen it in action more often than you imagine. Think of an app that you’ve released or one that you know of that was poorly tested. You likely spent the next few weeks firefighting, and got it to be functional faster than you thought possible. In this case, you were forced to test in production. But, what if you could hone the art of testing in production and use it to your advantage? What if you could spot and fix issues so much faster than you do today? What if you could influence development from start to end? What if you could do all this without risking the reputation of your team or organization? That’s the promise of TiP, and it’s worth a second glance.

The key reason to test in production is because of the real-world feedback it gives you. Today, with the plethora of devices and OS combinations you need to test your app against, it’s too expensive and complex to build a testing environment that closely resembles your production environment. Even if you can do this for an early-stage app, wait till you add more features, attract more users, or grow your development team. If you want real-world feedback (and every app needs it) your best bet is to master TiP.

Testing in production is necessary for teams that embrace continuous delivery. CI/CD is different from the traditional waterfall pipeline not just because of the way it facilitates collaboration across teams, but because it breaks releases down to small, uneventful, continuous streams of updates to the app. In the older model, you could test your app till it meets expectations before the next major rollout, but in a DevOps pipeline, you don’t have a major release to plan for. This fundamental change in the way releases happen has a ripple effect on testing as well. Just as releases have become frequent and fragmented, so should testing.

Testing in production may sound like something that happens only after the app is released, but this is actually misleading. The concept of behavior-driven development (BDD) as Ashley Hunsberger wrote about, or test-driven development (TDD) if you will, is integral to DevOps. This means testing is no longer restricted to just QA, but is baked into Dev function right from the start. In this case, testing in production is just a logical extension of the modern QA process.

Apps used to be monolithic, but today, they’re a collection of microservices. A microservices architecture allows each service to be deployed and tested individually without affecting other services. This is done with the help of Docker containers. Docker helps build redundancy so failures are always isolated to a few containers and not the entire app. To keep tests small and decentralized, you need to leverage lightweight, portable Docker containers.

To be proactive about TiP, you should consider implementing something like Netflix’s Chaos Monkey. Intentionally killing parts of your system and building its resilience is the best way to ensure your system can handle testing in production. This will take a different mindset, especially for developers who hate discovering bugs, but it’s what QA should be about — pushing the app to its breaking point to further improve it.

A great way to start testing in production is to have a group of beta testers opt in to test unstable releases. This gives you a lot of room to test critical components safely, and the much-needed confidence to get started. You should, however, strive to go beyond this group to testing in production with your entire user base. Now is the right time to test in production. Teams are ready for it, tools are available, and various methodologies have been vetted. If you’re looking at a future of CI/CD for your team, you can’t ignore testing in production.

Chris Riley is a technologist who has spent 12 years helping organizations transition from traditional development practices to a modern set of culture, processes and tooling. In addition to being a research analyst, he is an O’Reilly author, regular speaker, and subject matter expert in the areas of DevOps strategy and culture. Chris believes the biggest challenges faced in the tech market are not tools, but rather people and planning.