The way in which applications are deployed, and the infrastructure that runs in production, both shape the way applications are tested. In some cases, such as when you are migrating from the private to the public cloud, there is little to no impact.
In others, however, the consequences are significant. Perhaps the greatest example of a new trend that is reshaping application testing is the rise of containers and microservices. In this article, I explore containers’ impact on functional testing strategy and execution.
Container-based applications for testing
The reason this topic is so interesting is because containers raise the question of whether they should be used beyond development and production, and for testing as well. The idea is that if your application runs on containers, by having a container testing grid, you are minimizing effort and leveraging the benefits of containers for testing. But the key benefits of containers do not relate to testing at all. Their true benefits are size, portability, immutability, and ease of deployment. The following three factors make using containers for a testing grid tricky:
Containers are not meant for GUI: Containers out-of-the-box are not meant to run user interfaces. And without a user interface, containers will be unable to load the browser UI in order to run Selenium tests—and especially will not support screenshots and video recording. It turns out that getting a container to run any sort of UI is a bit of a hack, and the result is that your container ends up looking a lot like a virtual machine, which immediately destroys all benefits of it being a container to begin with. One can argue that now, with Windows containers, running a UI is much easier, which is true. Obtaining a UI on a Windows container is easier, but that also is not its intention. The best Windows containers are running Windows Core or Linux so that they maintain their small size and portability. But if you do use a UI in a Windows container, then you have immediately limited yourself to testing only Windows, and a specific version of Windows and the supported browser.
Containers for testing are disparate from containers for production: There’s a perception that running your testing grid on containers can unify infrastructure. However, you will unify technologies, but not infrastructure. No matter what, the container infrastructure used for your testing grid needs to be different from the container infrastructure used for your production applications, and will likely run in a different location. This means testing infrastructure is managed separately. In most cases, the containers for your testing environment will have completely different lifecycles and orchestration requirements. It is not likely that your Ops team will be excited to take on the task of running two separate container environments with very different setups and maintenance requirements—which means that the Quality Engineering team may have to learn how to manage that infrastructure themselves. This reduces the value of unified technology, and in most cases will increase the amount of effort it takes to support it.
Limited variation support: The most common use case for containers is to run Linux-based applications. This immediately limits the breadth and depth of the type of testing grid you can build. With limited Windows support and no Mac support, you are not going to have the same number of variations in OSs and browsers (not even the most common) which you should be testing your application on.
Trying to leverage containers for something they were not designed for won’t result in gains. In the best case, you maintain, but add nothing, and it will likely introduce either more work, or less testing breadth and depth.
How containers benefit functional testing
This does not mean that the quality team should ignore the fact that containers exist. Containers are not just an infrastructure technology, they are also impacting application architectures. The most notable impact is on applications that are microservice-based. They are also impacting how applications are being deployed, where in some cases, deployments and operational tasks have shifted left to the developer.
Both the application architectures and process will impact how applications are tested. Here are the most obvious ways:
Microservices change when and what you test: In the most advanced microservice-based environments, each service has its own lifecycle. And because the service is a small component of the entire application, you can’t just test the service—you have to test the service in the context of the entire application. From the functional testing perspective, you are typically ignoring the individual services. But testing should still be run as each new service is deployed, which means when you test might change. Organizations might even consider testing their applications in production post-service deployment.
Testing shifts left, and right: QA teams will feel the impact of processes moving closer and closer to when code is created. This means that testing strategies need to support developers’ ability to run more robust tests on their own. And Quality Engineering needs to provide them with the process and test suite to do so. It also means that the idea of continuous testing, and testing in production, is becoming more popular. Shift-left is a side effect of the processes that containers are driving, but it has a huge impact on how testing teams are structured and how they support development and application releases.
Awareness is key
In many environments (perhaps the majority), containers as part of your application will not impact how you test at all. And if you already leverage a cloud-based robust testing grid as a service, like Sauce Labs, you will not think about infrastructure at all. However, how processes and deployment strategies are changing with infrastructure could impact what you do—in what you test, how frequently, and how you support your developers with a robust testing strategy that prevents QA from being a bottleneck, therefore boosting the quality of the application.
Containers may not be a good solution to run your functional tests, but they could impact how you test and how you build your testing strategies, in both direct and indirect ways. Quality Engineering should be aware of and understand how containers are being used in their application, and prepare for the potential impact. The best thing any testing team can do is forget about testing infrastructure altogether by deferring it to a cloud-based testing grid, and focus on decreasing effort to run tests, and increasing the depth and breadth of test suites.
Chris Riley (@HoardingInfo) is a technologist who has spent 12 years helping organizations transition from traditional development practices to a modern set of culture, processes and tooling. In addition to being a research analyst, he is an O’Reilly author, regular speaker, and subject matter expert in the areas of DevOps strategy and culture. Chris believes the biggest challenges faced in the tech market are not tools, but rather people and planning.
