Shift-left testing and continuous testing have both emerged recently as important concepts in the world of QA. What’s the difference between them? Should you embrace one of these strategies as opposed to the other, or can you use shift-left testing and continuous testing together?
This article defines shift-left testing and continuous testing, and identifies the differences between them.
What is shift-left testing?
Shift-left testing refers to the process of moving testing earlier in the development cycle. Shift-left offers tremendous gains when compared to the old order of testing only when the code is complete. Catching bugs earlier is more cost-effective, and saves time and effort—not to mention the benefits to user experience.
However, shift-left testing seems to put the focus squarely on development at the expense of not looking to the “right” enough. Development and testing need to be in sync from the start, but Dev and QA need to also look to the right to participate in the responsibility of the Ops team, who are traditionally guardians of the app in production. So, while shift-left is important when making the transition to DevOps, it shouldn't be an end in itself; rather, it should be followed by a shift-right effort. This approach of covering both ends of the spectrum is known as continuous testing.
What is continuous testing?
Ragnar Lonn writes on DevOps.com that continuous testing “synchronizes Testing/QA with Dev and Ops processes optimized to achieve business and development goals.” While shift-left focuses on integrating QA with Dev, continuous testing goes further to integrate Ops in the equation as well. While shift-left is centered on the software delivery cycle, especially the early parts, continuous testing goes beyond the SDLC to consider larger business goals, and real-world user experience.
Key differences between shift-left and continuous testing
In general, continuous testing delivers more thorough and effective testing results. We can summarize the differences between shift-left testing and continuous testing by saying that continuous testing is better in the following respects.
Enabling end-to-end testing: Shift-left focuses on the pre-production part of the SDLC, whereas continuous testing focuses on the post-production stages as well. Some bugs show up only in production and are caused by unique scenarios after release. The best example of this is mobile devices—The fragmentation in mobile device types and mobile OSes and carrier networks makes it impossible to catch all bugs related to these factors. It takes a focused effort to watch for and track issues with mobile apps in production separately. App error monitoring is vital as mobile app crashes can be elusive, and you may need to set up error reporting specifically for mobile apps. This is very different from how web apps work, and unlike shift-left, continuous testing factors in all these issues.
Achieving earlier insights: Continuous testing is able to gather additional (and more substantial) feedback from latter parts of the software delivery cycle, and relay it back to the initial stages of development and testing. While shift-left encourages better upfront planning, and closer coordination from the start, continuous testing includes these, but also adds in feedback on what is going wrong despite the best efforts at the start. Shift-left takes a top-down approach, which is necessary at the start of ideation and planning. Yet, over time, a bottom-up approach of continuous testing is required for testing to be closely applicable to real-world situations.
Better root-cause analysis: Continuous testing is able to add deeper, more specific commentary on bug reporting, and so assists better in root-cause analysis. A bug may be caused by a changed cloud resource authorization, incompatibility after a version upgrade, or a number of issues which can happen frequently in dynamic container-based systems. These issues would be more easily spotted by continuous testing that doesn't consider testing as “done” once the app is released.
Refined user experience: Shift-left focuses on UI testing before release, but continuous testing takes into account performance metrics, error logs, and actual user behavior when testing the UI in real-world situations. This results in a more refined user experience, as it is based on real usage data.
Being proactive: Shift-left is more proactive than traditional testing approaches that wait for development to be complete, and then begin testing; however, continuous testing is more aware of the obstacles and issues ahead, and is able to factor these in at the beginning of testing. This makes QA teams more proactive, helping them to remember which decisions cause issues downstream and avoid them.
Going all-in with DevOps: DevOps is the combination of continuous integration (CI) and continuous delivery (CD), but a complete approach is not possible without continuous testing. As the frequency of builds in CI and the frequency of deployments in CD increase, so should the frequency of tests in continuous testing. This shouldn't just happen in the initial CI phase alone—It is required even in the later CD phase as well.
Conclusion
As you look to take your testing to the next level, shift-left is a building block, but isn't enough. You need to transition to more holistic continuous testing where tests are frequent, distributed across the pipeline, and real-world usage data is included. As you go deeper in your DevOps journey, don't just shift to the left—instead, shift to continuous testing.
Twain Taylor is a Fixate IO Contributor and began his career at Google, where, among other things, he was involved in technical support for the AdWords team. His work involved reviewing stack traces, and resolving issues affecting both customers and the Support team, and handling escalations. Later, he built branded social media applications, and automation scripts to help startups better manage their marketing operations. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications.
