Imagine owning a car with a 25% failure rate. Would you trust it to get you to work each morning, that you would always make important appointments on time, or that you and your family could reach a vacation destination hundreds of miles away without getting stranded? With that level of unreliability, you’d be lucky if it held together just long enough to drive it to the junkyard.
But when it comes to automated testing, it’s not unusual to see testing failure rates approach 25% or more. Without a strategic approach to test automation, you’ll just get a higher test failure rate and an inordinate amount of noise.
Noise increases the risk of missing critical failures. Imagine that you have 800 tests, 200 of which are failing. You haven’t had time to analyze them, because new features are coming in and you have deadlines. After the next test run, the number goes to 201. The number’s been creeping up for a while, so do you even notice it? Unfortunately, failure #201 indicates that the connection to the credit card processor is broken–probably the most critical defect in a retail setting. How would that failure rise above the noise of the other 200?
Understanding why test failures happen
Let’s look at the chief culprit for high test failure rates: while writing and running tests takes resources, maintenance tends to dwarf the combined costs of creation and execution. If you don’t update your tests to reflect changes in the app that may inadvertently impact the test–moving a button, changing a workflow, making a field required that wasn't required before–you now have a test that is designed to fail.
While many tests can be updated with no more than a few minutes of work, minutes can add up. If you don’t fix all the new test failures today, and you get a new batch tomorrow and the day after, and so on, you’ll naturally start to fall behind. Every new test adds to your burden, and if the site or app still generally works, then it becomes even easier to ignore test failures–until you miss the one test failure that makes your app inaccessible, ruins the customer experience, causes the business to lose revenue, and invites regulatory scrutiny. Yikes.
To improve your signal-to-noise ratio, you need fewer false test failures so you can spend more time fixing bugs instead of tests. Here are four best practices you can use to optimize your test automation strategy.
1. More isn’t better
If you test something unimportant and then get a test failure, the time it takes to investigate the failure will take away time from identifying and fixing critical bugs. Each additional test increases the potential for a test failure, creating more noise to sift through.
Instead of trying to achieve 100% test coverage for every feature–a mile wide and an inch deep–you should instead focus on running the right tests all the time on all the devices your customers use most.
Prioritize your testing on the critical user journeys and touchpoints that customers spend most of their time on or are tied directly to revenue so that when you do find a test failure, you can be sure it is worth the time and resources to investigate.
Action: for every test you write, ask yourself, “if this test fails and reveals a bug in this feature, will we stop the release?” If the answer is “no”, don’t put this test into the critical path of a build. Keep it, and run it separately as part of a low-priority check. This does NOT mean you get to ignore it if it fails.
2. Only run tests that currently deliver value
The fewer tests you have, the more often you can run them. In addition to creating tests for new features, don’t forget to keep an eye out for existing tests you can delete.
Test code is every bit as important as production code. The moment you stop treating it as such, you start to let quality slip, and worse: you devalue your own work
Action: Conduct regular test audits to identify and remove outdated or redundant tests that no longer add value to your organization. If a test isn’t providing useful feedback, it’s time to let it go. Just as you leverage a continuous testing approach to improve your code, you should use a continuous improvement approach to ensure your testing strategy evolves along with your app.
3. Write atomic tests
Teams get in trouble by trying to test too much at once. When you do that, a testing failure can’t provide the insight you need to know what needs to be fixed or how.
Instead, leverage ‘atomic’ tests, which focus on testing one specific action at a time in isolation. This makes the test less likely to fail due to unrelated issues and dependencies, making it easier to debug, maintain, or delete if it no longer drives value.
For instance, instead of testing a complete checkout process in one test, break it into smaller steps: adding an item to the cart, applying a discount code, and completing the payment. Each step is tested independently to pinpoint the failure's source. By making your tests autonomous from each other, you can also run them in any sequence or in parallel.
Action: Go through your maintenance-prone tests. For each one, identify a single question the test is designed to answer (these are based on an e-tailer example).
Does the checkout icon accurately reflect the number of items in the cart?
Does the product information page contain a price?
Does the shipping information auto-fill for logged-in users?
I can easily imagine an automated script that attempts to capture all three of the questions above. Do you tests attempt to answer multiple questions? If so, separate them out!
4. Keep tests short
Atomic tests are faster to execute, making them less likely to encounter external issues like timeouts while enabling faster feedback loops. You’ll also be able to run more tests in the same amount of time, increasing the odds that you find a critical bug sooner. As a rule of thumb, aim to keep your test execution time under two minutes.
Action: Similar to above, take a random sampling of the tests you depend on. For this sample, identify the three activities a test needs to perform:
Arrange: assemble the data necessary in order to ensure the primary question of the test can be answered (create users, add items to the cart, etc – ideally you’d do this via API)
Act: Perform a single action (based on the “atomic” recommendations above)
Assert: Gather precisely enough data to answer the primary question of the test
There’s always room for improvement
Making the leap to test automation is only the first step. It takes diligence to ensure your test strategy keeps pace with the needs of your business. By shifting your focus away from brute code coverage and towards more focused testing, you can shift the noise-to-signal ratio back in your favor. For tips on automated your testing process make sure to read through 7 Automation Best Practices for Better Testing in 2025.
