7 Reasons Your Testing Strategy Hasn’t Reduced Your Bug Count

Bugs happen. Totally preventing bugs is not possible.

But it is possible to manage bugs with the help of software testing. The goal of any software organization is to have an automated testing strategy in place that allows all bugs to be identified and fixed before they are released into production.

If you have been pursuing that goal, but are still falling short of preventing bugs in production, it might be time to reevaluate your testing strategy. This article identifies common reasons why your testing process may not be as effective at catching bugs as you’d like.

Code coverage metrics are frequently used to measure how well-tested an organization’s code is. And having high test coverage metrics is a worthy goal for any organization. I was recently impressed when a young developer on my team increased the code coverage on a project by over 25%. That feeling quickly vanished when another engineer pointed out that none of the new test cases validated expected outcomes, and we had a productive team discussion on the difference between covering code and testing code.

Code coverage is a useful metric if you pair it with mutation testing, ensuring that test coverage is both effective and thorough.

Coding standards help ensure that all engineers on a project create code which uses a similar style and formatting. Standards improve readability of code, the creation of code reviews, bug fixing, and overall maintenance easier for developers.

Code standards by themselves won’t reduce your bug count, but they do help ensure that your project maintenance is more straightforward when bugs occur.

When we think of combining coding and testing, often, the result is that the development team will take a break from development activities and spend a few hours feverishly testing the application and finding as many bugs as they can. A rash of bug fixing follows, and then the team returns to regular coding activities.

This approach may temporarily uncover bugs and create a dip in the number of defects in the code, but it won’t have a marked effect on the development of high-quality source code for the long term.

Code reviews are one of the most effective ways to ensure that your team is producing code which meets requirements and mitigates potential problems. Unfortunately, when deadlines approach and stakeholders are getting anxious, the code review seems to be the first casualty of war.

If you asked an engineering team for a show of hands on who would like to develop new functionality and who would like to refactor existing code and improve it, you probably already know what the results will be. Engineers like the allure of new frontiers, rather than wading through someone else’s code. Teams which strike a balance between new development and maintenance of existing code will see benefits in both the maintainability of existing code and the number of bugs contained therein.

Refactoring code also encourages engineers to think about writing unit tests in such a manner as to support these activities and ensure that optimization efforts don’t break core functionality.

As an engineer, I used to believe that the most valuable person I could have on my team was a diligent and conscientious QA engineer. The reality is that this paradigm resulted in me paying less attention to the quality of the code because my QA person would find any mistakes.

Integrating QA and Development activities encourages engineers to think through their solutions thoroughly and consider test cases as part of the product. Integration fosters a sense of ownership among engineers as well, creating a transition from an “engineering vs. quality problem” to “our problem.”

We just talked about the problems with traditional testing. Once you’ve combined the processes of developing and testing, you need to facilitate the automation of all testing activities and execute your automated test suite as often as possible. Foster an environment where the entire test suite can be executed:

Finally, enable testing against your production environment. Thorough and automated test suites will catch a lot of bugs before they reach production, but you want to make sure that your production environment continues to operate optimally and remains available for all consumers.

A genuinely effective test strategy becomes part of the development culture in your organization. Engineers should be enabled with the tools to effectively test their solutions and should be encouraged to automate testing wherever possible. Fostering a culture of ownership and continuous improvement will yield exponential dividends in your quest to reduce your bug count.

Mike Mackrory is a global citizen who has settled down in the Pacific Northwest - for now. By day he works as a Senior Engineer on a Quality Engineering team and by night he writes, consults on several web based projects and runs a marginally successful eBay sticker business. When he’s not tapping on the keys, he can be found hiking, fishing and exploring both the urban and the rural landscape with his kids. Always happy to help out another developer, he has a definite preference for helping those who bring gifts of gourmet donuts, craft beer and/or single-malt Scotch.