In a perfect world, you'd have time to write an automated test for every piece of code that you add to your application. In the real world, that's not feasible. You have to be strategic in selecting what to automate and what not to automate.
You will also find yourself having to reevaluate your automated testing strategy from time to time. As your applications change, your testing strategy should change, too. The automated scripts and tools you were using may no longer be the best fit.
With these challenges in mind, this article discusses how to decide what to automate in the first place. It then explains best practices for keeping your automated testing strategy up-to-date as your applications evolve.
Software Testing: When to Automate, and When Not To
When developing an automated testing strategy, you quickly realize that it is not reasonable to write automated test scripts for every component making up your application. There would simply be too much overhead in terms of designing and developing these scripts and then maintaining them as your application changes over time.
So, instead of trying to automate testing for everything, you need to choose which tests to automate in an effort to make your continuous testing strategy as effective as possible. This decision starts with analyzing the feature you intend to test in order to determine if automated testing is the correct route in a particular situation.
For each individual feature, you should ask yourself a few simple questions in order to determine if automated testing is necessary. The first involves the importance of the component to the overall functionality of the application. It goes without saying that some components of an application are more important than others. If the component drives major functionality for the application, then it most likely requires automated testing to ensure with each build that the functionality is preserved. On the contrary, if the component does not contribute to significant functionality, then it may not be necessary to develop automated test scripts to verify that it’s in working order with each commit.
Another aspect to consider is the duplication of testing within your automated testing suite. Some features by the very nature of their function may be tested already by a previously developed script. Maybe the feature utilizes components from other parts of the application where automated testing has already been implemented.
It’s important that the developers recognize that test coverage may already exist in this case, and avoid cluttering the testing suite with duplicate testing. This will help to steer clear of the frustration that can arise due to a build failure where multiple tests need to be analyzed despite the fact that they failed due to a common cause. In addition, avoiding this type of duplication will limit maintenance associated with your testing suite. When a change to a component occurs, you will only have to make changes to one test rather than many.
Change is Inevitable
Over time, your application will evolve. And with this evolution, your testing suite will be forced to evolve as well. It’s possible that prior to an application change, a feature didn’t require automated testing, but now it does. Or the opposite could be true. Maybe after an application change, there are scripts that exist as part of your automated test suite that are no longer necessary.
What questions can we ask ourselves to make this determination?
Did a component just become a lot more complicated? One of the things to consider is the amount of time it takes to test something manually and the propensity for human error when performing manual testing. If a component of your application was altered in such a way that makes manual testing more complicated than it is worth (or even just a tedious process for the tester), then it might be time to break out your scripting skills and automate testing.
Was a component rendered obsolete or was its impact significantly reduced? It’s possible that a major refactoring of existing code resulted in a particular component being utilized less frequently, or even removed altogether. In the latter case, you can simply remove the associated automated test script. In the former case, where a component is used less frequently, you should still test it automatically if possible, but you may want to reconsider the priority that the component gets in your automated testing routine. Perhaps your test script for that component no longer needs to run as often, for example.
Tips for Managing Your Automated Testing Suite
Actively managing your testing strategy will ensure that it remains effective. Consider the following tips to manage your automated testing suite:
Perform analysis to ensure test coverage is adequate when making modifications. Analysis of automated test scripts should be performed related to modified components. It is not enough to just see if the current test passes on commit. You must make sure that the test still fully tests the component when the code backing it is altered. This will require some manual effort, but you can also look at data such as failed tests, or how long a test takes to complete, in order to get a sense of whether code changes have impacted the behavior of your test.
Design your test scripts to be independent from the UI. The user interface, or UI, is the part of your application that is likely to change the fastest. Test scripts should be able to withstand changes to the UI when the backing component that provides the functionality isn’t changed. You therefore want to make sure that your automated tests do not depend on a UI being present in order to run. (See this article for other design considerations for your test scripts.)
Retire obsolete test scripts. Take some time after each release to see if there are any test scripts that can be retired, and remove said test scripts from your automated testing process. If you fail to do so, you will only serve to lengthen the amount of time it takes to test each build, and you will probably end up wasting time debugging a test or two that isn’t even necessary! On a tight delivery schedule, time will likely be limited. You won’t want to be wasting any of it on things that don’t require your attention.
Conclusion
Your applications are not static beasts. They will change over time, requiring your team to remain aware of how each change impacts your testing strategy. By honing your skills in test management, you can ensure that your testing strategy remains effective and ensures a high level of application quality.
Scott Fitzpatrick is a Fixate IO Contributor and has over 6 years of experience in software development. He has worked with many languages, including Java, ColdFusion, HTML/CSS, JavaScript and SQL.
