In a perfect world, you’d be able to test everything, all the time. In real life, however, you’re often faced with huge products and large features, and you just can’t test them all. If you try, you’ll go bankrupt.
So, what do you do in the face of this challenge? The answer is to take a risk-based approach to testing. This process starts by talking about Risk Analysis with your scrum team.
Below, I explain what a risk-based approach to testing entails, and how you can execute it when faced with a huge amount of products and features to test.
Introducing the Risk Analysis Worksheet
The Risk Analysis Worksheet is a tool to guide prioritization of tests in order to reduce project risk. The idea is to facilitate conversation as a team, understand risk, and agree which tests will be maintained going forward and how they will be written. This should not be done by the “Tester” in isolation. The worksheet (that my teams use) is derived from Rex Black’s Managing the Testing Process, but we have simplified it to get teams started.
The heart of the tool lies in the Likelihood, Impact and Risk Priority # columns. Likelihood is an indicator of the probability that a given workflow will have a problem, Impact is a measure of the probable severity of the problem, should it materialize, and Risk Priority # is the product of Likelihood and Impact, which produces a number that indicates risk exposure (with a low number representing greater risk).
Now that we understand how the risk analysis worksheet works, let’s take a closer look!
Identify Potential Risks
To get started, the team should decide on the quality risks, or potential scenarios that could present a risk. Dependent on the feature, the total listed will vary. For example, a less complex feature may see only a few rows, while a more complex feature can have over two hundred. If you are working on an existing feature, sometimes it is helpful to list the existing tests you have (for example, unit, integration, E2E, manual), and add more if the team thinks things are missing.
Understand the Likelihood
Now, for each quality risk, it’s time to discuss the likelihood of a bug occurring in that area. We use a scale of 1-4, 1 being the most likely, 4 being the least. You can use whatever you want, but this has worked well for us to keep it simple for now.
Some questions to ask when calculating likelihood are:
How often do bugs in this test area occur? Using bug counts by component or summary search, you can get an idea of what has historically been problematic.
How often was a piece of code modified? If the same file was modified multiple times by more than two developers, there is a statistically higher chance that a bug will be introduced.
How complex is the code base? If the team deems that this code base has lots of complexity and points where it could fail, then the likelihood would be higher.
Evaluate Impact
Next, the team should evaluate IMPACT. What is the impact if a bug does occur?
Questions to ask when calculating this:
Is there a workaround available? Will our client be dead in the water if this doesn’t work? Or, is there another way they can achieve their goal?
How much of the customer base is affected? In my world at Blackboard, opening a course affects everyone from students to teachers. A possible ornamental setting buried deep in a feature doesn’t.
How critical is the area we are testing? If the bug was found in an area of such high importance, would we ship it even if it was marked as low priority? We get this from client data to understand what our users are doing in the system the most.
What’s the Risk?
Risk Priority # is the product of Likelihood and Impact, which produces a number that indicates risk exposure (with a low number representing greater risk).
As a team, you decide the threshold, and agree what you will test and maintain from here. For example, most of our teams tend to go with 6 and lower. However, since grading is such a critical feature, we may agree to set the threshold at 9 to include a bit more maintained coverage.
Decide How to Test
Now that you have agreed upon your risk threshold, discuss as a team how these risks can be tested. As always, aim for more lower-level tests, with fewer UI-based tests.
One key point: For every type of test indicated, you need to know how those tests are monitored and maintained, and by whom.
Some Closing Thoughts…
Do note that this is meant as a guide and a tool to help you understand what risks are present, and how to use them to drive testing.
Some things that may help along the way:
It's important to get a baseline. Find a test that constitutes a likelihood of 1 as well as a test that constitutes an impact of 1. That will help you compare your other risk items in the suite. Having a good visual for different priority values helps in making future judgements. A good scale example is the Modified Mercali Intensity Scale for measuring earthquakes.
Testing the most important tests vigorously is better than trying to test everything with less vigor. Use pairwise testing when possible, as it's been proven to effectively give you the best coverage for your time.
SO far, we have found this tool to be incredibly useful as we aim to test smarter, and know we cannot test everything. Don’t test all the things, test the least amount of things you can, in the smartest way possible.
SauceCon 2017 is right around the corner. Join us in San Francisco from June 6-8 for the first-ever Sauce Labs user conference. A three-day event filled with training, workshops, best practices, and visionary content from the leading minds in automated testing. Tickets are still available.
Ashley Hunsberger is a Quality Architect at Blackboard, Inc. and co-founder of Quality Element. She’s passionate about making an impact in education and loves coaching team members in product and client-focused quality practices. Most recently, she has focused on test strategy implementation and training, development process efficiencies, and preaching Test Driven Development to anyone that will listen. In her downtime, she loves to travel, read, quilt, hike, and spend time with her family.
