Testing any software project is an important step in order to find out how the software functions. Learning when the project acts as expected (and when it does not) is the ultimate goal of the testing process. Testing stops design errors from reaching production code. However, testing should not only happen before code is deployed.
Because so much can go wrong initially, Quality Assurance engineers have traditionally set up a staging area to handle testing, in hopes that problems will show up in this staging environment before they can affect users.
However, this staging area is not where users will be involved with the project. The test staging area can mimic, but not authentically recreate, the environment in which the project will actually be used.
Tests that use the production data rather than a staging area will give QA a much better idea of what actually happens when real users start to interact with their projects.
What is testing in production?
Testing in production (TIP) means that new code changes are tested on live user traffic rather than in a staging environment. It is one of the testing practices used by continuous delivery.
Continuous delivery gets code changes into production using tools to automate the deployments. Engineering teams will make changes to their software in short cycles, so that it can be tested and released more frequently.
Benefits of testing in production
TIP hopefully ensures code functionality in the environment that the code features occupy. At the end of the day, no one cares if features work in staging; they care if they work in production. The only way to know if they work in production is to test them in production.
By moving to TIP, QA can also avoid the wasted effort of having to set up a staging area where preliminary testing occurs. Effort is always required to create a staging test environment that reflects the actual use environment. By testing in the place where usage will occur, that sort of unnecessary effort can be eliminated.
Given the dependencies that are present in modern production systems as well as the many possible edge cases, production testing has become a growing part of devops and software testing. Companies such as Google, Netflix, and Amazon routinely test new features (albeit to a fraction of their user base) through the use of production testing.
Should you test in production?
TIP as a method comes with boundaries. To be able to use this approach, the production code affected needs to be resilient enough to rebound from any testing-induced mishap. Poorly documented and fragile legacy code that has ended up being used in production may not be resilient enough to avoid this pitfall.
If the code to be tested is limited in scope, TIP can add to the length of a test. As an example, if only part of the production code is affected by new code, the production code must be brought to the correct state of execution to allow a relevant test. This can increase the number of steps in the testing process that are needed just to attain that relevant code state.
How To Implement TIP
The tool used to implement TIP will usually be either Selenium (for browser UIs) or Appium (for mobile apps). Both are open-sourced with wide developer support.
Let’s start with Selenium. For running an end-to-end verification or the integration test of a specific UI feature, Selenium may be just the thing.
Packages.json is the first specification that will be needed. For Selenium, Chromedriver (Chrome is the test browser here) and a test runner (Nunit) it would look like:
id="NUnit" version="3.8.1"
id="NUnit3TestAdapter" version="3.8.0"
id="Selenium.WebDriver"
id="Selenium.WebDriver.ChromeDriver"
Getting the driver up is fairly straightforward:
public class ChromeTestBase
{
public IWebDriver _driver;
[SetUp]
public void ChromedriverSetup()
{
_driver = new ChromeDriver();
}
[TearDown]
public void TestCleanup()
{
_driver.Quit();
}
}
A utility class can maintain some layers of abstraction from the actual driver and the tests. Here, a utility can find an element via XPath. In the test it can be called with the right parameters instead of worrying about what the XPath looks like each time:
public static IWebElement FindElementByXpath(this IWebDriver driver, string tag, string attribute, string value)
{
return driver.FindElement(By.XPath($"//{tag}[contains(@{attribute},'{value}')]"));
}
A utility class can do just about any repeatable task. It may also contain any type of reusable data. If you plan on using something more than once, create this type of file.
NUnit will need a TestFixture class with some Test tags. It will get the tags from the TF class and run them.
It sounds simple, but each test should do the minimum required in order to test what needs to be tested. Should other steps be needed to get the code into the right state before the actual tests can be run (such as signing in), a utility class or method can be created to perform the repeatable step.
An example of a Test Fixture is:
[TestFixture]
class Testcases : ChromeTestBase
{
[Test]
public void ErrorMessageAppearsNoNavigation()
{
//Arrange
_driver.NavigateTo(Urls.WebsiteBase + Urls.SigninPage);
var startingUrl = _driver.Url;
//Act
_driver.FindElementByClass(PageElements.SignInButton).Click();
//Assert
Assert.AreEqual(startingUrl, _driver.Url);
}
}
Additional tests could be added in a similar manner in any one of the programming languages that Selenium supports.
Testing mobile
Appium can be thought of as Selenium for mobile devices. Notably, it shares Selenium's protocols and methodologies. Appium can be used to test Mobile Native App, Mobile Web App and Hybrid App (Native+Web). Appium tests can also run on multiple mobile platforms. Appium API allows testing on iOS, Android and Windows platforms. It uses vendor-provided automation frameworks for compilation tasks. The supported frameworks are:
For Android 2.3+ Google’s Instrumentation
For Android 4.2+ Google’s UiAutomator/UiAutomator2
For iOS 9.3 and lower Apple’s UIAutomation
For iOS 9.3 and higher Apple’s XCUITest
For Windows Microsoft’s WinAppDriver
Each flavor of the mobile devices will require different drivers and automation frameworks in the testing code, similar to how differing web browsers require different testing code. Because of these variants in code, a full review of them is beyond the scope of this blog. However, the principles used in Selenium TIP carry over to Appium tests.
Tests should be abstracted from drivers for best results. Tests themselves should typically not use driver commands directly.
Test files, framework setup, helpers, etc. should be in different files that reference each other. This allows easy additions to the test repository while avoiding monolithic files.
Appium Client Libraries implement the Mobile JSON Wire Protocol (Mobile JSONWP), which is the mobile-specific extension of the JSON Wire Protocol (JSONWP)) used by Selenium as well as elements of W3C WebDriver specification. The purpose of the clients is to provide an interface to a given language (like Java, Python, Ruby), while hiding the protocol details.
Thus, the structure of Appium tests will be familiar to Selenium users with appropriate substitutions in the code for mobile use cases.
Conclusion
Application testing is a high priority for any software development organization. Most organizations prioritize testing as much as possible prior to production deployments (and rightfully so) to ensure the smoothest transition post-deployment. But production testing can provide some unique benefits for a DevOps organization that should not be overlooked. From preparing the team to respond to disastrous circumstances in production to providing a better user experience for the customer, production testing is becoming more and more of an essential part of application testing as time goes on.
