Using BDD To Automate Testing Of Single Page Apps (SPA) In Clinical Trials

We test our e-Clinical Web Apps on the four major browsers: Chrome, Firefox, Safari and Internet Explorer on Linux, Mac OS X and several versions of Windows as well as Apple's mobile iOS. These days maintaining an in-house Server farm that spins up VMs with various OS and Browser combinations is costly, time consuming, and cumbersome. Using the Sauce Labs platform for cloud-based testing made these problems go away and let us focus on quickly testing mobile and web apps on whichever platforms we wanted our product to support. It makes running, debugging and scaling test suites easier, saving us both time and money. Sauce Labs also lets us run tests in parallel, considerably speeding up our test suites.

Sauce Labs features such as videos and screenshots help us identify issues faster and are essential for the auditability of our engineering process. Each software version is accompanied by documentation and videos that show evidence of testing and business control of the SDLC, both essential for releasing apps in a regulated environment.

Of course we had technical issues along the way. Who doesn't?! For example, we found that Linux browser windows are much smaller than in Windows, and there were networking issues. But the Sauce Labs team support was really helpful in resolving them.

We write tests in behave / behaving, a BDD (Behaviour Driven Development) technique and web application testing framework with a Gherkin-based language (similar to Cucumber, Lettuce, and Freshen) that is backed up by Python code. Below is an example scenario (taken from the examples showcased on the behaving website

We contribute back to the community with the test steps we develop and keep them as general as possible rather than making them app-specific so that others can benefit too.

We use JIRA to track User Stories and Bugs as well as Confluence for team collaboration and release documentation.

We use Mercurial to store and version our Test scripts in Bitbucket.This is obvious from an engineering perspective, but critical in the regulated industry where we operate as it helps us providing traceability from requirements through to the Scenarios used to test our releases.

We write tests in Behave Pro, a JIRA plugin which allows us to link each JIRA issue to at least one Test Scenario, to manage our tests and provide requirements traceability.

We export and version the Scenarios into Bitbucket and then finally, pass them to Sauce Labs for execution on various OS / Browser combinations.

Like Sauce Labs, Behave Pro's support team was very responsive and helped us out with the technical issues we've encountered.

We continuously evaluate and improve the testability of our Web Apps. Often this is as trivial as adding CSS classes or HTML IDs to elements. Then there are the other, more challenging cases, like testing form fields that POST their values in response to blurred / focus lost user events.

We review every test scenario to make sure that the test coverage is correct, readable, and easily understandable. We use lots of CSS and jQuery selectors and absolutely love it when these are short and read like natural language.

Any test steps that touch third-party apps (such as OAuth2 providers) we package in macros or custom BDD steps. This isolates test suites from any changes that the third party makes to their app. For example, when Google decides to change their OAuth2 login screens.

We have created a couple of custom BDD steps to give us macros so that sequences of steps can be reused and even take parameters. Macros are created within our BDD feature files just like any other scenario. Each web app has it's own library of macros which we've defined in a feature file distinct from the feature files of the test suite. The macro library is tagged with @MACRO.

Creating Macros relies on an I define "macro_name(variables)" custom step as follows:

[code]@MACRO Feature: Test macros library Scenario: Define macro When I define "set_date(field,day,month,year)" as """ When I set "$field .day" to "$day" And I set "$field .month" to "$month" And I set "$field .year" to "$year" Using Macros is as simple as an I execute "macro_name(values)" custom step, for example: Feature: Test macros Scenario: Execute macro When I execute "set_date(#page .field__birthdt, 19, 02, 1983)"[/code]

Macros are defined within a “.feature” file on its own and are available across all of our tests.

Tests are alive! We have a dedicated Test Team and we found that they are surprised and may become frustrated when perfectly good tests start failing because developers have implemented a change in the app. Be prepared and allocate time to update the tests to match the app changes as such breakages will happen with absolute certainty. Changes are inevitable to some degree but when these do happen, make sure you have countermeasures like macros or data-driven tables to support you in easily maintaining your tests.

Developers need to be asked to consider the testability of web apps. At first it is not natural for them to add CSS classes and IDs to HTML elements to reduce the length of CSS selectors used by tests.