Using Exploratory Testing to Drive Automation

Traditionally automation and exploratory testing have been seen as independent tasks that work in different ways and normally do not appear to be ideal partners. This article suggests that there is value in utilizing the work done when automating to add some exploratory testing effort.

In most cases there is some form of documentation or communication which indicates what the system being designed will do. This can be in the form of requirements, design specification, email, design meetings and so forth. An assumption has been made here that testers are involved in the design phase, if they are not for your project then they should be.

Within the context in which I work these documents/communications are analyzed for automated cucumber scenarios and exploratory charters. For the automation scenarios these are created using the Gherkin format:

Feature: Password management
Scenario: Forgot password
Given a user with email "someuser@example.com" exists
When I ask for a password reset
Then an email with a password reset link should be sent

Exploratory charters are created in the Exploratory Charter format as described by Elisabeth Hendrickson in the book Explore It!:

Explore (target)
With (resources)
To discover (information)

Where:

·                  Target: Where are you exploring

·                  Resources: What resources will you bring with you

·                  Information: What kind of information are you hoping to find?

Once the exploratory testing charters and cucumber scenarios have been initially defined the Test Engineer manually steps through the cucumber scenario to determine the pre-requite components needed to complete the scenario from an automation perspective, at the same time adding details to the Gherkin feature file. This may involve building the environment and integrating the components. It is at this point that some level of exploratory testing can be performed to gain more value from this manual effort. This manual effort can be recorded on a wiki or some other similar tool. The suggestion for using a wiki is the ease of access and use for cross regional teams. It is not mandated for teams to us a wiki but from experience it appears to be the best way to share information.

This will require a mindset shift for the test engineer instead of only focusing on the automation scenario they should be looking for future opportunities to test either automated or manually and these should be fed back to the team and added as backlog items. One other benefit is that as you explore the automation scenario you may uncover issues or defects earlier which are always a good benefit for the project.

Once the step definitions have been coded and implemented for the automation scenario they should feed into whatever automation reporting system you have selected. This should report scenarios pass or fail. It is expected that defects should not be found at this stage and if scenarios fails it is indication that your expectations of what the system does has changed and should be investigated before any possible defects are raised. This is a simple approach to close the link between test automation and exploratory testing and leverage the skills of the test engineers to their full value.

It should be noted that there will still need to be more effort to ensure that the exploratory charters are run manually and recording using the current approaches for your projects.  This then become a continuous cycle as more scenarios are discovered for automation then more exploratory testing can be done and more information about the system can be uncovered.  This approach makes exploratory testing and automation partners rather than as independent entities.  It hopefully reduces the checking vs testing debate and focuses on utilizing the skills of testers to deliver a product that customers will enjoy using.

The diagram below is a simple visually representation of this concept.