Using Exploratory Testing to Drive Automation

Traditionally automation and exploratory testing have been seen as independent tasks that work in different ways and normally do not appear to be ideal partners. This article suggests that there is value in utilizing the work done when automating to add some exploratory testing effort.

In most cases there is some form of documentation or communication which indicates what the system being designed will do. This can be in the form of requirements, design specification, email, design meetings and so forth. An assumption has been made here that testers are involved in the design phase, if they are not for your project then they should be.

Within the context in which I work these documents/communications are analyzed for automated cucumber scenarios and exploratory charters. For the automation scenarios these are created using the Gherkin format:

Feature: Password management
Scenario: Forgot password
Given a user with email "someuser@example.com" exists
When I ask for a password reset
Then an email with a password reset link should be sent

Exploratory charters are created in the Exploratory Charter format as described by Elisabeth Hendrickson in the book Explore It!:

Explore (target)
With (resources)
To discover (information)

Where:

·                  Target: Where are you exploring

·                  Resources: What resources will you bring with you

·                  Information: What kind of information are you hoping to find?

Once the exploratory testing charters and cucumber scenarios have been initially defined the Test Engineer manually steps through the cucumber scenario to determine the pre-requite components needed to complete the scenario from an automation perspective, at the same time adding details to the Gherkin feature file. This may involve building the environment and integrating the components. It is at this point that some level of exploratory testing can be performed to gain more value from this manual effort. This manual effort can be recorded on a wiki or some other similar tool. The suggestion for using a wiki is the ease of access and use for cross regional teams. It is not mandated for teams to us a wiki but from experience it appears to be the best way to share information.

This will require a mindset shift for the test engineer instead of only focusing on the automation scenario they should be looking for future opportunities to test either automated or manually and these should be fed back to the team and added as backlog items. One other benefit is that as you explore the automation scenario you may uncover issues or defects earlier which are always a good benefit for the project.

Once the step definitions have been coded and implemented for the automation scenario they should feed into whatever automation reporting system you have selected. This should report scenarios pass or fail. It is expected that defects should not be found at this stage and if scenarios fails it is indication that your expectations of what the system does has changed and should be investigated before any possible defects are raised. This is a simple approach to close the link between test automation and exploratory testing and leverage the skills of the test engineers to their full value.

It should be noted that there will still need to be more effort to ensure that the exploratory charters are run manually and recording using the current approaches for your projects.  This then become a continuous cycle as more scenarios are discovered for automation then more exploratory testing can be done and more information about the system can be uncovered.  This approach makes exploratory testing and automation partners rather than as independent entities.  It hopefully reduces the checking vs testing debate and focuses on utilizing the skills of testers to deliver a product that customers will enjoy using.

The diagram below is a simple visually representation of this concept.

The Size of the Automation Box

I have previously written an article about doing too much automation and have recently been involved in successfully implementing automation and exploratory testing in agile environments.  During discussions about how to implement the automation framework a work colleague made the following statement in regards to defining the automation boundaries:

“Defining the size of the black box is critical”.  

 It made me think about the importance of knowing where your automation boundaries are and how critical it is for successful deployment of an automation framework.   One point to keep in mind when you try and implement an automation framework is the principal of K.I.S.S (‘Keep it simple stupid’)

I have seen (and been involved in) several failed automation implementations and in retrospect the main reason for failure has been trying to automate everything that you can possibly think to automate. I discussed this problem in the article too much automation and the importance of thinking critically about what to automate. Time and time again people go rushing ahead to try and automate everything they can without thinking about the boundaries of what their automation framework should be covering. 

 So what is the solution? The rest of this article will offer some guidelines that may help you in your quest of implementing a successful automation framework

The first thing to look at when trying to implement automation is the context of the testing you are carrying out or attempting to automate.  Are you involved in sub system testing, component testing, end to end testing, customer driven testing or any other type of testing?  It is important to look at this and found how what do you want the automation to do for you.  

• At Customer level you may want to automate the common customer scenarios.
• At a system level you may want to validate the input and outputs to a specific set of components and stub out the E2E system.

So defining your context of the purpose of your testing from an automation viewpoint is important.  The reason for this is that links to the second point, defining the size, or scope, of your automation for the project.  Most automation implementations fail to take into account the limits of what you intend to cover with the automation.  As my colleague put it 

The size of this will change depending on the context of the type of testing that you are carrying out.  For your automation implementation to be successful it is important that you define upfront your boundaries of what the automation is expected to cover.  Automation should be about what you input into the system and what you expect as an output everything in-between this is your black box .  So knowing what your testing context is should help define the size and limits of your black box knowing this information before you start to implement a test automation framework should greatly improve the chances of your automation implementation being successful.

Example of defining black box at a Component level of testing

    

Shaded Gray Area = Black box

For this example the boundaries would be

• Create automated test inputs for Input A in relation to defined requirements and expectations
• Automate expected outputs Output 1 and Output 2 based upon requirements and your expectations
• What happens inside of Component X is of no interest since this is outside the boundaries of what we are automating

Another example of defining black box at a Component level of testing

    

• Create automated test inputs for Input B in relation to defined requirements and expectations
• Automate expected outputs Output 1, Output 2, Output 3 and Output 4 based upon requirements and your expectations
• What happens inside of Component Y is of no interest since this is outside the boundaries of what we are automating

Example of defining black box at a Product level of testing

    

• Create automated test inputs for Input A and Input B in relation to defined requirements and expectations
• Automate expected outputs Output 1, Output 2, Output 3 and Output 4 based upon requirements and your expectations
• What happens inside of Component Y is of no interest since this is outside the boundaries of what we are automating

I am aware that this appears to be common sense but time and time again people fail to take on board the context of what they expect the automation to be covering.

So next time you are involved in implementing an automation framework remember to define the size of your black box before you start your implementation that way you know the boundaries of what your automation will cover and check for you.

Defining Testing

I am about to run a couple of internal workshops on the Exploratory Testing approach which is based upon a lot of work done by Michael Bolton and James Bach. One of the concerns I have been having recently is what people within the organisation think testing is in comparison to what they actually doing. So I started to put together an article looking at these concerns and trying to see if there is a problem. This blog is based upon some of the points that I cover in the article.

WARNING

Disclaimer:

The views and definitions expressed in this article are my own and as such they may not match what a dictionary may say or agree with your views/definitions.

When I start to look at what we see as testing activities they appear to fall into three distinct categories:

• Validation
• Verification
• Testing

These terms may be familiar to some of the older readers of this blog. V V & T has been around for a long time and has it origins within the manufacturing industry. It has been the main process to provide quality control and assurance of manufacturing production lines. (http://en.wikipedia.org/wiki/Verification_and_validation)

It appears that these ‘manufacturing’ processes have been applied to software testing (http://en.wikipedia.org/wiki/Verification_and_Validation_(software))

This seemed to have lead to the appearance of process standards initially the ISO 9000 Quality Assurance standard, which was modified to become the ISO 9001 2008 standard which included software. These standards are very closely linked to manufacturing process and from a software testing perspective the quality control methods.

Talking to and observing various companies I have seen that lot of people’s perception of testing is as shown in the photo below.

http://brigitteofseon.wordpress.com/category/work-hard-no-go-slow/

Is it a problem to have this perception of software testing?

At the beginning of my career in software testing a lot of companies started to change from mainly hardware manufactures to both hardware and software manufactures. There was a need among these companies to have processes they could use to prove the ‘quality’ of their software products and the general consensus was what had worked in quality control for hardware surely could be applied to software testing.

The reasoning behind these was based upon some fairly flawed assumptions:

• All software was the same
• All software worked in the same way
• All users would follow the designed work flows.
• All users would behave in the same way

The main focus of these processes was to validate and verify what was already known about the product and its expected inputs and outputs. In my opinion following quality control and assurance processes is not really about testing Testers ‘normally’ do not control the quality (Yes there are approaches such as TDD which ‘may’ help). If there is crap in the system you are testing then there will be still crap in the system afterwards. Testers provide a service telling you there is crap in the system. Michael Bolton talks more about getting out of the QA business here

Validation and verification

will in the majority of cases NOT

Tell us anything new about the product

Make us ask questions of the product

So what do I mean when I talk about validation and verification?

Validation:

To me validation is about proving what you already know about the product. Confirming what the requirements say are correct and that the system is correct in accordance to what you believe it should be. The normal response when validating will be:

• true or false
• yes or no
• 0 or 1

I see Validation as a checking exercise (See article by Michael Bolton here on testing v checking) rather than a testing exercise, it still has some value within the testing approach but it will not tell you anything new about the system being tested. It will prove that what you already know about the system is correct and working (or not working) this is like testing requirements or validation of fields in a database/GUI – you know what the input is you know what outputs you expect according to specification/requirements so why not automate this?

The majority of ‘testing’ I see happening is validation and even though it has some value I would not count validation as testing since it does not tell you anything new about the system you are testing.

It should be noted that to interpret the results from validation ‘testing’/checking requires human interaction to work out if what happens what the correct expected response.

Verification:

When I look at the term verification I use it for when we are verifying any bugs that have been previously found. Someone has made a change to the product and I want to verify that the change made has fixed the problem I had seen before. Some verification tests can be automated – for example if you have run a test previously and found the problem you may be able to automate the steps you followed from testing so that you do not need to run the same test again.

Testing:

I see testing as a thinking exercise in which you need a person to use their skills (and brain) to ask questions of the system being tested. From asking this question they learn more about the system and its behaviour. They will not know the answer to the question but by investigating and tinkering with the product they can form some reasonable answer to the question they posed.When testing we act like crime investigators – you suspect foul play, but you need to ask questions and gather evidence to back up your theories and provide answers to your questions. Testing is not based upon requirements or specification but rather when the specification and requirements are not saying.

Testing is about asking the

• What
• Why
• How

Nassim Nicolas Taleb came up with the following interesting quote:

We are better at doing than learning. Our capacity for knowledge is vastly inferior to our capacity for doing things – our ability to tinker, play, discover by accident. http://www.fooledbyrandomness.com/notebook.htm

So after all of this is there really a problem?

Some of the problems I see within the software testing industry are:

• We spend far too much validating rather than testing
• We repeat the same validations (manually) time and time again
• Cover less of the system by only repeating the same validations
• Testing is checking exercise rather than a testing exercise
• Testers are not being engaging
• Testers are not being challenged
• Testers do not need to think
• People see testing as a boring thing to do
• Testers if used to manually validate are seen as robots

What can be done to improve this?

• Look to automate the validation (checking stuff)
• Improve coverage by changing the data sets used in validation
• Start to use exploratory testing approach – attend a rapid software testing course
• Look at using Session based testing
• THINK engage your mind and question the system.

We need to keep learning about testing and do more testing rather than keep repeatedly validating systems, it then becomes a much better and challenging role to be a tester.