Interviews and Videos

I thought I would start the new year with a brief post about what I have been up to over the holiday period.

I was approached by the editors of Teatime with Testers magazine seeking permission to publish my current series of posts on testing skills - the first one has been published and you can download the magazine for FREE from here - http://www.teatimewithtesters.com/ .

During December Matt Young got in touch with me from the company Functionize to see if I would be willing to do a podcast with their CEO Ray Grieselhuber about "Why automation will not replace manual testing."  This ended up being a really good discussion in which everything from why automation can be useful, to a philosophical debate about what being human means.  We ended up chatting for so long that the article had to be split into two parts.  The podcast and articles can be found at the links below.

• http://www.functionize.com/blog/John-Stevenson-Test-Automation
• http://www.functionize.com/blog/Automated-Testing-John-Stevenson-Ray-Grieselhuber
• https://soundcloud.com/functionize/john-stevenson-on-test-automation - Podcast

During this period Richard Bradshaw started a great new concept of Whiteboard testing.  Considering I do not live that far from Richard I decided to pop over and we recorded a couple of sessions.  Links to these session can be found below:

A look at the test automation pyramid
Test Execution Model

I intend to record a few more whiteboard sessions in the near future.  If you have any great ideas for whiteboard sessions then please do record them and help support this wonderful initiative.

*EDIT*

Oh and I will be speaking in Brighton on March 11th at Testbash.  Hope to see you there.

Test Execution Model - updated

This is a follow on to the post I recently did about the automation pyramid that Richard Bradshaw gave at MEWT.

Following on from this some in the testing Twitterverse asked if Richard and I would do a video for the new initiative created by Richard called Whiteboard testing.  Instead of one video we decided to do two.  The first looked at the history of the test automation pyramid and you can watch it here: A look at the test automation pyramid.  The second video was a chance for me to present my own model which I have called the test execution model and can be seen here: - John Stevenson Test Execution Model.


After the video was posted I had a message from Dan Ashby who wanted to discuss more with me about the  test execution model and what he felt was missing.  We arranged a chat and discussed the model and broke down a few assumptions we were both making.  The outcome from this chat was the need to add a few extra parts to complete the test execution cycle.  With thanks to Dan I would like to present the next generation of the model.

The changes that Dan suggested for the model are as follows:

• As you test you turn some of your tacit knowledge into explicit knowledge which then can be added to your checks and become known known information.  This is represented by the flow from the testing arrow to the checking arrow at the top of the diagram.
• As you create more checks these in turn become oracles which you can use as heuristics for your future tests.  This is represented by the flow from the checking arrow to the testing arrow.

This then gives a cycle of test execution were the testing feeds into the checking which in turn can feed back into the testing.  It should be noted that is model is for test execution which does not only apply to only testing the product deliverable.  Testing is carried throughout the development cycle and as such testing activities can occur during artitecture & design discussion and during coding. Testing activities can and should occur anywhere during the development and deployment of the product and this model should be used in that context with regards to test execution.

The Laws of Sport and Automation

I have had this idea in my mind and on my backlog for quite a while.  It was only after speaking at MEWT in Nottingham that I felt I really should get around to writing it. 

There are many debates in the software development world about ‘test automation’ and how we can ‘automate all of the testing’.  I am in the context of this article ignoring the difference between testing and checking, more details of this discussion can be found here - Testing and Checking Refined . However some of my ideas and concepts will touch on the difference between checking and testing. 

Many have put forward arguments about automating what we know and if we have defined requirements up front then it should be possible to automate these.  My counter to this is that in many sports there are well defined upfront requirements (laws) of how the game should be regulated.  For example the laws of football (Soccer to those outside of Europe) can be found online here: FIFA Laws of the Game.  If this is the case and these requirements are defined upfront then why do we not have automated referees? I asked this question on twitter and some of the responses gave reasons due to the psychical limitations, such as battery power unable to run and so forth.  My line of thought is on how these requirements can, and are, interpreted. 

Looking deeper in to the football laws of the game it can be seen there are many ambiguous statements which given the current state of AI, at the time of the publication of this article, I feel are impossible to automate. For example on page 39 it states the following as a reason for a player to be cautioned.

“unsporting behavior”

What does this mean?  Page 125 attempts to define this with a list of what constitutes unsporting behavior.  One of this in particular I found interesting, it is one based on human nature of trying to con or cheat

“attempts to deceive the referee by feigning injury or pretending to have been fouled (simulation)”

This I feel would be a common sense decision made by the referee. How could an automated system know if is fake or not?  Then again how would the ref know?  It it is a common sense decision, being made depending on a multitude of factors and contexts.

How about this one? 

“acts in a manner which shows a lack of respect for the game”

What would count as lack of respect?  A player who in the last second of the game lets in a goal that allows the opposition to win the title. The player shows human emotion and frustration, there is a fine line between emotion and respect or the lack of it?  

My issue with this automation debate is that at this time it is not possible to automate common sense and multiple contexts in the decision making process that a referee has to go though in their thinking process. 

For example a team is winning 20 – 0 a machine would continue to officiate the game in accordance to the strict letter of the law.  Whereas a human referee would allow some flexibility in the interpretation of the rules.   They will allow some aspects of empathy to be applied to the game.  Is it yet possible to automate empathy? 

James Christie made a valid point on twitter that the reason in the majority of sports they are called laws and not rules is that:

“rules are detailed and specific whilst laws can be based on vague principles, which require interpretation and judgment. “

This makes sense since most countries have courts where lawyers debate how the laws of the land can or should be interpreted.  Then a jury, judge or set of judges make a decision based upon the arguments presented. Another case of were the requirements are listed and known but given current AI limitations would be impossible to fully automate. Even though we know that human beings are flawed in the judgement that are made, would using an automated judgement machine be any less flawed, if at all possible to produce?

Returning back to the laws of sport and how ambiguous those laws are we can look at the laws of Rugby Union

Looking at the beginning of the laws on page 21 there is guidance on how the laws should be applied:

The Laws must be applied in such a way as to ensure that the Game is played according to the principles of play. The referee and touch judges can achieve this through fairness, consistency, sensitivity and, at the highest levels, management.”

How would you automate sensitivity in this context?

According to the Oxford English Dictionary this in this context is defined as:

“A persons feelings which might be easily offended or hurt”

Add into that equation “fairness”, we are now journeying down the automation rabbit hole.

Looking at the laws regarding fair play and the guidance that the document provides for foul play (Law 10) section m gives the following guidance.

“Acts contrary to good sportsmanship. A player must not do anything that is against the spirit of good sportsmanship in the playing enclosure”

What constitutes “the sprint of good sportsmanship”? How do you clarify between intentional and unintentional behavior?   Again I am uncertain if this kind of decision could be automated.

If we look at the laws of Rugby League we can see similar issues in how difficult it can be for the laws to be interpreted. Rugby league was one of the early adopters of video technology to help assist the referee in the game.  This is what Michael and James in their article would define as tool assisted testing.  In this case a video referee can review certain decisions via the use of video technology. 

Looking at the definition of a forward pass.

“is a throw towards the opponents’ dead ball line”

How do you define this in the context of a fast moving game?  Under the section 10 which offers some guidance there is a distinction between deliberate and accidental forward passes.  How do you make a distinction between these two actions? Also would an automated system be able to deal with factors such as the momentum of the player and the wind moving the ball.  Yes they could process information quicker than a human could but would it be right?

This is not to say that referees are not fallible and there are many instances in sport of them making mistakes; however people are aware of this and can accept that fact.  Would people be so willing to accept a machine making similar mistakes based upon our biases that machine are not fallible?

Many sports are implementing some level of automated systems which are used to aid the referees.  

• Tennis has been using Hawkeye since 2002 
• Football has started to implement goal-line technology
• Cricket uses the Umpire Decision Review System 

It is interesting to note that each of these automated systems have had some controversy regarding their accuracy and success especially with the cricket system.

To conclude when people discuss test automation and attempt to automate as much as possible there is a need to step back and think critically. Automation in software development has a place and is a useful tool to use, however, it should not be thought of as an alternative to testing as applied by a human being.  Even when you think you have the requirements nailed down they are words and as such are open to a multitude of interpretations.  Using a mixture of automation, tool assisted testing and human testing in a ratio that adds value to the quality of the product being delivered is a more thoughtful approach rather than the mantra of we can “automate all the testing effort.” Going forward we need to be thoughtful of what machines can do and what they cannot do.  This may change as technology progresses but as of the publication of this article there are big limitations in automation.

MEWT4 Post #1 - Sigh, It’s That Pyramid Again – Richard Bradshaw

This is the first in a series of posts I plan to write after attending the fourth MEWT peer conference in Nottingham on Saturday 10th October 2015.

Before I start I would like to say thank you all the organizers and for inviting me along and a BIG MASSIVE thank you to the AST for sponsoring the event,

Abstract: 

Earlier on in my career, I used to follow this pyramid, encouraging tests lower and lower down it. I was all over this model. When my understanding of automation began to improve, I started to struggle with the model more and more.

I want to explore why and discuss with the group, what could a new model look like?

___________________________

During the session Richard explained his thoughts about the test automation pyramid created by Mike Cohn in his book Succeeding with Agile and how the model has been misused and abused.

Richard talked about how the model has adapted and changed over the years from adding more layers..

...to being turned upside down and turned into an ice-cream cone.

Images taken from - http://watirmelon.com/2012/01/31/introducing-the-software-testing-ice-cream-cone/

Duncan Nisbet pointed out that this really is now an anti-pattern - http://c2.com/cgi/wiki?AntiPattern.  The original scope of the diagram by Mike was to demonstrate the need to have fast and quick feedback for your automation and as such focused the automation effort to the bottom of the pyramid. Where the feedback should be fast. The problem Richard has been experiencing is that this model does not show the testing effort or tools needed to get this fast feedback.  It also indicated that as you move up the pyramid less automation effort was needed or should be done.  The main issue for Richard was how the pyramid has been hi-jacked and used as examples of the priority of effort should be on automation rather than focus on the priority of both in given contexts.  

Richard presented an alternative model in which both testing and automation with the tools required could be shown on the ice-cream cone diagram.

With this diagram the sprinkles on the top were the tools and the flakes the skills.  He then in real time adjusted the model to say it would be better as a cross-sectional ice-cream cone with testing throughout the cone and the tools across all areas of the original pyramid.  Many attendees liked this representation of the model but some thought that it still encouraged the concept that you do less of certain testing activities as you move down the ice-cream cone.

At this stage I presented a model I had been using internally to show the testing and checking effort. 

Again people thought this indicated that we need to do less as we move up the pyramid and it went back to the original point being made by Richard that the pyramid should die.

After MEWT I thought about this problem and tweeted an alternative representation of the diagram. After a few comments and some feedback the diagram ended up as follows:

With this model the pyramid is removed. Each layer has the same value and importance in a testing context.  It shows that the further up the layers you go the focus should switch more from checking to testing and the lower down the focus should be on automating the known knowns. All of this is supported by tools and skills.  As a model it is not perfect and it can be wrong for given contexts, however for me it provides a useful starting point for conversations with those that matter.  It especially highlights that we cannot automation everything nor should we try to do so.

In summary the talk given by Richard was one of the many highlights of the day at MEWT and inspired me to look further into the test automation pyramid model and its failings.  I agree with Richard that this original model should die especially in the way it is often misused.  Richard provided some useful alternatives which could work and hopefully as a group we improved upon the original model.   Richard did clarify that his ice-cream cone model with sprinkles is not his final conclusion or his final model and he will be writing something more on this in the near future.  His blog can be found here - http://www.thefriendlytester.co.uk/.

Now it is over to you, please provide your feedback and comments on this alternative model.

The Dark side of Continuous Deployment

A short post!

In the software development world there is a drive towards continuous deployment of software which in the majority of cases in a good thing.  The good side is the reduction in costs, faster feedback from users and quicker release of features and bug fixes.

However my experiences of using software in which there are constant and frequent updates provides me with some concerns.  Have you ever downloaded an app for your device that crashes upon starting or has new updates each and every day?  How does this make you feel?  I for one get very frustrated when I get up in the morning and my phone says x number of apps have been updated.  Or that the same app I have updated 3 times previously in the last week!  Suffice to say this app have been deleted and I no longer use it.

There is a common phrase that appears to have been forgotten in the rush to continuous deployment and using users as testers and that is "First impressions count"  If you come across a piece of software which you have purchased and it is buggy or does not work in the way you expect you may feel cheated.  All software has bugs that is to be expected, my worry is that we are paying for software which has not been sufficiently tested by skilled testers.

There is a major movement to automate 'testing' within continuous deployment practices.  This implies to me that those who promote and suggest this may not have an understanding of what testing is and the value it provides to the quality of software.  This is a big assumption for me to make but I am experiencing more and more examples of software being released to users which is not good enough and with a little bit of testing could have provided a much better user experience.

I am not saying to stop doing continuous deployment but within the development cycle there should be some level of human testing to provide information about how the software is behaving.  This is not meant to say have a dedicated testing phase but ensure that testing activities are added into each and every release. Continuous deployment is a wonderful tool to enable quick and easy releases however it should not be used to release poor or sub-standard software that relies on users to provide details of issues, especially software that a user has purchased.

The key problem to me as someone who works in business is that if you release poorly tested software and it is noticed then the chances are they will not come back and you will lose their customer.  If this happens again and again you may find you do not have a business, social media is powerful in allowing those who experience bad software to make it know very quickly to a huge amount of people.  Can your business take that risk?

RANT over.....

Over to you the reader, what are your thoughts on this?

Further reading:

Testing vs Checking refined - James Bach
Continuous Deployment - Agile Alliance
Why continuous deployment - Eric Ries

Using Exploratory Testing to Drive Automation

Traditionally automation and exploratory testing have been seen as independent tasks that work in different ways and normally do not appear to be ideal partners. This article suggests that there is value in utilizing the work done when automating to add some exploratory testing effort.

In most cases there is some form of documentation or communication which indicates what the system being designed will do. This can be in the form of requirements, design specification, email, design meetings and so forth. An assumption has been made here that testers are involved in the design phase, if they are not for your project then they should be.

Within the context in which I work these documents/communications are analyzed for automated cucumber scenarios and exploratory charters. For the automation scenarios these are created using the Gherkin format:

Feature: Password management
Scenario: Forgot password
Given a user with email "someuser@example.com" exists
When I ask for a password reset
Then an email with a password reset link should be sent

Exploratory charters are created in the Exploratory Charter format as described by Elisabeth Hendrickson in the book Explore It!:

Explore (target)
With (resources)
To discover (information)

Where:

·                  Target: Where are you exploring

·                  Resources: What resources will you bring with you

·                  Information: What kind of information are you hoping to find?

Once the exploratory testing charters and cucumber scenarios have been initially defined the Test Engineer manually steps through the cucumber scenario to determine the pre-requite components needed to complete the scenario from an automation perspective, at the same time adding details to the Gherkin feature file. This may involve building the environment and integrating the components. It is at this point that some level of exploratory testing can be performed to gain more value from this manual effort. This manual effort can be recorded on a wiki or some other similar tool. The suggestion for using a wiki is the ease of access and use for cross regional teams. It is not mandated for teams to us a wiki but from experience it appears to be the best way to share information.

This will require a mindset shift for the test engineer instead of only focusing on the automation scenario they should be looking for future opportunities to test either automated or manually and these should be fed back to the team and added as backlog items. One other benefit is that as you explore the automation scenario you may uncover issues or defects earlier which are always a good benefit for the project.

Once the step definitions have been coded and implemented for the automation scenario they should feed into whatever automation reporting system you have selected. This should report scenarios pass or fail. It is expected that defects should not be found at this stage and if scenarios fails it is indication that your expectations of what the system does has changed and should be investigated before any possible defects are raised. This is a simple approach to close the link between test automation and exploratory testing and leverage the skills of the test engineers to their full value.

It should be noted that there will still need to be more effort to ensure that the exploratory charters are run manually and recording using the current approaches for your projects.  This then become a continuous cycle as more scenarios are discovered for automation then more exploratory testing can be done and more information about the system can be uncovered.  This approach makes exploratory testing and automation partners rather than as independent entities.  It hopefully reduces the checking vs testing debate and focuses on utilizing the skills of testers to deliver a product that customers will enjoy using.

The diagram below is a simple visually representation of this concept.

Are you ‘Checking’ or ‘Testing’ (Exploratory) Today?

Do you ask yourself this question before you carry out any test execution?

If not then this article is for you.  It starts with a brief introduction about the meaning of checking and testing in the context of exploration and then asks the reader to think about the question then evaluate the testing they are doing and determine from their answer if what they are doing has the most value.

There have been many discussions within the testing community about what the difference is between ‘checking’ and ‘testing’ and how it fits within the practice of test execution.

Michael Bolton started the debate with his article from 2009 on ‘testing vs checking’ in which he defined them as such:

• Checking Is Confirmation
• Testing Is Exploration and Learning
• Checks Are Machine-Decidable; Tests Require Sapience

At that time there were some fairly strong debates on this subject and in the main I tended to agree with the distinctions Michael made between ‘checking’ and ‘testing’  and used this in my approach when testing.

James Bach, working with Michael, then came along with another article to refine the definitions in his article ‘Testing and Checking refined.

• Testing is the process of evaluating a product by learning about it through experimentation, which includes to some degree: questioning, study, modelling, observation and inference.
• (A test is an instance of testing.)
• Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.
• (A check is an instance of checking.)

From this they stated that there are three types of checking

• Human checking is an attempted checking process wherein humans collect the observations and apply the rules without the mediation of tools.
• Machine checking is a checking process wherein tools collect the observations and apply the rules without the mediation of humans.
• Human/machine checking is an attempted checking process wherein both humans and tools interact to collect the observations and apply the rules.

The conclusion to me appeared to be that checking is a part of testing but we need to work out which would be best to use for the checking part is a machine better or a human?  This question leads to the reason for putting this article together.

James on his website produced a picture to aid visualisation of the concept:

© - James Bach - Testing and Checking refined

Since in my world checking forms a part of testing as I have interpreted the article by James we therefore need to pause for a moment and think about what we are really doing before performing any testing.

We need to ask ourselves this question:

Are we ‘checking’ or ‘testing’ (exploratory) today?

Since both form a part of testing and both could, depending on the context, be of value and importance to the project it is vital that we understand what type of testing we are doing.  If we ask ourselves that question and our answer is ‘checking’ we now need to find out what type of checking we are doing.  If the checking that is being performed falls under the category of machine checking then we need to think and find out why we are doing this as a human thinking person rather than getting a machine to perform it.  Things that could fall under this category could be validation or verification of requirements or functions in which we know what the answer will be.  If this is the case then you need to ask

WHY:

• are you doing this?
• is a machine not doing this?
• can a machine not do this for me?

The problem with a person carrying out manual machine checking is that is uses valuable testing time that could be used to discover or uncover information that we do not know or expect. * When people working in software development talk about test automation this is what I feel they are talking about.  As rightly stated by James and Michael there are many other automation tools that testers can use to aid their exploratory testing or human checking which could be classified as test automation.

*See my article on expectations and why we need to explore.

So even though checking is a part of testing and can be useful, it may not be the best use of your time as a tester.  Sometimes there are various reasons why testers carry out machine checking such as:

• Too expensive
• Too difficult to automate
• No time
• Lack of skills
• Lack of people
• Lack of expertise. 

However if all you are doing during test execution is machine checking then what useful information are you missing out on finding? 

If we go back to the title of this article are you ‘checking’ or ‘testing’ today?

You need to make sure you ask yourself this question each time you test and evaluate which you feel would have most value to the project at that time.  We cannot continue to have the same ‘must run every test check manually’ mentality since this only addresses the stuff we know and takes no account of risk or priority of the information we have yet to discover.

The information that may be important or useful is hidden in the things we do not expect or currently do not know about. To find this we must explore the system and look for the interesting details that are yielded from this type of testing. (Exploratory)

I will leave you with the following for when you are next carrying out machine checking:

..without the ability to use context and expectations to “go beyond the information given,” we would be unintelligent in the same way that computers with superior compututional capacity are unintelligent.

J. S. Bruner (1957) Going beyond the information given.

Further reading

• Michael Bolton - http://www.developsense.com/blog/2013/03/versus-opposite/
• Elizabeth Hendrickson - http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=16684
• Rahul Verma - http://www.testingperspective.com/?p=3143
• InfoQ - http://www.infoq.com/news/2009/12/testing-or-checking
• The Test eye - http://thetesteye.com/blog/2010/03/the-testing-vs-checking-paradox/
• Paul Holland - http://testingthoughts.com/blog/152
• Eric Jacobson - http://www.testthisblog.com/2013/08/exploring-vs-checking-almost-did-it-for.html

The Size of the Automation Box

I have previously written an article about doing too much automation and have recently been involved in successfully implementing automation and exploratory testing in agile environments.  During discussions about how to implement the automation framework a work colleague made the following statement in regards to defining the automation boundaries:

“Defining the size of the black box is critical”.  

 It made me think about the importance of knowing where your automation boundaries are and how critical it is for successful deployment of an automation framework.   One point to keep in mind when you try and implement an automation framework is the principal of K.I.S.S (‘Keep it simple stupid’)

I have seen (and been involved in) several failed automation implementations and in retrospect the main reason for failure has been trying to automate everything that you can possibly think to automate. I discussed this problem in the article too much automation and the importance of thinking critically about what to automate. Time and time again people go rushing ahead to try and automate everything they can without thinking about the boundaries of what their automation framework should be covering. 

 So what is the solution? The rest of this article will offer some guidelines that may help you in your quest of implementing a successful automation framework

The first thing to look at when trying to implement automation is the context of the testing you are carrying out or attempting to automate.  Are you involved in sub system testing, component testing, end to end testing, customer driven testing or any other type of testing?  It is important to look at this and found how what do you want the automation to do for you.  

• At Customer level you may want to automate the common customer scenarios.
• At a system level you may want to validate the input and outputs to a specific set of components and stub out the E2E system.

So defining your context of the purpose of your testing from an automation viewpoint is important.  The reason for this is that links to the second point, defining the size, or scope, of your automation for the project.  Most automation implementations fail to take into account the limits of what you intend to cover with the automation.  As my colleague put it 

The size of this will change depending on the context of the type of testing that you are carrying out.  For your automation implementation to be successful it is important that you define upfront your boundaries of what the automation is expected to cover.  Automation should be about what you input into the system and what you expect as an output everything in-between this is your black box .  So knowing what your testing context is should help define the size and limits of your black box knowing this information before you start to implement a test automation framework should greatly improve the chances of your automation implementation being successful.

Example of defining black box at a Component level of testing

    

Shaded Gray Area = Black box

For this example the boundaries would be

• Create automated test inputs for Input A in relation to defined requirements and expectations
• Automate expected outputs Output 1 and Output 2 based upon requirements and your expectations
• What happens inside of Component X is of no interest since this is outside the boundaries of what we are automating

Another example of defining black box at a Component level of testing

    

• Create automated test inputs for Input B in relation to defined requirements and expectations
• Automate expected outputs Output 1, Output 2, Output 3 and Output 4 based upon requirements and your expectations
• What happens inside of Component Y is of no interest since this is outside the boundaries of what we are automating

Example of defining black box at a Product level of testing

    

• Create automated test inputs for Input A and Input B in relation to defined requirements and expectations
• Automate expected outputs Output 1, Output 2, Output 3 and Output 4 based upon requirements and your expectations
• What happens inside of Component Y is of no interest since this is outside the boundaries of what we are automating

I am aware that this appears to be common sense but time and time again people fail to take on board the context of what they expect the automation to be covering.

So next time you are involved in implementing an automation framework remember to define the size of your black box before you start your implementation that way you know the boundaries of what your automation will cover and check for you.

Stop Doing Too Much Automation

When researching my article on testing careers for the Testing Planet a thought stuck me about the amount of people who indicated that ‘Test’ Automation was one of the main learning goals of many of the respondents.  This made me think a little about how our craft appears to be going down a path that automation is the magic bullet that can be used to resolve all the issue we have in testing.

I have had the idea to write this article floating around in my head for a while now and the final push was when I saw the article by Alan Page (Tooth of the Angry Weasel) - Last Word on the A Word  in which he said much of what I was thinking. So how can I expand on what I feel is a great article by Alan?

The part of the article that I found the most interesting was the following:

“..In fact, one of the touted benefits of automation is repeatability – but no user executes the same tasks over and over the exact same way, so writing a bunch of automated tasks to do the same is often silly.”

This is similar to what I want to write about in this article.  I see time and time again dashboards and metrics being shared around stating that by running this automated ‘test’ 1 million times we have saved a tester running it manually 1 million times and therefore if the ‘test’ took 1 hour and 1 minute to run manually and 1 minute to run automated it means we have saved 1 million hours of testing.  This is so tempting and to a business who speaks in value and in this context this mean costs.  Saving 1 million hours of testing by automating is a significant amount of cost saving and this is the kind of thing that business likes to see a tangible measure that shows ROI (Return on Investment) for doing ‘test’ automation.  Worryingly this is how some companies sell their ‘test’ automation tools.

If we step back for a minute and go back and read the statement by Alan.  The thing that most people who state we should automate all testing talk about the repeatability factor.  Now let us really think about this.  When you run a test script manually you do more than what is written down in the script.  You think both critically and creatively , you observe things far from the beaten track of where the script was telling you to go.  Computer see in assertions, true or false, black or white, 0 or 1, they cannot see what they are not told to see.  Even with the advances in artificial intelligence it is very difficult for automation systems to ‘check’ more than they have been told to do.  To really test and test well you need a human being with the ability to think and observe.   Going back to our million times example.  If we ran the same test a million times on a piece of code that has not changed the chances of find NEW issues or problems remains very slim however running this manually with a different person running it each time our chances of finding issues or problems increases.  I am aware our costs also increase and there is a point of diminishing returns.  James Lyndsay has talked about this on his blog in which he discusses the importance of diversity   The article also has a very clever visual aid to demonstrate why diversity is important and as a side effect it helps to highlight the points of diminishing return. This is the area that the business needs to focus on rather than how many times you have run a test.

My other concern point is the use of metrics in automation to indicate how many of your tests have you or could you automate.  How many of you have been asked this question?  The problem I see with this is what people mean by “How many of your tests?”  What is this question based upon?  Is it...

• all the tests that you know about now?
• all possible tests you could run?
• all tests you plan to run?
• all your priority one tests?

The issue is that this is a number that will constantly change as you explore and test the system and learn more. Therefore if you start reporting it as a metric especially as a percentage it soon becomes a non-valuable measure which costs more to collect and collate than any benefit it may try to imply.  I like to use the follow example as an extreme view.

Manager:  Can you provide me a % of all the possible tests that you could run for system X that you could automate.

Me:  Are you sure you mean all possible tests?

Manager: Yes

Me: Ok, easy it is 0%

Manager:  ?????

Most people are aware that testing can have an infinite amount of tests even for the most simple of systems so any number divided by infinity will be close to zero, therefore the answer that was provided in the example scenario above.  Others could argue that we only care about of what we have planned to do how much of that can be automated, or only the high priority stuff and that is OK, to a point, but be careful about measuring this percentage since it can and will vary up or down and this can cause confusion.  As we test we find new stuff and as we find new stuff our number of things to test increases.

My final worry with ‘test’ automation is the amount of ‘test’ automation we are doing (hence the title of this article) I seen cases where people automate for the sake of automation since that is what they have been told to do.  This links in with the previous statement about measuring tests that can be automated.   There need to be some intelligence when deciding what to automate and more importantly what not to automate. The problem is when we are being measured by the number of ‘tests’ that we can automate human nature will start to act in a way that makes us look good against what we are being measured. There are major problems with this and people stop thinking about what would be the best automation solution and concentrate on trying to automate as much as they can regardless of costs. 

What ! You did not realise that automation has a cost?  One of the common problems I see when people sell ‘test’ automation is they conveniently or otherwise forget to include the hidden cost of automation.  We always see the figures of the amount of testing time (and money) being saved by running this set of ‘tests’ each time.  What does not get reported and very rarely measured is the amount of time maintaining and analysing the rests from ‘test’ automation.  This is important since this is time that a tester could be doing some testing and finding new information rather than confirming our existing expectations.  This appears to be missing whenever I hear people talking of ‘test’ automation in a positive way.  What I see is a race to automate all that can be automated regardless of the cost to maintain. 

If you are looking at implementing test automation you seriously need to think about what the purpose of the automation is.  I would suggest you do ‘just enough’ automation to give you confidence that it appear to work in the way your customer expects.  This level of automation then frees up your testers to do some actual testing or creating automation tools that can aid testing.  You need to stop doing too much automation and look at ways you can make your ‘test’ automation effective and efficient without it being a bloated, cumbersome, hard to maintain monstrosity (Does that describe some peoples current automation system?)  Also automation is mainly code so should be treated the same as code and be regularly reviewed and re-factored to reduce duplication and waste.

I am not against automation at all and in my daily job I encourage and support people to use automation to help them to do excellent testing. I feel it plays a vital role as a tool to SUPPORT testing it should NOT be sold on the premise that that it can replace testing or thinking testers.

Some observant readers may wonder why I write ‘test’ in this way when mentioning ‘test’ automation.  My reasons for this can be found in the article by James Bach on testing vs. checking refined.

Why we need to explore

This article looks at why within testing we need to carry out exploratory testing.

It is based upon the following diagram that James Lyndsay  drew during a recent diversity workshop.

When we talk about expectations we mean something that is written upfront , or known about before delivery of the product.  For example requirements or specification documentation.  There are other types of expectation that depend on the person such as experiences, domain knowledge or business knowledge.

The deliverable is the product being delivered and is normally the software under test.

When the product is delivered sometimes the expectations are met by deliverable and this is the area shown on the diagram where the ovals overlap and other times they are not.  Where they do overlap we can say that the deliverable meets the expectations in this area.  This could be seen as what we would with the testing world call verifying requirements, user stories, stuff/information that is known or that we know about.

**Note the area where expectation meets deliverable is not to scale or representative of a real world situation it is for clarification purposes only

The expectations to the left of the overlap are where our expectations are not met by the deliverable and these could be defined as bugs, defects, issues or a misunderstanding of what was expected.

The stuff to the right of the overlap is stuff in the deliverable that you did not expect and is the stuff you need to use exploratory testing to discover what could be vital and important information to someone who matters.  This is the stuff you do not know about, nor could you have known about until you started to do some testing.

Outside the ovals is an area in which you did not have any expectations nor is in the deliverable and as such is not important with regards to testing for the purpose of this article.

The following diagram is a visual representation of what I wrote above.

With this as a model we can now start to think about what James Bach and Michael Bolton discussed with reference to the terms checking and testing.  I see that for the stuff we know about (the expectations) we should look at covering this with machine checking (automation). This does not mean we should automate ALL expectations we need to base our automation on many factors cost, priority, risk, etc.  (I plan a future article on not doing too much automation). If we do this we should have some confidence that our deliverable meets our expectations.  This then allows testers to start to do some 'testing' and uncover the useful information that exists in the deliverable that no one knows about until some testing is carried out.

This is why IMO the only way we can test is not by following scripts (even though they can be useful to aid some elements of meeting our expectations) but by exploring the deliverable and finding out what it is actually doing that we did not expect it to do.

My next article will extend on this model and revisit measuring test coverage.