Are you ‘Checking’ or ‘Testing’ (Exploratory) Today?

Do you ask yourself this question before you carry out any test execution?

If not then this article is for you.  It starts with a brief introduction about the meaning of checking and testing in the context of exploration and then asks the reader to think about the question then evaluate the testing they are doing and determine from their answer if what they are doing has the most value.

There have been many discussions within the testing community about what the difference is between ‘checking’ and ‘testing’ and how it fits within the practice of test execution.

Michael Bolton started the debate with his article from 2009 on ‘testing vs checking’ in which he defined them as such:

• Checking Is Confirmation
• Testing Is Exploration and Learning
• Checks Are Machine-Decidable; Tests Require Sapience

At that time there were some fairly strong debates on this subject and in the main I tended to agree with the distinctions Michael made between ‘checking’ and ‘testing’  and used this in my approach when testing.

James Bach, working with Michael, then came along with another article to refine the definitions in his article ‘Testing and Checking refined.

• Testing is the process of evaluating a product by learning about it through experimentation, which includes to some degree: questioning, study, modelling, observation and inference.
• (A test is an instance of testing.)
• Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.
• (A check is an instance of checking.)

From this they stated that there are three types of checking

• Human checking is an attempted checking process wherein humans collect the observations and apply the rules without the mediation of tools.
• Machine checking is a checking process wherein tools collect the observations and apply the rules without the mediation of humans.
• Human/machine checking is an attempted checking process wherein both humans and tools interact to collect the observations and apply the rules.

The conclusion to me appeared to be that checking is a part of testing but we need to work out which would be best to use for the checking part is a machine better or a human?  This question leads to the reason for putting this article together.

James on his website produced a picture to aid visualisation of the concept:

© - James Bach - Testing and Checking refined

Since in my world checking forms a part of testing as I have interpreted the article by James we therefore need to pause for a moment and think about what we are really doing before performing any testing.

We need to ask ourselves this question:

Are we ‘checking’ or ‘testing’ (exploratory) today?

Since both form a part of testing and both could, depending on the context, be of value and importance to the project it is vital that we understand what type of testing we are doing.  If we ask ourselves that question and our answer is ‘checking’ we now need to find out what type of checking we are doing.  If the checking that is being performed falls under the category of machine checking then we need to think and find out why we are doing this as a human thinking person rather than getting a machine to perform it.  Things that could fall under this category could be validation or verification of requirements or functions in which we know what the answer will be.  If this is the case then you need to ask

WHY:

• are you doing this?
• is a machine not doing this?
• can a machine not do this for me?

The problem with a person carrying out manual machine checking is that is uses valuable testing time that could be used to discover or uncover information that we do not know or expect. * When people working in software development talk about test automation this is what I feel they are talking about.  As rightly stated by James and Michael there are many other automation tools that testers can use to aid their exploratory testing or human checking which could be classified as test automation.

*See my article on expectations and why we need to explore.

So even though checking is a part of testing and can be useful, it may not be the best use of your time as a tester.  Sometimes there are various reasons why testers carry out machine checking such as:

• Too expensive
• Too difficult to automate
• No time
• Lack of skills
• Lack of people
• Lack of expertise. 

However if all you are doing during test execution is machine checking then what useful information are you missing out on finding? 

If we go back to the title of this article are you ‘checking’ or ‘testing’ today?

You need to make sure you ask yourself this question each time you test and evaluate which you feel would have most value to the project at that time.  We cannot continue to have the same ‘must run every test check manually’ mentality since this only addresses the stuff we know and takes no account of risk or priority of the information we have yet to discover.

The information that may be important or useful is hidden in the things we do not expect or currently do not know about. To find this we must explore the system and look for the interesting details that are yielded from this type of testing. (Exploratory)

I will leave you with the following for when you are next carrying out machine checking:

..without the ability to use context and expectations to “go beyond the information given,” we would be unintelligent in the same way that computers with superior compututional capacity are unintelligent.

J. S. Bruner (1957) Going beyond the information given.

Further reading

• Michael Bolton - http://www.developsense.com/blog/2013/03/versus-opposite/
• Elizabeth Hendrickson - http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=16684
• Rahul Verma - http://www.testingperspective.com/?p=3143
• InfoQ - http://www.infoq.com/news/2009/12/testing-or-checking
• The Test eye - http://thetesteye.com/blog/2010/03/the-testing-vs-checking-paradox/
• Paul Holland - http://testingthoughts.com/blog/152
• Eric Jacobson - http://www.testthisblog.com/2013/08/exploring-vs-checking-almost-did-it-for.html