A test is...

On twitter Santhosh Tuppad posted the following tweet:

It made me think about what 'a test' really is.

I made half an attempt to improve the statement Santhosh made and suggested the following:

This would make the statement read as following:

A Test is an experiment that tests to see if the testing yields observational information.

I have thought about this and it still does not appear quite right for what I would say is a test.

There have been many discussions on what testing is and how 'a test' relates to the testing.

• Testing vs Checking - Bolton 2009
• Testing vs Checking refined - Bach/Bolton 2013
• What is Software Testing - ISQTB
• Exploratory Testing - Kaner 2006

A test is a part of testing as some of the links above state, therefore the definitions used for testing could apply to a test.  However there is a gap if you are communicating with someone and they ask you 'What is a test?'  What is your reply?  'Oh, it is a part of testing'  Hence thanks to Santhosh I wanted to look a little more into the definition of a test.

My thoughts is that for a definition of a test we should borrow from the scientific method.

You start with a theory, you run a series of experiments against that theory, you observe what happens, you analyse what you saw, you report the findings and adjust your theory based upon your new knowledge.

There is no mention of finding bugs within that process, looking at a dictionary definition of test it gives the following:

a critical examination, observation, or evaluation :  trial; specifically :  the procedure of submitting a statement to such conditions or operations as will lead to its proof or disproof or to its acceptance or rejection a test of a statistical hypothesis (2) :  a basis for evaluation :  criterion 

 an ordeal or oath required as proof of conformity with a set of beliefs 

a means of testing: such as (1) :  a procedure, reaction, or reagent used to identify or characterize a substance or constituent (2) :  something (such as a series of questions or exercises) for measuring the skill, knowledge, intelligence, capacities, or aptitudes of an individual or group 

 a positive result in such a test 

:a result or value determined by testing

https://www.merriam-webster.com/dictionary/test

Wow so many different meaning for a simple word!  

To aid the definition it needs context.

We are focusing on a software test, this enables us to narrow the definition of 'a test'. Note we are not using the term 'test case'.  Using 'test case' I feel encourages limitations in the scope of what a test means.

To conclude the following is my definition of 'a test' within the context of software testing.

A test is a series of experiments performed against a theory that evolves based upon observational and behavioral information uncovered by the test.

The software testing part is the reporting of these evolving theories and the observation made.

What about you?  Could you improve this?  Do you have your own definition?

Testing Skills # 8 – Being a Skeptic

 “Science is the organized skepticism in the reliability of expert opinion.”  Richard Feynman

The reason for this article is based upon a recent twitter conversation regarding how someone sometimes avoids skepticism due to its negative nature.  I fully understand what they mean when skepticism is used in way to attack a person or their character (ad-hominem argument).  The purpose of this article is to provide information about how useful having a healthy dose of skepticism is for those involved in software testing.

Many people have a belief that the purpose of testing is to prove that the software works as expected.  This is really a fallacy since testing cannot prove that the software will always work the way you expect. There is always an element of doubt in proving that it works due to the nature of software and how it is used. This is the same as in the field of scientific research where someone comes up with a theory based upon their experiments and then their peers using a fair bit of skepticism try to prove the theory wrong.  This is the crux of any scientific based research method.  It is not about producing your theory it is about applying critical thinking to your theory as to why it can be wrong.   Testing software is similar in its approach it is very difficult to inform someone, who matters that the software is working. We can provide useful information about its behavior and what it actually doing.

As a tester you need to apply critical thinking to your testing and to the evidence you produce.   This is where being able to look at what you are doing and what information you find will a fair degree of skeptical thought.  The scientific method is a useful skeptic tool to apply to your testing and to any other information that people provide to you as ‘fact’.  One approach is the use of FiLCHers:

If you look at each of these headings you can see that it is about trying to find evidence where your theory and experiment could be wrong, acting as a skeptic.

This is a vital skill for testers to possess to ensure that their testing is unbiased and factually correct. If you feel the skeptic is lacking then look at ways you can improve it.

Here are some suggestions to help you become a better skeptic:

• Learn about logical arguments and fallacies
• This page has a substantial list of them - http://changingminds.org/disciplines/argument/fallacies/fallacies_alpha.htm
• This book is a wonderful illustrated book proving examples of fallacies  and bad arguments - http://www.amazon.co.uk/Illustrated-Book-Bad-Arguments/dp/1615192255/

• Practice using skepticism – apply it to your work and the work of others.
• Use James Bach’s’ , Huh, Really, So technique - http://itknowledgeexchange.techtarget.com/software-quality/james-bach-on-critical-thinking-huh-really-so/

• Use reflection to help improve your skeptic skills.
• See my previous post on reflection - http://www.steveo1967.blogspot.com/2015/10/testing-skills-7-reflection.html 

If you have some suggestions of your own to help others in their journey to being a great skeptic please let me know by using the comments section.

Is testing a manufacturing process or a scientific approach?

Question

Does testing need to move away from manufacturing processes and more towards scientific approaches?

If we take a look at the world of manufacturing you can find many standards (ISO and others) for various everyday things.  However even within the same subject field there can be many competing standards.  For example look at the standards for mains electricity around the world.  Which standard do you use? 

http://en.wikipedia.org/wiki/Mains_electricity_by_country

What about light fittings in your country how many different types are available?

http://en.wikipedia.org/wiki/Edison_screw 

http://en.wikipedia.org/wiki/Bayonet_mount

In some cases these standards have been changed to meet local and cultural differences in others it has been a question of business and not wishing to pay a royalty fee. However in all these examples context plays a part in the adopting of a standard. Maybe I am being unfair and using examples that are not typical.  From the research I have done I find the same within the QC field, depending on what is being manufactured the processes and standard can and do change for similar products. 

The xkcd comic strip shows this very clearly.

(c) http://xkcd.com/927/

The problem I have within the domain of software testing is that when we apply manufacturing processes to software testing we are making many assumption. The biggest one being

 all software is the same and behaves the same 

which is not true. 


As such we require different approaches to deal with this which is in context with the software being developed rather than trying to make it fit to processes designed around the assumption that everything will be done exactly the same way.

My concern is with the new ISO software testing standard ISO/IEC 2911  which appears to be based on manufacturing process and practices.

The question I have is why? 

Software testing in my opinion is not a manufacturing driven process but more a scientific experimental approach in which the tester has questions and theories that they wish to prove or disprove.So why try and tie a thinking process down to a checklist style document driven process?  Is it because 'management' can have a false belief that it more easily to manage or am I being sceptical?

Looking at the field of science and research - yes I acknowledge there are still some processes but these seem to be based upon what has been done and experienced rather than up front unnecessary documents.  I see very little about processes more a case of techniques and approaches to use. 

http://en.wikipedia.org/wiki/Wikipedia:Scientific_standards

If you then add in other fields of science such as social and anthropology you have a wide range of approaches that lend themselves to testing. I see testers more as scientists, researchers or investigators looking at debunking their theories, biases and understanding and trying to learn more about what they are testing.

One approach I stumbled across was the following: http://www.respectproject.org/code/cstds.php?id and I think this could lead to another blog article there is so much useful information here for testers.



Quoting a couple of them:

• ensure factual accuracy and avoid misrepresentation, fabrication, suppression or misinterpretation of data
• critically question authorities and assumptions to make sure that the selection and formulation of research questions, and the conceptualisation or design of research undertakings, do not predetermine an outcome, and do not exclude unwanted findings from the outset 
• ensure methodology and findings are open for discussion and full peer review

I found this code of practice very similar to the ethics I try to employ when carrying our exploratory testing, as I have already stated I may come back to this with another article at a later date.

I remember in school during science lessons we would:

• Start from a theory
• We would then test that theory
• Write a conclusion based upon what we did 
• Report what we found.

In my mind this is like exploratory testing

• We have a charter and a mission
• We try to prove the mission right or wrong
• We write about our discoveries and what we learn
• We report what we find
• We check to see if what we find matches our original mission

We prove sufficient information for our peers to be able to replicate what we did and see if they come to the same conclusions or not.  We do this by treading similar ground but sometimes not exactly the same steps.

The similarities between exploratory testing and scientific research appear to be many.

• Formulate a theory
• Test the theory
• Explore ways to prove the theory is incorrect (Peer testing)
• Report your findings

So to conclude I feel that exploratory testing and testing in general has more to do with the sciences than it has to do with manufacturing.  The sooner we can move testing away from a manufacturing process centric way of working to a more natural scientific way of working the better the world of testing will be.  We really must stop testing being forced into a process in which people can tick a box and say yes we have a document for that and for that and that and of course that.   


Testing is a thinking activity not a document creation, ticking boxes checking process.