A test is...

On twitter Santhosh Tuppad posted the following tweet:

It made me think about what 'a test' really is.

I made half an attempt to improve the statement Santhosh made and suggested the following:

This would make the statement read as following:

A Test is an experiment that tests to see if the testing yields observational information.

I have thought about this and it still does not appear quite right for what I would say is a test.

There have been many discussions on what testing is and how 'a test' relates to the testing.

• Testing vs Checking - Bolton 2009
• Testing vs Checking refined - Bach/Bolton 2013
• What is Software Testing - ISQTB
• Exploratory Testing - Kaner 2006

A test is a part of testing as some of the links above state, therefore the definitions used for testing could apply to a test.  However there is a gap if you are communicating with someone and they ask you 'What is a test?'  What is your reply?  'Oh, it is a part of testing'  Hence thanks to Santhosh I wanted to look a little more into the definition of a test.

My thoughts is that for a definition of a test we should borrow from the scientific method.

You start with a theory, you run a series of experiments against that theory, you observe what happens, you analyse what you saw, you report the findings and adjust your theory based upon your new knowledge.

There is no mention of finding bugs within that process, looking at a dictionary definition of test it gives the following:

a critical examination, observation, or evaluation :  trial; specifically :  the procedure of submitting a statement to such conditions or operations as will lead to its proof or disproof or to its acceptance or rejection a test of a statistical hypothesis (2) :  a basis for evaluation :  criterion 

 an ordeal or oath required as proof of conformity with a set of beliefs 

a means of testing: such as (1) :  a procedure, reaction, or reagent used to identify or characterize a substance or constituent (2) :  something (such as a series of questions or exercises) for measuring the skill, knowledge, intelligence, capacities, or aptitudes of an individual or group 

 a positive result in such a test 

:a result or value determined by testing

https://www.merriam-webster.com/dictionary/test

Wow so many different meaning for a simple word!  

To aid the definition it needs context.

We are focusing on a software test, this enables us to narrow the definition of 'a test'. Note we are not using the term 'test case'.  Using 'test case' I feel encourages limitations in the scope of what a test means.

To conclude the following is my definition of 'a test' within the context of software testing.

A test is a series of experiments performed against a theory that evolves based upon observational and behavioral information uncovered by the test.

The software testing part is the reporting of these evolving theories and the observation made.

What about you?  Could you improve this?  Do you have your own definition?

Testing Qualifications - Certification revisited

Let me start this article by saying I am not against testing qualification at all and they can be a useful resource. Note I use the word qualification; certification is getting to become a bit of a loaded word. Richard Bradshaw talked about his experience here.  I tend to agree with what Richard is saying, as a starting point ISQTB (Or whatever name they wish to use) is a good idea and concept.  For those who have never been involved in testing and wish to know more about ‘some’ of the techniques and methods and a little about the history of testing then it can be a useful building block.  My concern is that it is not being sold in this way and that it becomes a filter to be used by companies and employment agencies.  I have talked about this previously here.   Danny Dainton also talked about his views on ISQTB from the new tester perspective here 

The reason why I have revisited this discussion is due to the recent debate on twitter about test certification and especially ISQTB.  Part of the discussion was the way in which numbers appear to have been used in an advertisement for companies to ensure their testing teams get the ISQTB certification.

Advert can be seen here. Googling a little more provided me with this information 

The bit that concerns me with this sort of misleading propaganda is this:

Various studies estimate the cost of a post-production software defect in the range of $4,000 - $5,000.[1] If ISTQB Software Tester Certification can help a software tester to eliminate just one post-production defect in his or her career, the return on investment for an ISTQB exam could exceed 1500%. With the Volume Purchase Program, that ROI could exceed 2000%.

Where are these figures obtained from?  It is a fallacy that with the way software development works today that fixing a defect later costs so much more.  Details of the cost of defect fallacy can be found here and ANYONE working in software development please read this great  book – The leprechauns of software engineering

This then lead to a talk that Rex Black was doing at STPCON in which the phrase:

“…greatly lowers the cost of post-release defects.”

Can clearly be seen

This is what concerns me a great deal, everything within the ISQTB world appears to be focused on profit and how to maximize the most amount of money with the least amount of effort.  Now someone may correct me if I am wrong on that one.  I do hope that I am and that the people who provide this training do really care about the testing profession and the people paying their hard earned money is of something that is worthwhile to them.  The problem is when profit is involved in learning activities the needs and the best interests of the student are normally lower down the list than the ROI (sic) for the training company shareholders.

I am not against testing qualifications I am against the way in which they are being sold and used within the profession.  I do not like the use of multiple choice exams in which someone can learn by rote and then pass and not know anything really valuable about testing.

We need a system of learning in which we can learn the basics, practice them and be assessed on our thinking and reasoning. The problem is that this is too difficult to do en masse since it eats into profit hence my concerns about profit before learning.

There are other testing training opportunities such as the Rapid software testing course by James Bach and Michael Bolton is another alternative.

Or  black box software testing course created by Cem Kaner is a step in the right direction.

How come we do not hear more about these?

Is the lobbying and scaremongering of the ISQTB too big?  I really hope not there are many passionate testers in the world and we have a moral and ethical obligation to provide the correct training and learning opportunities for these people.  We need to stop using false data and information to scare companies and managers into making people attend these courses

We need to be truthful to both those attending such courses and those paying for these courses.   Maybe there should be a disclaimer on the ISQTB website?

• This foundation exam will teach you about some methods and techniques of testing however it will NOT be able to prove the testing competencies or abilities of those doing the examination.
• It will be able to tell you that the person doing the exam has a good ability to remember stuff or got very lucky when selecting multi-choice answers at random.

I think I would be more comfortable with ISQTB if they provided alternatives and did not sell the exam as a way to be competent tester.

Off track slightly – when I decided to become a rugby league coach many years ago I had to do the following

• exam
• a practical assessment with the trainer
• 3 assessments in the field with an assessor
• keep a training diary for a year, of new stuff I learnt or stuff I implemented 

Only after that year could I class myself as a competent rugby coach.   Hmmm that is an idea maybe ISQTB could do that kind of assessment and training?

We are all responsible for our own training and learning and there are many ways in which we can learn. The problem is I see time and time again many people who call themselves testers and have done no learning or training specifically about testing since sitting the ISQTB foundation course.  This is what really needs to change in our craft.  We need to have a passion for what we enjoy.

I will finish this article with something I tweeted during the discussion and something I really do believe in

Testing is about asking questions and using critical and creative thinking.  It cannot be measured with a simple pass/fail