Is testing a manufacturing process or a scientific approach?

Question

Does testing need to move away from manufacturing processes and more towards scientific approaches?

If we take a look at the world of manufacturing you can find many standards (ISO and others) for various everyday things.  However even within the same subject field there can be many competing standards.  For example look at the standards for mains electricity around the world.  Which standard do you use? 

http://en.wikipedia.org/wiki/Mains_electricity_by_country

What about light fittings in your country how many different types are available?

http://en.wikipedia.org/wiki/Edison_screw 

http://en.wikipedia.org/wiki/Bayonet_mount

In some cases these standards have been changed to meet local and cultural differences in others it has been a question of business and not wishing to pay a royalty fee. However in all these examples context plays a part in the adopting of a standard. Maybe I am being unfair and using examples that are not typical.  From the research I have done I find the same within the QC field, depending on what is being manufactured the processes and standard can and do change for similar products. 

The xkcd comic strip shows this very clearly.

(c) http://xkcd.com/927/

The problem I have within the domain of software testing is that when we apply manufacturing processes to software testing we are making many assumption. The biggest one being

 all software is the same and behaves the same 

which is not true. 


As such we require different approaches to deal with this which is in context with the software being developed rather than trying to make it fit to processes designed around the assumption that everything will be done exactly the same way.

My concern is with the new ISO software testing standard ISO/IEC 2911  which appears to be based on manufacturing process and practices.

The question I have is why? 

Software testing in my opinion is not a manufacturing driven process but more a scientific experimental approach in which the tester has questions and theories that they wish to prove or disprove.So why try and tie a thinking process down to a checklist style document driven process?  Is it because 'management' can have a false belief that it more easily to manage or am I being sceptical?

Looking at the field of science and research - yes I acknowledge there are still some processes but these seem to be based upon what has been done and experienced rather than up front unnecessary documents.  I see very little about processes more a case of techniques and approaches to use. 

http://en.wikipedia.org/wiki/Wikipedia:Scientific_standards

If you then add in other fields of science such as social and anthropology you have a wide range of approaches that lend themselves to testing. I see testers more as scientists, researchers or investigators looking at debunking their theories, biases and understanding and trying to learn more about what they are testing.

One approach I stumbled across was the following: http://www.respectproject.org/code/cstds.php?id and I think this could lead to another blog article there is so much useful information here for testers.



Quoting a couple of them:

• ensure factual accuracy and avoid misrepresentation, fabrication, suppression or misinterpretation of data
• critically question authorities and assumptions to make sure that the selection and formulation of research questions, and the conceptualisation or design of research undertakings, do not predetermine an outcome, and do not exclude unwanted findings from the outset 
• ensure methodology and findings are open for discussion and full peer review

I found this code of practice very similar to the ethics I try to employ when carrying our exploratory testing, as I have already stated I may come back to this with another article at a later date.

I remember in school during science lessons we would:

• Start from a theory
• We would then test that theory
• Write a conclusion based upon what we did 
• Report what we found.

In my mind this is like exploratory testing

• We have a charter and a mission
• We try to prove the mission right or wrong
• We write about our discoveries and what we learn
• We report what we find
• We check to see if what we find matches our original mission

We prove sufficient information for our peers to be able to replicate what we did and see if they come to the same conclusions or not.  We do this by treading similar ground but sometimes not exactly the same steps.

The similarities between exploratory testing and scientific research appear to be many.

• Formulate a theory
• Test the theory
• Explore ways to prove the theory is incorrect (Peer testing)
• Report your findings

So to conclude I feel that exploratory testing and testing in general has more to do with the sciences than it has to do with manufacturing.  The sooner we can move testing away from a manufacturing process centric way of working to a more natural scientific way of working the better the world of testing will be.  We really must stop testing being forced into a process in which people can tick a box and say yes we have a document for that and for that and that and of course that.   


Testing is a thinking activity not a document creation, ticking boxes checking process.