Thursday, 26 July 2012

Is testing a manufacturing process or a scientific approach?

Does testing need to move away from manufacturing processes and more towards scientific approaches?

If we take a look at the world of manufacturing you can find many standards (ISO and others) for various everyday things.  However even within the same subject field there can be many competing standards.  For example look at the standards for mains electricity around the world.  Which standard do you use? 

What about light fittings in your country how many different types are available?

In some cases these standards have been changed to meet local and cultural differences in others it has been a question of business and not wishing to pay a royalty fee. However in all these examples context plays a part in the adopting of a standard. Maybe I am being unfair and using examples that are not typical.  From the research I have done I find the same within the QC field, depending on what is being manufactured the processes and standard can and do change for similar products. 

The xkcd comic strip shows this very clearly.

The problem I have within the domain of software testing is that when we apply manufacturing processes to software testing we are making many assumption. The biggest one being

 all software is the same and behaves the same 

which is not true. 

As such we require different approaches to deal with this which is in context with the software being developed rather than trying to make it fit to processes designed around the assumption that everything will be done exactly the same way.

My concern is with the new ISO software testing standard ISO/IEC 2911  which appears to be based on manufacturing process and practices.

The question I have is why? 

Software testing in my opinion is not a manufacturing driven process but more a scientific experimental approach in which the tester has questions and theories that they wish to prove or disprove.So why try and tie a thinking process down to a checklist style document driven process?  Is it because 'management' can have a false belief that it more easily to manage or am I being sceptical?

Looking at the field of science and research - yes I acknowledge there are still some processes but these seem to be based upon what has been done and experienced rather than up front unnecessary documents.  I see very little about processes more a case of techniques and approaches to use. 

If you then add in other fields of science such as social and anthropology you have a wide range of approaches that lend themselves to testing. I see testers more as scientists, researchers or investigators looking at debunking their theories, biases and understanding and trying to learn more about what they are testing.

One approach I stumbled across was the following: and I think this could lead to another blog article there is so much useful information here for testers.

Quoting a couple of them:

  • ensure factual accuracy and avoid misrepresentation, fabrication, suppression or misinterpretation of data
  • critically question authorities and assumptions to make sure that the selection and formulation of research questions, and the conceptualisation or design of research undertakings, do not predetermine an outcome, and do not exclude unwanted findings from the outset 
  • ensure methodology and findings are open for discussion and full peer review

I found this code of practice very similar to the ethics I try to employ when carrying our exploratory testing, as I have already stated I may come back to this with another article at a later date.

I remember in school during science lessons we would:

  • Start from a theory
  • We would then test that theory
  • Write a conclusion based upon what we did 
  • Report what we found.

In my mind this is like exploratory testing

  • We have a charter and a mission
  • We try to prove the mission right or wrong
  • We write about our discoveries and what we learn
  • We report what we find
  • We check to see if what we find matches our original mission

We prove sufficient information for our peers to be able to replicate what we did and see if they come to the same conclusions or not.  We do this by treading similar ground but sometimes not exactly the same steps.

The similarities between exploratory testing and scientific research appear to be many.

  • Formulate a theory
  • Test the theory
  • Explore ways to prove the theory is incorrect (Peer testing)
  • Report your findings

So to conclude I feel that exploratory testing and testing in general has more to do with the sciences than it has to do with manufacturing.  The sooner we can move testing away from a manufacturing process centric way of working to a more natural scientific way of working the better the world of testing will be.  We really must stop testing being forced into a process in which people can tick a box and say yes we have a document for that and for that and that and of course that.   

Testing is a thinking activity not a document creation, ticking boxes checking process.  


  1. Hi Steve,

    The title of the post - I'd say testing could be applied in both (and more) ways, so it depends who the question is addressed to. Think: is testing exploratory or scripted(?) - it depends on how it is applied and by whom and when.

    Standards - yes, they're needed. They help interoperability testing.

    The new testing standard - yes, I'm with you on that - there is no visibility into the standard (unless you are in the working group) and so it seems very strange with the process when, arguably, a (great?) deal of test thinking / writing happens outside of that working group. Hmm, what to conclude from that….

    At Let's Test Christin Wiedemann gave an interesting presentation of "the" scientific method and software testing. The slides are available on the Lets Test site, [1].

    Your fear about software being developed in a uniform way is a sign of classic Taylorist thinking - and has many typical consequences - I wrote about some of these connected with testing, [2]. This is complicated when people come in the "best practices" avenue, and is combined with cognitive fluency, such as [3].

    I'd suggest we work more with hypotheses rather than theories.

    Test thinking and thinking about testing is evolving! And that's a good thing!


  2. Great response Simon and thank you for pointing me towards these useful articles.

    I like the thoughts on looking at hypothesis instead if theory and you are correct context plays a part in trying to answer the question.

    one point regarding standards for interoperability testing that is ok as long as it is of user benefit and not business driven thinking open standards for cable in US

  3. Superb write up, John!

    Let meadd my own point "Innovation comes with trying – and testing ideas outside the comfort zones, not piling up data well inside the comfort zones."