Thursday, 26 July 2012

Is testing a manufacturing process or a scientific approach?

Does testing need to move away from manufacturing processes and more towards scientific approaches?

If we take a look at the world of manufacturing you can find many standards (ISO and others) for various everyday things.  However even within the same subject field there can be many competing standards.  For example look at the standards for mains electricity around the world.  Which standard do you use? 

What about light fittings in your country how many different types are available?

In some cases these standards have been changed to meet local and cultural differences in others it has been a question of business and not wishing to pay a royalty fee. However in all these examples context plays a part in the adopting of a standard. Maybe I am being unfair and using examples that are not typical.  From the research I have done I find the same within the QC field, depending on what is being manufactured the processes and standard can and do change for similar products. 

The xkcd comic strip shows this very clearly.

The problem I have within the domain of software testing is that when we apply manufacturing processes to software testing we are making many assumption. The biggest one being

 all software is the same and behaves the same 

which is not true. 

As such we require different approaches to deal with this which is in context with the software being developed rather than trying to make it fit to processes designed around the assumption that everything will be done exactly the same way.

My concern is with the new ISO software testing standard ISO/IEC 2911  which appears to be based on manufacturing process and practices.

The question I have is why? 

Software testing in my opinion is not a manufacturing driven process but more a scientific experimental approach in which the tester has questions and theories that they wish to prove or disprove.So why try and tie a thinking process down to a checklist style document driven process?  Is it because 'management' can have a false belief that it more easily to manage or am I being sceptical?

Looking at the field of science and research - yes I acknowledge there are still some processes but these seem to be based upon what has been done and experienced rather than up front unnecessary documents.  I see very little about processes more a case of techniques and approaches to use. 

If you then add in other fields of science such as social and anthropology you have a wide range of approaches that lend themselves to testing. I see testers more as scientists, researchers or investigators looking at debunking their theories, biases and understanding and trying to learn more about what they are testing.

One approach I stumbled across was the following: and I think this could lead to another blog article there is so much useful information here for testers.

Quoting a couple of them:

  • ensure factual accuracy and avoid misrepresentation, fabrication, suppression or misinterpretation of data
  • critically question authorities and assumptions to make sure that the selection and formulation of research questions, and the conceptualisation or design of research undertakings, do not predetermine an outcome, and do not exclude unwanted findings from the outset 
  • ensure methodology and findings are open for discussion and full peer review

I found this code of practice very similar to the ethics I try to employ when carrying our exploratory testing, as I have already stated I may come back to this with another article at a later date.

I remember in school during science lessons we would:

  • Start from a theory
  • We would then test that theory
  • Write a conclusion based upon what we did 
  • Report what we found.

In my mind this is like exploratory testing

  • We have a charter and a mission
  • We try to prove the mission right or wrong
  • We write about our discoveries and what we learn
  • We report what we find
  • We check to see if what we find matches our original mission

We prove sufficient information for our peers to be able to replicate what we did and see if they come to the same conclusions or not.  We do this by treading similar ground but sometimes not exactly the same steps.

The similarities between exploratory testing and scientific research appear to be many.

  • Formulate a theory
  • Test the theory
  • Explore ways to prove the theory is incorrect (Peer testing)
  • Report your findings

So to conclude I feel that exploratory testing and testing in general has more to do with the sciences than it has to do with manufacturing.  The sooner we can move testing away from a manufacturing process centric way of working to a more natural scientific way of working the better the world of testing will be.  We really must stop testing being forced into a process in which people can tick a box and say yes we have a document for that and for that and that and of course that.   

Testing is a thinking activity not a document creation, ticking boxes checking process.  

Monday, 9 July 2012

Book Review Experiential Learning - Beginning

Experiential Learning Beginning - Gerald Weinberg - A Book review

Telling is not teaching

This is my first attempt at trying to review a book so please be patience with me.

Part of my role in work is to coach and mentor other testers and this can either be informal or formal half day/full day workshops.  I have been fortunate and been given opportunities to coach exploratory testing to other testing teams within our organisation all over the world.  As part of this I do a mixture of both lecturing and practical workshops.  Some of which have been successful and others less so, I have always wondered what makes some successful and others not so.

I follow the blog posts of Gerald M Weinberg and have noticed his comments about putting together some books on Experiential learning and after researching this a little more I found it could be very useful for what I do as a day to day job.

It was because of this research that I decided to purchase the first two books by Weinberg on experiential learning: (There is a third book currently in the process of being written)

This review mainly focus on the first book (I have not completely reading the second one yet.)  Before I start I should state that these books do not offer a step by step guide on how to implement experiential learning into your coaching style.  The book acts more like a tour guide and offers you points on how to think differently in how you organise and run workshops.  These books are about making you think about being creative in your workshops and encouraging people to learn by doing rather than by you telling them. 

This reminds me of a quote by Nassim Nicholas Taleb

We are better at doing than learning. Our capacity for knowledge is vastly inferior to our capacity for doing things – our ability to tinker, play, discover by accident.

I do wonder if this/was a catalyst for the books that Weinberg has written.

Experiential Learning – Beginning 

It is very difficult to do justice in a review of a book which by its design is quite a technical book so I do hope Weinberg will forgive me a little if I quote directly from the book and miss some sections out otherwise this review could be very long.  Any misunderstanding or otherwise from the material within the book are my own making and no way reflect on the views that Weinberg may have been attempting to get across.

The first part of the book gives the reader some examples on how to start to implement experiential learning into their workshops.  It uses a zero level example of dividing your group into two teams and getting them to learn by doing the exercise and seeing what they could do by using the alternative method/approach you are trying to teach them.   From this first chapter I was able to come up with 3 or 4 different zero level examples I could use in my workshops.  The chapter explains the importance of feedback (I agree with this so much and is one of my frustrations with testing we do not feedback often enough.)  Weinberg goes on to explain how to design an experiential class and for using an off the shelf starting point.  All these suggestions give you the reader valuable pointers that can be applied with some thought on your behalf to any workshop exercises that you need to deliver.

The next chapter deals with how to construct an experiential leaning program and introduces the three stages of experiential learning.

  1. Exploration
  2. Invention
  3. Application

Within this section Weinberg presents a quote from Piaget: - Learning was a process of constructing not receiving new knowledge

Which indicates that to learn we need to do and construct our own thoughts and theories with some adjustment from the trainer.  Hence we allow the class to explore, we allow them to invent solution and then we apply the solution to the problems found. (This sounds so much like exploratory testing)  The role of the trainer changes from a lecturing role to a supporting guiding role.

Weinberg then goes on in the next chapter to explain in detail how to create learning cycles within the workshops and getting students to be responsible for their learning. This is expanded in the next few chapters in which bad exercises can be used to create good idea, Evolution an Exercise and improving the exercises metaphor.

The next chapter on using progress observation is one which offered me some insights as to the role of the trainer.  I have come from a traditional background of learning and teaching in which I stand in front of the room and present.  However either from my own experiences or from some unknown source some of my workshops are 100% practical and I take a role in observing and supporting rather than telling the group what should happen.  I have found that these workshops are the most successful in getting the message across and this chapter backs this up with a few more examples on how I can improve what I already do.  I especially like the section on the importance of feelings.  Something I have talked about in the past within this blog.

The next few charters deals with moving your own workshops forward and starting to generate ideas , being able to lead exercises ad keeping  exercises simple.  There are some interesting concepts discussed here along with reference to Weinberg’s other excellent book on the secrets of consulting.  The key elements of these chapters is guidance on prevents you the trainer from jumping in and showing people how to get to the so0ltuons without them working out for themselves, a difficult skill to master.

These are followed by a chapter on the opening exercise in which Weinberg describes the course principals and their importance to successful learning.

These principals being.
  • Safe
  • Comfortable
  • Transparent
  • Fun
  • Considerate
  • Redundant
He then describes methods to form teams and gather requirements for the exercise.

This is followed by a few chapters on working with people who do not and do know each other with some methods that can be used to encourage team work.  It also gives some advice on dealing with conflict issues with the group that may arise. These sections were the only ones that I am not sure at this time offer me any benefits.

There is also in between these chapters a chapter on Virginia Satirs Three Questions in which Weinberg starts a workshop by looking for answers to the three big picture questions.

  • How do they happen to be here? (Past)
  • How do they feel about being here? (Present)
  • What would they like to have happen? (Future)

These provide really good pointers as to why people are attending your workshop and provide the trainer with some valuable knowledge that can be used to help tailor the training so that they want to learn (An important concept of experiential learning)

The book finishes with some pointers on using exercises to help deepen working relationships within the group which appears to naturally lead on to the next book – Invention.  Which I have currently started to read and find it very interesting……

This is not an easy book for a novice to get into and requires you to read and then reread chapters.  This I feel is the intention of the book, it is to encourage you to reflect and think before you act and is a powerful way of helping you to learn and understand something. Would I recommend the book to other?    Yes I would, it is another tool in the arsenal of testers who need to mentor; they provide some very useful techniques and approaches that you can adapt to meet your needs.    There are some section in the book which frustrated me since it appears to concentrate on getting teams to worth together rather than how to use experiential learning to encourage people to learn, but maybe I need to reflect and think more on the message that is being delivered.

They are not a blue print for showing you how to deliver testing workshops more a guide to help you understand that people do not learn from being lectured to using a bulleted slideshow. (Death by Power point)

What I found interesting is that in the past I have been fortunate to have attended a few workshops by Michael Bolton and I can see some of the techniques that Weinberg has suggested within the book being used by Michael.  I do wonder if Michael is a student of experiential learning.  These book could teach the ‘factory schools’ a thing or two about how to learn to do testing, it is by doing rather than telling. That maybe a topic for another blog post!

 As part of my journey learning more about experiential learning I came across the following resources that people may find of interest/use

David A. Kolb on experiential learning
International Consortium for Experiential Learning

**Some corrections made - thanks Phil - I should proof read before hitting the publish button!**