The Importance of Worth

I am going to start this article with a reflection of when we were children.

I want you to imagine that day in school when you was a very young child and you produced your first ever painting.  You took all day to produce it, making careful use of colour and getting exactly how you wanted it to look.  At the end of the day you took your painting home to show your parents.  You were so excited and full of joy and expectations of what your parents would say about all the hard work you had done.  You ran into the living room with your painting in your hand and shouted out “Look Look, what I have done today”.  Your parents come over and take great interest in what you have produced, commenting about how clever you are and how wonderful you are.  They say how proud they are of you and they place your artwork on the fridge in the kitchen where everyone can see it.

Now if you have recalled this scene in your mind and many of you will do so.  How are you feeling?  Did the thought make you happy?  Did you feel pride in what you did?  Are you smiling at this thought?

So now let us zoom toward to present date…

You spend days/weeks/months (Cross out which is applicable) creating test scripts based upon assumptions, writing them up in whatever test case management system you have been told to use.  You put all your effort and thinking into being creative creating these step by step instructions for ‘testing’ the system.

After you have done this you then get ready to start testing using the work you have spent so long creating.  Once you start testing you realise that most of what you have already done upfront, all that effort is not going to be used.   So all these test scripts which you sweated over creating and completing in step by step precise detail, get ignored, never see the light of day, the labour of your work, forgotten and not commented on.

How often when we are told we must follow a scripted testing approach does this happen?  If we are honest it does happen a lot, I know to me in the past over half the scripts I created never got reviewed or used.  Half of the work I did was just forgotten about and left to gather dust in the test case repository.  I should make it clear that I am not against test scripting and that with the correct context they have value but indiscriminatingly forcing people to do something without experiencing it is in my opinion is such a stupid and pointless exercise.

Let us step back to our story from earlier.

Now imagine as a child you rush home to your parents with your painting in hand and once at home your parents take your painting and without saying a word lock it in a drawer and carry on with what they were doing.  How would you feel now?  Place yourself into the mind of your child self and imagine how would you feel?  Upset?  Sad?  Hurt?  Worthless?

So why when we do something as creative as testing do we do this?  We create so much in the way of test scripts but never get the chance to be proud of what we have done, what we have achieved.  We lock it away never to be used again, never to be talked about.  Is it any wonder that so many testers feel sad, unhappy and worthless in what we are being asked to do.  It is a key aspect of human nature that we want to show people what we have done, what we feel proud about, we need feedback to know that the tasks we are performing are worthwhile.  We need confirmation that we are valuable, needed and wanted.  If we continue to carry on with this path of insisting on doing pointless and useless tasks in which we then ignore or and just throw it away then we deserve to feel the way we do.

There are alternatives, using the exploratory testing approach can help prevent this waste; based upon only doing what is necessary at that time, using context.  Session based testing can make sure feedback on what you are doing becomes a key element of the testing approach.

Let us start to feel that we are important to software development and that testers are a worthy addition to this.

Some useful reading:

Session Based Test Management

What is Exploratory Testing 

Exploratory Testing Resources

Principles of Context Driven

Great Expectations

I recently spent some time running exploratory testing workshops in India and found I had some free time to start to reduce the mountain of books I have on my Kindle. I managed to read two books by Dan Ariely

Predictably Irrational, Revised andExpanded Edition: The Hidden Forces That Shape Our Decisions

The Upside of Irrationality: The Unexpected Benefits of Defying Logic at Work and at Home


Within these books there are some great insights into how we think we behave and how we actually behave, Dan calls the work he does behavioural economics.  There are many interesting article he talks about in his books and most of them I can relate to software testing.  The one I want to pick up on for this blog article is how we can be easily influenced into following a certain path and making us act in a predictably way by manipulating our expectations.

The worlds of advertising and marketing have very clever ways in manipulating us into buying their products.  One of the ways they can do this is to ‘Prime’ us, by doing this they make us think of a subject or a product so that we unconsciously act in a way that makes us want that product and only that product.

For example if right now I asked you to come with words that are associated with being elderly, what words would you come up with?  If for the next 10 minutes I said think about this and you came up with a list of positive and negative words for elderly.  After you have done this I then ask you to perform some tasks.  When you perform this tasks you will be slower, take more time and notice little aches and pains you have.  All this is from just thinking about the term elderly.   The use of association has a very powerful effect on our unconsciousness.  Taking this a stage further if you have been primed your expectations have been manipulated so that you tend to have a bias towards the initial priming.  For example if you are told beforehand that a certain type of coffee is unique, expensive, has a secret ingredient and tastes wonderful.  You will at some point have to try it and once you do because of all the priming you have to like it (If you like coffee that is – replace coffee with chocolate, beer or whatever your favourite thing is) your expectations have forced you to enjoy it.  Even if your taste buds are saying it tastes vile, if you have paid a lot of money for it and have been told many times how wonderful it is it.  You will tell yourself it is wonderful and amazing.  Priming your conscious is a powerful bias that can override many other indicators.

Now what if I tell you that the secret ingredient is elephant dung now you have this knowledge your mind will be changed, what if I told you this before you decided to buy the product, would you still buy the product?

So how does all of this relate to testing?

Imagine if all you are doing when ‘testing’ is validating the requirements, your expectations have been primed and managed.  If you keep hearing that the software is bad by the development teams, that the model being used is poor. Then these all force you to be primed and you automatically assume the product is poor and that the requirements are what we should expect the product to match.  Can you see how dangerous this would be?  You are priming yourself to only confirm what the requirements are or what people are saying. 

One of the ways to help resolve this is by using an exploratory testing approach, which can help to reduce your expectations and assumptions of the product under test.  It tries to achieve this by the use of models, oracles, and heuristics to ensure that your beliefs, biases and expectations are constantly being challenged as you are testing. 

Michael Bolton at Developsense has recently wrote some articles on oracles and heuristics on his blog page.

Software Quality Characteristics Poster

Having just read an excellent post by Sigge on using the Software Quality Characteristics created by the Test eye people and based upon the James Bachs Heuristic Test Strategy Model. I remembered I have created a poster based upon the Software Quality Characteristics. I have uploaded the poster to Google docs here:

https://docs.google.com/open?id=0B8jc_cHKwbNoNGZnXzBJcUlpTlU

It is in PDF format and has been designed to print at A1 size (very big)

enjoy.

More randomness

I have just finished reading the excellent book ‘The Drunkard's Walk: How Randomness Rules Our Lives by Leonard Mlodinow and found lots of useful bits of information that could relate to what we could experience when testing. The premise of the book (spoiler alert ----) is that randomness affects our lives all of the time. It talks about why do some people become very successful when others who have similar talents are not as successful? It explains about our natural ability to form patterns where no pattern exists. It gives examples of how we form mental relationships between independent events where there is no relationship (better known as regression towards the mean) It is a really interesting book and ties in with my previous posts on how our cognitive biases can easily fool us and the connection to testing.

One of the best lessons I got from the book was when we have formed a theory how do we prove that theory to be correct or not.

The example used is as follows (see how well you do)

“Suppose I tell you that I have made up a rule for the construction of a sequence of three numbers and that the sequence 2, 4, 6 satisfies my rule. Can you guess the rule? A single set of three numbers is not a lot to go on, so let’s pretend that if you present me with other sequences of three numbers, I will tell you whether or not they satisfy my rule. Please take a moment to think up some three-number sequences to test

Now that you have pondered your strategy, I can say that if you are like most people, the sequences you present will look something like 4, 6, 8 or 8, 10, 12 or 20, 24, 30. Yes, those sequences obey my rule. So what’s the rule? Most people, after presenting a handful of such test cases, will grow confident and conclude that the rule is that the sequence must consist of increasing even numbers. But actually my rule was simply that the series must consist of increasing numbers. The sequence 1, 2, 3, for example, would have fit; there was no need for the numbers to be even. Would the sequences you thought of have revealed this?”

The Drunkard's Walk: How Randomness Rules Our Lives by Leonard Mlodinow

Did you note that the author used the term “test case” – these are like little test cases to prove a theory or idea you have. The author talks in great depth about why people get this wrong most of the time. The reasoning being that once we form an idea or theory we search for ways to prove our idea is correct rather than prove the idea is wrong. There are many more ways to prove a theory is wrong rather than it is right. This is called confirmation bias and I talked about this in my blog here

Does this sound familiar to what we do in testing? We do so much to prove the requirements are correct when really we should be trying to prove that the requirements are wrong. To me within the world of software testing there is a great deal of trying to confirm (validate) what we already know about the software rather than try to ‘test’ what we do not know. Having read this book I can see why it is so easy to fall into this trap, in most cases our natural instinct is to look for positive ways to prove our theory correct, rather than try and disprove them.

When we test it appears as if we are fighting our natural instincts and we can get feelings of uneasiness and hence why some people may appear to struggle to adopt a more exploratory testing approach and find it difficult to move from a confirmation style of testing (checking) towards a more asking questions in which I do not know the answer style of testing. This feeling is commonly known as Cognitive Dissonance ( I blogged about this here.

If we can understand that we as testers should feel uneasy and that it is part of our remit to fight our natural instincts we can then use it as a tool when testing to improve how we test.

The example used in the book is in my opinion a great example of what testing is and is one of the take-a-ways from the book.

Patterns from Nothing

Continuing on from my previous post on Cognitive Illusions I thought I would start with the ability we have as humans to be fooled into seeing patterns from nothing. It is common for people to find shapes or objects when starting at the clouds or to think that there is a pattern of luck associated with the game we are playing. We can look at a random set of data and without doubt we will naturally make a pattern. Ben Goldache talks about this in his book Bad Science[1] It is in our nature and we are over sensitive to making patterns when none exist. Look at the following example of tosses of a coin, H-Heads, T-Tails.

HHHHHHHHHHHTHHHHHHHHHHHH

Now what conclusion would you make from this set of results?

Have you come up with any?

If you have come up with a conclusion that is your natural instinct and intuition to create a pattern and a cognitive illusion. Given that the coin is true the chances that the sequence above would happen is the same as any other sequence. Take this one step further and I ask you to say what the result will be on the next coin toss.

What would you answer?

Why would this be your answer?

Using statistics the possibility of it being H or T is 50/50 or equal chance. This is the reason casinos make so much money they know we are all fallible and use that against us. We make the mistake that there is a pattern and that our luck must change. I am sad to inform you but there is no luck the chances are still the same and within a casino the odds will always be against you.

The same can be applied to those who follow sport and come across the phrase that someone is on a lucky streak; this again is our natural bias to create a pattern when none exist. For example a soccer player has the follow goal scoring record. (X means scored in the game – O – means did not score.

XXXXXOOXXXXXXXXXXOOOOOOOO

Our tendency to create a pattern means that we will take that data and say the player has had 2 lucky streaks of scoring and is currently having a dip in form. With such simple data it is so easy to create and formulate assumptions and make patterns where there is no pattern. This is especially easy to do if there is no context. The simple example above proves the need to have some context. If I gave some more information that the player above has for the last ten games been playing in the senior side instead of the juniors, would that make a difference to your conclusion?

So how does this apply to testing?

There is a talk within testing that we should trust our intuition (I am one of these people to talk about this) and go with our gut feelings. Malcolm Gladwell in his book Blink [2] describes this to great effect. However we need to be aware that our intuition can try and fool us and try to create patterns when we are carrying out our testing. The problems come when we start to see these patterns and this causes us to miss other information that may be important.

For an example of this watch the following video (Information provided by Gordon Pitz [3] )

http://www.youtube.com/watch?feature=player_embedded&v=vJG698U2Mvo

[7]

Have you watched the video?

No?

Please go and watch it, it will help you understand the rest of this article.

Did you see the Gorilla? [4]

No?

This might be due to being distracted and focused on a task. Noticing patterns and forming inconclusive assumptions when there are none can cause the same effects and as such it does show the point that our minds can be easily distracted and miss important information. It is important when we are testing that we do not spend too much of our time looking and investigating patterns since out natural instinct is to see patterns we could end up missing a lot more important information.

This is vital when we are testing using the exploratory testing approach where it is very easy to go off track and away from our mission to investigate what we think is a pattern of behaviour within the system under test. It is best in these situations to make a note of it and continue on track.

Sometimes it is difficult to go against what is natural and some find it near impossible and this could be one of the reasons why the exploratory testing approach may not be suitable for them or they find it too difficult. I hope that this article will encourage those who have struggled to have another go knowing that sometimes that could be fighting against their own instincts and as such making it appear more difficult for them.

So are there are techniques that can be used to help resolve this bias?

The problem is that since this is a natural built in instinct, and because we are aware of it, it does not necessarily mean we can resolve it.

“Knowing that it exists does not remove it”

Gordon Pitz [3]

There are few techniques that could help

One previous described when using session based testing is to keep to your mission and make a note of interesting patterns that you think are emerging. Later when you do a feedback session to others explain your thoughts about the pattern and see if others see the same pattern. If they do not it could be a case that you see a pattern when there is none

.

Another way which may help to prevent this bias is to use paired testing, there is gathering evidence that social facilitation [5] can help to reduce cognitive bias and paired testing is one way to make use of social facilitation. We seem to be more attentive and aware when we are being observed. It should be used with caution since if the task is complex and difficult people will perform badly. So this can only really be used when the task is not over complex.

One more technique that I have found invaluable is the use of testing framing as mentioned by Michael Bolton [6]. I attended a course on this and I do recommend that people read the article on his website. Using this approach helps the tester to focus on the purpose of the test but it also has a cool side effect that it can help to remove this bias to see patterns when there are none. It works especially well when you have to justify your reasoning.

The next article will look at the cognitive illusion of regression to the mean and its possible impact on testing.

References:

[1] Ben Goldache – Bad Science - http://www.amazon.com/Bad-Science-Quacks-Pharma-Flacks/dp/0865479186/

[2] Malcolm Gladwell – Blink - http://www.amazon.com/Blink-Power-Thinking-Without/dp/0316010669

[3] The Deceptive Nature of Intuition – Gordon Pitz- http://www.unc.edu/~gpitz/pdf/Chabris-Simons%20review.pdf

[4] The invisible Gorilla - Christopher Chabris and Daniel Simons - http://www.amazon.com/Invisible-Gorilla-How-Intuitions-Deceive/dp/0307459667

[5] Social Facilitation - http://en.wikipedia.org/wiki/Social_facilitation

[6] Test framing – Michael Bolton - http://www.developsense.com/blog/2011/05/ive-been-framed/

[7] http://www.theinvisiblegorilla.com/videos.html

Cognitive Illusions

or how your mind plays tricks on you.

People who regularly read my blog may be aware that I have a keen interest in psychology and how it can relate to testing. If you have not read my blog before wow welcome first timer I hope you enjoy and come back for more articles in the future.

I have in the past written a few articles about bias (here, here, here and here) and how it can be dangerous when we are testing. Having just read an excellent book called Bad Science by Ben Goldache I thought I would revisit this subject since Ben has a whole chapter on this very subject called

‘Why Clever People Believe Stupid Things’

It is a very interesting chapter and it made me re-think about the need to be careful when we are testing and reporting what we believe has happened. The human mind is a tricky beast and there are various methods it uses to try and trick us into believing things which are not true.

For example take a look at the following picture by French artist Felice Varini (the site is in French) This is a fantastic anamorphic illusion in which our mind joins all the pieces together to make us see something that in reality is not real.

http://www.varini.org/02indc/33indcq10.html

Looking at it from a different perspective shows us this.

http://www.varini.org/doshpdv/dos2010/18-10-hd.html

An important lesson in testing is not to look at things from only one point of view. See how our mind tricks us in to thinking something is real when it is not.

Ben Goldache manages to breakdown some of the common tricks our mind plays into the following:

# Randomness

# Regression to the Mean
# The bias towards positive evidence
# Biased by our prior beliefs
# Availability
# Social influences

Which he concludes with the following statements

1 - We see patterns where there is only random noise.

2 - We see causal relationships where there are non

3 - We overvalue confirmatory information for any given hypothesis.

4 - We seek out confirmatory information for any given hypothesis.

5 - Our assessment of the quality of new evidence is biased by our previous beliefs.

6 - Our assessment of the quality of new evidence is biased by our social influences.

(I added the 6th one myself)

Once we become aware of these illusions that our mind plays on us we can start to put practices in place they helps to try and remove them. I should warn you it is impossible to remove them entirely since we are only human after all, but being aware that they exist is a good start.

Over the next few blog articles I will be taking each one of these topics and applying it to testing

The Purpose of Testing

I have a strong passion for psychology and the social sciences and their connection to software testing. I currently have a few books on the go on these subjects and hope to write up my thoughts on these books and their connection to testing in the future within the blog.
For those that are interested the books I am currently reading (and re-reading – to make sure that things I assumed from within the books are correct) are:

• Qualitative Data Analysis A user friendly guide for social scientists - Ian Dey
• Thinking fast and slow – Daniel Kahneman
• Research Methods for Everyday Life – Scott W Vanderstoep

One interesting quote I found in the book by Ian Dey was the following:

Exploration, description and explanation are the three purposes of social science research.
Earl Babbie

Looking at this quote made me think about what the purposes of testing are and I came to the conclusion that this is the same as for social science research as quoted by Earl Babbie.

If we break this down we have:

• Exploration: This is done using exploratory testing, charters, missions etc
• Description – Let us describe what we are doing and what we have done when testing
• Explanation – Let us explain to managers, peers, stakeholders what we found when testing and our findings.

There are lots of articles, discussions, books about the purpose of testing and how very complex it is, this single sentence quote in my opinion sums up everything about the purpose of testing

I made a note to have a look at what Earl Babbie has got to say and found he has written lots of articles and books some of which could/may apply to software testing, it looks like I have added a few more books to my ever expanding reading list.