I have just finished reading the excellent book ‘The Drunkard's Walk: How Randomness Rules Our Lives by Leonard Mlodinow and found lots of useful bits of information that could relate to what we could experience when testing. The premise of the book (spoiler alert ----) is that randomness affects our lives all of the time. It talks about why do some people become very successful when others who have similar talents are not as successful? It explains about our natural ability to form patterns where no pattern exists. It gives examples of how we form mental relationships between independent events where there is no relationship (better known as regression towards the mean) It is a really interesting book and ties in with my previous posts on how our cognitive biases can easily fool us and the connection to testing.
One of the best lessons I got from the book was when we have formed a theory how do we prove that theory to be correct or not.
The example used is as follows (see how well you do)
“Suppose I tell you that I have made up a rule for the construction of a sequence of three numbers and that the sequence 2, 4, 6 satisfies my rule. Can you guess the rule? A single set of three numbers is not a lot to go on, so let’s pretend that if you present me with other sequences of three numbers, I will tell you whether or not they satisfy my rule. Please take a moment to think up some three-number sequences to test
Now that you have pondered your strategy, I can say that if you are like most people, the sequences you present will look something like 4, 6, 8 or 8, 10, 12 or 20, 24, 30. Yes, those sequences obey my rule. So what’s the rule? Most people, after presenting a handful of such test cases, will grow confident and conclude that the rule is that the sequence must consist of increasing even numbers. But actually my rule was simply that the series must consist of increasing numbers. The sequence 1, 2, 3, for example, would have fit; there was no need for the numbers to be even. Would the sequences you thought of have revealed this?”
Did you note that the author used the term “test case” – these are like little test cases to prove a theory or idea you have. The author talks in great depth about why people get this wrong most of the time. The reasoning being that once we form an idea or theory we search for ways to prove our idea is correct rather than prove the idea is wrong. There are many more ways to prove a theory is wrong rather than it is right. This is called confirmation bias and I talked about this in my blog here
Does this sound familiar to what we do in testing? We do so much to prove the requirements are correct when really we should be trying to prove that the requirements are wrong. To me within the world of software testing there is a great deal of trying to confirm (validate) what we already know about the software rather than try to ‘test’ what we do not know. Having read this book I can see why it is so easy to fall into this trap, in most cases our natural instinct is to look for positive ways to prove our theory correct, rather than try and disprove them.
When we test it appears as if we are fighting our natural instincts and we can get feelings of uneasiness and hence why some people may appear to struggle to adopt a more exploratory testing approach and find it difficult to move from a confirmation style of testing (checking) towards a more asking questions in which I do not know the answer style of testing. This feeling is commonly known as Cognitive Dissonance ( I blogged about this here.
If we can understand that we as testers should feel uneasy and that it is part of our remit to fight our natural instincts we can then use it as a tool when testing to improve how we test.
The example used in the book is in my opinion a great example of what testing is and is one of the take-a-ways from the book.
