When I started to look at how the human mind works and the traps that it continually falls into I did not realise what a huge area of psychology this is. The subject of bias and the human mind is fascinating and every tester should be aware that every decision we make when testing a product will be subjected to our cognitive biases.
I have previously touched upon how bias can affect our judgement when I wrote the blog post about confirmation bias and cognitive dissonance. We need to have awareness that what we think could be wrong and subjective to our own biases. There are ways to try and reduce cognitive biases by the use of pairing and debriefing however that is not the subject of this blog. The purpose of this blog is to look at some of the common cognitive biases in relation to their effect on testing.
I shall start by defining the term cognitive bias:
A cognitive bias is a mistake in reasoning, evaluating, remembering, or other cognitive process, often occurring as a result of holding onto one's preferences and beliefs regardless of contrary information.
Within the psychology field of cognitive biases there are many different types of biases some of which I have previously discussed. Within this article I will look at a few more which could have an effect on our testing. The whole area of cognitive bias is huge and I could write many more blogs on different types of bias and it is something I may return to at some point. Since it such a large area I may not go into great detail about each type of bias but give enough information for people reading this blog to be aware of the failings of our human minds.
One bias that intrigues me is called the conjunction effect.
A definition of this bias is described below:
When two events can occur separately or together, the conjunction, where they overlap, cannot be more likely than the likelihood of either of the two individual events. However, people forget this and ascribe a higher likelihood to combination events, erroneously associating quantity of events with quantity of probability.
An example of this can be seen when using the experiment that Amos Tversky and Daniel Kahneman carried out:
Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.
Which is more probable?
A) Linda is a bank teller.
B) Linda is a bank teller and is active in the feminist movement.
B) Linda is a bank teller and is active in the feminist movement.
In the experiment 86% of people answered B even when using mathematics it can be proven than A is more probable. This is the conjunction fallacy in action. When your mind tricks you into believing something is more probable than it is.
Another experiment from Tversky and Kahneman during 1983 two different experimental groups was asked to rate the probability of two different statements, each group seeing only one statement:
- A complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983.
- A Russian invasion of Poland, and a complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983.
Even though the probability of each happening was low there was a significant difference in people choosing that the second statement was more likely.
The moral? Adding more detail or extra assumptions can make an event seem more plausible, even though the event necessarily becomes less probable.
How does this all relate to testing?
Within testing we have to look at statements and judge what is most probable to happen.
For example we see the following requirements:
Req1:If the user is a software engineer then screen ‘is engineer’ must be shown.
Req2: If the user is a software engineer and likes to listen to classical music then screen ‘music engineer’ must be shown.
Now if we look at the following user story.
David went to university to train as a classical violinist. Once leaving university David retrained as a software engineer ad started to write code for a major software house.
Which is more likely to be true?
Most people will see Req2 as the most likely to be true, and ignore Req1. However the probability that Req2 is more likely in the given user story is far less than Req1.
Now with these two requirements you can see a conjunction effect, in which one of the statements appear to more likely than the other and as testers we should be aware of this. However our minds might not notice that there is a conjunction and our human bias takes over so when we test we assume that only req2 is valid and ignore req1. However the probability of req2 is less than req1, we need to be aware of this bias and try to eliminate it. This is why context is important, we need to apply context to the situation and to the test to ensure it is valid, correct and nothing is being missed.
The issue we face as testers is that some requirements and the resulting test ideas we come up with could be subjected to the conjunction effect were our minds are telling us that the probability of event x happening is greater than the probability of event y even when if we look at the mathematics the probability between each event is the same or event x actually has a lower probability.
How can we prevent his bias?
Experiments that have tried to repeat the Tversky and Kahneman bank teller fallacy noted that if before the decision was made people were allowed to discuss and communicate their thoughts with others then conjunction fallacy occurrence was significantly lower.
This indicates that it is possible to reduce the chance of conjunction fallacy occurring just be simply communicating and talking with other people.
When researching this area for the blog I found that there are a lot of links to cognitive framing:
The problem being that the way something is worded (framed) can lead people to be subjective to conjunction fallacy. Maybe this is something that architects and technical authors need to be aware of when creating design and requirement documents? How a requirement is framed could influence a developer to write code in certain way and become subjected to the conjunction fallacy giving more probability to some event or requirement that is actually true, developers please be aware of this.
What fascinates me on this particular area is the idea that the way things are worded (framed) and how our mind understands these words (conjunction fallacy) could be the reason that a lot of bugs are created in code. I would love to gather data on this and see if there is some correlation.
So if we look at framing and how it can influence our thought process and cause bias. The problem with framing is that it can be subjective. If we frame a sentence in such a way as to force people to believe that a certain fact is true then there will be some people in which it will have no influence. Framing is used a great deal within politics and advertising to encourage people to belief in a certain policy or product. It can be very powerful as a tool. Michael Bolton ran a workshop at Eurostar 2010 on test framing (add link) and this was very useful to help the tester to think if a test needs to be run or why it was not run however it was only after Eurostar that I had a thought that framing can be used in a opposite direction and become dangerous as a tool. It could be used to force people to think that the wrong view is the correct view. An example of this is to have a list within a test plan of 1000s of test cases but to frame it in such a way that to the casual reader every single test case must be run and is of high value. This is before the tester has actually touched or used the product to be tested. The framing is used to justify wasted effort and work.
For example look at the following statement:
I prefer to code using Java than C++ because it is easier with the development suite I have installed and C++ does not work within the development environment I have set up.
The framing bias here is that the person thinks Java is superior to C++ because of how they have set up their environment. It maybe that using C++ would make it easier for the developer on the project they are working on but they are trying to give reasons why they are working in Java instead of C++.
There are many more examples in the world of software development. I prefer OS x to OS y because of z. Machine type ‘a’ is far better than machine type ‘b’ because it does ‘xyz’. It does not matter that the person making the statement has never used OS y or machine type ‘b’. They are forming a bias viewpoint and using framing to justify it. IMO this is why the context driven way of testing is so important. When looking at requirements and statements it is easier to think of ‘in which context’ to verify if the requirement is just and sound.
As testers we need to be aware of this and when we report our findings during testing be aware of how we frame the words.
Part 2 of this article will look at belief projection and how it may hinder our testing efforts.
its great sharing i really found this very help full and i hope that every one will get help, thanks to share and we hope that you will keep it update and share information with us
ReplyDeleteresult