The previous articles in this series have looked at what is critical and creative thinking, defining the stages of testing, looking at the thinking required for documentation review and for test planning This article looks in depth at the test execution stage and the style of thinking that may be best for this stage as described in the diagram from the first article of this series
Test Execution
When it comes to actually carrying out test execution and by that I mainly mean manual testing in an exploratory way by the use of session based test management. There is a need to have a fairly even split between critical and creative thinking. When you are doing test execution you are carrying out
Carrying these simple things out may help to remain focused on your thinking process when testing.
It is not possible to do two thinking tasks at the same time not matter what you believe you think you can do. See here, here and here.
So you need to be creative in thinking what you may want to test next based upon what you are experiences whilst testing. At the same time you are learning about the system and need to think critically about what assumptions you are coming up with from what you see and deciding if what you are doing is the best use of your time.
One method that can be used when testing is to look at test framing as a critical thinking approach that can be useful during the test execution phase. It provides you some questions that can engage your critical thinking side such as:
It should be noted that these questions can be used to during any of the testing stages. I have found that test framing from my own experiences provides the most value during the test execution phases since the problem you are trying to frame is at the forefront of your thinking.
When carrying out your testing session it is important to think critical about the priority. This is something that we tend not to think too much about when testing and mainly think about this during planning or when reporting bugs. I feel when you are in the test execution phase it is important to use the framing techniques mentioned above to adjust your priorities based upon what you are uncovering and the new information you are being presented with. It is not a race to complete your testing sessions and slowing down just a little to think critically about the testing you are doing and thinking about the value of what you are doing at that moment and adjusting the priority based upon your thinking. There are dangers within this approach in that your thinking could be clouded by your many biases , however this can be mitigated later during the test analysis phase.
On the creative side it is important to remember that one of the key parts of exploratory testing is to look for new opportunities to test and the best way to do this is to thinking creatively and as you uncover new stuff that you had not thought about testing before to make a note of this. Note taking is important for effective testing. If you do not make notes you will have a strong chance of forgetting that great idea or concept you had thought about at the time to test. It is important to capture your thoughts and your ideas for future use. This can be done in many ways from the creative use of video , capture and replay tools, wikis , mind maps , session recording tools (Session tester Rapid reporter) to good old pen and paper . If using pen and paper a good way to make your creative ideas stick is the use of drawing and images, I highly recommend the book ‘The back of the Napkin’ by Dan Roam for more information on this. I also suggest talking to Andy Glover (78) (@cartoontester) since he has a lot of great thoughts and ideas on the use of creativity in testing.
How you capture is not important, it is what you capture that is important. One issue I have had over the years of doing exploratory testing is how much is enough details for my notes. One creative method I have found useful is at the end of each day I email my session notes to myself and the first email the next morning I read is my session notes. If I cannot understand them at that time, less than 24 hours after I wrote them, how can I expect other people to understand them? This has over the years enabled me to refine the amount of details I have in my session notes and become better at note taking.
As human beings we seem to be very afraid of failure and try to avoid being in situation in which we are wrong in our thoughts and ideas. In my opinion the best part of using exploratory testing is the ability to be able to test lots of your thoughts and ideas which you have just thought about when testing and not being too bothered if your ideas are right or wrong. It is about proving your theories and it costs very little if they prove to be wrong. We need to have and test lots and lots of our ideas to help us understand more about the system in doing this we are going to lots of failures of our ideas and in many cases we are going to be proved wrong.
I have wrote previously about placing many bets and how useful this is for testing. Franz Johansson in his book ‘The Click Moment’ explains more about the theory being this thinking. To be creative we need to embrace failure and see the learning opportunities this provides us. There is an interesting article on this concept of failure helping us to learn more which has been given the title of ‘Productive failure’. It appears that we can learn more and become better by being allowed to fail rather than only being correct. There are other articles on the fact that allowing us to be confused is good and encourages us to learn more. So when you are testing it might be good sometimes to be confused about the software since it may help you learn more.
So to summarise the execution phase as a tester you need to use both styles of thinking equally. You need to be creative in how you test and what ideas you come up with to test at that time and in the future. Alongside this you need to think critically about what you are currently testing and is what you are doing right there and then the right thing and do your biases influence your approach. It is not easy being to switch between these two styles of thinking during a period of testing and maybe this explains why some people when testing feel mentally exhausted after a testing session since there brain have been engaged in a constant battle between creative and critical thinking.
The next section will look at the style of thinking needed to analyse the testing that has been done and the evidence you have gathered.
Test Execution
When it comes to actually carrying out test execution and by that I mainly mean manual testing in an exploratory way by the use of session based test management. There is a need to have a fairly even split between critical and creative thinking. When you are doing test execution you are carrying out
“Simultaneous test design, execution and learning” (James Bach)When in the test execution phase the spilt between the two styles of thinking is evenly match there are times when you will need to think creatively and time when you need to think critically. Switching between these different styles is not easy and it is a hard skill for a tester to master. The more you practice the easier it does become. However session based test management can be very powerful to help with your thinking styles since it encourages the use of focused and uninterrupted periods of time which is vital for both styles of thinking to be fully utilised. To do this correctly it may be a good idea when doing test execution to do the following:
- Switch off email
- Unplug phone
- Put up a sign saying “please do not disturb thinking testing in progress.”
Carrying these simple things out may help to remain focused on your thinking process when testing.
It is not possible to do two thinking tasks at the same time not matter what you believe you think you can do. See here, here and here.
So you need to be creative in thinking what you may want to test next based upon what you are experiences whilst testing. At the same time you are learning about the system and need to think critically about what assumptions you are coming up with from what you see and deciding if what you are doing is the best use of your time.
One method that can be used when testing is to look at test framing as a critical thinking approach that can be useful during the test execution phase. It provides you some questions that can engage your critical thinking side such as:
- Why are you running (did you run, will you run) this test (and not some other test)?
- Why are you running that test now (did you run that test then, will you run that test later)?
- Why are you testing (did you test, will you test) for this requirement, rather than that requirement?
- How are you testing (did you test, well you test) for this requirement?
- How does the configuration you used in your tests relate to the real-world configuration of the product?
- How does your test result relate to your test design?
- Was the mission related to risk? How does this test relate to that risk?
- How does this test relate to other tests you might have chosen?
- Are you qualified (were you qualified, can you become qualified) to test this?
- Why do you think that is (was, would be) a problem?
It should be noted that these questions can be used to during any of the testing stages. I have found that test framing from my own experiences provides the most value during the test execution phases since the problem you are trying to frame is at the forefront of your thinking.
When carrying out your testing session it is important to think critical about the priority. This is something that we tend not to think too much about when testing and mainly think about this during planning or when reporting bugs. I feel when you are in the test execution phase it is important to use the framing techniques mentioned above to adjust your priorities based upon what you are uncovering and the new information you are being presented with. It is not a race to complete your testing sessions and slowing down just a little to think critically about the testing you are doing and thinking about the value of what you are doing at that moment and adjusting the priority based upon your thinking. There are dangers within this approach in that your thinking could be clouded by your many biases , however this can be mitigated later during the test analysis phase.
On the creative side it is important to remember that one of the key parts of exploratory testing is to look for new opportunities to test and the best way to do this is to thinking creatively and as you uncover new stuff that you had not thought about testing before to make a note of this. Note taking is important for effective testing. If you do not make notes you will have a strong chance of forgetting that great idea or concept you had thought about at the time to test. It is important to capture your thoughts and your ideas for future use. This can be done in many ways from the creative use of video , capture and replay tools, wikis , mind maps , session recording tools (Session tester Rapid reporter) to good old pen and paper . If using pen and paper a good way to make your creative ideas stick is the use of drawing and images, I highly recommend the book ‘The back of the Napkin’ by Dan Roam for more information on this. I also suggest talking to Andy Glover (78) (@cartoontester) since he has a lot of great thoughts and ideas on the use of creativity in testing.
How you capture is not important, it is what you capture that is important. One issue I have had over the years of doing exploratory testing is how much is enough details for my notes. One creative method I have found useful is at the end of each day I email my session notes to myself and the first email the next morning I read is my session notes. If I cannot understand them at that time, less than 24 hours after I wrote them, how can I expect other people to understand them? This has over the years enabled me to refine the amount of details I have in my session notes and become better at note taking.
As human beings we seem to be very afraid of failure and try to avoid being in situation in which we are wrong in our thoughts and ideas. In my opinion the best part of using exploratory testing is the ability to be able to test lots of your thoughts and ideas which you have just thought about when testing and not being too bothered if your ideas are right or wrong. It is about proving your theories and it costs very little if they prove to be wrong. We need to have and test lots and lots of our ideas to help us understand more about the system in doing this we are going to lots of failures of our ideas and in many cases we are going to be proved wrong.
I have wrote previously about placing many bets and how useful this is for testing. Franz Johansson in his book ‘The Click Moment’ explains more about the theory being this thinking. To be creative we need to embrace failure and see the learning opportunities this provides us. There is an interesting article on this concept of failure helping us to learn more which has been given the title of ‘Productive failure’. It appears that we can learn more and become better by being allowed to fail rather than only being correct. There are other articles on the fact that allowing us to be confused is good and encourages us to learn more. So when you are testing it might be good sometimes to be confused about the software since it may help you learn more.
So to summarise the execution phase as a tester you need to use both styles of thinking equally. You need to be creative in how you test and what ideas you come up with to test at that time and in the future. Alongside this you need to think critically about what you are currently testing and is what you are doing right there and then the right thing and do your biases influence your approach. It is not easy being to switch between these two styles of thinking during a period of testing and maybe this explains why some people when testing feel mentally exhausted after a testing session since there brain have been engaged in a constant battle between creative and critical thinking.
The next section will look at the style of thinking needed to analyse the testing that has been done and the evidence you have gathered.