The previous articles in this series have looked at
what is critical and creative thinking,
defining the stages of testing, looking at the thinking required for
documentation review,
test planning and test execution. This article looks at the test analysis phase and the styles of thinking that may be best for this stage as described in the diagram from the
first article of this series
Test Analysis
So you have finished your testing session and gather together all sorts of evidence of what you have uncovered, discovered and learnt. So now is the time to look in detail at what you have done and found and apply some critical thinking to the task.
This is one stage within testing which I feel is understated and often not a great amount of time and effort spent on it. However I think it is one of the most valuable for the tester who carried out the testing since it allows them to analyse what they have done and to think critically about themselves and see if there improvements they can made.
It is interesting if you ‘Google’ test Analyst and ‘definition’ the variety of the responses that are returned. A selection of abstracts is shown below:
“A test analyst reviews the results of process tests in a company's operating systems or manufacturing processes. The analyst also researches potential defects and works in tandem with engineers to provide solutions.”
(Ehow.com – Test Analyst Job Description)
“In the position of test analyst, a person fulfils an important role by reviewing results from process tests in a business’s manufacturing or operating systems. The analyst will also research potential deficiencies and work together with engineers in order to provide solutions”
(Job is Job Test Analyst job description)
“The Senior Test Analyst is responsible for designing, developing, and executing quality assurance and control processes, test strategies, test plans and test cases that verify a software conformance to defined acceptance criteria (i.e. System behaviours) and feature design documents, as well as application standards”
(Aus registry job specification Senior Test Analyst)
“Works with the Testing team to assist with the preparation of test plans and the testing of software in line with company guidelines and standards”
(Vector UK Job description Test Analyst)
What I find interesting that there appears to be two main definitions of a test analyst one who analysers what has been tested and one that plans, executes and analysers the test results. It appears over time that what could or may have been a specialist role has now become one which is interchangeable with the title of ‘tester’. There is nothing wrong with this and it may slightly digress from the purpose of this section but I thought it was a useful comparison of what the definition of what a test analyst is.
In my own world it is a stage that all testers need to be proficient in and having the necessary skills and thinking to carry out the task. The analytical skills of testers IMO appears to be a forgotten skill or one in which less importance is being placed.
DEBRIEF
The first thing that should be done after you have completed your test execution phase is to reflect on all you have done and the best way to do this is to debrief and talk to other people. There are many ways that you can do this but one model within the session based testing management frame is by the use of
PROOF (expanded to
PROOF LA). This is a very useful and quick way to explain what you have done, what you need do and what you plan to do by the use of some simple questions.
- Past. What happened during the session?
- Results. What was achieved during the session?
- Obstacles. What got in the way of good testing?
- Outlook. What still needs to be done?
- Feelings. How does the tester feel about all this
- Learning. What have you learnt? What could you do better?
- Adaptation. What do you need to adapt or change?
If you working in small teams then it may not be possible to do this style of review however there is nothing to stop you doing some self-reflection and using these question to debrief yourself. You may notice that these questions require both critical and creative thinking in equal measures. The questions will make you either think critically about what you have done or need to do against what you could do better and improvement ideas that your creative thinking would help generate.
WHAT DID YOU LEARN?
When you have completed the debrief it is important to spend a little more time thinking about the ‘new’ stuff you have learnt. This is valuable information that could be used not just for you but for other people. Using the evidence you have gathered you could put together some ‘how to guides ‘if there were parts of what you did that was surprising or difficult. This also aids your own memory and helps to reinforce your learning. The added benefit is that others can look at this and use it to aid their own learning and understanding. The way I implement this is by the use of a wiki in which for the project we are working on we have a section in which we link or create useful information.
DEFECTS
Looking at your testing evidence the next thing you may want to do is think critically about the issues you found which could be defects. You may want to first try and repeat the issue to ensure it is reproducible. (Some might not be) You may want to talk to a developer, customer or architect to discuss if it is really a defect. If after thinking about the issue you may then want to raise it as a defect and attach the evidence you captured during the test execution phase. To create a good defect report requires a lot of critical thinking on the part of the tester and I would highly recommend that you participate in the
Bug Advocacy course or at least work your way through the ‘free’
course material. If you really want to get your defects fixed you need to present a compelling argument as to why this defect needs to be fixed, this course can help you achieve this.
*To do the Bug Advocacy course you need to complete the
Black Box Software Testing foundations course first.
AUTOMATION
Once you have raised your defects you could now use a mixture of creative and critical thinking to see what of the evidence you have gathered what would prove to be useful to automate. In this context I am talking about the use of
BDD test frameworks. At the same time it could be worth using some creative thinking to see what automation tools could help support your exploratory testing. It is useful to remember that automation is not just a checking exercise but can provide invaluable to exploratory testing to aid their testing. Michael Bolton wrote a
blog post on this subject where he talks about manual and automated testing and the meaningless use of labels.
FUTURE IDEAS
One area in which people when carrying out exploratory testing appear to miss out on is looking for future opportunities to test.
We forget the definition of exploratory testing especially the “test design” part
“Simultaneous test design, execution and learning”
(Exploratory Testing explained)
If you did not make any notes of future things to test when carrying out the test execution phase then you may soon have no job to do! This is an important aspect of exploratory testing and one in which you need to remain focused on when testing.
If you do have a list of ideas, this is the time to use some more critical thinking and see which ideas have value within the context of the project you are currently testing. You can if you like give each idea a priority and a risk value if this helps your critical thinking of the value of the idea. Other ideas that may help to critical evaluate your test ideas is to discuss them other members of the team. You could also apply some
testing framing. It should be noted that when you are critically thinking your future test ideas your creative thinking side may become active to come up with other novel ideas that you may be able to test. You should also note these down as well since they could prove to be valuable.
Improvement
We are not perfect and constantly look for ways in which we can improve. An aspect of the test analysis is to reflect on what you have done and think critically about what improvements you could make.
The field of social science and ethnographic research can be helpful here and I wrote an
article on this. From this article I put together a way that tester can use reflection to help improve testing An abstract of which can be seen below:
Reflect
Personal reflection:
- Could you have done things better if so what? (Both from a personal and testing perspective)
- Have you learnt new things about the product under test (That are not documented)?
- Has your view of the product changed for better or for worse? Why has your view changed?
‘Epistemological reflexivity’ (What limits did we hit?)
- Did your defined tests limit the information you could find about the product? (Did you need to explore new areas that you had not defined)
- Could your tests have been done differently? If yes how?
- Have you run the right tests?
- If you did things different what do you think you would have found out about the product?
- What assumptions have you uncovered to be true/false?
- Did the assumptions you make impede or help your testing?
The University of Plymouth produced an article of
critically thinking reflection which has some useful ideas that may help when you come to reflect on your own improvements. They have also produced a nice
critically thinking poster.
As you can see from the above for the test analysis phase you need mainly to be critical thinking with some creative thoughts since the majority of the work you do during this phase is to reflect on what you have done, need to do and how well have you done it.
The next section will look into the test reporting phase.
