This is the final post in this series on creative and creative thinking and testing. It has been a journey of discvoery for myself and along the way I have found out that there is more to how we think when testing than even I first thought and all of this came about from an initial posit-it note diagram. Along this journey we have:
This final post of the series will look at the styles of thinking required when we are reporting our testing
Test Reporting
So after the planning execution and analysis you are now ready to report your finding. The style of thinking required for this phase appears to be obvious in that you need to be creative in how you are going to present the information you have found. You need to make sure that it is clear and easy for your reader to understand without any possible chance of it being misunderstood or more dangerously misused. To do this you will need to think a little bit critically and ask yourself can the following about the information you are presenting:
There are many more questions that you can ask yourself but the key of this level of critical thinking is to ensure you are objective about what you report and unbiased as possible in your reporting. There are a few methods that can used to help with reporting and we can learn a little from the medical world and how they use critical thinking in helping with the medical research reporting
It is important to think about what should be not included since this aids clarity for the reader.
Returning to creative thinking one effective and creative way to represent your testing information is by the use of dashboards.
Dashboards
James Bach talked about a low tech testing dashboard on the rapid software testing course and has some information about it on his website
Del Dewar went slightly further and presented the dashboard as shown below.
More information on what each of the columns numbers or colours mean can be found on the links – in other words I want you to do a bit of research into dashboard and how useful they may be for your own test reporting.
From my own experience of using these styles of dashboards for test reporting what I found was that it gave a very quick overview of the project and of the issue but was not good at reporting progress which is something that test management required and this leads to storytelling and metrics.
Session Notes and Wiki Links
One more thing to add here is that when I tried the above with senior management teams there was a request for access to the data of what was actually tested, in other words they wanted access to the session notes and actual evidence of what had been done. To solve this at the top level of each dashboard we provided a link to the wiki sessions where we kept all the session notes. I encourage you to have full transparency of testing effort and allow access to all who wants to see what testing has taken place and I feel it helps if there are no barriers or logins in the way for people to be able to access the raw data.
If as we described earlier in this document we are using session based test management then we should be producing evidence of what we have tested and the information we have found as we go along and test and we are using whatever is the best method for capturing this, video, screen capture, wiki. This should be in a place in which all have access to and everyone (who matters) is aware of its location.
Storytelling
The next thing that you need to do with your test reporting is to tell a story or two. This again requires some deep critical thinking. Michael Bolton says that test reporting is about the telling of three stories at three levels. I provide a quick summary of this below for full details refer to the original article link available here.
Michael has a lot more information about test reporting in a great series of articles:
Markus Gärtner summaries this in his article titled “Tell a story at the daily stand up”.
As can be seen from the articles published by Michael Bolton you quickly switch from one style of thinking to the other depending on the context of the story you are telling. This is a difficult skill for a tester to master but once you practice it you can become an expert test report story teller.
Another way in which you can be creative and report your testing as well as your test planning is by using mind maps. Darren McMillan produced a great article on this and it can be found here.
It is important also at this stage to remember about your test plan and look at what information you will need to update in this. From what you found out during testing and how you risks and priorities may have changed need to be reflected in your test plan.
Qualitative vs Quantitative
There have been many discussions within the testing community about qualitative and quantitative measurements some of which I will share here as useful information. It is very easy to fall into the trap that numbers tells the whole story and since they are easy to collect will provide all the information that management require. I think we need to be careful of this and use our critical thinking to make a judgement on what the numbers really provide.
Cem Kaner has an excellent article on the validity of metrics here and the thing I most noted about this was the following:
I agree with this but we need to use both our critical and creative thinking to provide the story to go with the metrics. I think we all agree that quantitative measures are flawed but we need to be able think creatively to resolve this and provide the information the customer requires in a way which is simple and easy for them to understand without misleading anyone.
Some of the discussions within testing community of test metrics
NEXT
So you have got to the end of this article and hopefully have a understanding that the different stages of testing requires different types of thinking at different levels. So what do you do now? First of all this is not the end of the journey. You now go back to the start and continue until told not to at the same time you can continue to practice some of the lessons and examples given in this document. Improve them, be creative and create your own, adapt them to fit your needs. This is not a document of best practice it is a springboard to help you, a reference guide, if you like, that can be modified and act as a starting point for you to improve and learn more about what style of thinking you may need when testing.
The important lesson is that testing requires a great deal of thinking and if you are not thinking when you are involved in any testing activity then maybe, just maybe, you may not be testing.
Enjoy.
John Stevenson
- looked at what is critical and creative thinking,
- and tried defining the stages of testing,
This final post of the series will look at the styles of thinking required when we are reporting our testing
Test Reporting
- Can it be interpreted in different ways?
- Is the most important information clearly shown (in the context of what is important to the reader)?
- Have I made the context clear?
- If using numbers am I using this to back up a story?
- Have I made the main risks and issues very clear?
- Is what I am reporting ethnically and morally correct?
There are many more questions that you can ask yourself but the key of this level of critical thinking is to ensure you are objective about what you report and unbiased as possible in your reporting. There are a few methods that can used to help with reporting and we can learn a little from the medical world and how they use critical thinking in helping with the medical research reporting
“Students should also be encouraged to think about why some items need to be reported whereas others do not”
It is important to think about what should be not included since this aids clarity for the reader.
Returning to creative thinking one effective and creative way to represent your testing information is by the use of dashboards.
Dashboards
James Bach talked about a low tech testing dashboard on the rapid software testing course and has some information about it on his website
Low Tech Testing Dashboard Example – James Bach - taken from these slides
Del Dewar went slightly further and presented the dashboard as shown below.
More information on what each of the columns numbers or colours mean can be found on the links – in other words I want you to do a bit of research into dashboard and how useful they may be for your own test reporting.
From my own experience of using these styles of dashboards for test reporting what I found was that it gave a very quick overview of the project and of the issue but was not good at reporting progress which is something that test management required and this leads to storytelling and metrics.
Session Notes and Wiki Links
One more thing to add here is that when I tried the above with senior management teams there was a request for access to the data of what was actually tested, in other words they wanted access to the session notes and actual evidence of what had been done. To solve this at the top level of each dashboard we provided a link to the wiki sessions where we kept all the session notes. I encourage you to have full transparency of testing effort and allow access to all who wants to see what testing has taken place and I feel it helps if there are no barriers or logins in the way for people to be able to access the raw data.
If as we described earlier in this document we are using session based test management then we should be producing evidence of what we have tested and the information we have found as we go along and test and we are using whatever is the best method for capturing this, video, screen capture, wiki. This should be in a place in which all have access to and everyone (who matters) is aware of its location.
Storytelling
The next thing that you need to do with your test reporting is to tell a story or two. This again requires some deep critical thinking. Michael Bolton says that test reporting is about the telling of three stories at three levels. I provide a quick summary of this below for full details refer to the original article link available here.
- Story of the product – this is where you tell the story of the product using qualitative measures, using words and creative description of the product and what it does, did. What you found interesting.
- Story about testing – this is used to back up your story of the product, what did you do when testing, what did you see and what did you cover. It is about where you looked and how you looked and where you have as yet not looked.
- Story about progress – the final story is the one about why you think the testing you did was good enough and why what you have not tested was less important (critical thinking)
Michael has a lot more information about test reporting in a great series of articles:
Markus Gärtner summaries this in his article titled “Tell a story at the daily stand up”.
As can be seen from the articles published by Michael Bolton you quickly switch from one style of thinking to the other depending on the context of the story you are telling. This is a difficult skill for a tester to master but once you practice it you can become an expert test report story teller.
Another way in which you can be creative and report your testing as well as your test planning is by using mind maps. Darren McMillan produced a great article on this and it can be found here.
It is important also at this stage to remember about your test plan and look at what information you will need to update in this. From what you found out during testing and how you risks and priorities may have changed need to be reflected in your test plan.
Qualitative vs Quantitative
There have been many discussions within the testing community about qualitative and quantitative measurements some of which I will share here as useful information. It is very easy to fall into the trap that numbers tells the whole story and since they are easy to collect will provide all the information that management require. I think we need to be careful of this and use our critical thinking to make a judgement on what the numbers really provide.
Cem Kaner has an excellent article on the validity of metrics here and the thing I most noted about this was the following:
“If our customers demand these metrics then we provide them but we have a moral & ethical duty to inform that they are flawed”
I agree with this but we need to use both our critical and creative thinking to provide the story to go with the metrics. I think we all agree that quantitative measures are flawed but we need to be able think creatively to resolve this and provide the information the customer requires in a way which is simple and easy for them to understand without misleading anyone.
Some of the discussions within testing community of test metrics
- Erik Brickarp – EAST meet up – Talking Metrics
- John Stevenson – Measuring with Stories
- Ilari Henrik Aegerter - Defending the Qualitative Approach against the Quantitative Obsession
- Michael Bolton - Issues about Metrics and Bugs
- Robert Watkins – 4 things to remember (about metrics from Cem kaner)
NEXT
So you have got to the end of this article and hopefully have a understanding that the different stages of testing requires different types of thinking at different levels. So what do you do now? First of all this is not the end of the journey. You now go back to the start and continue until told not to at the same time you can continue to practice some of the lessons and examples given in this document. Improve them, be creative and create your own, adapt them to fit your needs. This is not a document of best practice it is a springboard to help you, a reference guide, if you like, that can be modified and act as a starting point for you to improve and learn more about what style of thinking you may need when testing.
The important lesson is that testing requires a great deal of thinking and if you are not thinking when you are involved in any testing activity then maybe, just maybe, you may not be testing.
Enjoy.
John Stevenson
No comments:
Post a Comment