http://clarotesting.wordpress.com/2010/07/21/but-how-many-test-cases/
and several people blogged a response
Simon Morley added his view here: http://testers-headache.blogspot.com/2010/07/test-case-counting-reflections.html
Jeroen Rosink added his here: http://testconsultant.blogspot.com/2010/07/repsonse-on-how-many-test-cases-by.html
and Abe Heward directed people to a similar blog he had wrote earlier: http://www.abeheward.com/?p=1
Each of these blogs makes very valid points about how non-useful measuring test cases are for indicating the progress or coverage of testing effort.
The aim of this blog is to try and expand upon these posts and see if there are ways in which we could measure testing effort and progress within resorting to using numbers.
To start with we shall take a look at a made up situation.
You are testing on a project which is has a two week testing cycle, your manager has requested that you need to report each day the following:
- How many test cases you have
- How many have been run
- How many have passed
- How many have failed.
(Does this seem familiar to anyone?)
So before you start testing you report to your manager that you have 100 test cases to run over the two week cycle
At the end of day one you report the following
- Test cases ran: 60
- Test cases pass: 59
- Test cases fail: 1
- Defects raised: 1
- Test cases still to run: 40
So management think cool we are ahead with testing, 60% done in one day.
At the end of day 2 you report:
- Test cases ran: 62
- Test cases pass: 61
- Test cases fail: 1
- Defects raised: 1
- Test cases still to run: 138
Management now thinks how come you only ran two test cases today, why are you running slowly? WHAT!!!! Where did those other 100 test cases come from? Did you not do your job correctly to begin with?
However the two you ran today had lots of dependencies and very complex scripts.
Plus your testers noticed that there appeared to be new features that had not been documented or reported, you have now had to add another 100 test cases. Also your testers actually think when they are testing and thought of new edge cases and ways to test the product whilst they were testing.
Management starts to panic – you reported on day one that 60% of testing had been completed. Now you are saying only 30% of the testing has been completed, Stakeholders are not going to happy when we report that we have only covered 30% when the day before I reported to them that 60% had been completed.
This continues, your testing team are really good testers and find more and test ideas which are turned into test cases. So at the end of day seven you report the following:
- Test cases ran: 1200
- Test cases pass: 1109
- Test cases fail: 91
- Defects raised: 99
- Test cases still to run: 10000
So at the end of the first week you have only completed 8% of all the test cases. You get fired for incompetence and the project never gets released.
Many people reading this may have experienced something similar to the above, what worries me that there are still people stating the best way to measuring testing is by the use of test cases!
The question now is that if measuring by the use of test cases is not a good way to measure then what can we do?
The following suggestions are my own and what I apply within my test approach, it does not mean it will work for everyone nor am I saying it is the best approach to take. However the purpose of my blog is to offer suggestions about testing that could be useful to some people.
I work in the following testing environment:
- Agile based – 2 week iterations
- Customer changing requirements frequently
- Code delivered daily
- Functions and features added without supporting documentation
- Use a mixture of scripted and exploratory testing
If I tried to report the testing effort using the traditional test case scenario it would be of little (or zero) value, since the test case number would be constantly changing.
What we do is split functions, features etc into test charters, as per the exploratory testing approach, these ‘Test Charters’ are known as the test focus areas of the software. If a new function or feature is discovered a new charter is created.
We then use the Session Based Test Management approach (James and Jon Bach - http://www.satisfice.com/sbtm/) and implement sessions based upon mission statements and test ideas. During the testing session the testers are encouraged to come up with new test ideas or new areas to test, these are captured either during the session or during debrief.
The reporting of progress is done at the test charter (test focus area) level. The test manager reports in the following way.
Test focus area 1: -Testing has started – there are a few issues in this area:
Description of Issue x, issue y, issue z.
Which need to be resolved before there is conference that is area is fit for its purpose.
Test focus area 2 – has been tested and is fit for it purpose
Test focus area 3 – test has started and some serious failures have been found defect 1, defect 2, defect 3
And so on.
Some people may ask but how will this tell us if we meet the deadline for testing? I am sure it will NOT tell you if you will finish ALL of your testing before the deadline since testing is an infinite thing, we as testers will carry on testing until we meet a stop heuristic (See Michael Bolton article on stopping heuristics: http://www.developsense.com/blog/2009/09/when-do-we-stop-test/).
The problem with testing is that it is not a yes or no when it comes to the question of have you completed your testing yet. Every time a skilled tester looks at the software they can come up with more and more test areas and test ideas that they could carry out. These may or may not add benefit to the suitability of the software and if it is fit for its purpose. What is required is a test manager that talks to and listens to their test team and see which test areas are the most important and MANAGE test sessions based upon what is critical – basically do some good old prioritizing. The test manger needs to ask the difficult questions of the stakeholder and project managers.
- What features can you do without?
- What are the critical areas that are required?
- Function abc has many serious problems – it can cause problems x,y,z for your users. Do you need function abc?
- We have tested all the key functions and found the following problems x,y,z. You want to release tomorrow, are you OK with these known issues?
In return the stakeholders and project managers must trust the test team and accept that when they report that an area has been ‘sufficiently’ tested they believe them.
To summarize – instead of reporting on a small area of testing such as test cases, move a couple of level ups and report on the progress for test areas/functions./features based upon the importance of the feature. This may not tell you if you will compete the testing before the deadline but it will show you how well the testing is progressing in each functional area at a level that stakeholders can relate to and understand. The trust your stakeholders will have in you should improve since you are giving them a story about the progress of the testing effort without trying to hide things using numbers.
Great post! What I like about it is that you don't suggest that your approach is managing to keep the number of test cases under control. You don't suggest that at the end of 7 days you will have run more tests than the people who counted their scripts (although I believe you will).
ReplyDeleteInstead you point out that through discussion and careful management as opposed to counting you ran the important tests.
This is exactly what I'd like to hear and read more about. That testers provide value to the business by providing the information that the customers need, rather than throwing numbers at them and hope that some are useful.
It seems to me that when people want test case counts, they're using that as a surrogate measure of test coverage. This requires us to understand coverage and to report on it:
ReplyDeleteGot You Covered: Excellent testing starts by questioning the mission. So, the first step when we are seeking to evaluate or enhance the quality of our test coverage is to determine for whom we're determining coverage, and why.
Cover or Discover: Excellent testing isn't just about covering the "map"—it's also about exploring the territory, which is the process by which we discover things that the map doesn't cover.
A Map By Any Other Name: A mapping illustrates a relationship between two things. In testing, a map might look like a road map, but it might also look like a list, a chart, a table, or a pile of stories. We can use any of these to help us think about test coverage.
And here's why counting doesn't cut it:
http://www.developsense.com/articles/2007-11-WhatCounts.pdf
---Michael B.
Excellent. My post was something of a rant, but this picks up the problem and offers a constructive response. Unless testers can offer that constructive response to the tyranny of meretricious metrics then we're just going to be annoying outsiders, howling at the moon. We have to be influential insiders.
ReplyDeleteRegards, James Christie
Michael,
ReplyDelete"It seems to me that when people want test case counts, they're using that as a surrogate measure of test coverage."
All metrics are surrogates for what people really want to know. They are attempts to represent an idea as a number.
Test case counts may be surrogates for test coverage. But they are more likely surrogates for "How are we doing?" and "Will we get done on time?"
Test coverage (perhaps expressed as a percent?) is also a surrogate. Perhaps for "How effective is our testing?"
Great post, John. If we know what is important to our customers we can use our reports to inform them in a far more directed way. Giving out numbers is pretty meaningless.
ReplyDeleteEven metrics such as Defect Density Percentage which can be used to highlight areas of the product that have a higher proportion of defects is meaningless without an understanding of the significance of the individual defects and the impact on the customer experience when those defects are taken together.
I therefore like the fact that you are taking the trouble to report on the significance aspect and giving the stakeholder reasons why these should affect his or her decisions.
Thanks.
Stephen
Thank you for all the comments. The issue of measuring has been one of the major problems I have faced as a tester. Trying to justify why we now have 10x 20x 100x the original number of test cases to project managers has taken up a lot of time in which I could be doing some useful testing. I feel we still have a long way to go to get the message across that testing is not something that you can plan 100% upfront, there will always be things that testers will find or uncover, because we are intelligent, free thinking individuals.
ReplyDeleteThanks for your post James since it spurred me to finalize an idea I had about measuring.
I must say a big thank you to Mr Michael Bolton - his ideas and challenging questions have made me challenge myself and the way testing is being done. The thoughts within this post have come from questions and challenges that Michael has set both in person and online. The seeds for this post were formed during a conversation in a bar at Eurostar - before the singing and music! If anyone ever gets the chance to meet Michael - or better attend his Rapid Testing Course then I can do nothing more than fully endorse this as a way of thanks for the time and effort he has given up freely to help me.
I really liked your approach for tracking testing. Your idea about test focus area really gives me another perspective to how data can be represented so it makes more sense. I am currently in a system where I use planned test cases as a basis to show where we are with testing. And it has always been hard to justify why the numbers changed so much. I have always stated that when you pull a number out of a hat .. you really shouldnt ask me why I pulled on particular number.
ReplyDeleteRecently my focus has been by feature for test cases. I like the test focus area much better.
Do you use use cases and tracking testing based on those? I would like to understand test focus a little more and if you do have time would love to discuss it with you.
Hi Shipa
ReplyDeleteThank you for your questions. I have sent you an email so that you can communicate with me more and I can then try to answer your questions in more details.
In brief - I use an exploratory testing approach based upon James and Jon Bachs excellent Session Based Test Management (http://www.satisfice.com/sbtm/)method. So if you change test focus area to test charter you can then match test ideas with test focus areas and mission statements. Within Exploratory testing I try to avoid using test cases and concentrate on test ideas which may contain many test cases. You could if you require map test cases to test areas (functions/features) or even to customer requirements if you have a requirements document. IMO it still comes down to the telling of a story and saying that we have tested x,y,z and found issues 1,2,3. It depends on if management TRUST you as a professional and RESPECT that if you say it has been tested as fully as you can in the time given then it has been tested. Anyway get in touch on email and we can discuss this further.
You have mail. Thank you
ReplyDelete