Friday 4 November 2011

Defining Testing

I am about to run a couple of internal workshops on the Exploratory Testing approach which is based upon a lot of work done by Michael Bolton and James Bach. One of the concerns I have been having recently is what people within the organisation think testing is in comparison to what they actually doing. So I started to put together an article looking at these concerns and trying to see if there is a problem. This blog is based upon some of the points that I cover in the article.
WARNING
Disclaimer:

The views and definitions expressed in this article are my own and as such they may not match what a dictionary may say or agree with your views/definitions.

When I start to look at what we see as testing activities they appear to fall into three distinct categories:
  • Validation
  • Verification
  • Testing

These terms may be familiar to some of the older readers of this blog. V V & T has been around for a long time and has it origins within the manufacturing industry. It has been the main process to provide quality control and assurance of manufacturing production lines. (http://en.wikipedia.org/wiki/Verification_and_validation)

It appears that these ‘manufacturing’ processes have been applied to software testing (http://en.wikipedia.org/wiki/Verification_and_Validation_(software))

This seemed to have lead to the appearance of process standards initially the ISO 9000 Quality Assurance standard, which was modified to become the ISO 9001 2008 standard which included software. These standards are very closely linked to manufacturing process and from a software testing perspective the quality control methods.

Talking to and observing various companies I have seen that lot of people’s perception of testing is as shown in the photo below.

http://brigitteofseon.wordpress.com/category/work-hard-no-go-slow/

Is it a problem to have this perception of software testing?

At the beginning of my career in software testing a lot of companies started to change from mainly hardware manufactures to both hardware and software manufactures. There was a need among these companies to have processes they could use to prove the ‘quality’ of their software products and the general consensus was what had worked in quality control for hardware surely could be applied to software testing.

The reasoning behind these was based upon some fairly flawed assumptions:

  • All software was the same
  • All software worked in the same way
  • All users would follow the designed work flows.
  • All users would behave in the same way

The main focus of these processes was to validate and verify what was already known about the product and its expected inputs and outputs. In my opinion following quality control and assurance processes is not really about testing Testers ‘normally’ do not control the quality (Yes there are approaches such as TDD which ‘may’ help). If there is crap in the system you are testing then there will be still crap in the system afterwards. Testers provide a service telling you there is crap in the system. Michael Bolton talks more about getting out of the QA business here

Validation and verification

will in the majority of cases NOT

Tell us anything new about the product

Make us ask questions of the product

So what do I mean when I talk about validation and verification?

Validation:

To me validation is about proving what you already know about the product. Confirming what the requirements say are correct and that the system is correct in accordance to what you believe it should be. The normal response when validating will be:

  • true or false
  • yes or no
  • 0 or 1

I see Validation as a checking exercise (See article by Michael Bolton here on testing v checking) rather than a testing exercise, it still has some value within the testing approach but it will not tell you anything new about the system being tested. It will prove that what you already know about the system is correct and working (or not working) this is like testing requirements or validation of fields in a database/GUI – you know what the input is you know what outputs you expect according to specification/requirements so why not automate this?

The majority of ‘testing’ I see happening is validation and even though it has some value I would not count validation as testing since it does not tell you anything new about the system you are testing.

It should be noted that to interpret the results from validation ‘testing’/checking requires human interaction to work out if what happens what the correct expected response.

Verification:

When I look at the term verification I use it for when we are verifying any bugs that have been previously found. Someone has made a change to the product and I want to verify that the change made has fixed the problem I had seen before. Some verification tests can be automated – for example if you have run a test previously and found the problem you may be able to automate the steps you followed from testing so that you do not need to run the same test again.

Testing:

I see testing as a thinking exercise in which you need a person to use their skills (and brain) to ask questions of the system being tested. From asking this question they learn more about the system and its behaviour. They will not know the answer to the question but by investigating and tinkering with the product they can form some reasonable answer to the question they posed.When testing we act like crime investigators – you suspect foul play, but you need to ask questions and gather evidence to back up your theories and provide answers to your questions. Testing is not based upon requirements or specification but rather when the specification and requirements are not saying.

Testing is about asking the

  • What
  • Why
  • How

Nassim Nicolas Taleb came up with the following interesting quote:

We are better at doing than learning. Our capacity for knowledge is vastly inferior to our capacity for doing things – our ability to tinker, play, discover by accident. http://www.fooledbyrandomness.com/notebook.htm

So after all of this is there really a problem?

Some of the problems I see within the software testing industry are:

  • We spend far too much validating rather than testing
  • We repeat the same validations (manually) time and time again
  • Cover less of the system by only repeating the same validations
  • Testing is checking exercise rather than a testing exercise
  • Testers are not being engaging
  • Testers are not being challenged
  • Testers do not need to think
  • People see testing as a boring thing to do
  • Testers if used to manually validate are seen as robots

What can be done to improve this?

  • Look to automate the validation (checking stuff)
  • Improve coverage by changing the data sets used in validation
  • Start to use exploratory testing approach – attend a rapid software testing course
  • Look at using Session based testing
  • THINK engage your mind and question the system.

We need to keep learning about testing and do more testing rather than keep repeatedly validating systems, it then becomes a much better and challenging role to be a tester.

4 comments:

  1. Hi John,

    Great post. I agree with your general view on Verification, Validation and Testing. So I am with you there. I do however have a slight difference on definitions.

    For me Verification is the process of checking whether or not the software meets the specification both in following the specified, or rather derived from specification, correct and incorrect paths. Therefore I view this as a checking process that extends to and includes the verification of defects.

    Validation differs from this by focussing not on the mere correctness of the implemented code. It looks if the software delivers what was intended by the described functionality.

    Testing then not only takes an essential step further. It still includes the previous two steps by executing these skill-fully. It ought to do this automated as much as possible (but not necessarily). The real strength of testing however comes from its investigative nature in searching to provide information about the software that enables the customer to make an informed decision about it.

    So to conclude. Although I differ a bit in my definitions I agree with the essence of your view.

    Regards,

    Jean-Paul

    ReplyDelete
  2. Thanks for the comment Jean-Paul

    The problems I had putting this together has been that everyone has a slightly different view on what validation and verification means hence my warning at the start ;o)

    With regards to your point about verification you are correct in the context you have placed it and as such I agree with you. The same goes for your comment about validation. Both of these do have a place in the world of software testing and should not be ignored but they should not be abused in the way they seem to be at the moment.

    ReplyDelete
  3. Hi John,
    a lot of your post resonates with me as i'm currently working at an organisation where everyone values testing, but i'm seeing little evidence of it happening. What is occurring is some V&V.
    I was first exposed VV&T 14 years ago, while working for British Telecom. VV&T was de rigueur there, and it was explained to me as "ensuring we built the right thing, that we built the thing right and that it will play nicely with the rest of its integration points", in that domain, an integration point could be a telephone subscriber .
    Im now looking at introducing continuous inspection, and this has its roots firmly in electronics manufacturing. Without this, even when using continuous integration, we are only sampling at discrete points, and i have seen defects enter the system and go undetected through sampling alone.

    Regards,

    Stuart.

    ReplyDelete
  4. I wrote a piece related to this some time ago Investigation vs. Validation (http://www.perftestplus.com/resources/014PeakPerf.pdf)

    ReplyDelete