A Competent Tester

You can teach a student a lesson for a day; but if you can teach him to learn by creating curiosity, he will continue the learning process as long as he lives. ~Clay P. Bedford

I started writing this blog article in draft about a month ago to have a little rant about how certification does not make you a competent tester based upon my experiences and frustrations whilst trying to recruit testers. I was prompted by a post by Rob Lambert to revisit the article and try to complete it.

Recently I have been trying (and I emphasis the word trying, it has been really trying and challenging to find the right people) to recruit testers to work with me on some quite technical projects. None of the projects have any real UI interfaces nor are they web based applications, this is real hardcore technical testing at a binary level.I became very frustrated and even got to the stage that I felt my standards were too high.

I lot of the CV and candidates that were interviewed all stated that they had ISEB or ISTQB certification and from that I assumed they would have a basic grasp of boundary and edge cases. How wrong was I!!! To make it worse I asked about their views on any of the current new approaches and techniques in testing, even having to prompt to what do you think of context driven testing and all I got back in return was a blank look. I asked what articles or books have you read recently about software testing or any book that you could relate to software testing and again all I got was blank looks and shrugs of shoulders.

I am not against certification in any shape or form and I do not have a view if people try to make money out of it that is not the issue. The issue I have is that in some (most) cases these schemes are being sold on the basis that once you have completed them you are now a ‘skilled’ tester and know everything there is to know about testing.

PASSING A MULTI-CHOICE EXAM DOES NOT MAKE YOU COMPETENT!!!!!

Rob Lambert in his article explains in great depth what makes testers ‘skilled’ and I have to agree with him. Those who read my blog know I have a major interest in the social sciences and psychology and how this can collate to testing. More importantly how it can help make you a better tester.

So if you want to become competent at testing you have to read more, interact more with the testing community, become self-learning. My ethos is always to keep on learning and never stop doing so.

Taking any of the certification courses can be, for someone new in the world of testing, a good STARTING point, grounding in SOME of the techniques and skills. These courses will NOT teach you about how testing fits into Agile and about exploratory testing. Nor will they teach you how to test and make you think ‘outside the box’ Only getting involved within the testing community will do that. I good start is the Software Testing Club – read some of the articles and blogs there, subscribe to the excellent RSS feed. I am not being paid by the Software Testing Club to say this I am promoting it because again it is a good starting point for those would wish to learn more about testing and want to improve so they can become competent testers.

I will finish with a quote about learning.

It's what you learn after you know it all that counts. ~Attributed to Harry S Truman

Mentoring a New Tester - First test challenge

The following is the results from setting Matthew the eBay challenge which was created by Jon Bach for a weekend testing session. The reported produced by Matthew including his thoughts and thinking – anything which is in red italics are my comments.

Once the session had finished I arranged a debrief and the results of that can also be found in this blog article.

The Mission:

Use the eBay website and find the most expensive product, the most unusual and to get just one hit

Most expensive:

My initial thoughts on this is to just leave the product description blank and go for the greatest product price given to me in the cost field.

Most Unusual:

I'm thinking on this one to just enter in the product field, a few random letters and numbers. What is unusual…? I suppose that’s up to me to decide.

Just one Hit:

This has stumped me. How do you get just one hit without knowing what is on offer. Maybe if I knew the exact product name I could enter it, but even then a search engine will look at all the words entered and come up with a few other options for me, defeating my challenge. Trial and error? Or just entering one random word that I think will be a rarity on eBay, like an ‘AAD’ – Mentor comment What is AAD? I think I will try the latter.

Most expensive:

Straight away no field for price, so clicked on advanced. Mentor comment – could this have been planned/scripted before using the product?

Mentor comment – notice the use of ebay.co.uk – would ebay.com give difference results? I did not specify which to use (assumption was made) Good example of ambiguous requirements?

Typed in 10,000,000

Error

So now I am going to also enter into the keyword/item number field the number ‘1’ as a I imagine most item numbers will have a 1 in it. – Mentor comment – why number not letter? The use of wildcards?

Chose the field ‘all categories’ as this was the most likely category for the most expensive as it will search the whole site.

As well as entering the min price of 10.000.000 and leaving the max price because I don’t want to limit myself considering what I am trying to find.

Click search

It looks like I might have found the upper limit for eBay with only three items all coming in at the same price. –Mentor comment – interesting when this task was done for weekend testing there was no such limit found on ebay.com – is this just a co.uk limit? Is it a legal issue?

But are these the most the most expensive.

Have now entered ‘2’ into the keywords field. –Mentor comment – good exploring – testing your theories based upon results.

Same 3 as above were found plus another, reinforcing the fact that eBay may have an upper limit of £12,869,224.17

Have now entered ‘3’ into the keywords field.

Nothing else found except what has already been found.

Now going to change the min price to 13,000,000 and fill the keywords field with ‘2’, as this was the one with the most hits.

Nothing found.

There is no individual most expensive item, only that eBay has - mentor comment – has is strong word – are you sure? Appears to have may be better an upper price limit of £12,869,224.17 on a number of items. - Mentor comment – as per previous comment does this apply to all eBay sites?

Most Unusual:

Entered a few random letters and numbers into the keyword field on the homepage as shown below.

Found mainly stamps.

Unusual hobby, but not unusual enough for me. - Mentor comment – you need to expand on your definition of what is unusual to you.

Typed in unusual and found mainly ornaments. Not there yet.

Another word for unusual… ‘Quirky’, entered that, found mainly clothes. - Mentor comment – good trying to find different meaning – a dictionary/thesaurus could be useful here

Still not there.

Maybe ‘weird, entered that and found pens in the style of syringes, more disturbing than unusual.

So I thought that this may take to long with me just entering random words. So I thought what might be unusual to me.

I went back to the home page and I looked through the drop down menu.

And thought that the industrial field is something I know nothing about, and may provide me with some unusual items. So chose that field and left the keywords field blank.

Before I had a chance to look at these results I saw excavator. Unusual to me, can also be quite funny at times, so I decided to choose that from the related searches bar under the search tool.

Scrolled down the results page and mainly saw digging machinery.

Mentor comment – love the thinking process being described on how you move from one way to find a solution to the next

Until I saw this

This to me is unusual. Never seen it nor am I likely to see it anywhere else funny to boot too.

Mentor comment – Great result I would have also pointed out the obvious about the digger being a pick up only and delivery being free

Just One Hit:

I thought I would try just entering one random word that I think will be a rarity on eBay, like an ‘AAD’.

So I put ‘aad’ in the keywords field and searched all categories

No good. The word ‘aad’ can be placed into any word of the thousands of products out there.

So I am now thinking of narrowing that field so I chose the advanced search button on the home page.

Chose the fields below to try and narrow the sites search options.

Not what I was looking for. I narrowed the search options that much that the site could not find anything, so it decides to ‘help’ me and remove some of my search options to give me a few items for me to look at.

I’m thinking now that you cannot randomly try and narrow the search as the search engine will just try and ‘help’. So maybe if I know an item to be there that I type into the keywords field the exact spelling of that known item. I am going to use one the most expensive items (previously found) as an example. As shown below.

Mentor comment – love this way of thinking – very thoughtful analytic mind

Eureka!

Is this cheating though? – Mentor comment – hmmm NO – you achieved the mission the system allowed you to do this - so it is not cheating just manipulating the system to achieve a result.

All I was asked to do was to get just one hit. I have that one hit. So is this testing the system? The search engine did not try and ‘help’ me on this one. It might know that I am not randomly searching and therefore did not want to help me? In this case then, I have tested the system to get just one hit, so long as you know that an item is there.

Mentor comment – useful information to get one hit you need to know about an item first - good piece of knowledge to be made aware of

Debrief:

During the debrief with Matt I loosely used the PROOF (LA) method

We talked about what had happened during the testing session and Matt replied that it a challenge since he had not done anything like this before in the sense of calling it ‘testing’

He enjoyed the session and felt a sense of achievement in meeting the missions but felt his lack of experience and knowledge of testing techniques hindered him and it was something he felt he needs to do some more learning on.

Matt compared the way he went through the list of missions as similar to army briefs in the sense of some upfront planning of what needs to be done to achieve the goal, he felt his experience of this type of context helped a great deal. During the recording of the session and making his notes he stated that there appeared to be a comparison to recording physic experiments. Where you have a theory to prove/disapprove, you then describe your results and form a conclusion based upon the analysis of the results.

The things that hindered Matt were a lack of clear objectives – something the army instils and some testing knowledge. As part of his homework I asked Matt to research wildcard searches and boundary analysis.

Overall Matt enjoyed the session and is looking forward to the next one.

From my experience it is so easy for those of us who have been in the testing field for awhile to forget how much of the techniques and practises we do by instinct and from our inbuilt repository. Something we should remember when starting with someone new to testing.

Until the next challenge…..

Mentoring a New Tester

A couple of days ago my son in law was talking to my wife with regards to what he would like to do after he completes his service in the Army and what options he has. From this conversation my wife had an idea about him entering the world of software testing and as such asked me to have a word with him to see if he would like to join this crazy world.

Matt (@Matt_Wellington) is a member of the Royal engineers and has done two tours of Afghanistan has a member of the EOD bomb search team. He spends his days searching for IEDs and making sure it is safe for colleagues and local people to walk safely on the roads. He currently has two years (plus) left in the Army and is looking for new challenges; well software testing is a challenge and one I feel he could be great at.

I mentioned on twitter that I was thinking of mentoring Matt and any advice would be welcome.

I got a sudden flurry of encouraging tweets.

@ola_hylten

@steveo1967 Why not dig into the exercises in rapid software testing? I don't think @jamesmarcusbach or @michaelbolton would mind that!

@PeterWalen

@steveo1967 Who is he with and when does he get out? I might suggest some intro stuff "Testing Computer Software" tied with non comp. books.

@michaelbolton

@steveo1967 Give him something to test. Observe him doing it. Feed back. Repeat

@destruise

@steveo1967 It gives them a good insight on the 'before' and 'after', and by writing cases from the start they are productive! win/win :)

@ola_hylten

@steveo1967 Then you can move on and do weekend testing with him observing first and then you do pair testing with him at the keyboard.

@michaelbolton

@steveo1967 "How do you recognize a bug?" has many parallels with "How do you recognize a bomb?"

@steveo1967 So let me get this straight: he searches for things that might blow up harmfully, and you say he has no testing experience?! ;)

@ola_hylten

@michaelbolton @steveo1967 Good point Michael :) Software testing has less personal risks, hopefully, most of the time ;)

@PeterWalen

@steveo1967 agree w/ @michaelbolton He's got training in critical thinking now apply it differently. 1/2

@PeterWalen

@steveo1967 @michaelbolton If he's not ovrseas maybe test meetups may help? Software generally safer than ordinance ;)

So after explaining what testing involves (Briefly – some may think how can you do this briefly!!!)

I first gave him the challenge as suggested by Michael Bolton ‘How do you search/recognise a bomb?”

Matt explained about planning before hand, then re-evaluating when at the site, following safety procedures and using local and own knowledge to search the area. Then when something looks suspicious use techniques learnt during training and on the job experience to confirm the presence of an explosive. Marking the area as such if it is dangerous or the fact it has been searched and then reporting to his senior office the results of what was found.

After hearing this I explained the comparisons with software testing and his answer was is that all this is to it then? Oh the joys of the innocent……

I then gave Matt some material to read, wrongly or rightly the CD from the ISEB course, I feel this is a good way to start and to get to know some of the basic methods of testing, boundary, static etc. I also gave him copies of some excellent software testing books.

The art of software testing by Myers

A Practitioner's Guide to Software Test Design by Lee Copeland

Both which I still use as references

My plan for mentoring are as following

• Basic techniques – the toolset every tester needs
• Scripted Testing
• Exploratory Testing
• Test planning
• Test reporting/debrief
• Weekend testing
• Paired testing

The aim is to have lots and lots of practical/hands on.

Matt does have limited time since in a cpl of weeks he will be working 7 days a week so I need to use an approach which is not overloading and allows him to build up his testing skill set and increase his confidence of the craft.

To this aim I gave Matt his first testing exercise based upon a weekend testing session organized by Jon Bach

The mission was to test the ebay website and find:

• The most expensive
• A search that returned only one result
• A search that returned an unusual item

I asked him to document what he did, what he found and most important his thought process.

The results of this first testing session are amazing – I think I have found I natural tester…… I will publish the results of this first testing effort in my next article.

If anyone wants to help/encourage/support or advise Matt then please get in touch with him on twitter (@Matt_wellington)

An update or two.

I noticed that I have not written a blog article in awhile so I thought I would put together a short article on what I have been up to so that regular readers can be sure that I am still alive and well.

On the personal front we have had a few health scares over the past month hence my lack of tweeting or blogging.

On the work front I have been very busy and involved in a few different and exciting projects while continuing to look at different ways in which we can improve.

During this period I have been looking more and more into ethnographic research and its connection to testing. I find this area of social science fascinating and how much it appears to collate to testing. Since there does appear to be a connection to this I am current running a couple of case studies internally based upon methods from ethnographic research as mention by Richardson in their article for Qualitative Inquiry: Evaluating ethnography

The findings for this case study will be presented at the UNICOM Next Generation Testing conference 18/19 May 2011

If you cannot make this event I do intend to give a very basic/quick introduction to this approach the Software Testing Club meet up in Oxford on the 14th April 2011 This event will be used as a world premier for the approach I have been working on so that definitely makes it worth attending. Or the fact that Lisa Crispin and Rob Lambert will be there should tick everyone’s box.

Without giving away too much detail before the meet up here is a brief summary of the approach I have been investigating

• The concept is based upon questioning the tester as much as you question the product being tested.
• It is check-list that can be used on an individual basis and should take between 5 and 10 minutes. The idea is to look at what you are doing and checking it is the right thing and see if you missing anything.
• I will be giving away the check-list on the evening of the meet up. (wow a freebie)

Have I given away too much information, not enough or left you wanting more?

If you want to know more then I suggest you sign up to attend the meet-up or the UNICOM conference

Is Context Driven Testing a gimmick?

My inspiration for this article has been a comment I received with regards to trying to organise an internal Rapid Software Testing course .

Someone made a comment that they felt that this ‘smacks of a gimmick’ but would be interested in finding out what people discover/get out of this in relation to the way we work.

The views expressed in this article are solely my own and are based upon my own experiences and knowledge of the testing profession.

Currently within the testing world there appears to be two schools of thought:

The traditional (standards) approach – driven by the ISTQB examination board (formerly the ISEB)

And there is the

Context Driven Testing concept– driven by such people as Cem Kaner, James Bach and Michael Bolton.

This article is not about entering a debate to say one approach is better than the other and IMO I think a good balance is a mixture of the various approaches. There are many other approaches and concepts to testing than the two mentions above such as the Agile and the analytical approaches but the main discussions within the testing community appear to mainly refer to the two 'schools' listed above.

The principles and concept of context driven testing is that the emphasis is about thinking, experiencing and doing rather than assuming and making interpretation of what people believe the system should do. It is more about ‘hands-on’ and learning about the system as you test.

A common misconception is that there is no planning involved within rapid software testing and that it is based upon ‘free’ testing and it is without structure or discipline. In my experience and from using the material from the rapid software testing course there is more planning, structure and focused based testing than with any other approach. The introduction of session based testing in which testers have a mission and a goal to aim for during their testing session ensures that testing remains focused and on track.

The difference between the two approaches in that the standard approach is mainly used to define testing (test cases) before you have actually have access to the system under test. There is an inherent weakness in this approach in that assumptions are made that requirements, design and user needs are correct, accurate and not missing.

Once actual testing has started the majority of testers revert to working in a context driven way. They adapt the scripts they have written; they think of new ones and make decisions on what not to run. The context driven approach is to have some lightweight upfront planning which is ambiguous and allows the tester the freedom to approach the testing by using their logic and adapting as they learn more about the system. This allows the tester to build up knowledge of the system while executing, creating new tests and recording what they are testing. This is the basic definition of exploratory testing. Time is not wasted on creating tests that will never be run or maintaining a list of test scripts with incorrect test steps. It is about recording what is happening at the time testing happens and storing the results of that testing session. This then can if possible be automated and as a test never run manually again.

The difference between the approaches is that rapid software testing requires testers to think as they test and not just tick boxes. It forces the tester to question what they see and allows them to freely explore and discover new things about the system. It uses triggers (heuristics) to keep asking the question "Is there a problem here?"

It does this by comparing the product with similar product, looking at the history of the product or the claims made by the product. These are tests in which there is no yes or no answer it depends on the context and the thinking of the tester.

Is context driven testing a gimmick?

IMO it is not

It is a natural way to test products and it is the way testing has been done since it became a career choice. However people have not admitted (or will not admit) to following this approach or be aware that they are doing this.

The whole concept and approach of rapid software testing is to give it a name and provide useful skills and tools to improve this methodology which follows the thinking of context driven testing.

_______________________________________________________

FOOTNOTE:

After a lively discussion on twitter with James Bach I feel I need to clarify some misuse of definitions.

James gave a great description to show the differences between Rapid Software Testing and Context Driven:

Rapid Testing is a testing methodology that is context-driven.

But context-driven testing is not Rapid Testing.

After this 'revelation' I have made some minor changes to the original post.

Measuring Testing

I saw a couple of tweets by @Lynn_Mckee recently on the metrics that are used in testing.

There are many great papers on #metrics. Doug Hoffman's "Darker Side of Metrics" provides insight on behavior. http://bit.ly/gKPHcj #testing

Ack! So many more that are painful... Scary to read recent papers citing same bad premises as papers from 10 - 15 yrs ago. #testing #metrics

And it made me think about how we measure testing.

This article is not going to be

'This is how you should measure testing’

or

offer any ‘best practice’ ways of measuring

My concern with any of the ways in which we measure is that it is done without context or connection to what question you which to have answered with the numbers. It is a set of numbers devoid of any information to their ‘real’ meaning. There are many and various debates within the software testing field about what should and should not be measured. My take on all of this is:

Can I provide useful and meaningful information with the metrics I track?

I still measure number of test cases that pass and fail and number of defects tests and fixed.

Is this so wrong?

If I solely presented these numbers without any supporting evidence and a story about the state of testing then yes it is wrong it can be very dangerous.

I view the metrics that are gathered during testing to be an indication that something might be correct or wrong, working or not working, I do not know this just from the metrics it is from talking to the team, debriefing and discussing issues.

I capture metrics on requirement coverage, focus area coverage, % of time spent testing, defect reporting, system setup. So I have a lot of numbers to work with which on their own can be misleading, confusing and misinterpreted. If I investigate the figures in detail and look for patterns I notice missing requirements, conflicting requirements and what is stopping me executing testing.

So what is this brief article saying?

Within the software testing community I see that we get hung up on metrics and how we measure testing and I feel we need to take a step back.

It is not too important what you measure but how you use and present what measurements you have captured. It is the stories that go with the metrics that are important, not the numbers.

What you believe might not be true. (Part 2)

The first part of this article looked at conjunction bias and framing bias and how it can influence our thinking towards incorrect assumptions all under the heading of cognitive bias.

The next part of this article investigates other forms of bias and how they influence our decisions and thought processes. One of the first I will touch upon in this article is belief bias.

Belief bias has many similarities to confirmation bias and in some ways both are closely linked. If someone has very strong beliefs they can use arguments that back up their beliefs in such a way that only evidence that supports their beliefs are used giving a confirmation bias to their beliefs. There are many examples of this in the world from the belief in the existence of aliens to the range of conspiracy theories that are abound on the internet.

So what is belief bias?

People will tend to accept any and all conclusions that fit in with their systems of belief, without challenge or any deep consideration of what they are actually agreeing with.

http://changingminds.org/explanations/theories/belief_bias.htm

Belief bias is the conflict a person incurs when their beliefs do not match the logic of what is presented to them

The danger with belief bias is that it can quickly turn to belief projection:

Psychological projection is a form of defence mechanism in which someone attributes thoughts, feelings, and ideas which are perceived as undesirable to someone else.

The problem now is that the beliefs of someone on a team could become fostered on other team members using belief projection even if what they believe is unfounded. Within software development we all have our own views and beliefs on what a piece of software is expected to do.

How does this have an impact on software development and especially testing? Imagine a situation in which a tester has a very firm belief on how an interface should interact. They then test that interface and find it is not behaving as they believe it should be. A bug report is now raised and passed back to the development team. It is found that the bug was raised in error and that the interface interacts as designed and described in the requirements. This is simple case in which regardless of what requirements, design specifications and others are saying the testers strong belief bias is saying everyone is wrong and what they believe in is correct.

In the world in which we as testers operate I doubt that the above would happen since we are now in situations in which developers and testers communicate and there is no more throwing it over the wall way of releasing. However if you still work in teams in which there is a lack of communication and talking then belief bias can have a large negative affect on testing.

Another issue is when you do work in a team and belief projection comes into the equation. If someone on your team subconsciously has a belief that the developers think the testers are a waste and not necessary (negative personality trait) then could project this on to other members of the team and start to cause a barrier of resentment to build up between teams. It is impossible to prevent people having opinions and thoughts about other members of a team but having an environment in which everyone is allowed to express their views and thoughts in an open discussion can help to remove this type of bias. Within one company in which I worked as a team lead I would have an open session in which nothing was recorded or written down but people could express views and thoughts on what was really happening within the project. Sometimes it would be heated and people would get emotional but it managed to clear the air. One important part of this method was that a mediator was always in charge to prevent it getting into a naming calling situation.

Another bias which could have an impact on testing is illusory correlation in which people form a connection between two events even when all evidence shows that there is no such connection or relationship. A good example is people who have arthritis believe that their condition worsens depending on the weather. Redelmeier and Tversky conducted an experiment in which they took measurements based upon the patient’s view of their condition and at the same time noted details meteorological data. Even though nearly all the patients believed that their condition got worse during bad weather the actual results shown that there was a near zero correlation between the two.

Wikipedia defines illusory correlation as:

Illusory correlation is the phenomenon of seeing the relationship one expects in a set of data even when no such relationship exist.

http://en.wikipedia.org/wiki/Illusory_correlation

It is easy to see the effect this can have on software development. Imagine if a developer creates an illusory correlation between two variables that do not have any real correlation therefore introducing bugs into the project. There has been a study on the reasons for software errors and it has been found that illusory correlation does play a part. Details of this study can be found here.

Stereotypes are normally defined by the use of illusory correlation. Someone who came from a small town where everyone was kind makes an assumption that everyone from a small town is kind therefore when they go out into the world and meet a kind person they correlate that the person much be from a small town since they are kind even if the correlation is not true or does not make any sense.

How does this help or hinder with regards to software testing?

The problem occurs when testers work in isolation and form they own methods and create their own hypotheses of what should happen when they test the product under certain conditions. The danger is that the testing becomes one sided searching for evidence that matches their current hypothesis of how the product should react. The resulting factor of this bias within testing is that conditions are tested which meets the illusory correlation of the tester but conditions which do not meet the expected assumptions are not tested. This could then cause significant bugs to be missed due to flows not meeting the expected correlation being tested.

It is very difficult to avoid falling in to the illusory correlation trap since the human mind tries to take the easiest path and groups’ objects together for easier recall, hence the existence of stereotypes. To help to avoid this cognitive bias it is again important to not work in isolation and to involve others in both your planning for testing (kick offs), the execution of your testing (pairing) and the result of your testing (debriefs)

There are many other biases that I have yet to touch upon and some I might save for future articles including one or two that could have a positive affect when it comes to testing

In the meantime while you wait for my next post @Qualityfrog tweeted a link to a whole bunch of fallacies and their meaning here:

http://www.fallacyfiles.org/taxonomy.html

That should keep you occupied for awhile.

I wonder how many of these fallacies affect your day to day testing?

On a positive note since developers will also suffer from these fallacies when coding, there will always be a need for testers…….