Tuesday, 19 April 2011

Mentoring a New Tester - First test challenge

The following is the results from setting Matthew the eBay challenge which was created by Jon Bach for a weekend testing session. The reported produced by Matthew including his thoughts and thinking – anything which is in red italics are my comments.

Once the session had finished I arranged a debrief and the results of that can also be found in this blog article.

The Mission:

Use the eBay website and find the most expensive product, the most unusual and to get just one hit

Most expensive:

My initial thoughts on this is to just leave the product description blank and go for the greatest product price given to me in the cost field.

Most Unusual:

I'm thinking on this one to just enter in the product field, a few random letters and numbers. What is unusual…? I suppose that’s up to me to decide.

Just one Hit:

This has stumped me. How do you get just one hit without knowing what is on offer. Maybe if I knew the exact product name I could enter it, but even then a search engine will look at all the words entered and come up with a few other options for me, defeating my challenge. Trial and error? Or just entering one random word that I think will be a rarity on eBay, like an ‘AAD’ – Mentor comment What is AAD? I think I will try the latter.

Most expensive:

Straight away no field for price, so clicked on advanced. Mentor comment – could this have been planned/scripted before using the product?

Mentor comment – notice the use of ebay.co.uk – would ebay.com give difference results? I did not specify which to use (assumption was made) Good example of ambiguous requirements?

Typed in 10,000,000


So now I am going to also enter into the keyword/item number field the number ‘1’ as a I imagine most item numbers will have a 1 in it. Mentor comment – why number not letter? The use of wildcards?

Chose the field ‘all categories’ as this was the most likely category for the most expensive as it will search the whole site.

As well as entering the min price of 10.000.000 and leaving the max price because I don’t want to limit myself considering what I am trying to find.

Click search

It looks like I might have found the upper limit for eBay with only three items all coming in at the same price. Mentor comment – interesting when this task was done for weekend testing there was no such limit found on ebay.com – is this just a co.uk limit? Is it a legal issue?

But are these the most the most expensive.

Have now entered ‘2’ into the keywords field.Mentor comment – good exploring – testing your theories based upon results.

Same 3 as above were found plus another, reinforcing the fact that eBay may have an upper limit of £12,869,224.17

Have now entered ‘3’ into the keywords field.

Nothing else found except what has already been found.

Now going to change the min price to 13,000,000 and fill the keywords field with ‘2’, as this was the one with the most hits.

Nothing found.

There is no individual most expensive item, only that eBay has - mentor comment – has is strong word – are you sure? Appears to have may be better an upper price limit of £12,869,224.17 on a number of items. - Mentor comment – as per previous comment does this apply to all eBay sites?

Most Unusual:

Entered a few random letters and numbers into the keyword field on the homepage as shown below.

Found mainly stamps.

Unusual hobby, but not unusual enough for me. - Mentor comment – you need to expand on your definition of what is unusual to you.

Typed in unusual and found mainly ornaments. Not there yet.

Another word for unusual… ‘Quirky’, entered that, found mainly clothes. - Mentor comment – good trying to find different meaning – a dictionary/thesaurus could be useful here

Still not there.

Maybe ‘weird, entered that and found pens in the style of syringes, more disturbing than unusual.

So I thought that this may take to long with me just entering random words. So I thought what might be unusual to me.

I went back to the home page and I looked through the drop down menu.

And thought that the industrial field is something I know nothing about, and may provide me with some unusual items. So chose that field and left the keywords field blank.

Before I had a chance to look at these results I saw excavator. Unusual to me, can also be quite funny at times, so I decided to choose that from the related searches bar under the search tool.

Scrolled down the results page and mainly saw digging machinery.

Mentor comment – love the thinking process being described on how you move from one way to find a solution to the next

Until I saw this

This to me is unusual. Never seen it nor am I likely to see it anywhere else funny to boot too.

Mentor comment – Great result I would have also pointed out the obvious about the digger being a pick up only and delivery being free

Just One Hit:

I thought I would try just entering one random word that I think will be a rarity on eBay, like an ‘AAD’.

So I put ‘aad’ in the keywords field and searched all categories

No good. The word ‘aad’ can be placed into any word of the thousands of products out there.

So I am now thinking of narrowing that field so I chose the advanced search button on the home page.

Chose the fields below to try and narrow the sites search options.

Not what I was looking for. I narrowed the search options that much that the site could not find anything, so it decides to ‘help’ me and remove some of my search options to give me a few items for me to look at.

I’m thinking now that you cannot randomly try and narrow the search as the search engine will just try and ‘help’. So maybe if I know an item to be there that I type into the keywords field the exact spelling of that known item. I am going to use one the most expensive items (previously found) as an example. As shown below.

Mentor comment – love this way of thinking – very thoughtful analytic mind


Is this cheating though? Mentor comment – hmmm NO – you achieved the mission the system allowed you to do this - so it is not cheating just manipulating the system to achieve a result.

All I was asked to do was to get just one hit. I have that one hit. So is this testing the system? The search engine did not try and ‘help’ me on this one. It might know that I am not randomly searching and therefore did not want to help me? In this case then, I have tested the system to get just one hit, so long as you know that an item is there.

Mentor comment – useful information to get one hit you need to know about an item first - good piece of knowledge to be made aware of


During the debrief with Matt I loosely used the PROOF (LA) method

We talked about what had happened during the testing session and Matt replied that it a challenge since he had not done anything like this before in the sense of calling it ‘testing’

He enjoyed the session and felt a sense of achievement in meeting the missions but felt his lack of experience and knowledge of testing techniques hindered him and it was something he felt he needs to do some more learning on.

Matt compared the way he went through the list of missions as similar to army briefs in the sense of some upfront planning of what needs to be done to achieve the goal, he felt his experience of this type of context helped a great deal. During the recording of the session and making his notes he stated that there appeared to be a comparison to recording physic experiments. Where you have a theory to prove/disapprove, you then describe your results and form a conclusion based upon the analysis of the results.

The things that hindered Matt were a lack of clear objectives – something the army instils and some testing knowledge. As part of his homework I asked Matt to research wildcard searches and boundary analysis.

Overall Matt enjoyed the session and is looking forward to the next one.

From my experience it is so easy for those of us who have been in the testing field for awhile to forget how much of the techniques and practises we do by instinct and from our inbuilt repository. Something we should remember when starting with someone new to testing.

Until the next challenge…..

Friday, 8 April 2011

Mentoring a New Tester

A couple of days ago my son in law was talking to my wife with regards to what he would like to do after he completes his service in the Army and what options he has. From this conversation my wife had an idea about him entering the world of software testing and as such asked me to have a word with him to see if he would like to join this crazy world.

Matt (@Matt_Wellington) is a member of the Royal engineers and has done two tours of Afghanistan has a member of the EOD bomb search team. He spends his days searching for IEDs and making sure it is safe for colleagues and local people to walk safely on the roads. He currently has two years (plus) left in the Army and is looking for new challenges; well software testing is a challenge and one I feel he could be great at.

I mentioned on twitter that I was thinking of mentoring Matt and any advice would be welcome.

I got a sudden flurry of encouraging tweets.

@steveo1967 Why not dig into the exercises in rapid software testing? I don't think @jamesmarcusbach or @michaelbolton would mind that!

@steveo1967 Who is he with and when does he get out? I might suggest some intro stuff "Testing Computer Software" tied with non comp. books.

@steveo1967 Give him something to test. Observe him doing it. Feed back. Repeat

@steveo1967 It gives them a good insight on the 'before' and 'after', and by writing cases from the start they are productive! win/win :)

@steveo1967 Then you can move on and do weekend testing with him observing first and then you do pair testing with him at the keyboard.

@steveo1967 "How do you recognize a bug?" has many parallels with "How do you recognize a bomb?"
@steveo1967 So let me get this straight: he searches for things that might blow up harmfully, and you say he has no testing experience?! ;)

@michaelbolton @steveo1967 Good point Michael :) Software testing has less personal risks, hopefully, most of the time ;)

@steveo1967 agree w/ @michaelbolton He's got training in critical thinking now apply it differently. 1/2

@steveo1967 @michaelbolton If he's not ovrseas maybe test meetups may help? Software generally safer than ordinance ;)

So after explaining what testing involves (Briefly – some may think how can you do this briefly!!!)

I first gave him the challenge as suggested by Michael Bolton ‘How do you search/recognise a bomb?”

Matt explained about planning before hand, then re-evaluating when at the site, following safety procedures and using local and own knowledge to search the area. Then when something looks suspicious use techniques learnt during training and on the job experience to confirm the presence of an explosive. Marking the area as such if it is dangerous or the fact it has been searched and then reporting to his senior office the results of what was found.

After hearing this I explained the comparisons with software testing and his answer was is that all this is to it then? Oh the joys of the innocent……

I then gave Matt some material to read, wrongly or rightly the CD from the ISEB course, I feel this is a good way to start and to get to know some of the basic methods of testing, boundary, static etc. I also gave him copies of some excellent software testing books.

Both which I still use as references

My plan for mentoring are as following

  • Basic techniques – the toolset every tester needs
  • Scripted Testing
  • Exploratory Testing
  • Test planning
  • Test reporting/debrief
  • Weekend testing
  • Paired testing

The aim is to have lots and lots of practical/hands on.

Matt does have limited time since in a cpl of weeks he will be working 7 days a week so I need to use an approach which is not overloading and allows him to build up his testing skill set and increase his confidence of the craft.

To this aim I gave Matt his first testing exercise based upon a weekend testing session organized by Jon Bach

The mission was to test the ebay website and find:

  • The most expensive
  • A search that returned only one result
  • A search that returned an unusual item

I asked him to document what he did, what he found and most important his thought process.

The results of this first testing session are amazing – I think I have found I natural tester…… I will publish the results of this first testing effort in my next article.

If anyone wants to help/encourage/support or advise Matt then please get in touch with him on twitter (@Matt_wellington)