The expected result was 42. Now what was the test?

The Purpose of Testing

2012-01-13T07:46:00.000-08:00

I have a strong passion for psychology and the social sciences and their connection to software testing. I currently have a few books on the go on these subjects and hope to write up my thoughts on these books and their connection to testing in the future within the blog.
For those that are interested the books I am currently reading (and re-reading – to make sure that things I assumed from within the books are correct) are:

One interesting quote I found in the book by Ian Dey was the following:

Exploration, description and explanation are the three purposes of social science research.
Earl Babbie

Looking at this quote made me think about what the purposes of testing are and I came to the conclusion that this is the same as for social science research as quoted by Earl Babbie.

If we break this down we have:

Exploration: This is done using exploratory testing, charters, missions etc
Description – Let us describe what we are doing and what we have done when testing
Explanation – Let us explain to managers, peers, stakeholders what we found when testing and our findings.

There are lots of articles, discussions, books about the purpose of testing and how very complex it is, this single sentence quote in my opinion sums up everything about the purpose of testing

I made a note to have a look at what Earl Babbie has got to say and found he has written lots of articles and books some of which could/may apply to software testing, it looks like I have added a few more books to my ever expanding reading list.

Apprenticeship schemes at Test Conferences

2011-12-09T04:38:00.000-08:00

A quick blog on a thought I have had.

I read an article today about how we could try and fix the IT skills gap that exists within the UK, this may also apply around the world, by getting young adults into apprenticeships. I have a view that for some people academia study is not for them and they would better suited to a vocational training course instead of a university degree. I never went to university and as such I do not have a degree. Do I feel as if I have missed out? I do not think so but I have not experienced university life so cannot be sure if I missed out something I may have liked.

I think within our profession of testing we have an opportunity to mentor and help create the next generation of testers (not discounting coders, architects) and allowing them to build their skills and knowledge up by learning from experience rather than studying non relevant subjects at university (How many universities do testing as a degree?) As Nassim Nicholas Taleb has said we as human beings are far better at learning from doing rather than from books. I have over the past year been mentoring two people in our craft of testing one is still on-going the other has managed to secure a tester role within a company, neither have been involved in testing beforehand. I feel we within our community should be trying to do this and encourage young adults by maybe taking them under our tutorage, it does not require a large amount of personal investment, a few hours per week. Or maybe within our companies we should all start looking at trying to introduce apprenticeship schemes, let’s try to tap into this vast resource who in my opinion feels they have been abandoned by the educational system.

On the other side I want to call out to those who run conferences, EuroSTAR, CAST, Lets test, UNICOM and say let’s advertise for young adults who may have an interest to come along as an apprentice for the length of the conference. They would not pay a fee but would be expected to produce a report on their thoughts and what actions they intend to take away for the future. I am have not finalized these thoughts but it would give these young adults to get engagement in a craft which I myself feel very passionate about.

Maybe the organizations that run the conferences could look at running an apprenticeship competition, vetting process. I am sure there are many vocational colleges (Both UK and around the world) who would be willing to get involved in this. It has the added effect that it will start to raise in the minds of the next generation of influential people the value of testing and put testing out there as a forwarding thinking craft that people want to get involved with.

What do others think?

I would especially love some feedback from conference organizers to see how feasible these ideas are.

Recipe Knowledge

2011-12-08T04:25:00.000-08:00

This is my response to an blog article written by Paul Gerrard.

http://gerrardconsulting.com/index.php?q=node/599

I was going to post this as a comment but thought it would be better as a separate blog article

.I am not sure if I agree with entirely what Paul was saying but that is the point of the good blog article. I will say I do entirely agree with his conclusion that we have to have our eyes open and your brain switched on. There are methods that can be used to prevent the ‘quitting’ process and the rambling around in the dark approach to exploratory testing but I think that would be an entirely different article.

However I would suggest people try searching for article on avoiding (or being aware of) bias, cognitive research methods, focusing and defocusing skills. Another one to look at is Air Traffic Control work patterns; they work in time boxed shifts is this similar to session based testing?The point I want to make is the issue Pauls makes about domain knowledge and the usefulness that scripts may bring is an important one.

I am not in the camp that we should abandoned scripts and a lot of the people I communicate with are not saying that either. I feel there are a lot of Chinese whispers with regards to views of some people on the use of scripted tests. I cannot recall anyone saying to me that we must abandon scripts in favour of just doing exploratory testing (Is that a bias and I am deliberately missing or not noticing information?) We also can train ourselves not to quit using a variety of cognitive processes especially the use of checklists and heuristics. These ‘tools’ enable us to counter the quitting instinct but triggering new paths, observations and comparisons.

Testing is not just about finding things it is about asking questions and forming theories based on the answers (evidence) given while experiencing the software. This may lead to more questions and further evaluation and even more re-evaluation of what you already thought debunking and disproving your theories. Finding bugs is a side effect of this approach, a very useful side effect but it is not the sole purpose of testing.

There is a term used within society (especially the social science community) which is ‘recipe’ knowledge, this is often devalued by academics since it is a step by step instruction for learning something. In the everyday world context recipes tell you what to use, what you will need (ingredients) and exactly what procedures to follow, this sound familiar to scripts in the testing world. These recipes can provide important foundations for acquiring or developing skills or as we would say in the software development world learning domain knowledge? People using a recipe as we know may not follow it exactly, they may taste the product and adjust it for their own personal taste so the will move away from the script. However we should not pretend that learning a recipe is the same as learning a skill.

If we for example look at baking, this requires a ‘knack’ which can only come from experience (if you have tried baking bread you will understand this) like qualitative analysis, baking also permits creativity and the development of your own styles.

The skilled tester at some point, like the experienced chef, may stop using the recipe book and start to experiment and explore different tastes and ways to discover more and hopeful improve their skills. At the same time the recipe (script) remains a useful tutorial for the newcomer to the art.

Some of the content used here is taken from the following book:

Qualitative Data Analysis: A User-friendly Guide for Social Scientists - Ian Dey

A ‘title’ is of value to someone who matters

2011-12-07T12:37:00.000-08:00

Recently I attended the Eurostar Testing Conference in Manchester and came away with some mixed messages and thoughts about the content of the conference. Some of the presentations and tracks were really good whilst others appeared to repeat the same old information. I hope to write a few blog articles on some of the positive messages I got from the conference along with lots of ideas I have with regards to social science and how it can be used within testing, these may have to wait until after the holiday period.

The reason for writing this blog post is due to the, what appeared to be a negative, message coming from some of the key note presentations, this is my opinion and how I understood the messages in the context of my views on testing and testers. The one point I wish to raise (and maybe rant) is one of the messages that James Whittaker made.

“at Google ‘Tester’ has disappeared from people’s job titles. People who were ‘testers’ are now ‘developers’ and are expected to code regularly”

Now my thoughts on this may be taking the point James was making out of context however I am not sure in what other context this could be made.

James during the presentation made the point that testers should be part of the team and not get bogged down in who has what role and I whole hearty agree with that.

However from a social and status perspective people need to be able to identify with a title and there has been a lot of talk within the development community about removing titles especially the title of tester. Take the following scenario:

You go out on a social evening with a group of friends and their partners would you work with a project manager, a developer, a business analyst and a tester, As the evening proceeds each person is asked by a non-team member what they do at work.

The developer could reply: I write code and create applications

The tester could reply that they test to ensure the system works

The project manager could reply that they make sure everyone knows what target they have to meet

The Business analyst could say they provide information on what the customers who will use the application need

Each person answering this question I would say would be proud of their job title and what they do.

So my take on making a statement in which we say get rid of the of the title of tester and call everyone a developer is a little insulting and makes me personally feel unappreciated and unvalued. I feel I have been working as a tester for a long period of time now and whilst I can understand that within a team people can have a variety of roles and responsibilities why should I have to give up something that I feel passionate about? I wonder what would be said if at a developers conference everyone is now going to be called a business analyst since we all provide something that the customer wants.

Why does everyone have to be a developer within a project? My concern is why has the word ‘tester’ become such a dirty word? It is if we should be ashamed of what we are and what our title is.

I AM A TESTER AND PROUD OF IT!

Defining Testing

2011-11-04T08:15:00.000-07:00

I am about to run a couple of internal workshops on the Exploratory Testing approach which is based upon a lot of work done by Michael Bolton and James Bach. One of the concerns I have been having recently is what people within the organisation think testing is in comparison to what they actually doing. So I started to put together an article looking at these concerns and trying to see if there is a problem. This blog is based upon some of the points that I cover in the article.

WARNING

Disclaimer:

The views and definitions expressed in this article are my own and as such they may not match what a dictionary may say or agree with your views/definitions.

When I start to look at what we see as testing activities they appear to fall into three distinct categories:

Validation
Verification
Testing

These terms may be familiar to some of the older readers of this blog. V V & T has been around for a long time and has it origins within the manufacturing industry. It has been the main process to provide quality control and assurance of manufacturing production lines. (http://en.wikipedia.org/wiki/Verification_and_validation)

It appears that these ‘manufacturing’ processes have been applied to software testing (http://en.wikipedia.org/wiki/Verification_and_Validation_(software))

This seemed to have lead to the appearance of process standards initially the ISO 9000 Quality Assurance standard, which was modified to become the ISO 9001 2008 standard which included software. These standards are very closely linked to manufacturing process and from a software testing perspective the quality control methods.

Talking to and observing various companies I have seen that lot of people’s perception of testing is as shown in the photo below.

http://brigitteofseon.wordpress.com/category/work-hard-no-go-slow/

Is it a problem to have this perception of software testing?

At the beginning of my career in software testing a lot of companies started to change from mainly hardware manufactures to both hardware and software manufactures. There was a need among these companies to have processes they could use to prove the ‘quality’ of their software products and the general consensus was what had worked in quality control for hardware surely could be applied to software testing.

The reasoning behind these was based upon some fairly flawed assumptions:

All software was the same
All software worked in the same way
All users would follow the designed work flows.
All users would behave in the same way

The main focus of these processes was to validate and verify what was already known about the product and its expected inputs and outputs. In my opinion following quality control and assurance processes is not really about testing Testers ‘normally’ do not control the quality (Yes there are approaches such as TDD which ‘may’ help). If there is crap in the system you are testing then there will be still crap in the system afterwards. Testers provide a service telling you there is crap in the system. Michael Bolton talks more about getting out of the QA business here

Validation and verification

will in the majority of cases NOT

Tell us anything new about the product

Make us ask questions of the product

So what do I mean when I talk about validation and verification?

Validation:

To me validation is about proving what you already know about the product. Confirming what the requirements say are correct and that the system is correct in accordance to what you believe it should be. The normal response when validating will be:

true or false
yes or no
0 or 1

I see Validation as a checking exercise (See article by Michael Bolton here on testing v checking) rather than a testing exercise, it still has some value within the testing approach but it will not tell you anything new about the system being tested. It will prove that what you already know about the system is correct and working (or not working) this is like testing requirements or validation of fields in a database/GUI – you know what the input is you know what outputs you expect according to specification/requirements so why not automate this?

The majority of ‘testing’ I see happening is validation and even though it has some value I would not count validation as testing since it does not tell you anything new about the system you are testing.

It should be noted that to interpret the results from validation ‘testing’/checking requires human interaction to work out if what happens what the correct expected response.

Verification:

When I look at the term verification I use it for when we are verifying any bugs that have been previously found. Someone has made a change to the product and I want to verify that the change made has fixed the problem I had seen before. Some verification tests can be automated – for example if you have run a test previously and found the problem you may be able to automate the steps you followed from testing so that you do not need to run the same test again.

Testing:

I see testing as a thinking exercise in which you need a person to use their skills (and brain) to ask questions of the system being tested. From asking this question they learn more about the system and its behaviour. They will not know the answer to the question but by investigating and tinkering with the product they can form some reasonable answer to the question they posed.When testing we act like crime investigators – you suspect foul play, but you need to ask questions and gather evidence to back up your theories and provide answers to your questions. Testing is not based upon requirements or specification but rather when the specification and requirements are not saying.

Testing is about asking the

What
Why
How

Nassim Nicolas Taleb came up with the following interesting quote:

We are better at doing than learning. Our capacity for knowledge is vastly inferior to our capacity for doing things – our ability to tinker, play, discover by accident. http://www.fooledbyrandomness.com/notebook.htm

So after all of this is there really a problem?

Some of the problems I see within the software testing industry are:

We spend far too much validating rather than testing
We repeat the same validations (manually) time and time again
Cover less of the system by only repeating the same validations
Testing is checking exercise rather than a testing exercise
Testers are not being engaging
Testers are not being challenged
Testers do not need to think
People see testing as a boring thing to do
Testers if used to manually validate are seen as robots

What can be done to improve this?

Look to automate the validation (checking stuff)
Improve coverage by changing the data sets used in validation
Start to use exploratory testing approach – attend a rapid software testing course
Look at using Session based testing
THINK engage your mind and question the system.

We need to keep learning about testing and do more testing rather than keep repeatedly validating systems, it then becomes a much better and challenging role to be a tester.

If Testers were Paranormal Investigators

2011-10-24T03:04:00.000-07:00

Image: Witthaya Phonsawat / FreeDigitalPhotos.net
http://www.freedigitalphotos.net/images/view_photog.php?photogid=3116

I thought considering it is getting close to the spooky time of year (All Hallows Eve) I would put together a tongue in cheek article about what would happen if exploratory testers were paranormal investigators.

It was a dark moonlit night as the certified and exploratory tester approached the dark imposing building that their project manager had asked them to look at. The project manager wanted them to report back on if the building was suitable for him to move into and that there were no hidden surprises. They had heard stories that the building was full of bugs and other scary stuff.
The testers used the (pass) keys to enter the building and slowly walked into the main hallway of the building. As they started to walk around the room suddenly got cold as the temperature dropped.

The certified tester says it must death being down the temperature.

The exploratory tester looks around the room and notices that there appeared to be a draft coming from just outside the room. They go to explore the draft since it has now interested them as something that could answer a question. Once outside the room they notice that one of the windows has come open due to a broken catch. They close the window and make a note to contact a handy man in the morning. The temperature of the room returns back to normal.
The testers slowly move towards the kitchen when all of sudden an overwhelming disgusting smell overpowers their senses.

The certified tester is certain that this is the smell of death coming to get them.

The exploratory tester is not sure and starts to use their sense of smell to see if they locate where the smell is coming from. They notice that the smell appears to get stronger in the direction of the fridge. They open the fridge door and note that the fridge does not appear to be working (even noticing that it is plugged in) inside the fridge there is a bottle of gone off milk which appears to be the source of the smell. They make another note to contact a fridge repair person in the morning.

The testers now move to the next floor in the building when suddenly they appear to see something move on the stairwell.

The certified tester is sure that this is a sign of sprits from beyond the grave.

The exploratory tester takes a moment to think about possible reasons for the movement before realising that the window has come open again causing the light fitting to swing and cast different light patterns on the stairwell giving the impression of movement.

They now move towards the main bedroom on the first floor and suddenly they hear unnatural sounds and what appears to be a creature from another world.
The certified tester is certain that this is souls from the other side warning them to leave and now starts to panic.

The certified tester is not so sure and even though they are starting to get a little scared they open the door to the main bedroom and the noise gets louder and louder. With their heart pounding they enter the room and see a large irregular shaped mass on the bed from where the noises are coming from. Slowly they move forward get closer, closer still and even closer……..

Suddenly the mass moves and the exploratory tester notice that it is their project manager fast asleep and snoring, the snoring which is causing the supernatural noises.

THE END.

Loosely based upon the following article: http://www.wired.com/magazine/2011/09/pl_screenghosthunters

Disclaimers:
• All characters appearing in this work are fictitious. Any resemblance to real persons, living or dead is purely coincidental.
• I wish to stress that this blog in no way endorses a belief or non-belief in the occult or anything of a supernatural nature.

Sat Navs and Maps

2011-10-20T00:40:00.000-07:00

The following blog article is based upon a lightening talk I gave at the Software Testing Club Meet Up in Winchester on the 19th Oct 2011.

I have recently been on holiday touring the Yorkshire dales, moors – covering over 1000 miles in one week. The car is fitted with a sat nav which is great when we want to get from A to B but also I have in the car a large scale map of the UK. I started to think about how we use these two ‘tools’ and how this could be used within testing to show the difference between following a set of instructions (scripts) and exploring the countryside (ET)

An example is that both have the same goal (mission) I want to get from A to B. However we use Sat Navs to show us the most direct, quickest, fastest way (some Sat Navs do now have an option for scenic route)

So I set off into the Yorkshire moors using the large scale map (my wife being the navigator), we knew where we wanted to end up, but the route we took was through the back of beyond. (In fact one the roads we ended up on did not even appear on the Sat Nav map (saying we must return to a digitized area – bug?) We explored the areas and when we noticed things that appeared interesting we took a detour and explored these areas. It was wonderful experience and we found places of interest that were outstanding in natural beauty along with all the seasons in one day (sun, rain, hail). At the end of the journey we had discovered some great things but still ended up at the place we wanted to be, yes it took more time (slightly) but we found out more.

My point is that if you stick to using the sat nav you end up at the same place but you may miss so much that is interesting. Now can we compare this to testing? Yes a script ‘may’ be useful from getting you from A to B but how much will you discover, how many surprises will you find? Yes I could repeat the same journey again since we have the map and know the route I took. Would I want to repeat the exact same route? I am sure that if I went to that area again I would be tempted to go a slightly different way since there could be things around the corner that may interest me.

Rob Lambert pointed out the following to me:

“I find the sat nav is a safety and re assurance aid also in that i can explore but then turn the sat nav on, or refer to it, to then return to a known route.”

I would question this is the sense that it could lead to a false sense of security. What happens if the map gets corrupted, or the electronics fail? I would tend to think of the paper map as the safety, reassurance rather than the sat nav which may have a tendency to fail.

With regards to the meet up in Winchester - I wish more people would have come along they missed a great evening of testing discussions with Michael Bolton being on top form. There are plans to have a regualr bi-monthly meet up in Winchester in the near future - watch out for an announcement via the software testing club soon.

Risky Business

2011-09-18T02:27:00.000-07:00

Within the testing profession we are all aware of risk and in the majority of cases we adjust our testing based upon risk. Is this the wrong approach to take? What models do you use to assign risk to elements within the project?

In my experience in most situations the risks we apply are based upon things we known could go wrong or disrupt the testing we are going to carry out. Most risk assessment is done before hand and upfront. It is normally based upon the probability of what could occur to a system based upon someone’s experiences, viewpoint and biases at a given time. I am not sure this is the correct approach to take within testing.

Testing is not an exact science there are some elements where we can predict the outcomes and risks, yet there are far more where it is unpredictable. The thoughts behind this blog post are to look at this unpredictability and how we can try to include that into our testing approach.

Nassim Nicholas Taleb (1) within his book The Black Swan (2) talks about the highly improbable and its impact on the stock market. He states that the majority of investments are based upon risk and use models in which known risks are taken into account. What these models do not include are the improbable risks things such as natural disasters (3) or individuals/countries (4) do something that cannot be predicted.

In conclusion Taleb says that most models are based upon using top down predictions using experiences of what has already happened which is a high risk strategy rather than plan against the unpredictable the things that cannot be planned for.

So how can this apply to testing?

How many times within testing have we seen a last minute showstopper, just before go live? Or a showstopper discovered in the live system when some, what appears to be a totally random set of circumstances happen (multi failure of various unconnected components – recent power failure within the USA (5)). Could this have been predicted as a risk? Would people have built this into their models? IMO I doubt this.

Do we need to change the way we use risk within our testing? Taleb talk about using stochastic tinkering (6) which to me is fascinating since it appears to match closely to the exploratory testing approach. As an example look at the following two statements:

Thus stochastic tinkering requires experimenting in small ways, noticing the new or unexpected, and using that to continue to experiment.

The general principle is: Do as little as possible unless the system shows you have to do more, then do only as much as you need to keep the process going.

If we change the wording of these statements so that they apply to testing:

Thus stochastic tinkering requires TESTING in small ways, noticing the new or unexpected, and using that to continue to TEST.

The general principle is: Do as little as possible unless the system shows you have to do more TESTING, then do only as much as you need to keep the TESTING going.

Does the exploratory testing approach (by design or accident) do this already? To me it appears as if by using exploratory testing instead of using detailed, well planned, risk assessed test scripts we are more likely to discover the ‘black swans’

Food for thought…

References:

(1) http://www.fooledbyrandomness.com /

(2) http://www.amazon.co.uk/Black-Swan-Impact-Highly-Improbable/dp/0141034599/ref=sr_1_1?ie=UTF8&qid=1316335755&sr=8-1

(3) http://www.yourmoneysite.com/news/2011/mar/market-sees-ripple-effect-of-japan-tsunami-key-indices-lose-over-1-each.html

(4) http://www.marketoracle.co.uk/Article30457.html

(5) http://www.businessweek.com/news/2011-09-09/san-diego-utility-restores-power-to-california-households.html

(6) http://www.cleanlanguage.co.uk/articles/blogs/45/Stochastic-tinkering.html

Professional Qualifications and Bodies

2011-08-05T07:44:00.000-07:00

I saw an interesting tweet from James Bach (@jamesmarcusbach) the other day:

@testingclub What counts as certification? What's a "professional qualification?" Why is schooling confused with education?

Which was in reply to seeing the following post from the software testing club (@testingclub) about a survey of testers?

@jamesmarcusbach you may be interested in the Education for Testers survey results http://www.thetestingplanet.com/2011/07/infographic-education-for-testers/

Whilst the data within the survey may be of interest to some people what really got me thinking were the questions James was asking and within this blog article I am going to attempt and answer some of them from my perspective. It does not necessary mean that my view is correct and I encourage people to debate and correct points that I make, however it is important to remember that the context of this, it is my own personal view of the testing world.

One of the key points that James states to the testing community is that testing is context driven, I feel the answer to these questions are also dependant on context and as such the answers to the questions are context driven.

The first question I intend to try and answer is “What’s a professional qualification?”

The context I am using to answer this is within the UK and Europe where they appear to be very well defined.

Professional qualifications in the UK are generally awarded by professional bodies in line with their charters. These qualifications are subject to the European directives on professional qualifications. Most, but not all, professional qualifications are 'Chartered' qualifications, and follow on from having done a degree (or equivalent qualification).
(http://www.wordiq.com/definition/British_professional_qualifications)

However the important point to note here is the word ‘generally’ to me this does not mean all professional qualification are awarded by professional bodies.

So ‘generally’ professional qualifications are awarded by professional bodies – but what are professional bodies? How do you become a professional body? It appears that it is simple to set up a professional body, all you need to do is:

Get a group of people interested in the same subject

Produce a charter which describes your aims and ethos

Have regular meetings

One interesting point that is made about profession qualifications and bodies that I found was:

Membership of a professional body does not necessarily mean that a person possesses qualifications in the subject area, or that they are legally able to practice their profession.

Some professional bodies can be cartel in which anyone who is not a member cannot practice legally in that domain. Examples of this are within the field of Medicine doctors need to register with the BMA and nurses with the RCN to be able to practice.

So professional qualification in this context indicates that you are proficient in your field and some professional bodies only allow you to practice if you continue to keep up to date with current practices and methods and publish new findings for your peers to review. Without doing this you lose your right to practice. IMO this is the direction tester should be going in. We need to be continuing to learn, read articles, publish articles and enter into debates about the course we take.

ISEB and the other certification schemes are ok as a starting point but it is not the end of learning. We need to adapt these schemes so that they are not static and become out-dated as they currently are. The problem comes that for the people who run these schemes to do this would not make it cost effective and as such it is not in their interest to change. This goes against the reasoning for having these ‘professional qualifications’ the bodies that are saying they represent us on a professional level are not adhering to two KEY parts of being a professional body.

Protecting its fellow professionals

Looking after public interest by maintaining and enforcing standards of training and ethics

Without this happening I have little confidence in the current testing ‘professional qualifications’

Moving on to James question about confusing schooling and education

I find this interesting since seen both sides of the education system (formal schools) having been to school up to the age of 18 and from working within an education system. I think I see what James is getting at. Formal education worked and did not work for me, due to my circumstances up to a certain age I was away from school more than I was there by my own choice I just did not go. Once I did settle into going to school regularly I found it offered me some fantastic grounding in key subject skills (maths, science, history, English) – I really struggled with English and still do according to my wife! It also gave me social skills in being able to share, communicate, listening to others, letting others have their view point which may not agree with mine. I feel lucky in the schools I attended, they may not have been the highest achieving schools but they taught great life skills they I am always thankful for. (Pity think more about league tables than the students). So how does this differ from schooling? The confusion I think comes from the fact that most definitions of schooling see schooling as part of being at school and formal education.

http://www.yourdictionary.com/schooling
http://www.merriam-webster.com/dictionary/schooling

I find this definition worrying since I see schooling as something slightly different. It can mean the education you get at school. However what about ‘home’ schooling, self-schooling? In which you embark on a different style of learning which is not institutional.

The other context here could be that James could be referring to the differing schools of testing. This does not sit right with me and I do have a problem with having different ‘schools’ of testing. I see testing as one big thing not lots of different fragmented schools. Since each school has some strong views and ideas that the others do not agree with we end up in heated debates in which no one side wishes to back down. I am not sure how that helps the testing profession, debates are ok, but constant fighting is not good and at some point a middle ground should be found even if it does not sit easy with all sides. Sometimes it is better for the good of the all rather than for the good of the individual.

My thoughts on these different schools and professional bodies etc. is that maybe just maybe all sides should come together and look at forming a learned society.

What is a learned society?

A learned society is an organization that exists to promote an academic discipline or group of disciplines

http://en.wikipedia.org/wiki/Learned_society

I think this would be a wonderful way forward and maybe the software testing club could be (form) the society? I am not sure nor have I investigated what would be needed but it looks like that they do some of it already publication of articles etc. I would be most interested in what the people at the software testing club think of this and what the general community feels within all of the different schools.

Finally to answer the last question by James:

What counts as certification?

There are many definitions of certification the main being one in which an organization recognise individual/company etc. that meet certain criteria. These criteria could be passing exams, years of experience, publication of articles and so on.

However this really does not ask the question that James asked. Within the survey it shows how many people hold a certification. However as correctly noted by James it does not say which certification. I would have expected this to be much higher. I have many certificates, PAT testing, rugby coach and first aid. None of these are really relevant for my day to day job of testing so I still no sense in the results as they are displayed. However even if it said testing certification which would it mean? ISEB? AST? Etc etc. This one question really stumped me since I could not find an answer that sat easy with me. If I write regular testing article (blog, magazine) and publish should I be certified? If I get my work colleagues to write a report on how competent I am at testing would make me certified? I really do not have an answer for this one and as James did on twitter open up this question to the community.

So the challenge is set:

In your opinion:

What counts as certification?

Is Product Knowledge essential for effective testing?

2011-08-05T04:13:00.001-07:00

I might not be blogging or being online as much as I have been this is due to family life, those close to me know what I mean but here is a new article. I do have lots of ideas and thoughts it is difficult finding time to put them together. I will be at Eurostar in Manchester this year and I hope lots of you will be attending.

It has been awhile since I posted a new blog article so here is a new one.

I recently read an excellent article by Paul Gerrard about all testing being exploratory http://gerrardconsulting.com/index.php?q=node/588 and thought it was so good I posted within our company intranet. I got an unexpected reply which made me think about testing and the skills that are required to be an effective tester. This reply is the reasoning behind this blog post.

The reply I got was as follows:

The interesting challenge to the basic idea is that the tester needs to have knowledge , and very good knowledge of how the system is supposed to work. Only with that knowledge in place is it then really possible to 'intuitively' carry out testing that will be good exploratory testing. Without that deep knowledge it turns into 'random' testing, which, while it has its place in a test approach, I’m not sure it could form the bedrock of a test plan.

The challenge then becomes how to get that knowledge to the tester/test team. I can see how for long term projects/products, the tester becomes truly expert in his Component Under Test, but for new things, or new people to that test team, the ramp up time and 'completeness' of such an approach is questionable and a bit difficult to scale.

For sure exploratory has a part to play - but hard to see how its 'all'.

This made me think about product knowledge and is it really essential for effective testing. So I posted the question on twitter:

Interesting discussion about needing 2 have product knowledge 2 do good exploratory #testing and without this becomes random testing. (1/2)

I have my views on this but would love 2 have #testing community opinions, views, counter views on this. Might do blog post on this (2/2)

I got some replies to this very quickly (as I would expect from such a dynamic community) – sorry if the time order appears a little wrong – I wish twitter would let me do this easier than a cut and paste job

Radionotme

@steveo1967 before gaining experience and understanding I'd have agreed that my testing was more random than exploratory.

@steveo1967 I've recently been exposed to Microsoft AX and found that with experience my exploratory testing is becoming more fruitful.

@mgaertne

@steveo1967 @QualityFrog Well, I would argue that if I know how the software behaves, I don't need to test at all, do I? :)

@steveo1967 @QualityFrog "what I believe it should do" is not knowing to me. :)

@QualityFrog

@steveo1967 @mgaertne product behaviour can be observed in testing. It is implementation, not requirement.

@can_test

@steveo1967 if you know how to do Exploratory Testing well then no prior product knowledge is required. Someone on the team needs it tho!

@steveo1967 IMO that is a very good post on ET. I see nothing there that says prior product knowledge required. (cc @paul_gerrard)

Which were very interesting since they were opposing views about the statement made…

@Radionotme stated that he found product knowledge useful to prevent random testing while mgaetne and qualityfrog stated that it was not necessary and that you could learn about the system whilst exploring.

I countered this with:

@steveo1967

@QualityFrog @mgaertne the counter claim made is that knowing how the product behaves is essential to test quicker and save time

By this time @Michaelbolton had joined the debate

@michaelbolton

@steveo1967 @mgaertne @qualityfrog Whether knowledge or belief, where do you obtain it? From /testing/ what you know.

steveo1967 It takes exploration to develop a decent strategy, tactics, and checks. To me,

@steveo1967 One can develop product knowledge, learning through exploratory #testing. Running scripts helps to suppress that learning.

Trying to keep up with all the threads I replied:

@steveo1967

@mgaertne @QualityFrog not really you would still test that what you believe it should do it actually does do.

@michaelbolton they are not saying ET is not useful they are saying it is more effective when you have product knowledge. Do you agree?

@Radionotme interesting experience with some product knowledge making your #testing more efficient would like 2 know more.

@Radionotme was it random because u had no structure? SBTM for example helps to structure ET or is it ur ET skills improved?

@mgaertne @QualityFrog v true but the view expressed states have knowledge of the product and how u expect it to behave is essential

More people started to enter the debate:

@can_test

@steveo1967 last thought: is Jazz music random? To the untrained ear, perhaps it is. To the experienced, you see it takes great skill.

@Veretax

@can_test @steveo1967 What is what we perceive as random, really isn't random at all, just we lack sufficeent perception to see its order?

@steveo1967

@can_test if someone on the team needs product knowledge why can this not be the tester?

@can_test

@steveo1967 I didn't say it _couldn't_ be the tester. I said it doesn't _have_ to be the tester. Financial Svcs jobs are bad about this.

@steveo1967

@can_test very well put about random being undisciplined that is my point ET without discipline is random

@can_test

@steveo1967 that's what it sounds like to me anyway. "Effectiveness" is meaningful in a certain context. Don't blame the tools.

@steveo1967 I think it's about Trust, or lack thereof. That is, I will trust your ET if you are a product expert, otherwise no, its random

@steveo1967 experience in anything increases your testing efficiency. ET usually looks random to those who don't understand it.

@steveo1967 IMO, good testing is about changing your perspective on the system. That's harder when you are the SME too.

@steveo1967 what does "random testing" mean to them? I may have many hypotheses I want to test that are off the main path. Is that random?

@steveo1967 the statement I disagree with is that Exploratory Testing requires prior industry/product knowledge. That's not true.

@javandervlis

@michaelbolton @steveo1967 @mgaertne @qualityfrog You either belief there’s milk in the fridge or you don’t be…

@michaelbolton

@steveo1967 In addition, exploration implies that someone intends to discover something. Knowledge can never be known to be complete.

@steveo1967 Biases can't be eradicated, but they can be recognized, controlled, and managed in a number of ways. It starts with awareness.

@WadeWachs

@michaelbolton @steveo1967 I am currently the 'fresh eyes'. Old and new eyes both find bugs, though they can be different bugs. 1/?

@Steveo1967

@WadeWachs @michaelbolton very good point. My concern with knowing product is having bias expectations of behavior and not seeing problems

@michaelbolton

@steveo1967 More effective /for what/? More product knowledge /vs. more what-else/? Heuristic: fresh eyes find failure. #testing

So from this lively debate what can we conclude?

Some people think that you do not need to have any product/domain knowledge to be able to carry out exploratory testing since one of the principals of ET is that you learn about the system as you test. Other people say that you would be just doing random testing if you had no knowledge of the product since your expectations of how it should work can guide your testing.

My own personal view is very much in the middle it has been known for me to say that I can test any product without any prior knowledge (domain or otherwise of the product), however the important word missing there is ‘effective’ How effective is my testing without domain knowledge? Does it suddenly become hit and miss and as stated by @radionotme more random. I am currently working on a product which is very niche without understanding how certain packets are formed and transmitted you could spend a lot of time testing unnecessary stuff (there is a counter to this that no testing is unnecessary – in that it exercises the system in nonstandard ways – true but doing too much of this soon makes it less effective)

So to conclude I do not think there is an obvious answer to this. In some cases I feel domain/product knowledge could be essential to make the testing efficient. l it does not mean that a competent tester could not learn this domain knowledge very quickly and start to be effective and efficient at testing the product. However it needs to be recognised that when someone joins a team and comes without domain knowledge there will be some ramp up time for them to become familiar with the domain. In my opinion this where exploratory testing comes into its own, as an approach to use for someone to get on board with a system and learn about the system it is the most effective way especially if you can afford to do paired exploratory testing.

Domain/product knowledge is not essential to do effective testing but it can certainly help.

A Competent Tester

2011-05-29T02:02:00.000-07:00

You can teach a student a lesson for a day; but if you can teach him to learn by creating curiosity, he will continue the learning process as long as he lives. ~Clay P. Bedford

I started writing this blog article in draft about a month ago to have a little rant about how certification does not make you a competent tester based upon my experiences and frustrations whilst trying to recruit testers. I was prompted by a post by Rob Lambert to revisit the article and try to complete it.

Recently I have been trying (and I emphasis the word trying, it has been really trying and challenging to find the right people) to recruit testers to work with me on some quite technical projects. None of the projects have any real UI interfaces nor are they web based applications, this is real hardcore technical testing at a binary level.I became very frustrated and even got to the stage that I felt my standards were too high.

I lot of the CV and candidates that were interviewed all stated that they had ISEB or ISTQB certification and from that I assumed they would have a basic grasp of boundary and edge cases. How wrong was I!!! To make it worse I asked about their views on any of the current new approaches and techniques in testing, even having to prompt to what do you think of context driven testing and all I got back in return was a blank look. I asked what articles or books have you read recently about software testing or any book that you could relate to software testing and again all I got was blank looks and shrugs of shoulders.

I am not against certification in any shape or form and I do not have a view if people try to make money out of it that is not the issue. The issue I have is that in some (most) cases these schemes are being sold on the basis that once you have completed them you are now a ‘skilled’ tester and know everything there is to know about testing.

PASSING A MULTI-CHOICE EXAM DOES NOT MAKE YOU COMPETENT!!!!!

Rob Lambert in his article explains in great depth what makes testers ‘skilled’ and I have to agree with him. Those who read my blog know I have a major interest in the social sciences and psychology and how this can collate to testing. More importantly how it can help make you a better tester.

So if you want to become competent at testing you have to read more, interact more with the testing community, become self-learning. My ethos is always to keep on learning and never stop doing so.

Taking any of the certification courses can be, for someone new in the world of testing, a good STARTING point, grounding in SOME of the techniques and skills. These courses will NOT teach you about how testing fits into Agile and about exploratory testing. Nor will they teach you how to test and make you think ‘outside the box’ Only getting involved within the testing community will do that. I good start is the Software Testing Club – read some of the articles and blogs there, subscribe to the excellent RSS feed. I am not being paid by the Software Testing Club to say this I am promoting it because again it is a good starting point for those would wish to learn more about testing and want to improve so they can become competent testers.

I will finish with a quote about learning.

It's what you learn after you know it all that counts. ~Attributed to Harry S Truman

Mentoring a New Tester - First test challenge

2011-04-19T08:19:00.001-07:00

The following is the results from setting Matthew the eBay challenge which was created by Jon Bach for a weekend testing session. The reported produced by Matthew including his thoughts and thinking – anything which is in red italics are my comments.

Once the session had finished I arranged a debrief and the results of that can also be found in this blog article.

The Mission:

Use the eBay website and find the most expensive product, the most unusual and to get just one hit

Most expensive:

My initial thoughts on this is to just leave the product description blank and go for the greatest product price given to me in the cost field.

Most Unusual:

I'm thinking on this one to just enter in the product field, a few random letters and numbers. What is unusual…? I suppose that’s up to me to decide.

Just one Hit:

This has stumped me. How do you get just one hit without knowing what is on offer. Maybe if I knew the exact product name I could enter it, but even then a search engine will look at all the words entered and come up with a few other options for me, defeating my challenge. Trial and error? Or just entering one random word that I think will be a rarity on eBay, like an ‘AAD’ – Mentor comment What is AAD? I think I will try the latter.

Most expensive:

Straight away no field for price, so clicked on advanced. Mentor comment – could this have been planned/scripted before using the product?

Mentor comment – notice the use of ebay.co.uk – would ebay.com give difference results? I did not specify which to use (assumption was made) Good example of ambiguous requirements?

Typed in 10,000,000

Error

So now I am going to also enter into the keyword/item number field the number ‘1’ as a I imagine most item numbers will have a 1 in it. – Mentor comment – why number not letter? The use of wildcards?

Chose the field ‘all categories’ as this was the most likely category for the most expensive as it will search the whole site.

As well as entering the min price of 10.000.000 and leaving the max price because I don’t want to limit myself considering what I am trying to find.

Click search

It looks like I might have found the upper limit for eBay with only three items all coming in at the same price. –Mentor comment – interesting when this task was done for weekend testing there was no such limit found on ebay.com – is this just a co.uk limit? Is it a legal issue?

But are these the most the most expensive.

Have now entered ‘2’ into the keywords field. –Mentor comment – good exploring – testing your theories based upon results.

Same 3 as above were found plus another, reinforcing the fact that eBay may have an upper limit of £12,869,224.17

Have now entered ‘3’ into the keywords field.

Nothing else found except what has already been found.

Now going to change the min price to 13,000,000 and fill the keywords field with ‘2’, as this was the one with the most hits.

Nothing found.

There is no individual most expensive item, only that eBay has - mentor comment – has is strong word – are you sure? Appears to have may be better an upper price limit of £12,869,224.17 on a number of items. - Mentor comment – as per previous comment does this apply to all eBay sites?

Most Unusual:

Entered a few random letters and numbers into the keyword field on the homepage as shown below.

Found mainly stamps.

Unusual hobby, but not unusual enough for me. - Mentor comment – you need to expand on your definition of what is unusual to you.

Typed in unusual and found mainly ornaments. Not there yet.

Another word for unusual… ‘Quirky’, entered that, found mainly clothes. - Mentor comment – good trying to find different meaning – a dictionary/thesaurus could be useful here

Still not there.

Maybe ‘weird, entered that and found pens in the style of syringes, more disturbing than unusual.

So I thought that this may take to long with me just entering random words. So I thought what might be unusual to me.

I went back to the home page and I looked through the drop down menu.

And thought that the industrial field is something I know nothing about, and may provide me with some unusual items. So chose that field and left the keywords field blank.

Before I had a chance to look at these results I saw excavator. Unusual to me, can also be quite funny at times, so I decided to choose that from the related searches bar under the search tool.

Scrolled down the results page and mainly saw digging machinery.

Mentor comment – love the thinking process being described on how you move from one way to find a solution to the next

Until I saw this

This to me is unusual. Never seen it nor am I likely to see it anywhere else funny to boot too.

Mentor comment – Great result I would have also pointed out the obvious about the digger being a pick up only and delivery being free

Just One Hit:

I thought I would try just entering one random word that I think will be a rarity on eBay, like an ‘AAD’.

So I put ‘aad’ in the keywords field and searched all categories

No good. The word ‘aad’ can be placed into any word of the thousands of products out there.

So I am now thinking of narrowing that field so I chose the advanced search button on the home page.

Chose the fields below to try and narrow the sites search options.

Not what I was looking for. I narrowed the search options that much that the site could not find anything, so it decides to ‘help’ me and remove some of my search options to give me a few items for me to look at.

I’m thinking now that you cannot randomly try and narrow the search as the search engine will just try and ‘help’. So maybe if I know an item to be there that I type into the keywords field the exact spelling of that known item. I am going to use one the most expensive items (previously found) as an example. As shown below.

Mentor comment – love this way of thinking – very thoughtful analytic mind

Eureka!

Is this cheating though? – Mentor comment – hmmm NO – you achieved the mission the system allowed you to do this - so it is not cheating just manipulating the system to achieve a result.

All I was asked to do was to get just one hit. I have that one hit. So is this testing the system? The search engine did not try and ‘help’ me on this one. It might know that I am not randomly searching and therefore did not want to help me? In this case then, I have tested the system to get just one hit, so long as you know that an item is there.

Mentor comment – useful information to get one hit you need to know about an item first - good piece of knowledge to be made aware of

Debrief:

During the debrief with Matt I loosely used the PROOF (LA) method

We talked about what had happened during the testing session and Matt replied that it a challenge since he had not done anything like this before in the sense of calling it ‘testing’

He enjoyed the session and felt a sense of achievement in meeting the missions but felt his lack of experience and knowledge of testing techniques hindered him and it was something he felt he needs to do some more learning on.

Matt compared the way he went through the list of missions as similar to army briefs in the sense of some upfront planning of what needs to be done to achieve the goal, he felt his experience of this type of context helped a great deal. During the recording of the session and making his notes he stated that there appeared to be a comparison to recording physic experiments. Where you have a theory to prove/disapprove, you then describe your results and form a conclusion based upon the analysis of the results.

The things that hindered Matt were a lack of clear objectives – something the army instils and some testing knowledge. As part of his homework I asked Matt to research wildcard searches and boundary analysis.

Overall Matt enjoyed the session and is looking forward to the next one.

From my experience it is so easy for those of us who have been in the testing field for awhile to forget how much of the techniques and practises we do by instinct and from our inbuilt repository. Something we should remember when starting with someone new to testing.

Until the next challenge…..

Mentoring a New Tester

2011-04-08T01:16:00.000-07:00

A couple of days ago my son in law was talking to my wife with regards to what he would like to do after he completes his service in the Army and what options he has. From this conversation my wife had an idea about him entering the world of software testing and as such asked me to have a word with him to see if he would like to join this crazy world.

Matt (@Matt_Wellington) is a member of the Royal engineers and has done two tours of Afghanistan has a member of the EOD bomb search team. He spends his days searching for IEDs and making sure it is safe for colleagues and local people to walk safely on the roads. He currently has two years (plus) left in the Army and is looking for new challenges; well software testing is a challenge and one I feel he could be great at.

I mentioned on twitter that I was thinking of mentoring Matt and any advice would be welcome.

I got a sudden flurry of encouraging tweets.

@ola_hylten

@steveo1967 Why not dig into the exercises in rapid software testing? I don't think @jamesmarcusbach or @michaelbolton would mind that!

@PeterWalen

@steveo1967 Who is he with and when does he get out? I might suggest some intro stuff "Testing Computer Software" tied with non comp. books.

@michaelbolton

@steveo1967 Give him something to test. Observe him doing it. Feed back. Repeat

@destruise

@steveo1967 It gives them a good insight on the 'before' and 'after', and by writing cases from the start they are productive! win/win :)

@ola_hylten

@steveo1967 Then you can move on and do weekend testing with him observing first and then you do pair testing with him at the keyboard.

@michaelbolton

@steveo1967 "How do you recognize a bug?" has many parallels with "How do you recognize a bomb?"

@steveo1967 So let me get this straight: he searches for things that might blow up harmfully, and you say he has no testing experience?! ;)

@ola_hylten

@michaelbolton @steveo1967 Good point Michael :) Software testing has less personal risks, hopefully, most of the time ;)

@PeterWalen

@steveo1967 agree w/ @michaelbolton He's got training in critical thinking now apply it differently. 1/2

@PeterWalen

@steveo1967 @michaelbolton If he's not ovrseas maybe test meetups may help? Software generally safer than ordinance ;)

So after explaining what testing involves (Briefly – some may think how can you do this briefly!!!)

I first gave him the challenge as suggested by Michael Bolton ‘How do you search/recognise a bomb?”

Matt explained about planning before hand, then re-evaluating when at the site, following safety procedures and using local and own knowledge to search the area. Then when something looks suspicious use techniques learnt during training and on the job experience to confirm the presence of an explosive. Marking the area as such if it is dangerous or the fact it has been searched and then reporting to his senior office the results of what was found.

After hearing this I explained the comparisons with software testing and his answer was is that all this is to it then? Oh the joys of the innocent……

I then gave Matt some material to read, wrongly or rightly the CD from the ISEB course, I feel this is a good way to start and to get to know some of the basic methods of testing, boundary, static etc. I also gave him copies of some excellent software testing books.

The art of software testing by Myers

A Practitioner's Guide to Software Test Design by Lee Copeland

Both which I still use as references

My plan for mentoring are as following

Basic techniques – the toolset every tester needs
Scripted Testing
Exploratory Testing
Test planning
Test reporting/debrief
Weekend testing
Paired testing

The aim is to have lots and lots of practical/hands on.

Matt does have limited time since in a cpl of weeks he will be working 7 days a week so I need to use an approach which is not overloading and allows him to build up his testing skill set and increase his confidence of the craft.

To this aim I gave Matt his first testing exercise based upon a weekend testing session organized by Jon Bach

The mission was to test the ebay website and find:

The most expensive
A search that returned only one result
A search that returned an unusual item

I asked him to document what he did, what he found and most important his thought process.

The results of this first testing session are amazing – I think I have found I natural tester…… I will publish the results of this first testing effort in my next article.

If anyone wants to help/encourage/support or advise Matt then please get in touch with him on twitter (@Matt_wellington)

An update or two.

2011-03-31T01:22:00.000-07:00

I noticed that I have not written a blog article in awhile so I thought I would put together a short article on what I have been up to so that regular readers can be sure that I am still alive and well.

On the personal front we have had a few health scares over the past month hence my lack of tweeting or blogging.

On the work front I have been very busy and involved in a few different and exciting projects while continuing to look at different ways in which we can improve.

During this period I have been looking more and more into ethnographic research and its connection to testing. I find this area of social science fascinating and how much it appears to collate to testing. Since there does appear to be a connection to this I am current running a couple of case studies internally based upon methods from ethnographic research as mention by Richardson in their article for Qualitative Inquiry: Evaluating ethnography

The findings for this case study will be presented at the UNICOM Next Generation Testing conference 18/19 May 2011

If you cannot make this event I do intend to give a very basic/quick introduction to this approach the Software Testing Club meet up in Oxford on the 14th April 2011 This event will be used as a world premier for the approach I have been working on so that definitely makes it worth attending. Or the fact that Lisa Crispin and Rob Lambert will be there should tick everyone’s box.

Without giving away too much detail before the meet up here is a brief summary of the approach I have been investigating

The concept is based upon questioning the tester as much as you question the product being tested.
It is check-list that can be used on an individual basis and should take between 5 and 10 minutes. The idea is to look at what you are doing and checking it is the right thing and see if you missing anything.
I will be giving away the check-list on the evening of the meet up. (wow a freebie)

Have I given away too much information, not enough or left you wanting more?

If you want to know more then I suggest you sign up to attend the meet-up or the UNICOM conference

Is Context Driven Testing a gimmick?

2011-03-10T04:05:00.000-08:00

My inspiration for this article has been a comment I received with regards to trying to organise an internal Rapid Software Testing course .

Someone made a comment that they felt that this ‘smacks of a gimmick’ but would be interested in finding out what people discover/get out of this in relation to the way we work.

The views expressed in this article are solely my own and are based upon my own experiences and knowledge of the testing profession.

Currently within the testing world there appears to be two schools of thought:

The traditional (standards) approach – driven by the ISTQB examination board (formerly the ISEB)

And there is the

Context Driven Testing concept– driven by such people as Cem Kaner, James Bach and Michael Bolton.

This article is not about entering a debate to say one approach is better than the other and IMO I think a good balance is a mixture of the various approaches. There are many other approaches and concepts to testing than the two mentions above such as the Agile and the analytical approaches but the main discussions within the testing community appear to mainly refer to the two 'schools' listed above.

The principles and concept of context driven testing is that the emphasis is about thinking, experiencing and doing rather than assuming and making interpretation of what people believe the system should do. It is more about ‘hands-on’ and learning about the system as you test.

A common misconception is that there is no planning involved within rapid software testing and that it is based upon ‘free’ testing and it is without structure or discipline. In my experience and from using the material from the rapid software testing course there is more planning, structure and focused based testing than with any other approach. The introduction of session based testing in which testers have a mission and a goal to aim for during their testing session ensures that testing remains focused and on track.

The difference between the two approaches in that the standard approach is mainly used to define testing (test cases) before you have actually have access to the system under test. There is an inherent weakness in this approach in that assumptions are made that requirements, design and user needs are correct, accurate and not missing.

Once actual testing has started the majority of testers revert to working in a context driven way. They adapt the scripts they have written; they think of new ones and make decisions on what not to run. The context driven approach is to have some lightweight upfront planning which is ambiguous and allows the tester the freedom to approach the testing by using their logic and adapting as they learn more about the system. This allows the tester to build up knowledge of the system while executing, creating new tests and recording what they are testing. This is the basic definition of exploratory testing. Time is not wasted on creating tests that will never be run or maintaining a list of test scripts with incorrect test steps. It is about recording what is happening at the time testing happens and storing the results of that testing session. This then can if possible be automated and as a test never run manually again.

The difference between the approaches is that rapid software testing requires testers to think as they test and not just tick boxes. It forces the tester to question what they see and allows them to freely explore and discover new things about the system. It uses triggers (heuristics) to keep asking the question "Is there a problem here?"

It does this by comparing the product with similar product, looking at the history of the product or the claims made by the product. These are tests in which there is no yes or no answer it depends on the context and the thinking of the tester.

Is context driven testing a gimmick?

IMO it is not

It is a natural way to test products and it is the way testing has been done since it became a career choice. However people have not admitted (or will not admit) to following this approach or be aware that they are doing this.

The whole concept and approach of rapid software testing is to give it a name and provide useful skills and tools to improve this methodology which follows the thinking of context driven testing.

_______________________________________________________

FOOTNOTE:

After a lively discussion on twitter with James Bach I feel I need to clarify some misuse of definitions.

James gave a great description to show the differences between Rapid Software Testing and Context Driven:

Rapid Testing is a testing methodology that is context-driven.

But context-driven testing is not Rapid Testing.

After this 'revelation' I have made some minor changes to the original post.

Measuring Testing

2011-02-21T06:50:00.000-08:00

I saw a couple of tweets by @Lynn_Mckee recently on the metrics that are used in testing.

There are many great papers on #metrics. Doug Hoffman's "Darker Side of Metrics" provides insight on behavior. http://bit.ly/gKPHcj #testing

Ack! So many more that are painful... Scary to read recent papers citing same bad premises as papers from 10 - 15 yrs ago. #testing #metrics

And it made me think about how we measure testing.

This article is not going to be

'This is how you should measure testing’

offer any ‘best practice’ ways of measuring

My concern with any of the ways in which we measure is that it is done without context or connection to what question you which to have answered with the numbers. It is a set of numbers devoid of any information to their ‘real’ meaning. There are many and various debates within the software testing field about what should and should not be measured. My take on all of this is:

Can I provide useful and meaningful information with the metrics I track?

I still measure number of test cases that pass and fail and number of defects tests and fixed.

Is this so wrong?

If I solely presented these numbers without any supporting evidence and a story about the state of testing then yes it is wrong it can be very dangerous.

I view the metrics that are gathered during testing to be an indication that something might be correct or wrong, working or not working, I do not know this just from the metrics it is from talking to the team, debriefing and discussing issues.

I capture metrics on requirement coverage, focus area coverage, % of time spent testing, defect reporting, system setup. So I have a lot of numbers to work with which on their own can be misleading, confusing and misinterpreted. If I investigate the figures in detail and look for patterns I notice missing requirements, conflicting requirements and what is stopping me executing testing.

So what is this brief article saying?

Within the software testing community I see that we get hung up on metrics and how we measure testing and I feel we need to take a step back.

It is not too important what you measure but how you use and present what measurements you have captured. It is the stories that go with the metrics that are important, not the numbers.

What you believe might not be true. (Part 2)

2011-02-02T02:49:00.000-08:00

The first part of this article looked at conjunction bias and framing bias and how it can influence our thinking towards incorrect assumptions all under the heading of cognitive bias.

The next part of this article investigates other forms of bias and how they influence our decisions and thought processes. One of the first I will touch upon in this article is belief bias.

Belief bias has many similarities to confirmation bias and in some ways both are closely linked. If someone has very strong beliefs they can use arguments that back up their beliefs in such a way that only evidence that supports their beliefs are used giving a confirmation bias to their beliefs. There are many examples of this in the world from the belief in the existence of aliens to the range of conspiracy theories that are abound on the internet.

So what is belief bias?

People will tend to accept any and all conclusions that fit in with their systems of belief, without challenge or any deep consideration of what they are actually agreeing with.

http://changingminds.org/explanations/theories/belief_bias.htm

Belief bias is the conflict a person incurs when their beliefs do not match the logic of what is presented to them

The danger with belief bias is that it can quickly turn to belief projection:

Psychological projection is a form of defence mechanism in which someone attributes thoughts, feelings, and ideas which are perceived as undesirable to someone else.

The problem now is that the beliefs of someone on a team could become fostered on other team members using belief projection even if what they believe is unfounded. Within software development we all have our own views and beliefs on what a piece of software is expected to do.

How does this have an impact on software development and especially testing? Imagine a situation in which a tester has a very firm belief on how an interface should interact. They then test that interface and find it is not behaving as they believe it should be. A bug report is now raised and passed back to the development team. It is found that the bug was raised in error and that the interface interacts as designed and described in the requirements. This is simple case in which regardless of what requirements, design specifications and others are saying the testers strong belief bias is saying everyone is wrong and what they believe in is correct.

In the world in which we as testers operate I doubt that the above would happen since we are now in situations in which developers and testers communicate and there is no more throwing it over the wall way of releasing. However if you still work in teams in which there is a lack of communication and talking then belief bias can have a large negative affect on testing.

Another issue is when you do work in a team and belief projection comes into the equation. If someone on your team subconsciously has a belief that the developers think the testers are a waste and not necessary (negative personality trait) then could project this on to other members of the team and start to cause a barrier of resentment to build up between teams. It is impossible to prevent people having opinions and thoughts about other members of a team but having an environment in which everyone is allowed to express their views and thoughts in an open discussion can help to remove this type of bias. Within one company in which I worked as a team lead I would have an open session in which nothing was recorded or written down but people could express views and thoughts on what was really happening within the project. Sometimes it would be heated and people would get emotional but it managed to clear the air. One important part of this method was that a mediator was always in charge to prevent it getting into a naming calling situation.

Another bias which could have an impact on testing is illusory correlation in which people form a connection between two events even when all evidence shows that there is no such connection or relationship. A good example is people who have arthritis believe that their condition worsens depending on the weather. Redelmeier and Tversky conducted an experiment in which they took measurements based upon the patient’s view of their condition and at the same time noted details meteorological data. Even though nearly all the patients believed that their condition got worse during bad weather the actual results shown that there was a near zero correlation between the two.

Wikipedia defines illusory correlation as:

Illusory correlation is the phenomenon of seeing the relationship one expects in a set of data even when no such relationship exist.

http://en.wikipedia.org/wiki/Illusory_correlation

It is easy to see the effect this can have on software development. Imagine if a developer creates an illusory correlation between two variables that do not have any real correlation therefore introducing bugs into the project. There has been a study on the reasons for software errors and it has been found that illusory correlation does play a part. Details of this study can be found here.

Stereotypes are normally defined by the use of illusory correlation. Someone who came from a small town where everyone was kind makes an assumption that everyone from a small town is kind therefore when they go out into the world and meet a kind person they correlate that the person much be from a small town since they are kind even if the correlation is not true or does not make any sense.

How does this help or hinder with regards to software testing?

The problem occurs when testers work in isolation and form they own methods and create their own hypotheses of what should happen when they test the product under certain conditions. The danger is that the testing becomes one sided searching for evidence that matches their current hypothesis of how the product should react. The resulting factor of this bias within testing is that conditions are tested which meets the illusory correlation of the tester but conditions which do not meet the expected assumptions are not tested. This could then cause significant bugs to be missed due to flows not meeting the expected correlation being tested.

It is very difficult to avoid falling in to the illusory correlation trap since the human mind tries to take the easiest path and groups’ objects together for easier recall, hence the existence of stereotypes. To help to avoid this cognitive bias it is again important to not work in isolation and to involve others in both your planning for testing (kick offs), the execution of your testing (pairing) and the result of your testing (debriefs)

There are many other biases that I have yet to touch upon and some I might save for future articles including one or two that could have a positive affect when it comes to testing

In the meantime while you wait for my next post @Qualityfrog tweeted a link to a whole bunch of fallacies and their meaning here:

http://www.fallacyfiles.org/taxonomy.html

That should keep you occupied for awhile.

I wonder how many of these fallacies affect your day to day testing?

On a positive note since developers will also suffer from these fallacies when coding, there will always be a need for testers…….

What you believe might not be true. (Part 1)

2011-01-26T06:21:00.000-08:00

When I started to look at how the human mind works and the traps that it continually falls into I did not realise what a huge area of psychology this is. The subject of bias and the human mind is fascinating and every tester should be aware that every decision we make when testing a product will be subjected to our cognitive biases.

I have previously touched upon how bias can affect our judgement when I wrote the blog post about confirmation bias and cognitive dissonance. We need to have awareness that what we think could be wrong and subjective to our own biases. There are ways to try and reduce cognitive biases by the use of pairing and debriefing however that is not the subject of this blog. The purpose of this blog is to look at some of the common cognitive biases in relation to their effect on testing.

I shall start by defining the term cognitive bias:

A cognitive bias is a mistake in reasoning, evaluating, remembering, or other cognitive process, often occurring as a result of holding onto one's preferences and beliefs regardless of contrary information.

http://www.cramster.com/definitions/cognitive-bias/784

Within the psychology field of cognitive biases there are many different types of biases some of which I have previously discussed. Within this article I will look at a few more which could have an effect on our testing. The whole area of cognitive bias is huge and I could write many more blogs on different types of bias and it is something I may return to at some point. Since it such a large area I may not go into great detail about each type of bias but give enough information for people reading this blog to be aware of the failings of our human minds.

One bias that intrigues me is called the conjunction effect.

A definition of this bias is described below:

When two events can occur separately or together, the conjunction, where they overlap, cannot be more likely than the likelihood of either of the two individual events. However, people forget this and ascribe a higher likelihood to combination events, erroneously associating quantity of events with quantity of probability.

http://changingminds.org/explanations/theories/conjunction_fallacy.htm

An example of this can be seen when using the experiment that Amos Tversky and Daniel Kahneman carried out:

Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.

Which is more probable?

A) Linda is a bank teller.
B) Linda is a bank teller and is active in the feminist movement.

In the experiment 86% of people answered B even when using mathematics it can be proven than A is more probable. This is the conjunction fallacy in action. When your mind tricks you into believing something is more probable than it is.

Another experiment from Tversky and Kahneman during 1983 two different experimental groups was asked to rate the probability of two different statements, each group seeing only one statement:

A complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983.
A Russian invasion of Poland, and a complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983.

Even though the probability of each happening was low there was a significant difference in people choosing that the second statement was more likely.

The moral? Adding more detail or extra assumptions can make an event seem more plausible, even though the event necessarily becomes less probable.

How does this all relate to testing?

Within testing we have to look at statements and judge what is most probable to happen.

For example we see the following requirements:

Req1:If the user is a software engineer then screen ‘is engineer’ must be shown.

Req2: If the user is a software engineer and likes to listen to classical music then screen ‘music engineer’ must be shown.

Now if we look at the following user story.

David went to university to train as a classical violinist. Once leaving university David retrained as a software engineer ad started to write code for a major software house.

Which is more likely to be true?

Most people will see Req2 as the most likely to be true, and ignore Req1. However the probability that Req2 is more likely in the given user story is far less than Req1.

Now with these two requirements you can see a conjunction effect, in which one of the statements appear to more likely than the other and as testers we should be aware of this. However our minds might not notice that there is a conjunction and our human bias takes over so when we test we assume that only req2 is valid and ignore req1. However the probability of req2 is less than req1, we need to be aware of this bias and try to eliminate it. This is why context is important, we need to apply context to the situation and to the test to ensure it is valid, correct and nothing is being missed.

The issue we face as testers is that some requirements and the resulting test ideas we come up with could be subjected to the conjunction effect were our minds are telling us that the probability of event x happening is greater than the probability of event y even when if we look at the mathematics the probability between each event is the same or event x actually has a lower probability.

How can we prevent his bias?

Experiments that have tried to repeat the Tversky and Kahneman bank teller fallacy noted that if before the decision was made people were allowed to discuss and communicate their thoughts with others then conjunction fallacy occurrence was significantly lower.

http://www.econ.ohio-state.edu/levin/wpapers/linda_geb_2nd_revision1-1.pdf

This indicates that it is possible to reduce the chance of conjunction fallacy occurring just be simply communicating and talking with other people.

When researching this area for the blog I found that there are a lot of links to cognitive framing:

http://www.adsavvy.org/the-power-of-framing-effects-and-other-cognitive-biases/

The problem being that the way something is worded (framed) can lead people to be subjective to conjunction fallacy. Maybe this is something that architects and technical authors need to be aware of when creating design and requirement documents? How a requirement is framed could influence a developer to write code in certain way and become subjected to the conjunction fallacy giving more probability to some event or requirement that is actually true, developers please be aware of this.

What fascinates me on this particular area is the idea that the way things are worded (framed) and how our mind understands these words (conjunction fallacy) could be the reason that a lot of bugs are created in code. I would love to gather data on this and see if there is some correlation.

So if we look at framing and how it can influence our thought process and cause bias. The problem with framing is that it can be subjective. If we frame a sentence in such a way as to force people to believe that a certain fact is true then there will be some people in which it will have no influence. Framing is used a great deal within politics and advertising to encourage people to belief in a certain policy or product. It can be very powerful as a tool. Michael Bolton ran a workshop at Eurostar 2010 on test framing (add link) and this was very useful to help the tester to think if a test needs to be run or why it was not run however it was only after Eurostar that I had a thought that framing can be used in a opposite direction and become dangerous as a tool. It could be used to force people to think that the wrong view is the correct view. An example of this is to have a list within a test plan of 1000s of test cases but to frame it in such a way that to the casual reader every single test case must be run and is of high value. This is before the tester has actually touched or used the product to be tested. The framing is used to justify wasted effort and work.

For example look at the following statement:

I prefer to code using Java than C++ because it is easier with the development suite I have installed and C++ does not work within the development environment I have set up.

The framing bias here is that the person thinks Java is superior to C++ because of how they have set up their environment. It maybe that using C++ would make it easier for the developer on the project they are working on but they are trying to give reasons why they are working in Java instead of C++.

There are many more examples in the world of software development. I prefer OS x to OS y because of z. Machine type ‘a’ is far better than machine type ‘b’ because it does ‘xyz’. It does not matter that the person making the statement has never used OS y or machine type ‘b’. They are forming a bias viewpoint and using framing to justify it. IMO this is why the context driven way of testing is so important. When looking at requirements and statements it is easier to think of ‘in which context’ to verify if the requirement is just and sound.

As testers we need to be aware of this and when we report our findings during testing be aware of how we frame the words.

Part 2 of this article will look at belief projection and how it may hinder our testing efforts.

Are testers’ ethnographic researchers?

2011-01-18T06:23:00.000-08:00

People who follow me on twitter or via this blog might be aware that I have a wide range of interests in areas outside my normal testing job. I like to research and learn different things, especially psychology, and see if it may benefit and improve my skills and approaches during my normal testing job. One area I have being looking at for awhile is the social science of ethnography. The approaches used when carrying out research appears to have many similarities to software testing and I feel we could benefit and maybe improve our testing skills by examining ethnography.

IMO there are two areas in which we can learn from ethnography:

To improve our understanding of users and how they differ by using ethnographic methods

Use ethnographic methods to test software in an exploratory way.

I should start by explaining what my understanding of ethnography is:

Wiki attempts to define it here:

http://en.wikipedia.org/wiki/Ethnography

The free dictionary attempts to give a definition here:

http://www.thefreedictionary.com/ethnography

A better definition can be found here:

http://www.brianhoey.com/General%20Site/general_defn-ethnography.htm

The problem with trying to describe and define ethnography is that it has wide and varied meanings.

To me it is a branch of the study of humanity (anthropology) in which the researcher actively gets involved and participates with the study group rather than just sitting back and observing. The reporting is doing using qualitative (words) measurements rather than rely on quantitative (numbers) measurements.

One of the key factors when approaching ethnographic research is to be aware that participation, rather than just observation, is one of the keys to the approach. Does this not sound familiar to testing, especially exploratory testing? Actively using the software under test to find out about its characteristics and behaviour are similar to a ethnographic researcher living within a community and participating with that community to learn about its beliefs and characteristics. There appears to be very close parallels between ethnographic research and exploratory testing. Wikipedia states:

One of the most common methods for collecting data in an ethnographic study is direct, first-hand observation of daily participation.

How similar is that to testing software?

Another approach within ethnography is the use of grounded theory to explain the results from the participation. This is when the data is used to provide theories about the data. This is different from grand theory in which the theory is defined without the use of real life examples and therefore has a danger of not fitting the actual data gathered afterwards (is this similar to scripted and exploratory, grand theory vs grounded theory?)

Grounded theory is a constantly evolving set of conclusions that can continue indefinitely based upon the changing data being obtained by the ethnographic researcher. One of the questions that are asked about ethnographic research is:

When does this process end?

One answer is: never! Clearly, the process described above could continue indefinitely. Grounded theory doesn't have a clearly demarcated point for ending a study. Essentially, the project ends when the researcher decides to quit. (http://www.socialresearchmethods.net/kb/qualapp.php)

How similar is this to testing?

When do we stop testing?

Many articles have been written on this subject and mainly we stop when we can learn nothing new, no time or ran out of money. See this article by Michael Bolton for more information

I feel that ethnographic research stops because of similar reasons.

One interesting section I saw within the wiki article was about the process of ethnographic research in which to aid the researcher areas were split and the research asked questions.

Substantive Contribution: "Does the piece contribute to our understanding of social-life?"
Aesthetic Merit: "Does this piece succeed aesthetically?"
Reflexivity: "How did the author come to write this text…Is there adequate self-awareness and self-exposure for the reader to make judgements about the point of view?"
Impact: "Does this affect me? Emotionally? Intellectually?" Does it move me?
Expresses a Reality: "Does it seem 'true'—a credible account of a cultural, social, individual, or communal sense of the 'real'?"

I thought about this and started to change the context to be about software testing:

Substantive Contribution: "Does the testing carried out contribute to our understanding of the software?"
Aesthetic Merit: "Does the software succeed aesthetically?" Is it suitable for the end user?
Reflexivity: "How did the author come to write this test…Is there adequate self-awareness and self-exposure for the reader to make judgements about the point of view?"
Impact: "Does this affect me? Emotionally? Intellectually?" Does it move me?
Expresses a Reality: "Does it seem 'true'—a credible account of a requirement'?"

By doing this I found I suddenly had a set of heuristics to measure against the software testing that has been carried out, yet again more similarities between the two crafts.

Another area in which ethnographic research can be useful to software testing is when you need to test software that has a lot of UI interactions. Using the methods of ethnography a tester could go visit the users and observe and participate in their daily routine to find out the common tasks carried out and what oddities are seen. The oddities are the things of greatest interest since these are the things that would not normally be planned for and without active participation with the users would normally not be uncovered until it is too late.

There are many studies being carried out to determine if ethnographic research should be used when designing software system, however my concern with this is that it appears to be stuck in the design up front way of working which is not a flexible iterative approach, in my view it is easier, quicker and cheaper to ensure that testers use ethnographic methods when testing to ensure the design is suitable for users or even better get the users involved earlier and observe them earlier.

The more I have delved into the study of ethnography the more and more I have seen similar patterns to software testing. This makes me aware that software testing is not solely a hard science but a craft that encompasses many disciplines outside of the typical number crunching and algorithm creating world of software development.

Within the testing profession we need to look outside of the box and find approaches, methods, structures that can improve the discipline. To ensure our craft grows we need to ensure we do not narrow out field of vision or thought.

Remember you’re a Tester

2011-01-14T01:59:00.000-08:00

I want you to remember one word in the following list:

Bug
Insect
Ant
Dragon Fly
Ladybird
Crane Fly
Beetle
Bee
Wasp
Hornet
Cockroach
Earwigs
Termite
Grasshopper
Flea
Mosquito

My previous post was about debrief and how important it is to testing.

The problem I have come across during debrief has been trying to remember all the things that happened during the day or during the session(s). Maybe this is a, me getting older thing, and my memory is going.

One thing whilst I was reading recently about cognitive bias seemed to be a bias that could be helpful both during testing and during the debrief sessions. This was called the Von Restorff Effect and basically it was how our brains remember things that stand out.

http://changingminds.org/explanations/memory/von_restorff.htm

The above link uses an example of a list of words in which one word is in a different colour, our brains are more likely to remember the word that is a different colour and stands out.

You might be asking what connection does this have to testing.

Michael Bolton via twitter pointed me towards Adam White who has an interest in the Von Restorff effect. In his blog Adam states the following:

I use the Von Restorff effect in testing all the time. I frequently notice what doesn’t belong and it tends to be what I remember the most. http://www.adamkwhite.com/2007/09/30/using-heuristics-to-cook/

Part of our skills as a tester is noticing:

When something does not appear to fit – we notice
When something appears out of place – we notice
When something appears not quite right – we notice.

Could it be that testers have a strong Von Restorff cognitive bias? Maybe this is the missing ‘thing’ that people say testers have. You can not describe it but you just know it is a skill you have.

Going back to this article…..

How can this help during debrief?

My thoughts on this are that to remember something that is important for the debrief when we are working within SBTM. We should make a note of what it is and ensure we highlight it in a different way to make it stand out and ensure that we remember it later. Maybe some people already do this (the use of the highlighter pen).

Maybe the excellent tool for recording sessions, Rapid Reporter by Shmuel Gershon be expanded. Can it have an option to highlight certain things to make them stand out. I know it can do rtf and bold but that is not enough for me. I need highlighting and colouring, plus an option to do freehand doodles.

Why doodles you may ask?

One of the side effects of the Von Restorff effects is that we remember words better if associated with a picture. If I need to remember a URL is not working I would doddle a chain with a link missing. Or an interface that is failing to communicate I could draw a face with a plaster over the mouth. Just little things that help me remember problems that occurred. By the way I am rubbish at drawing not on the same level as the cartoon tester.

To conclude this article I think as testers we already have a cognitive bias to remembering things that stand out but within testers it appears we notice these things a lot more, either via our continuing training or a natural skill we possess. We need to ensure that important things we need to remember for debriefs are made to stand out during our testing sessions to ensure we do not forget them.

Which word did you remember from the list at the beginning?

Was it termite?

The Feedback Loop

2011-01-11T07:39:00.000-08:00

One of the critical elements of following the session based test management (http://www.satisfice.com/sbtm/) approach is the use of quick feedback. To achieve this it is suggested that a debrief should be done at the end of each session/day. Jon Bach (http://www.satisfice.com/articles/sbtm.pdf) suggest the use of PROOF

Past. What happened during the session?
Results. What was achieved during the session?
Obstacles. What got in the way of good testing?
Outlook. What still needs to be done?
Feelings. How does the tester feel about all this?

This approach is excellent for communicating what has happened during the testing session(s), however I keep hearing that people are not doing the debrief . There are many reasons why these are not being done, lack of time/resource or see no benefit are a few of the reasons given. This blog post is why it is important to carry out these debriefs and ensure they are done sooner rather than later.

I am looking at this from a psychology viewpoint to highlight the way our minds work and to keep reminding readers that software testing is a human sapient process and not an automated ticking of boxes process.

There are various studies that have indicated that the longer you take to act upon information the less you are able to recall that same information at a later date. During Eurostar 2010 Graham Freebur stated that unless you act upon information you had digested at the conference then within 72 hours that information would start to be lost and fade. The crucial part of this is that as humans we are fallible and lots of different psychological biases start to play with our minds so unless we can talk and pass on the information we have as soon as possible the more likely that the data we have will become clouded.

It is important that we debrief to someone to ensure that any error in our interpretation of the system under test can be corrected. The reasoning behind this is when we are testing a complex system we make assumptions as we test and the system may appear to confirm our assumptions and as such fuel what could be incorrect interpretations of the system. A computer system will never be able to inform you that your assumptions are wrong or right it could indicate a bias one way or another. The only way to repair errors in interpretations is to interact with a human being. This is the reasoning why debrief is very important so that any assumptions can be challenged and if necessary corrected.

As humans we are very good at being adaptive and changing our viewpoint and opinion when presented with new information but to do this effectively it needs to be a conversational setting, we are very bad at dealing with delayed feedback and the longer it is left the more likely we will keep our initial bias and interpretations.

The point of this rather short blog post is to explain why debrief after a testing session is important and that it needs to be done as soon as possible. Delays and excuses only cause more assumptions and incorrect information to appear to be the correct answer.

Make the time to debrief, plan for it and use it, it is crucial element of testing.

Autistic Software

2011-01-05T06:50:00.000-08:00

When we start to test a system as testers we normally start with a lot of assumptions predicting how the software will act dependant on certain inputs. However how the software reacts will be interpreted differently depending on who is doing the testing. This is especially true when testing UIs or systems that depend on human interaction.

I recently read an * article by Danah Boyd in which they described a problem in the way in which software is developed without much thought being placed upon what the user needs or practices. This article was first published in 2004 and I still feel it is very relevant to software development today.
*(http://www.danah.org/papers/AutisticSocialSoftware.pdf)

More and more software is being developed which requires interactions with humans and of course as we as software testers know humans are fallible and unpredictable. The problem with developing software is that it is very simple and easy to do the complex mathematical formula stuff since this is what a computer is good at, in reality a glorified calculator. As testers finding problems in these logical areas should be simplistic. This is not to disrespect the skill of testing, but knowing the formula used and without any UIs it is a fairly easy task for a skilled tester to determine if a problem exists. IMO the problems start to appear when we develop software that requires interaction with humans. I am not aware of any research done in this field of software development but I wonder how much time is spent creating and fixing UI systems. If we look at the characteristics for a given piece of software we can see that it is very good with numbers but has poor social interaction skills. This is very similar to the characteristics of autism:

• "Socialising doesn't come naturally - we have to learn it."

People with autism often have difficulty recognising or understanding other people's emotions and feelings, and expressing their own, which can make it more difficult for them to fit in socially. They may:

• not understand the unwritten social rules which most of us pick up without thinking: they may stand too close to another person for example, or start an inappropriate subject of conversation
• appear to be insensitive because they have not recognised how someone else is feeling
• prefer to spend time alone rather than seeking out the company of other people
• not seek comfort from other people
• appear to behave 'strangely' or inappropriately, as it is not always easy for them to express feelings, emotions or needs.
• Difficulties with social interaction can mean that people with autism find it hard to form friendships: some may want to interact with other people and make friends, but may be unsure how to go about this.
(http://www.autismsussex.org.uk/training/WhatIsAutism/characteristics_of_autism)

So as testers what can we do to try and improve the poor human interaction skills of software? Are there any approaches that can help this? Another area I have been researching is Ethnography (http://en.wikipedia.org/wiki/Ethnography) and how similar this is to software testing. (More on this in a future blog)

One of the many things I liked in the article by Danah was how the software was being designed to encourage people to have different identities so that they could protect their identity. This is so much against human nature and Danah expressed their concerns with the following:

Why on earth should we encourage people to perform a mental disorder in the digital world?

Quite a statement!!!

I think from a testing perspective that we need to ensure that software being developed meets the practices and needs of the user rather than the technology needs of the software. We really should be trying to make the software more socialable and be able to interact with a variety of different human types. We should look to understand the user and what their needs are rather than force them to work in a particular way. We should look at what is most suitable for the user. If we can manage to do this then we can start to produce and release software that does not frustrate and annoy users. This maybe a Utopia belief but as testers I feel we are the ones that need to drive this way of thinking. So the next time you are testing a UI and it starts to frustrate you ask yourself why. Do not put up with autistic software, teach it to become more socially aware.

Sour Grapes

2010-12-14T01:02:00.000-08:00

People who follow my blog may already know that I have an interest in psychology and how it can impact our thinking in relation to testing. Previously I have written articles on emotions , feelings and confirmation bias. I intend within this article to look at Cognitive Dissonance and how much of a help or a hindrance it can be for a tester.

There have been many studies on Cognitive Dissonance and if it really does exist however it would be best to describe what it is first. Wikipedia describes in it very simple terms:

Cognitive dissonance is an uncomfortable feeling caused by holding conflicting ideas simultaneously.

(http://en.wikipedia.org/wiki/Cognitive_dissonance)

The history of Cognitive Dissonance is accredited to Aesop’s tale of the Fox and the Grapes where the fox who wants to eat the grapes cannot reach them and as such to remedy the conflict that he really does want the grapes decides that the grapes are either not ripe or too sour, hence the title of this blog.

My thoughts on this conflict of beliefs or opinions and how we adjust to resolve the conflict is something that I feel could benefit testers especially when carrying out exploratory testing.

As testers I think it is important we recognise when we are experiencing Cognitive Dissonance and learn to ask questions before we make a decision to resolve the conflict. The problem with experiencing cognitive dissonance is that it is very easy to change our opinion and belief and suddenly we are drawn into the trap of confirmation bias.

As humans we do not like the feeling of conflict within our minds and we try to make a decision to resolve this conflict. Once we have made that decision we will try to justify the reasoning for making the choice we did.

For example if you had two areas to test and each area had the same level of important and you choose area A instead of B. You will now subconsciously favour choice A more than choice B. Now if someone comes up to you and says Area B is more important you have a conflict and your mind needs to give reasons as to why you choose A. You may make statements such as well at least Area A has been tested, it was important for me to test it. You will try to justify the reasoning for making what appears in your mind is the wrong choice.

When testing if we get two conflicting oracles and you need to make a decision on what is correct or the product owner needs to make a decision, cognitive dissonance could come into play. A decision could be made which is wrong. Only the problem now comes at a later date when you need to justify why you made that decision you will change your opinion or belief to make sure that the decision you made was the correct one.

It can become worse if you use a rating system to rate identical items. Imagine you are in a team and the team is given the task to rate features to be tested or developed. These ratings are then used to determine the order in which features for the product are developed. You then make a choice to test/develop your most highly rated feature, at a later date you get to rate the list again the item that you rated as next best has suddenly become of low importance even though it has the same value as the previous highly rated item. Making a choice after rating affects your value/belief/opinion of the item of same identical value and you start to score it lower to stop your uneasy feeling or cognitive dissonance. Project manager and team leads need to be aware of this since something of high importance could be downgraded by a team because of the issues of cognitive dissonance.

So how else can this affect testing?

I believe it can actually harm what we are testing. If the belief we have for the software we are testing in what it should be doing contradicts what it is doing and we adjust the conflict to justify what the software is doing there is a danger we could miss an important bug.

Further reading:
http://tip.psychology.org/festinge.html
http://www.colorado.edu/communication/meta-discourses/Theory/dissonance/
http://arstechnica.com/science/news/2010/12/this-is-your-brain-undergoing-cognitive-dissonance.ars

___________________________________________________________________________________

This is going to be my last post before the holiday season so I would like to say to all readers old and new have a happy holiday and look forward to writing again in the New Year.

Sorting the chaff from the wheat

2010-12-09T07:55:00.000-08:00

One of the questions posed at Eurostar 2010 hot topics panel session (link) was:

What’s the most important skill for a tester?

Michael Bolton (Link) gave a witty reply of

Recognizing there is no most important skill.

Whilst I agree with Michael that testers need a wide variety of skills it got me thinking about skills that all testers need or should have. One of these is being able to deal with the vast amount of data that everyone has to deal with. How do we deal with?

I have recently been re-reading the classic HG Wells story “The Time Machine (Link) and started to think about the two communities with the story, the Eloi and the Morlocks and how society in general is getting so much information that it is starting to make us dumb, this IMO is a dangerous thing for testers.

Are we as a society becoming like the Eloi?

They have access to amazing technology that helps with all their needs however they come across in the book as dumb and lack curiosity. They see no need to be thinkers or philosophers. It appears technology is making things easier and easier for us.

When I was young (in the very old days) to find anything out I used to have to read a book. Since books were an expensive item I used to have a list of books I wanted for my birthday or Christmas and I used to visit the local library every week to sit and read and learn new fantastic things. I would get lost in a world of fantasy and knowledge; even then I had a thirst for learning which fortunately has never left me. It is such a shame that local libraries all over the world are shutting due to technology. (Do a Google search for News and Libraries and Closing)

Now information is available at the click of a button. We can find information on how an airplane works or the theory of relativity in an instance. Technology has made all this information easier to get, however IMO it has made us think less.

Do we still question all this information?

As testers we know we should be questioning everything, we learn to sort the chaff from the wheat as the title of this blog post implies. (Link). With such a wealth of information that is so easy to gather the skill is to be able to collate this information and remove the distractions. How easy as testers do we find this? I find it a natural thing I appear to do without thinking until I started to write this blog article. My concern is that with so much information do I end up throwing away something that is later proves to be vital or important.

Does anyone out there in the testing community have a method they use to help with this?

How do we not forget everything?

This then leads on to the topic of self learning – how do you select what to read and what not to read? How do you ensure you do not miss a really important article that has been blogged?

One approach I use for self learning is to use twitter and the software testing club (Link) within these communities’ people we talk about blogs they have read or recommend to read, the power of the crowd. This helps to reduce the amount of information I have to process. Another approach is to actually talk to people; humans are wired to be better at absorbing information via speech than from the written word, it is more likely to be remembered.

My other concern with all this information is our ability to remain focused, another important testing skill. With so much information to digest it is so easy to get into the habit of just scanning the information and not reading the whole article (I wonder how many people will get to this part of the article?). It is so easy to just start an article and get distracted by some other piece of information and not return to the original article. I sometimes think I should not add any hyperlinks to my articles and just add them to the end but I want to credit the people who inspire me or provide me with information as I write the article, it is one of those things which is important to me. A perfect example of this was the recently article Michael Bolton wrote about estimation (Link) which was a five part article, how many people read the whole of the article? It is so easy to skip or scan and miss an important point within an article and the same can be applied to testing. If we scan and miss something it could be that the thing we missed will cost us a lot of money.

My other concern is that we are becoming a society of 24x7 learners, we never switch off.

Are you one of these people?

My concern on this came from a conversation in which I stated I do hobby as a job, this scared me. I have a passion for testing and learning but am I not in danger of burning myself out or forgetting valuable knowledge unless I switch off?

How many others reading this blog switch off and pursue other interests outside of technology?

If you take nothing from this article please do switch off. I have hobbies that have nothing to do with computers. I enjoy being creative and take photographs, spending time outside at stupid o’clock catching sunsets and sunrises and landscapes. I enjoy growing things and spend time in my garden. I am fortunate to have a very large garden in which to grow and nurture things. I also have a family and I am a grand parent and spending time with my granddaughter is such a wonderful thing to do. We call her our little time waster – since time can go so quickly when you are engaged in playing. After all these hobbies it is surprising that I have time to do my job or to keep learning but I come back to my work more energized and ready to learn more.

Since starting to write this article I have found a couple of other blogs that mention the problems of attention span and remaining focused they can be found here:

http://cultureandcommunication.org/f09/tdm/sara-hardwick/attention-span-in-the-internet-age-information-overload-memory-and-teal-deers/

http://www.theatlantic.com/magazine/archive/2008/07/is-google-making-us-stupid/6868/
I do recommend people reading them.

PS I will leave it to your imagination who I think the Morlocks are :o)

The Human Element

2010-12-03T10:43:00.000-08:00

I attended the Eurostar Testing Conference (http://www.eurostarconferences.com/conferences/2010/) in Copenhagen, Denmark this year and met a large group of very interesting people. A few highlights for me were:

Meeting the Cartoon tester (http://cartoontester.blogspot.com/) in person, a friendly unassuming guy with a quick sense of humour.

The other highlight was amount of ‘real life’ examples of exploratory testing and session based testing management. One of the best things I took away was from Carsten Feilberg’s talk on Session-Based Testing in Practice (http://carstenfeilberg.blogspot.com/) in which he reframed the wording of SBTM to Managing Testing Based upon Sessions. It was a why did we not think of that before!!!!

One of the keynotes was by Stewart Reid on “When Passion Obscures The Facts: The Case for Evidence-Based Testing” in which he looked at what testing could learn from Evidence Based Medicine (http://en.wikipedia.org/wiki/Evidence-based_medicine) . During the presentation I thought I could see many flaws in the argument he was trying to put together but could not quite work out what it was. One thing I have found out since and one point that Stewart did appear to miss was the work of the GRADE Working Group which is a newer system (and appearing to gain ground). The principles here are based upon Extrapolations (http://en.wikipedia.org/wiki/Extrapolation).

To quote from Wikipedia:

Extrapolations" are where data is used in a situation which has potentially clinically important differences than the original study situation. Thus, the quality of evidence to support a clinical decision is a combination of the quality of research data and the clinical 'directness' of the data.

Interestingly the data gather for extrapolation are more based upon human experience rather than just a set of numbers. Is it just me or is this like running a set of known tests then exploring afterwards? See my previous post on Hybrid testing (http://steveo1967.blogspot.com/2010/09/hybrid-testing.html)

So why have I called the title of this blog post “The Human Element”?
I was having a conversation with my wife(Tracy) after the conference since she is a retired theatre nurse and understand the medical arena very well and she came up with a wonderful phrase. It is all well and good having all these numbers and statistics but you cannot ignore the human element. She gave an example of this in which a nurse working in Intensive Care has a lot of machinery (with installed software) at her disposal however none of this equipment can tell her if the patient is feeling happy or sad or is uncomfortable.

Tracy said the problem is no machines have a soul they do not care how the patient is feeling, the machine could be saying everything is ok but the nurse and their compassion knows and understands how the patient is. I asked my wife to have a talk with Stewart and some other testers including Lynn Mckee (http://www.qualityperspectives.ca/)

This provided a wonderful insight to me in that we as testers forget that there are lots of people who we should be using as oracles for when we test a system we should not be forgetting about the human element.

During her conversation Stewart started to mention the use of statistics as evidence and for making healthcare decisions (Cochrane Library - http://www.thecochranelibrary.com/view/0/index.html?s_cid=citation) and Tracy said that to get to the point of making a decision still requires the GP to ask questions and to explore all possibilities. At the end of the day it is just statistics said Tracy and it does not help in a situation in which a perfectly healthy 20 year old is prescribed a drug for a problem and then dies due to an undetected heart problem. No amount of journals, evidence can account for this, since it is on a personal level between the patient and the medical expert.

The final conversation I remember Tracy having was with Lynn and a few other and it is very useful for testers when they come up against the problem of ‘It should do this.’
Tracy talked about Dr Spock and the book about the development of children (http://en.wikipedia.org/wiki/Benjamin_Spock) and that at certain age’s children should be doing this and that. This book causes major worries in parents when their child does not meet the timescales within the book for talking, sitting up, walking etc. Tracy then made a point which caused a great amount of laughter. “People seem to forget that babies have not read the book – they will develop at their own pace”

I found this a wonderful piece of insight, we seem to forget that everyone is different and if we apply this to software and the development of software we start to realise that every piece of software is different and that we need to explore the software and play with it to get the full potential out of the software.

To conclude this post I would like to say a big thank you to my wife Tracy for her encouragement and support in what I do and for giving us testers a lesson in remembering about the human element in what we do.

STC: A tester is for life, not just for Christmas

2010-11-09T06:46:00.000-08:00

I have completed my form have you?

The Software Testing Club will be creating an ebook to help raise money for Oxfam this Christmas.

It needs your input as a tester – so if you have not yet completed the form go to the website and do so.

You can help by filling in this form, promoting the book and/or donating.

Then you can make a donation to OXFAM and promote this worthwhile cause by any communication means you have at your disposal.

For example
twitter,
your blog
email,
telephone,
over drinks at the bar,
Whilst whispering sweet nothings to your sweetheart.
(oops strike the last one – that might get you in trouble)

but I hope you get the message the more people that know about this the better.

Exploring the World

2010-11-02T02:29:00.000-07:00

I have noticed that I have been a little remiss with my blog recently, this has been due to a combination of different things such as workload, home life and not having a great amount to say, which is fairly unusual for me. I don’t want to blog for the sake of blogging I want to blog when I feel I have something to say about the testing world.

I will soon be on my travels again to talk about exploratory testing and testing skills this time in Israel as part of an internal company workshop. I find it interesting that again I will adjust my material to match my audience on a cultural level, see my previous blog about training in India (http://steveo1967.blogspot.com/2010/06/training-in-india.html).

I wonder how many of us do this and how many of us just keep the same material and just recycle it regards of the audience?

This brings me to the point of this blog, if we treat software as different cultures and we try to explore and communicate with these cultures in the exact same way each time without making adjustments for the cultural differences.

Are we going to get to know anything about this culture?
What will we learn?
Will this culture give us any useful information back?

If we compare this the approach I use when presenting you can see that I learn about the culture. I am exploring by communicating with it and finding out about all the subtle differences there are. I try to avoid the traps and fopars that can cause offence by being ignorant of the culture. I consult oracles that have knowledge of the culture; I use heuristics when presenting to test the reaction of the audience.

Does it respond well to what I am saying?
Is it losing interest?

I then adapt my presentation on the fly to try to re-engage with the audience.
I am using the exploratory approach when presenting and I do this naturally. I very much doubt that most people who read this blog do not change their material, communication methods and approach when working with different cultures.

So why do we as a testing profession still insist that we can test software with scripts that do not change or adapt to the slight differences in culture? Yes there is an argument that says something’s do not change regardless of the culture and that is true. However if you go to a different culture and you are ignorant about their values and beliefs and you are unwilling to learn then you will leave that culture none the wise or richer in experience and understanding.

Do not be ignorant about exploring software, yes you can use the same techniques and methods you have gathered over the years to explore the software but do not fall in to the trap of following things by rote.

Hopefully there will be a few events coming up soon in which I can get some more topics to blog about.

If anyone is interested I will be at Eurostar (http://www.eurostarconferences.com/) this year in Copenhagen Denmark and it would be nice to meet up with like minded people and hopefully have some great discussions over beer of course.

Hybrid Testing

2010-09-27T05:58:00.000-07:00

There has been a lot of talk within the testing community about the scripted v non-scripted approach to testing. I have read and heard from people aligned to each school of thought trying to debunk the other schools approach. This can be very confusing to those that work in the profession of software testing. There are arguments on either side with people on either side presenting their point of view. I thought I would blog about my experiences of using both approaches in my day to day testing job.

When I first started in testing I worked for many companies which had adopted the Prince 2 methodology of software development and loosely followed a V-model process. This meant that requirements and specifications were gathered before any development work started. Using these documents as a tester I would do some gap analysis from a testing perspective to see where requirements contradicted each other and design specifications did not meet requirements. These were very heavyweight documents and it was a laborious task that engaged me as a tester to a certain point. Using these documents I would start to create scripted tests and build up a repository of test suites. Once the software started to be developed and I could gain access to certain features my engagement as a tester increased. I would run through my scripted tests and find that a large amount of them would need altering since I had made the wrong assumptions or the requirements and specification did not meet what was delivered. As I ‘explored’ the software I found more and more test ideas which would become test cases. The amount of discussions I had with senior management on why the number of test cases was increasing is another story altogether. I would spend a large amount of time adding detailed steps to the test scripts and then when we had another drop of the software run them again as a regression pack. I tried to automate the tests which for some easy parts worked and for others did not. What I did not know at the time was I was carrying out exploratory testing without knowing I was doing so. Once I had the software it was the most engaging time as a tester, it was what made me feel like I had done a good job by the end of the day.

So let us jump forward to today: - we have TDD, agile and a multitude of different approaches to software development. It is all about being flexible and developing software the customer needs quickly and efficiently and being able to adapt quickly when customer needs change. As testers we get to see and explore the software a lot sooner.

A lot has changed from a tester perspective we are now engaged more in the whole process, we are expected to have some knowledge of coding, IMO not always necessary but a good tool to have. We get to see the software a lot sooner and able to exercise and explore the software and to engage our testing minds to what the software should, could or may do. However have things changed that radically?

What has made me think about writing this blog has been the debates that have been going on about scripted vs. non-scripted. I am currently working on a new project in which there are many dependencies on internal components and external 3rd parties all of which are working to different timescales. Some of the components can be simulated which others cannot due to time constraints and other technical problems. We have some pretty good requirement documents and some design specifications. What we do not have at the moment is fully working end to end software. So I am back creating scripted test cases to meet the requirements, finding discrepancies in the documents and asking questions. The difference is that now I do not fully step out my scripts I create pointers on how to test the feature, I note test ideas that could be interesting to look at when the software arrives, I make a note of any dependencies that the software would require before testing that feature. So I create a story about testing the feature rather than create a step by step set of instructions. It is more a testing checklist rather than a test script. So with this I am combining both scripted and the non-scripted approach. I am sure a lot of readers will read this and think that they are doing the same.

The people who talk about exploratory testing have never said to my recollection that there is no need for scripted tests. Some of the requirements I have are fixed they will not change; the results should be as stated; so those requirements I can script or automate if possible. It does not mean you are doing anything wrong nor does it mean that you are not following an exploratory approach. Exploratory testing is just that an approach, it is not a method, it is not a do this and nothing else. It is a tool to use to enable you to test and hopefully engage you in testing rather than just being a checking robot. If you still create detailed step by step scripts then there is nothing wrong in doing that, I still do when required.

Exploratory testing can be used without the software, you can look at available documents and explore them for test ideas and new creative ways to test what the documents are stating, you can question, you can analysis you can make suggestions and improvements, you can use your brain.

We test the system and gather

2010-09-08T00:38:00.000-07:00

I have noticed that it has been awhile since I did a blog due to family commitments and vacation time. I had an idea to blog about the importance of gathering evidence when testing especially when using an exploratory testing approach. I decided to take the example of why it is importance from an internal workshop that I run on exploratory testing.

So are you sitting comfortably?

Here is the story of Timmy the Tester and Elisa the Explorer

Timmy Tester

Timmy Tester has been given a new build to test and decides he is going to test it using the exploratory approach. Timmy writes that his mission statement is to test function x has been implemented.

He installs the release and starts to enter different values and pressing buttons around the new feature at random. He does this for about half an hour and then all of a sudden the application crashes.

Timmy takes a screen shot of the crash and a system dump of the crash and goes to talk to the developer. The first question the developer asks is

“Have you tried to reproduce the problem?”

At this point Timmy says no and goes back to try to reproduce the problem.

Two days later Timmy has been unable to reproduce the problem and now thinks it could have been one of those strange things.

3 months later the application is now live on the customer site. Within 30 minutes there are a large number of calls to support stating the application is crashing. The problem gets passed back to Timmy who notices that the problem appears to be the same as the one they saw when carrying out ‘exploratory’ testing….

Elisa the Explorer

Elisa has been given a new build to test and decides she is going to test it using the exploratory approach. Elisa creates a mission statement stating they are going to be testing the new function.

Elisa installs the new application and starts to enter different values around the new feature. As she is doing this Elisa has another computer in which she makes notes and takes screenshots at relevant points to aid clarity of each step that she has carried out. At certain points Elisa finds behaviour of the system which does not seem correct so she starts another mission statement to look into that behaviour. Elisa then starts to examine the strange behaviour in more detail making notes of the steps she is carrying out at the same time. All of a sudden when pressing a button the application crashes.

Elisa makes some notes, takes a screen shot and a system dump of the crash.

Elisa then resets the application back to a clean system and repeats the last set of steps which she had made a note of. The crash happens again.

Elisa then goes to see the developer and states she has managed to produce the problem more than once and here are the steps.

Elisa sits with the developer while they go through the steps together and the developer sees the crash.

Two days later Elisa has been given a fix for the crash. She now has an automated test for the crash and runs it straight away. The test passes and Elisa continues with the rest of her testing.

It may seem like common sense but I have seen more cases of Timmy than Elisa from people who have said they are using exploratory testing. It is extremely important to record everything and remember exploratory testing does not remove any of the principles of testing:

“All tests are repeatable”
“All problems are reproducible.”

There are many ways we can gather evidence of our testing sessions and there are a large amount of tools available to the exploratory tester. In much the same way that when the first humans decided to explore the North Pole they took tools with them that could help and support their efforts exploratory testers can do the same when exploring the system under test.

Maybe I should look at some of these tools and write a blog about them – or even better people who read this blog might be able to suggest some good tools that they have had experience of using.

The Certification Filter

2010-08-02T03:34:00.000-07:00

I had a few moments to myself the other day and decided to have a bit of fun and do some research about what agencies required when you apply for testing roles. I was surprised (or not) by the number of roles that stated “will be ISEB certificated”

How can they get it so wrong? The ISEB is now defunct it should be ISTQB (sic) so all those who hold an ISTQB qualification need not apply………

Anyway back to the point I am trying to make.

I did some quick research and found that within the last seven days of the 122 testing roles listed I would be excluded from 28 of them since they stated …will be ISEB certified. If you add in the ones that add ‘would prefer candidates with ISEB certification. You get to over half of the roles advertised that I could not apply for.

@Mheusser made the following tweet @steveo1967 - certification might just get you the job you don't want to have.

That maybe be true but I got thinking and maybe it is not the company using the agency that are mandating this but the agency adding its own filter. The company could be missing out on some great testers because of the agency filtering system.

The following are some imaginary conversations with an agency (Sadly I am sure some people may have already experienced this…)

************************************************************************************

Me: Hi I am interested in being put forward for the role of chief tester as per the advert you posted this morning on your website.

Agency: Sure, have you already sent us your CV, Resume?

Me: Yes

Agency: OK, what is your name, let us see if we can find your CV

Me: My name is Timmy Tester

Agency: Wow cool name sure matches your profession

Me: Yes (Sigh) I get that all the time..

Agency: OK Got your details – just looking at them now, wow 20 years in software testing, that is impressive, I see you have worked at some very well known companies. Oh wait a minute…. You do not list that you hold the ISEB qualification.

Me: No I do not.

Agency: I am sorry Timmy I cannot put forward your application – we only accept people who are ISEB qualified.

Me: Why? I have more than 20 years in software testing.

Agency: Yes I can see that, however the ISEB qualification means that you know how to test and is mandatory for any roles you apply for with us. Sorry but I cannot put you forward for this. I suggest you go and sit the exam and get back to us. Goodbye.

************************************************************************************

Me: Hi I am interested in being put forward for the role of chief tester as per the advert you posted this morning on your website.

Agency: Sure, have you already sent us your CV, Resume?

Me: Yes

Agency: OK, what is your name, let us see if we can find your CV

Me: My name is Ivor Certificate

Agency: OK Got your details – just looking at them now, you have been in testing for 8 years now, I see you also hold the ISEB qualification. That means you must know a lot about testing.

Me: No I do not.

Agency: Sorry, you said you do not?

Me: Yes I did say, no I do not know anything about testing

Agency: Then how come you have a ISEB Certification?

Me: Because I noticed that if I had this I could apply for any testing job.

Agency: So you must know something about testing?

Me: Well it is an interesting story. I paid to do the multi choice exam, sat down and completed it in 10 minutes by just randomly answering the questions. No one checks if I have any competence at testing, by luck I managed to get the pass mark required to get the certificate. Hence I am now classed as someone who must know about testing and how to test.

Agency: But it says on your CV that you have been testing for the last 8 years.

Me: Yes I have but I just go in and let others, who are not certified do all the work while I just copy what they have done and claim credit for it. So are you going to put me forward for the role?

Agency: Yes I will you meet the selection criteria, so I cannot see any problems in putting you forward.

Me: Thank You.

************************************************************************************

This may seem like a silly situation but I am sure it could happen in reality. I am not against qualification and people trying to improve themselves but when those qualifications are then used as a filter to exclude people from applying from jobs, it makes me see red.

There are some good examples of proving your ability as a good tester. You can talk to previous companies that you have worked for. They could interview you and talk to you about testing and your thoughts, problem solving ability. However this would take too much time.

I have not as yet found any roles that have stipulated that they require that you attended a Rapid Software Testing Course or The Black Box Software Testing course through the Association for Software Testing. Why is this so? Is it that these types of courses do not have the huge budget to promote themselves? Or that they try to be non profit making?

What is the solution to all of this?

I think within the testing community we need to start educating agencies and companies about how to sort out good from bad testers, how we go about this I am not sure.

Should we have a dedicated website that we can direct agencies to, to explain about certification? I feel this could be a good start and would need someone with far better web skills than myself to get running and also it would need to be unbiased as possible.

Then mail shot the CEOs at each agency when we hit this problem directing them to the website.

Do we try to do presentations at employment agency conferences?

I feel there is a need to educate agencies and companies that are looking to employ testers and give advice on how to spot the good from bad candidates but they need to get rid of the certification filter.

Another thought I had would it be a good idea to have a vetting service for testers and agencies? There could be a one stop service for agencies to verify testers, their abilities and obtain a list of people who would vouch for them.

I tried to think at the weekend if this could work or not. I have a lot of concerns about it being misused and ‘gamed’ by people who have a moral compass that is slightly off balance. How would it be funded? Would it become a monster of its own making? How would testers be vetted and vouched for? Would testers be vetted and vouched based upon their online presence? For example I am sure I could ask a few people online to vouch that I am a good tester however none of these people have worked with me and seen me carry out testing. I could be just saying the right thing at the right time to impress people – how would anyone know unless they have worked with me?

This really brings us back to the beginning of the article, agencies and companies need someway to vet testers and get some guarantee that they know about testing (regardless of which school of thought they follow). So using the certification method is an ideal way to sort out candidates quickly no matter how flawed the certification may be. Do I just bite the bullet and sit the exam if I do not want to be excluded for any testing job? Does anyone have any other methods that agencies or companies can use when they are looking for skilled testers?

Reponse to How many test cases by James Christie

2010-07-28T06:31:00.000-07:00

James Christie wrote a great blog about his concern on using test cases to measure testing

http://clarotesting.wordpress.com/2010/07/21/but-how-many-test-cases/

and several people blogged a response

Simon Morley added his view here: http://testers-headache.blogspot.com/2010/07/test-case-counting-reflections.html

Jeroen Rosink added his here: http://testconsultant.blogspot.com/2010/07/repsonse-on-how-many-test-cases-by.html

and Abe Heward directed people to a similar blog he had wrote earlier: http://www.abeheward.com/?p=1

Each of these blogs makes very valid points about how non-useful measuring test cases are for indicating the progress or coverage of testing effort.

The aim of this blog is to try and expand upon these posts and see if there are ways in which we could measure testing effort and progress within resorting to using numbers.

To start with we shall take a look at a made up situation.

You are testing on a project which is has a two week testing cycle, your manager has requested that you need to report each day the following:

How many test cases you have
How many have been run
How many have passed
How many have failed.

(Does this seem familiar to anyone?)

So before you start testing you report to your manager that you have 100 test cases to run over the two week cycle

At the end of day one you report the following

Test cases ran: 60
Test cases pass: 59
Test cases fail: 1
Defects raised: 1
Test cases still to run: 40

So management think cool we are ahead with testing, 60% done in one day.

At the end of day 2 you report:

Test cases ran: 62
Test cases pass: 61
Test cases fail: 1
Defects raised: 1
Test cases still to run: 138

Management now thinks how come you only ran two test cases today, why are you running slowly? WHAT!!!! Where did those other 100 test cases come from? Did you not do your job correctly to begin with?

However the two you ran today had lots of dependencies and very complex scripts.

Plus your testers noticed that there appeared to be new features that had not been documented or reported, you have now had to add another 100 test cases. Also your testers actually think when they are testing and thought of new edge cases and ways to test the product whilst they were testing.

Management starts to panic – you reported on day one that 60% of testing had been completed. Now you are saying only 30% of the testing has been completed, Stakeholders are not going to happy when we report that we have only covered 30% when the day before I reported to them that 60% had been completed.

This continues, your testing team are really good testers and find more and test ideas which are turned into test cases. So at the end of day seven you report the following:

Test cases ran: 1200
Test cases pass: 1109
Test cases fail: 91
Defects raised: 99
Test cases still to run: 10000

So at the end of the first week you have only completed 8% of all the test cases. You get fired for incompetence and the project never gets released.

Many people reading this may have experienced something similar to the above, what worries me that there are still people stating the best way to measuring testing is by the use of test cases!

The question now is that if measuring by the use of test cases is not a good way to measure then what can we do?

The following suggestions are my own and what I apply within my test approach, it does not mean it will work for everyone nor am I saying it is the best approach to take. However the purpose of my blog is to offer suggestions about testing that could be useful to some people.

I work in the following testing environment:

Agile based – 2 week iterations
Customer changing requirements frequently
Code delivered daily
Functions and features added without supporting documentation
Use a mixture of scripted and exploratory testing

If I tried to report the testing effort using the traditional test case scenario it would be of little (or zero) value, since the test case number would be constantly changing.

What we do is split functions, features etc into test charters, as per the exploratory testing approach, these ‘Test Charters’ are known as the test focus areas of the software. If a new function or feature is discovered a new charter is created.

We then use the Session Based Test Management approach (James and Jon Bach - http://www.satisfice.com/sbtm/) and implement sessions based upon mission statements and test ideas. During the testing session the testers are encouraged to come up with new test ideas or new areas to test, these are captured either during the session or during debrief.

The reporting of progress is done at the test charter (test focus area) level. The test manager reports in the following way.

Test focus area 1: -Testing has started – there are a few issues in this area:

Description of Issue x, issue y, issue z.
Which need to be resolved before there is conference that is area is fit for its purpose.

Test focus area 2 – has been tested and is fit for it purpose

Test focus area 3 – test has started and some serious failures have been found defect 1, defect 2, defect 3

And so on.

Some people may ask but how will this tell us if we meet the deadline for testing? I am sure it will NOT tell you if you will finish ALL of your testing before the deadline since testing is an infinite thing, we as testers will carry on testing until we meet a stop heuristic (See Michael Bolton article on stopping heuristics: http://www.developsense.com/blog/2009/09/when-do-we-stop-test/).

The problem with testing is that it is not a yes or no when it comes to the question of have you completed your testing yet. Every time a skilled tester looks at the software they can come up with more and more test areas and test ideas that they could carry out. These may or may not add benefit to the suitability of the software and if it is fit for its purpose. What is required is a test manager that talks to and listens to their test team and see which test areas are the most important and MANAGE test sessions based upon what is critical – basically do some good old prioritizing. The test manger needs to ask the difficult questions of the stakeholder and project managers.

What features can you do without?
What are the critical areas that are required?
Function abc has many serious problems – it can cause problems x,y,z for your users. Do you need function abc?
We have tested all the key functions and found the following problems x,y,z. You want to release tomorrow, are you OK with these known issues?

In return the stakeholders and project managers must trust the test team and accept that when they report that an area has been ‘sufficiently’ tested they believe them.

To summarize – instead of reporting on a small area of testing such as test cases, move a couple of level ups and report on the progress for test areas/functions./features based upon the importance of the feature. This may not tell you if you will compete the testing before the deadline but it will show you how well the testing is progressing in each functional area at a level that stakeholders can relate to and understand. The trust your stakeholders will have in you should improve since you are giving them a story about the progress of the testing effort without trying to hide things using numbers.

DANGER - Confirmation Bias

2010-07-27T03:46:00.000-07:00

In my previous blog I touched upon a term called Confirmation Bias and how as testers we should be aware of this. I stated that I would put a blog together on the subject so here it is.

I should start by defining what confirmation bias is.

Confirmation bias refers to a type of selective thinking whereby one tends to notice and to look for what confirms one's beliefs, and to ignore, not look for, or undervalue the relevance of what contradicts one's beliefs:- http://www.skepdic.com/confirmbias.html

The reason I started to look more into confirmation bias was due to the following article in Ars Technica - http://arstechnica.com/science/news/2010/07/confirmation-bias-how-to-avoid-it.ars

A good example of this is if you are thinking of buying a new car and all of a sudden you seem to notice lots and lots of the model of the car you was thinking of purchasing. You mind is conditioning itself to notice this make and model of car and making you notice them more, even if there are no more than there was before – you appear to be seeing them everywhere.

Another example is if you start talking to a friend about a certain film and actor and then suddenly notice lots of coincidences, the actor is on a advert, the film is being shown again on TV, a support actor is in another film you just started to watch. The following gives a good example of this. http://youarenotsosmart.com/2010/06/23/confirmation-bias/

If there was no such thing as confirmation bias there would be no conspiracy theories. Conspiracy theories are based upon information which proves the theory correct; those who believe in the theory ignore the evidence that debunks that theory.

So why is there any concern for testers?

Let us start with an example.

You are working closely with the development team and you start to ask them questions about the release you are about to test. You ask their viewpoint on which areas they feel are the most risky and which they feel are the most – so you can adjust your priorities as required, a pretty standard exchange between developers and testers. You now start testing beginning with the area of high risk and work your way to the low risk areas.

You find a few serious bugs in the high risk areas (as expected) and you find no problems in the low risk areas.

After release a major bug is reported in the low risk area you tested. How did you miss the bug? Did you see the bug but your thinking was that everything was working alright? Did confirmation bias play a part? Did your subconscious hide the bug from you? Now this gets very scary, most people who work in software testing know that some bugs try to hide from you, we expect them to hide in the software. What happens if they decide to hide in your brain?

So how can we try and prevent confirmation bias?

The quick and easy way to try and prevent confirmation bias is to ensure that more than one tester tests the same feature, they may bring in their own confirmation bias but hopefully it will be different from the previous testers bias. There is more chance that it will be different if the testers have not discussed the area under test beforehand.

Another way to try and prevent confirmation bias is to do ‘paired testing’ either with a software engineer, another tester or a user. That way you can question each other with regards to what is true and what is false. There is a chance that you could cross contaminate each other with your own confirmation bias, but the risk should be less than if your are working on your own.

It is not easy to remove confirmation bias since it is infectious. The way of working on a software development project requires testers to communicate more and more with other areas of the business and at each stage and with each conversation confirmation bias could be introduced.

So should we lock ourselves away in a dark room with no communication with anyone else on the team? I think I would get out of testing as a career if that happened, the Social Tester (@Rob_Lambert) would now be the anti-social tester, time to get him a ASBO (For our non-UK readers - http://en.wikipedia.org/wiki/Anti-Social_Behaviour_Order)

My view is that there is no realistic way to prevent confirmation bias due to the way software development projects work and that there is a need for everyone to be able to communicate with each other. However if testers are aware that there is such a thing as confirmation bias then they can try and take steps to ensure it does not creep into their testers. That is the whole concept and point of this blog – to help to raise awareness of confirmation bias and how it can effect your testing.

The Emotional Tester (Part 2)

2010-07-19T02:54:00.000-07:00

The first part of this blog looked at how our emotions could affect how we test. This second part will look at how we could capture our feelings when testing and could this provide us with any useful information about the product we are testing. Could it prove to be a useful oracle when testing?

On twitter @testsidestory said the following:

That is done regularly in usability labs: capture emotions and facial expressions of the users as they use the s/w

This was in response to a question that I posted on twitter:

…. - what I am thinking is that we need to capture our mood when #testing it could indicate a problem in the s/w…

The concern with this is that it would be very expensive to implement for the majority of people. I thought how we could implement a system that could capture emotional state and be effective and inexpensive.

One idea I had was to use a concept from the book Blink by Malcolm Gladwell, in which Malcolm talks about how important our initial emotion/reaction is when we first encounter something. There is a discussion about how often our ‘gut reaction’ proves to be correct and he uses an example of a statue that a gallery had bought after a lot of scientific experts, who had tested the statue, had said the statue was genuine. A couple of art experts who got to see the statue before it was unveiled in private viewings had a ‘feeling; that there was something wrong about the statue, their initial gut reaction was telling them it was a fake. Several months latter it was discovered to be a fake.

The above is a concise retelling of the story within the book, however why did the scientific experts get it so wrong? Could it be that conformation bias played a part? The scientific experts wanted so much to believe that it was real and not fake they caused bias in the results or avoided obvious facts that pointed to it being a fake. I think confirmation bias is a great subject and one I will look at from a testing perspective sometime in the future.

So can we use this ‘gut reaction’ concept in testing?
Would it be of any value?

I should state that I have not tried any the following ideas and that if anyone would love to volunteer within their organizations to ‘trial’ the ideas out I would be most interested. Due to circumstances I currently do not have the ability to try this out on a large scale.

The first problem we face is how we capture out initial reaction to what we are testing. The requirements for this are that it is:

Easy to capture
Simple
Quick

My thought is to use different smiley’s which are simple and quick to create and capture thus covering all the requirements.

My idea would be to use three different smiley’s:

Happy
Neutral
Unhappy

Why use smiley’s?

The idea as to why use smiley’s is that anyone can draw them no matter how artistic and from the perspective of measurements it is very easy to recognize and see pasterns when using such well known symbols. The other longer term thought was that it is easy to extend to add sad, angry, and extremely happy if you wish to improve the range of emotions and feelings.

Capturing the initial feeling/emotion.

If you are working in an environment in which you are carrying out exploratory testing and following mission statements (Session based testing) then this is very simple to implement. The idea is that when the tester starts their mission (session) they should within the first couple of minutes (5 at a max) record their emotion/feeling of the software by the use of the smiley’s.

If this was done for every session being run and captured in such a way that it would be easy to see at a glance which areas (test charters) testers are unhappy with it could provide some useful information.

So you now have a whole set of data with regards to the testers initial feeling about the software there are testing, what does this information tell you?

For example a certain test focus area shows that all the testers are unhappy in that area would this indicate a problem? I feel it could indicate something wrong in that area but you would need to talk to the testers and gather more information (obtain context) I think the great thing about capturing initial feelings towards the software could help the development teams to focus on areas where there could be implied problems based upon initial feeling.

This approach could be taken a step further and get the testers to add another smiley when they have finished the session to see how they feel about the software after they have finished their session. You now have two sets of data and can compare any discrepancies with the two.

What would you think if the majority of testers were happy about a certain test focus area but at the end of the session they were unhappy?

Does this indicate a problem?

Or what if it was the opposite mostly unhappy and at end of session they were happy?

Also if they were unhappy at the beginning and at the end, their gut reaction proves to be correct, does this give an indicator that there are some major issues within that area?

Could this indicate frustration with the system, lack of knowledge maybe?

In my opinion this approach could provide to be a very useful oracle to the quality of the software.

What do think?

Could this prove to be useful?

I would love some feedback on this idea - good or bad.

The Emotional Tester (PART 1)

2010-07-16T03:39:00.000-07:00

This blog is going to be in two parts, the first will focus on the question of do emotions affect the quality of testing. The second will look at ways in which we can gather information about how we feel about the product we are testing to see if there is any value in capturing this information.

I have an amateur interest in psychology and how some of the ideas and thoughts from this area of science can be used in software testing. I was reading ‘The Psychology of Problem Solving’ by Janet E. Davidson & Robert J. Sternberg and it had a section on how emotions affect the way we think and focus.

So I decided to tweet a question based on some of the information I had read:

Emotions and #Testing:-Do we find more bugs when we are in a bad mood? Psychology research shows we are more negative when in bad mood.

It would be interesting to have feedback from #testing community on this - Does this mean a good tester has to be a grumpy so and so... :o)

It was not long before I started to receive replies on this.

@Rob_Lamber: @steveo1967 I don't really attribute negativity to being good at finding bugs. Positive attitude, passion, inclination...not negativity

@nitinpurswani I @steveo1967 i cannot think when i am in bad mood and i guess sapient testers think

@ruudcox @steveo1967 This article might help: Mood Literally Affects How We See World. http://www.medicinenet.com/script/main/art.asp?articlekey=100974

This turned in to a lively debate on which mood is better for testing.

After reading various articles there appeared to be some common ground on how we think and see things based upon our emotions and mood.

Looking at the article suggested by @ruudox this suggested that when in a good mood we can see the whole picture and when in an unhappy mood we narrow our focus.

This appears to be backed up by research from Foless & Schwarz

Individuals in a sad mood are likely to use a systematic, data driven bottom-up strategy of information processing, with considerable attention to detail In contrast, individuals in a happy mood are likely to rely on pre-existing general knowledge structures, using a top-down heuristic strategy of information processing, with less attention to detail (foless & Schwarz, 1999;).

This now leads to some complex dilemmas, and the whole point of this blog.

Which mood is best for someone whose is a professional tester?

Which mood is more than likely to find more bugs when testing?

What other influences can affect our ability to test well?

My thoughts indicate from the information and research I have read that to be really good at testing and finding defects we need to be in a sad or unhappy mood.

Research concludes that when in a sad or unhappy mood we are more than likely to focus in on the task and step though in a data driven way. When happy we are more than likely to see the whole of the picture and look at the task from a top down approach.

Now in my opinion both of these traits are needed to be excellent testers. So do testers need to have split personalities that they can switch on or off?

The point made by @nitinpurswani about being in a bad mood stops him thinking and that to be a sapient tester he needs to think. This got me thinking and I asked him a question back.

@nitinpurswani I like that idea. However if you're in a bad mood with what u are #testing would it make you want to break it more?

My thought behind this is that if something is annoying me or irritating me I feel I am more than likely work harder to find out why it is annoying me. I become deeply focused on the problem in front of me. Does this mean I am in a bad mood? Not necessarily so – it could be I am annoyed at what I am testing but not in a bad mood in general.

When in a happy mood when testing it is easy to just let things go, we unconsciously think well that is not too much of problem we can forget about it. This is a dangerous attitude to have as a tester because this simple little problem can come back to be huge problems. Someone in an unhappy mood is more than likely to investigate why this thing is annoying and find the underlining cause.

@Rob_Lambert made a very valid point that there are environmental issues that could come into play. How many testers when testing listen to music? Rob suggested that the type or style of music you are listening to can influence the mood you are in and as a side effect the way you are thinking. I had not thought about this very much but going deeper than this – if you are working in a open office and everyone around you is having a laugh and joking would this make your testing better or worse? What if a tester and a developer are having a heated debate about something that has just been tested? Will this influence your testing?

Does any of this article back up my earlier tweet that testers need to be grumpy so and sos?

However I think this view is too simplistic. I am often asked about testers and how they are different from developers. (There is still a big drive within testing that developer and tester can be the same person and be able to switch between the different roles). I have a feeling that some of the best testers can switch between different psychological emotional states when testing. They have the best of both worlds. Able to remain focused when something is bugging them and then when they have solved what is bugging them able to switch to a whole picture view of the system they are testing.

When I started to write this article I thought it would be very simple to come to a conclusion about how emotions can affect our ability to test and what is the best mood to be in to get the best out of testing. It has proven more difficult than I thought and I still have not come to any firm conclusion about which is the best.

The one interesting point that should be made is that as professional testers we need to be aware of our emotions and how they can affect the quality of the testing we are doing. Part 2 of this blog will be looking at how we can capture our emotion and feelings about the product we are testing and see if this could provide useful information.

Managing Exploratory Testing with Mercury Quality Center

2010-07-11T14:30:00.000-07:00

I thought I would write about my experiences of using Mercury Quality Center (MCQ) to help manage my exploratory testing sessions

When carrying out exploratory testing I use the James and Jon Bach approach of Session based testing (http://www.satisfice.com/sbtm/). What I found is that the tool provided did not match the needs of the company and was hard to sell to management since we already had commercial tools for capturing testing effort (MQC). I had to re-think how I could get buy-in from management on using the exploratory testing approach whilst making use of the tools we already had.

One of the first things I did was implement a structure within the test plan section of MCQ. So I defined the following folder structure for each project

Project Name

Test Charter

Mission Statement

Test Ideas

e.g.

Project Name -->

Test Charter -->

Mission Statement -->

Test ideas(s)

So under the planning section testers can define a folder name for the test charter they are working on and then add a folder for each mission statement and then add their test ideas.

The thinking behind this was at a glance anyone can see what has been covered under each test charter and see if their any gaps. Reports can be pulled off and used during debrief sections to act as focus points when discussing the testing that has been done.

I created a Test Plan Hierarchy using a standard numbering scheme for the folder and test idea names. This helped with traceability and navigation around the test plan.

e.g.

Project Name -

01 – Test Charter 01 -->

01.01 – Mission Statement 01

01.01.01 – Test Idea 01

01.01.02 – Test Idea 02

01.02 – Mission Statement 02

01.02.01 – Test Idea 01

01.02.02 – Test Idea 02

02 – Test Charter 02 -->

02.01 – Mission Statement 01

02.02.01 – Test Idea 01

02.02.02 – Test Idea 02

MQC is setup for a formal test case and test step scripted form of testing, I have not found a way to get around this however instead of test cases I use test ideas and needed a quick way to create new test ideas without being bogged down in writing details about lots of steps. So I suggested that each test idea has ONLY the following information:

Test Idea Name
Test Idea description (This should be as descriptive as possible – include any models/heuristic thinking/problem solving ideas)
A single test step - This is required by MQC so that the user can run the test and record its status (Pass/Fail etc)

Since we use a different system for capturing defects (Don’t ask!) I also added a folder to each project called 99- Defects – so that I could trap any defects that needed testing.

The next step was to have a structure for the test lab (this is where details of tests are run)

I implemented the following structure:

Project Name -->

Project Release Version X.Y -->

01.01 - Mission Statement 01 -->

01.01.01 - Test idea 001

01.01.02 - Test case 002

01.02 - Mission Statement 02 -->

01.02.01 - Test idea 001

01.02.02 - Test case 002

It is recommended that X.Y numbers in Project Release Version name are provided as a multiple digit left zero padded integers. This is to ease sorting by name. This was basically copied over from the test plan section.

For exploratory testing I suggested that as a minimum the following columns are included when recording the execution of the test idea.

Plan.Test Name
Result
Defect (For recording CQ defects raised within that test script)
Priority * (How important is this test idea , what risk is it to the project by not doing this test idea)
Status
Execution Date

Once this had been setup it was then easy to run a session based upon a mission statement for the session I was running. Each mission statement had multiple test ideas. I found this very useful since it was very quick to create test focus areas based upon test charter names and mission statements. These could then very simply be turned into session sheets within MQC test lab.

One of the key elements of session based testing is to capture what all the evidence of the exploratory testing session. I implemented the following to capture details of what went on the testing session. Each test idea was run from within MQC and recorded if that test idea passed or failed. (I am aware this can be very subjective and depends on context however to ease transient to ET it is necessary to have some familiar ways of recording progress). I ensured that all session notes, log captures, screen prints, videos etc were captured by attaching them to the test idea.

THIS was very IMPORTANT – since if anyone needs to follow your test idea in the future they now have a record of what and HOW you executed your test idea. This is an issue with biases here and people carrying out testing afterwards could just follow your notes and repeat what you did which is not really exploratory testing but that can be mitigated by mentoring.

You now have a tool in which you can capture what you have done during your exploratory testing sessions.

There are a few issues I find with MQC and I am sure people out there in the testing community may have the answers. I want to use MQC to record the time spent on each session (As short, medium, long). I also I wanted to capture how much as a percentage of that time was spent on:

Test execution
Bug reporting and investigation
Test environment set up
Test data setup

This would help in the telling of the story of what is stopping the testers actually testing. I am sure there is a way to do this is MQC and I just need to do some more investigation. I hope readers of this find it useful, I know it has helped me to persuaded management to take exploratory testing seriously.

To finish this is working for me, it is not perfect and I am investigating other ways/tools that can make this more efficient. Looking at using a java application to create the session sheets and report back via the MQC API directly – but that is in the future. I am also investigating ways to customize MQC so that I can have the columns I wish to have. I will let you know it that works.

Testers are the bearers of Bad News

2010-06-25T01:06:00.000-07:00

I read an interesting blog by James Christine yesterday (24-06-2010) (http://clarotesting.wordpress.com/2010/06/23/challenging-the-culture/) in which organizations which promote a positive/good news culture could be doing themselves harm by trying to encourage people not to report any bad news and how dangerous this is. I loved the alternative take on this and it got me thinking about a blog post I intended to do about how testers are perceived as the bearers of bad news and how we could change this perception. The article by James has spurred me to put the blog together.

I have an amateur interest in psychology and human behaviour and as such I am fascinated by reading articles in which as a person we can learn to adjust our outward persona to help benefit ourselves and those around us. For example Beth Lane wrote an interesting article on perception checking: http://improving-relationships.suite101.com/article.cfm/improve_your_relationships

From such a small article I learnt a lot about myself and how others may perceive me. I use many methods to ensure I can communicate with others to the best of my ability. I apply this to my job as a software tester (note homage to Michael Bolton here not a QA - http://www.developsense.com/blog/2010/05/testers-get-out-of-the-quality-assurance-business/)

I try to have a good working relationship with software engineers/developers/programmers (still struggling with what these highly talented people want to be known as) since most of the time I go and speak to them it is to say something is not working or it has crashed.

I had an eureka moment one day and took a step back to see why the relationship between the testing team and the software development teams were fragile and highly strained. I put myself in the shoes of the development team and how they perceived the testing team I asked the team how they felt about the testers and the responses I received all seemed to have a common ground.

‘I get a feeling of here we go again whenever a testers phones or comes to see me’

‘I dread it when a tester comes to see me’

‘They are always complaining that something does not work’

‘They only phone me when something goes wrong’

It got me wondering about how as testers we could improve this critical relationship and form a much better relationship. I thought that if everyday the same people are visiting/phoning me and giving me bad news I would soon develop a negative perception of those people.

So what can be done to change this?

I decided to try something a little different – someone once said from small acorns mighty oaks grow. I decided that instead of phoning or visiting the software development team when I had a problem I found the time to go and visit and ask how their weekend was or how the family is or what they thought of such and such in the news just a general chit-chat. One important thing I made sure I never did was to start to talk about general things and then say ‘Oh by the way … such and such does not work’ I cannot emphasize enough NOT to do this. My reasoning behind this was to build up a relationship and stop the feeling of dread when I turned up that something was wrong again.

The effect of this was amazing, the development team soon started to say hey have you seen this we are working on and start to talk with a passion about what they were working on. From a testing viewpoint this is valuable knowledge gathering. The attitude of the development team changed, when I did contact the team with a problem or something was wrong they would listen, emphasize and take a real interest in the problem rather than just dismiss it. The relationship between the teams improved tremendously.

So to conclude as a tester you do not need to always be the bearer of bad news to the development team. Take an interest in them as a person, take an interest in their lives and what they enjoy, take the time to learn about the people you work with. The benefits could be outstanding.

A word of caution on this – it has to be genuine – you really do need to be interested when you are talking to people about their personal lives. Otherwise you will come across as being cynical and shallow. If this happens then I am afraid you will cause an even bigger resentment and maybe even hatred of you.

Training in India

2010-06-24T01:35:00.000-07:00

I recently ran an exploratory testing workshop in India and I thought I would blog about this experience.

There are many differing views about Indian software development teams, some which are unfounded and some that are characteristics of the working style of Indian teams.

From my experience of working with many different teams around the world the statement above can really be applied to any team no matter where they are in the world how people interact and their style of working is dependent on their culture and way of working.

The workshop I was running had a lot of interaction and required engagement from those attending otherwise it is hard to gauge if the audience understands what you are trying to deliver. I was worried due to what I had been informed about the culture within India that there would be little if any engagement and everyone would agree with what I was saying, even if what I was saying was wrong. (I like to set little traps in my presentations and say things which anyone in testing will know is stupid and start a debate.)

I remembered an article that Jon Bach had written about his viewpoint on working with Indian testers and how he ended up making an apology. (http://jonbox.wordpress.com/2009/12/17/to-india-an-apology/). This article was KEY in how I ended up presenting the workshop to the teams in India.

So taking on board the lessons Jon had learnt I started to change my presentation a little to become more personal more about who I was rather than what I was trying to deliver.

I changed the start of the presentation and included a lot of personal information about myself including photos which I had of my family. When I started to run the workshop I explained that this was just an approach, a possible way of working I was not going to say to anyone attending that this is how you MUST do things and if you disagree with anything I am saying then please let me know. I then spent the next 20-30 minutes explaining about myself and my family. I think that this part was the key element – the need to reach out to the India team on a personal level, family in India culture is very important (Joint Family). I talked about our daughter and granddaughter coming to live with us when her husband was away for six months with the army and lots more. I then asked people attending to talk about themselves and their families. Suddenly the atmosphere in the room changed it become more relaxed and people appeared more receptive.

Can this one little change make such a difference?

So I begin delivering the workshop and found the engagement and interaction of those attending to be amongst the best I have come across whilst I have been delivering this workshop. There was passion, interaction, thoughtful questions and in some cases surprising answers. In my opinion it was one the best workshops I have ever been involved with.

Was I just lucky?

If I had not changed the start of the presentation would I have still ended up with the same reaction and interaction?

I cannot really say since I have no comparison – this was a one chance to deliver to a team in India, I was on a tight schedule, so it was important to get it right.

I really must say a big thank you to Jon Bach, since without reading about his experience I think I would have blindly gone and presented and not got the response I required nor would anyone have really learnt anything.

So the tip for anyone working or dealing with teams from India is to make sure you can engage with them on a personal level , open up and let people know who you really are.

The story of organizing a charity event.

2010-06-10T06:44:00.000-07:00

*********

I think I may have been a little remiss over the past couple of month by not updating my blog as much as I should be. I see posts by other great bloggers appearing every week or two but mine appear about once a month. I thought I would take time away from testing issues and blog the reason why I have not been as actively involved in the testing community as I would like to have been. This is a subject close to my heart and some may read and feel it is a little self indulgent however the cause IMO is more than worthwhile.

On Saturday 5th of June 2010 my wife and I organized in conjunction with the Amateur Poker Players League Europe (APPLE)

http://www.appleholdem.com/poker/

a poker tournament at the Prince of Wales Pub, Bishopstoke, Eastleigh to raise money for the Help for Heroes Charity.

http://www.helpforheroes.org.uk/

Apart from running the main poker tournament we had a pool tournament, a raffle and various other fun events. The support of local business was outstanding and overwhelming they could not do enough to help and given the current economic climate it was extremely humbling It was a different story with the large companies who I shall not name here who were not interested at all so when you think you need to pop out to get a pint of milk or buy something try to think of your local community businesses first rather than the big uninterested corporations.

The reasoning behind this is that our son-in-law (Lance Corporal Matthew Wellington) who is in the Royal Engineers returned from his tour of Afghanistan. His role with the Royal Engineers is with the EOD (Explosive Ordnance Disposal,) which as you can imagine is a highly dangerous and stressful job. He has a daughter who is now 2 years old and unfortunately has only seen her daddy for about 1 year of her life since this is Matthews’s second six month tour of Afghanistan within two years.

He has done his duty whilst the family at home, apart from the natural worry, felt helpless, so organized this day to help provide something back to those who are serving and the unfortunate ones who return injured. During his current tour he had to go through the trauma of losing some colleagues and a few who came back suffering from horrific injuries.

So you can imagine my wife Tracy and I had lots to organize and do, which took our minds away from the worry of our son-in-law whilst he was on tour, dreading watching the news and of hearing another member of the armed forces had been injured or killed. It has been a very stressful time and to be able to do something good has helped a great deal. At the end of the day the final amount we raised for this cause was over £1500.00 not bad for a single day event.

Part of this blog is to raise awareness of Help for Heroes and all that they do. They are not politically motivated and are doing a wonderful job and ensuring members of the UK armed forces are rehabilitated in an environment suitable for such heroes. SO if nothing else after reading this blog please visit the Help for Heroes website and maybe just maybe make a small donation.

http://www.helpforheroes.org.uk

Bangalore Testers meet-up (15-May-2010)

2010-05-16T22:55:00.000-07:00

I thought I would write a brief blog about a meet up of testers whilst I was in Bangalore, India.

It started with me sending a tweet out asking if any testers in Bangalore were interested in meeting up at the weekend. After a short while Santhosh Tuppard (@santhoshst ) responded by putting an invite on his blog site (http://tuppad.com/blog/2010/05/13/bangalore-testers-meetup-–-may-15th-2010/) then all of a sudden there was a flurry of activity on twitter as people asked about timing and where and when.

So on the Saturday I made my way down to the forum and met up with seven testers from Bangalore.

Pradeep Soundararajan (@testedtested - http://testertested.blogspot.com/)

Ajay Balamurugadas – (@ajay184f - http://www.enjoytesting.blogspot.com/)

Dhananjay Kumar (@dhantester)

Santhosh Tuppad (@santhosht - http://tuppad.com/blog/)

@nitinpurswani

+ two others – if you can remember who they add please add their details as a comment.

We had an agenda or some points to talk about:

Should we record our emotions when testing?

Exploratory Testing – good and bad points – how can we solve the bad parts?

Measuring Testing quality – How?

If we kept to these themes was a different matter.

After some brief introductions everyone was keen to know about me. After explaining I had been in IT for over 24 years which was older than some attending!!! Pradeep pointed out I did not look that old!!!!

We then started to discuss some testing issues and the following is what I remember of the conversations that took place. I am sure some of the others who attended will correct any details I have got wrong :o)

One testing issue we looked at is when teams are being managed remotely and managers at not in the same location as the testing team. Pradeep says this becomes a problem because managers want a breakdown of the testing activity all the time and the testing team spend a great deal of time answering questions from managers about what they are testing.

My thoughts on this was that anything that is actually stopping a tester from testing should be recorded and reported so at the end of each test phase/cycle when reporting back to management if a lot of time is being spent reporting test activities to managers rather than testing then this can be clearly seen and a ‘good’ manager will try to rectify it. One idea suggested was that the managers should become more hands on and take part in the testing activities. The exploratory testing approach makes this very easy to do.

There was a lot of interest in reporting what is stopping testing rather than any thing else – this lead on to a discussion of measuring testing and testing quality. This is always a difficult question and I am not sure we fully answered it in our discussions. I mentioned that when testing should we record our feelings and emotions about what we are testing. I suggested that we use a happy, sad face system so when we feed sad about testing we have a sad face when we feel angry an angry face and so on, It would then be very easy to see a trend for a area under test. If there were lots of angry faces then someone will spot and trend and start asking questions of the testers about why they feel this way. Pradeep pointed out that developers may try to influence the testers to always make happy faces. I stated that testers should be still independent of developers and that we should support and help developers but we would not tell them how to write an algorithm so why should they tell us how to test?

On the subject of measuring testing the discussion revolved around looking for trends rather than looking at numbers. The numbers could still be wrong but that is where talking helps to understand what the numbers and trends are saying.

Another point was made about using Tech support people as a resource for testers. I posted this on twitter as a question and got the following responses: (may not be in time/date order)

testingqa: @steveo1967 Tech support people hold some of the skills, but I wouldn't immediately say it implies they will be a good tester too #testing

testingqa @QualityFrog @michaelbolton My 1st role in IT was Tech Support role,those I worked alongside were slack & passed along cases for me to solve

testingqa @QualityFrog @michaelbolton So my faith in Tech Support staff isn't great ;) But I do agree that a good tech support person can hold an... appropriate background useful for testing, but even then Tech Support doesn't require an eye for detail often.

qualityfrog @michaelbolton @testingqa I've seen good ppl move fr tech supt 2 testing & vice-versa. there r similarities in foundation in what makes good

qualityfrog @michaelbolton @testingqa And, sadly, some in tech support are about as astute at faking support as some testers are in faking testing.

michaelbolton @testingqa Not automatic, but the skills and experience of support greatly overlap those for #testing. Great foundation, I'd argue.

michaelbolton @QualityFrog @testingqa I took *good* tech support person as read. Your pessimistic interpretation is also valid, alas.

So even here there are two sides to the argument but it appears that ‘good’ tech support could be valuable resource to testing. Maybe I am a little biased on this since my background is technical support and I believe that if you have a good grounding of how users think and technical knowledge as well (preferably at a coding level) then you have the right attributes to look at becoming a tester.

It was interesting that whilst talking to the group and I was aware I was doing a lot of the talking how much interest was shown in the subjects we were discussing, more importantly how much passion all the people at the meeting had for testing. I could see that in everyone’s faces a passion for learning and understanding. Key attributes of ‘excellent’ testers. I twittered about meeting the future teachers of software testing and I still believe that these people hold the key to the next 10 – 20 years of software testing.

The next subject that came up was one of certification – I explained that on the way to the office the other day I saw a sign that said “Learn software testing in two weeks”, everyone in the group laughed. We talked about the fact that software testing is not like programming in which once you have learnt the foundations of the language you then have the building blocks to create code. Software testing is not computer science but a mixture of many types of science including psychology and philosophy. It is not an exact science and there is plenty of scope to get it very wrong which we have all seen. The question of certification seems to be a very interesting one considering the latest blog posts coming from James Bach and Stewart Reid. (http://www.satisfice.com/blog/archives/457, http://www.eurostarconferences.com/conferences/session-details.aspx?sessionId=209)

I do not understand the need why fellow professionals need to attack each others ideas, it is getting very political. This is just my own personal viewpoint and nothing to do with the meet up. I believe in the need to debate and discuss ideas and opinions but when it appears that someone is personally attacking someone it then appears like a personality problem and distracts from a meaningful debate on the issues – I really want to attend Eurostar now just to hear the debates.

This nicely leads us to the subject of certification that was brought up during the meet up. We discussed certification and all of us agreed that it appeared to be a money making scheme that gave no benefit to experienced testers. The concern of the group was that agencies would now demand that testers have this certification and people will take the exam worried that if they did not they would not get a job in testing. The group felt this was very wrong, to be pressurised in taking an exam because there is no other measurable way to prove you are an ‘excellent’ testers. As a group we came up with some ideas that may be a way forward and needs everyone in the testing community to push forward.

All testers should have an online presence
They should be involved in writing blogs.
Be actively involved in testing discussion (software testing club, twitter)
Should try to meet fellow testers a couple of times a year at testing meet up – the internet allows this very easily as this meet up has proved

Once this is done when you apply for a job and the agency asks for your certification – point to your active online presence in testing ask them to talk to peers who have met you and who can vouch for you. If we ALL did this then the need to pay organisations to prove you can test goes away. Let us as a testing community certify each other.

So with this last thought I did a little experiment on the people in the meet up involving the calculator experiment as taught by Michael Bolton – if you have experienced this from Michael then you will know what I mean if not then you need to find someone who knows because I am not going to spoil it by explaining on here. Thank you again Michael for giving me a useful way to demonstrate a key point.

So all too quickly the meet up had to end. Did we all learn something? I hope so. I know I did and I had a wonderful time and left feeling encouraged and motivated. I still think I talked far too much and for that I apologise hopefully if there is a next time I will encourage others to speak even more – you have been warned……

The Online Testing Community

2010-05-06T01:57:00.000-07:00

I have worked in the field of IT for a long time and seen many changes but none have been as significant as the one that has happened online. The rise in the number of fellow professionals now blogging and tweeting (or should that be twittering) is amazing. The online testing community is increasing more and more as testers take the plunge and start to write and debate about testing online.

We have online testing clubs: http://www.softwaretestingclub.com/

magazines: (http://blog.softwaretestingclub.com/magazine/)

testing knowledge exchange : (http://testing.stackexchange.com/)

Organizations trying to get more of the testing community together online: (http://blogs.stpcollaborative.com/stpcollab/)

I would count myself as a late starter in this revolution only being brave enough to start twittering and blogging late last year. Why do I say brave enough? I am sure I am not the only one who feels that there are so many peers who we read about online and who we, deep down, admire. We may feel that we can never be good enough to write articles about testing or that what we write will be dismissed by the community. In my case I am well aware that my grammar is not the best in the world but all I am doing is writing down what I am thinking and I hope it comes across in a good way. There can never be enough of us online talking about our own opinions and valuable experiences. I had a fear that people would not be interested in what I had to say or worse still would think what I had to say is silly and I would feel rejected and humiliated. Surprising the testing community has not been like that.

I have had some wonderful debates and discussions on testing issues. I have been coached online by Michael Bolton (@michaelbolton) on using Exploratory Testing, in which he gave up his own time for free and is wonderful patience person who really makes me think differently. I have been introduced to such great testing thinkers as Rob Lambert (@Rob_Lambert) who has been a leading character in organizing the online testing community. I have had some great comments on my blog about subjects that I find interesting. I have found it a great outlet for my thoughts and ideas on testing that I once kept to myself, afraid that what I was thinking would not be of interest to anyone. I have also found it to be a wonderful resource for information and ideas about testing and how much people are eager to help.

I would like to say to anyone who is reading this and does not have a presence online to just go for it, start a blog, start a discussion on twitter or join the software testing club and start a debate. I have found it has given me a new lease of professional life, it has made me more aware that whatever problems I come across other are coming across the same problems. It has encouraged me to start writing about my experiences of testing and that I may have some useful information that others want to hear or read about. It has encouraged me to come out of my shell and talk to people about my passion for testing and that cannot be a bad thing……

Where do all the ‘old’ (experienced) software engineers go?

2010-04-14T05:51:00.000-07:00

I was having a conversation with a colleague the other day whilst in the kitchen area making a coffee. When they stated the following:

“You never see many engineers over the age of 40 working in a development environment, actually writing code or testing”

I thought this was a interesting statement and it made me think. I have been in the IT business for a long time and yes I am over 40 and I am still actively involved in cutting edge software development projects. However when really thinking about the statement, how many more people did I know or have known who work in software development as developers and testers are still actively involved?

We had a discussion about it and came up with some reasons why.

They move up further in the company (VP, CEO, etc) and take a less active role
They switch careers becoming technical architects etc.
They give up working in IT

One other point was made

“Software engineering is a young person’s career”

Quite a controversial statement!!!!

However could this be true?

Another colleague mentioned a point that as we get older we lose our mental ability or cognitive processing. However this article seems to debunk this: http://www.healthandage.com/html/min/afar/content/other6_1.htm

The article does state that we do lose our attentional ability and processing speed – key elements for software engineers.

What did interest in the article was the following:

“In general, memory tasks that are complex and require manipulating a lot of new information quickly become more difficult with age. Facts, names, and events that are not often accessed may become more difficult to retrieve from memory. However, knowledge that has been accumulated over a lifetime, which is repeatedly accessed and expanded, is generally retained. Well-practiced skills and abilities remain intact. And vocabulary usually continues to increase throughout life.”

So we may become slightly slower mentally as we get older but we retain our well practiced skills and abilities.

One important point made in the article which is related to my discussions on the telling of stories and is a key skill of an excellent tester: “….vocabulary usually continues to increase throughout life”

So my question and the real point of this article is:

“Do companies make a conscious or unconscious decision to remove older software engineers?”

It would be a shame if this is happening since currently I feel like I am in my prime. I am still discovering new and wonderful things about software development each and every day. I still have the same passion for my chosen career as I did when I first started with the added advantage that I have years of experience to fall back on as well.

I would love to hear from other people who, once they look around their respective companies, notice the same trend. Or from anyone who has any more theories on this.

Measuring with Stories

2010-03-24T02:03:00.000-07:00

In my experience one of the most difficult tasks I have is to try and change the way we measure testing. When I first started out in the profession of testing there was very little thought given to how we should report the quality of the testing that has been carried out. The most common way was to record the number of test cases and report how many passed and failed.

Then the managers wanted more information such as:

The number of defects raised
The number of defects fixed
The severity of the defects.

Does this sound familiar to anyone?

Sadly even today when testing as a profession has started to mature, managers still measure the quality of testing by using these figures and metrics.

I read the following article the other day: Defect Detection By Developers

The article talks about how developers can discover more defects in the same amount of time frame as a tester.

I always have problems when I see articles like this.

How is the statement:

'The number of defects detected by the developer is of the same order as detected by a test engineer in the same time frame'

Quantifiable or Qualitative?

There is no mention of the type of defect and the measure of risk of the project if the defect was not found. What happens if say 90% of the defects found using this method were purely cosmetic? Would this indicate this method is better than using a skilled tester during the same time frame? The skilled tester may find less defects during the same timescale but they may (and normally do) find the difficult to detect defects. Or having the tester and the developer work together during the development phase using continuous build to check as they go?

The approach discussed is one which many companies should already be using

In my experience one of the most difficult tasks I have is to try and change the way we measure testing. When I first started out in the profession of testing there was very little thought given to how we should report the quality of the testing that has been carried out. The most common way was to record the number of test cases and report how many passed and failed.

Then the managers wanted more information such as:

Code reviews
Peer reviews
Documentation reviews (What happens if the project is a prototype project in which no documentation exists? - Maybe BDD could cover this?)
Unit tests etc.

The article misses vital approaches such as continuous integration? How many defects are trapped during this method?

I think the article has some valid points however I strongly object to the statement that a developer can detect the same quality of defects as a skilled tester can in the same period of time.

I do not like 'measure by defects' to prove quality - quality is proven by the telling of a story to indicate that the product is of the right quality for its purpose.

This leads me nicely into the title of this blog.

Why does management insist on measuring the quality of testing using numbers?

Are managers measured using numbers?

In my role as a test manager I have never, in my experience, been appraised solely using numbers nor as far as I believe has any senior stake holders been measured solely using numbers. When I talk to my superiors about my management goals I try to tell stories and avoid using numbers. For example if I said to my line manager ‘I have met 70% of my targets’ and left it at that. What information would my line manager get? Would that be good or bad? Compare that to some tester reporting to their manager that 70% of their testers have passed. What information can you get from that?

I understand these are extremes but unfortunately these extremes are common when it comes to measuring the quality of testing.

So if I spoke to my line manager and said that over the last six months I have been involved in addressing x, y and z, which are the most important tasks, I still have problems with a and b and I have not had the time to deal with T and S. I have enjoyed the challenge of x y and z and feel I have achieved the best I can in these areas, a and b are ongoing and I am sure that I can complete them soon as long as I get support for b. My estimates where too ambitious for me to even start T and S but these were not too important so I gave them a low priority for completion.

Does this give you a better understanding of what has been happening? Do you notice there is no use of numbers in the above? I feel it gives far more information.

So if we now do the above again and convert it to a story about testing:

For the last six months I have been involved in testing areas x, y and z, which are the most important areas of the product. I still have problems with test areas a and b which appear not to be working as expected and I have raised defects for these issues which are reported below. I have not had the time to test areas T and S. Test areas x y and z are of sufficient quality for release. Currently test areas a and b are not of sufficient quality but once the defects in b are fixed I am confident that the quality will improve. Due to the time taken to address the issues with a and b my estimates where too ambitious for me to even start test areas T and S but since these were not too high priority the product can work without them as long as it is documented.

How does this sound? If you as a manager read this would you have a better understanding of the quality of testing?

Management is normally measured by the telling of a story, so why not measure the quality of testing by the telling of a good story?

I was on twitter the other day I noticed an article by BJ Rollison talking about a similar talking in how we measure the quality of testing. Meaningful Measures

There was an interesting line he came up with:

'At one time I naively believed that there was a core set of metrics that all teams should be collecting all the time that we could put into a ‘dashboard’ and compare across teams. In retrospect that was really a bone-headed notion. Identifying these measures is not easy, and there is no cookie-cutter approach. Each project team needs to decide on their specific goals that may increase customer value or impact business costs. Testers should ask themselves, “why are we measuring this?” “What actions will be taken as a result of these measures?” And, “if there is no actionable objective associated with this measure, then why am I spending time measuring this?'

Some very valid points, testers should be asking why are we measuring this, is it quantifiable? Will it help lay people understand the quality of the product?

When we measure the quality of testing it should be clear, concise and in a language that anyone can understand. Blinding people with numbers does not aid clarity. Trying to measure different teams on different projects with the same metrics (code coverage, defect counts, test case pass/fail) does not indicate one team is better than the other all it does is help management pretend that they understand the quality of testing that took place.

With this article I am not saying we should abandon all numerical metrics to measure the quality of software testing but we need to look more at the story behind the numbers since these can give you far more information on the quality of testing.

Some other useful articles on Software Metrics:
meaningful-metrics by Michael Bolton
Metrics by Kaner and Bond

Does manual testing really lose its value as Companies encouraging more and more automation?

2010-03-03T04:11:00.000-08:00

A colleague the other day asked me for my views on the automation/manual debate and asked the following question:

Does manual testing really lose its value as Companies encouraging more and more automation?

I thought this is a very interesting question and decided to blog my response.

I should start by saying I am not split into either camp on the automation vs. manual testing debate. I can see the benefit of both depending on circumstances you are in. If you have an old legacy project in which you are 100% sure nothing will change then automation could be the answer. If you involved in a system in which changes could be made and you want to CHECK that the functionality or the business rule that is in place is still giving the exact same result with no deviation then automation could work.

My thoughts on the question that I was asked (and the title of this article) are completely the opposite. Automation requires no sapience or thinking to be executed, so once the automated checks have been written (which does require sapience) you can run and forget.

The problem is that the world in which we work in is all about changing, adapting and making things better (normally) especially in an agile environment where change is embraced.

I should state that my definition of manual testing is not of following pre scripted tests but of being a test explorer, searching in all the nooks and crannies, trying to discover new and intriguing things about the software. If it is pre scripted then automate it, do not waste good tester intelligence and skill on running a check list, your testers deserve better.

So I would ask the following questions on any company who want to encourage more automation at the expense of manual testing.

Is it cost effective to write lots of automated checks compared to carrying out manual exploratory testing?
Which method would in the same time period give the most test coverage?
Which would be the most easy to adapt to major changes?
Which would uncover the most problems or issues?

I feel there is some value in using automation at unit level and build level, continuous integration with acceptance checks is a useful tool for the software tester since it lets them have a early look at changes with some confidence that what they have been given will at least have a chance of working. Sure beats the good old days of rejecting x releases in day because of a typo in an install script or a missing dll from the build.

You hopefully can see that I am not against using automation; however there appears to be a view in the testing world that automation can replace manual testing or make its value less. This view does worry me since if the corporate suits think they can get more value and better quality using automation then the message about the art software testing is not being broadcast well enough.

Manual and automation can co-exist very well together; however:

manual testing can exist without automation
but automation cannot exist without manual testing.

There is an interesting podcast with Jon Bach and Michael Bolton with their viewpoints on the difference between checking and testing available here:

http://www.quardev.com/blog/2010-02-02-1123487836

Exploratory Testing and Scrums

2010-02-15T06:47:00.000-08:00

I was having a conversation with Michael Bolton (www.developsense.com) and we started to discuss how ET can be incorporated effectively within agile scrums. After a few minutes of chatting Michael suggested that I blog my thoughts on this since it appeared to be a good idea.

So here we have a short article on how we could use ET within a scrum.

The structure of a scrum in my experience is

What you have done?
What you intend to do?
What is stopping you?

In most cases a scrum normally turns into a technical debate – this may not always be the case but it is something that I have experienced. My thoughts are why not use the time in the scrum to the benefit of testers?

Instead of just saying what you have done, tell the story of what charters you have followed and what you have found interesting. Talk about the test ideas you followed and what other test ideas (test coverage?) you have thought about while testing.

The crucial difference is the next part:

Instead of just saying what charters or testing ideas you intend to carry out for the next days work why not start a discussion on what test areas they feel you could be missing – this would involve the programmer, the product owner and any others in attendance at the scrum. Imagine the benefit in using the experience and knowledge of other people on the project to help with your test coverage.

I am sure people could say why not just go and talk to them individually for their ideas? My view is that people are more than likely to bounce ideas off each other and trigger responses in others when they are together as a team rather than as individuals.

I have attempted to do this in my own scrums with mixed success. Sometimes people within the team have pointed out huge areas of coverage that the test team had not even thought about whilst in other cases there has been a muted silence. I feel for this to work depends on the dynamics of the team and how they interact.

The next stage for this experiment would be to try and take the user stories during the planning sessions and open a discussion on test coverage/ideas. I really would like to try to use something like HICCUPPS (http://www.developsense.com/articles/2005-01-TestingWithoutAMap.pdf ) or SFDPOT (http://www.developsense.com/articles/2005-10-ElementalModels.pdf) with all the development team to see if we as a team could improve test coverage.

I know I promised an article on my experience of being coached by Michael Bolton on Exploratory Testing and my discussions with Jon Bach (http://jonbox.wordpress.com/) on managing ET. However I thought I should blog this while it is still in my mind. The article will happen I promise. I just want to wait until I have finished the coaching sessions.

Plus shrinik – I have not forgotten that I will blog an article measuring management with a story. – I hope to put this together sometime in March.

Child’s Play

2010-02-05T03:29:00.000-08:00

This article is based upon some thoughts that I have had over the past year while watching my granddaughter playing and learning new things, she will be two years old in April. It is amazing how quickly children of that age learn to do tasks without being taught such as walking, the beginnings of communication and how to play and explore.

I am an exploratory tester and my thinking is how we can as tester harness what children do naturally? Some may say that they do exploratory testing and they feel it is natural. If that is the case why do so many testers have difficulty adapting to exploratory testing and keep falling back to scripted testing?

Peter (unlicensed testers) asks a lot of questions about children and learning in his blog article here: http://007unlicensedtotest.blogspot.com/2009/11/what-do-you-get-if-you-cross-7-month.html

Watching my granddaughter the other day I observed that she was trying to put her trousers on. At first she managed to get both legs into the same leg hole, she noticed that this did not feel right so started again but this time she tried to put both legs into one of the small leg holes and found that this did not work either. After this she then managed to get the trousers the correct way around and one leg in each hole but did not pull them up and ten tried to walk and fell over.

What can we learn from this?

We can see that she tried following different options and observing the results, she then thought she had completed the task but found that it was not really complete. So if we convert this to software testing we can see that she is using heuristics to determine how to do the testing, the trial and error approach. She is using her emotions and feelings that something is not correct and she is doing a lot of noticing, which is something that every good exploratory tester should be doing. There are also examples of mentally noting future areas to test, the fact that when she tried to walk she fell over. The next time she tried to put on her trousers she did manage to pull them all the way up before setting off to walk.

There are many other examples of trial and error that children appear to do when playing. If something does not feel right they will suddenly change the approach to the problem or in some cases they just give up.

What can we learn from children playing? I have observed that this exploring behaviour appears to start diminishing once children start to attend full time school. Why is this so?

Is it because schools start to impose on children their own ethos and standards and re-model children to not take risk?

One of the main elements of testing IMO is the taking of risks.

We all do the ‘Let us try this to see what happens’, ‘Let us try something else and see what happens.’ Many corporations are risk adverse and as such when testers are brought in they have to provide a return on investment (a hot topic on twitter at the moment) so they are less likely to follow a risky approach. Some may argue that exploratory testing is not risky and I would intend to agree with them. However the business world does not seem to afford the time to be able to cope with let us try this and see what happens and then let us try that and see what happens. They require order and structure and no risk.

Children on the other hand when learning and playing do not take risk into account they try and if it does not work try something else and if that does not work they continue trying until they get a result they are happy with. They remain focused on the task at hand but appear to be able to solve problems without the fear of failure.

So what happens as we get older? Why do we lose this ability to explore and learn without fear of failure? It appears to be a natural human instinct that somehow is gradually removed as we get older.

Is it to do with the education system and how they remove the risk factor and make everyone fear failure and taking risks? Do we become institutionalize in to conforming to the known path, to stop asking the probing questions and to stop playing? I am not sure I have the answer to these questions however I am sure as testers there are some valuable lessons we could learn from children and how they explore, learn and play. (Simultaneous learning, test design and test execution)

I wonder if James would not mind if we change the definition of exploratory testing to learn, explore and play?

I think everyone who wants to learn more about exploratory testing should take some lessons from children. Do not be afraid to explore, if you make a mistake learn from the mistake to improve the next time and have fun. Testing should be about having fun and enjoyment it should not be a chore. If it becomes a chore get a different job……

Does anyone have any interesting games for testers? If so I would love to hear from you.
___________________________________

My next blog should be on my experiences of being coached by Michael Bolton and Jon Bach on using and managing Exploratory Testing.

Why Testers do not add Value

2010-01-26T07:45:00.000-08:00

This is an article I published internally last year when there was a debate going on about the value of testers within the testing community so I thought would re-publish it here for people to have a read.

___________________

This may seem like a strange statement coming from a tester; however the purpose of this article is an attempt at justifying the need for testers on a project. Financial people might use traditional cost/benefit comparison models when recruiting people for a team or project which may not necessarily be applicable when resourcing testers.

What made me think about writing this article was something I happened to read on the Software Testing Zone website by Debasis Pradhan 1 and from Michael Bolton at the Developsense website .2

There are two comments within these articles that appear to come from a developer’s point of view about the role of testers within software projects.

The first is from Debasis blog:

Software Testing is a worthless process. It does not add any value to the project!

And one from Michael’s blog

But testers don't add value to a project;

These comments made me think about the way testers are perceived on projects and whether or not they really do add any value to a project. This led to researching how accurate these statements were and the production of this article.

The problem I have found when carrying out the research for this article is that the term ‘value’ can have different meanings depending on the context in which it is used.

If I were to ask you what is the value of a gift a loved one gave you who is no longer here?

What value would you put on that item?

In money terms it might not be worth that much but in emotional or sentimental terms, to you, it could be priceless. It is very important when reading this to use the correct context for the meaning of value. For the purpose of this article value will be measured in terms of monetary value otherwise your emotional viewpoint may get in the way of what is being said.

So the definition of value for this article will be:

Value implies monetary value.

If we take a look at a typical upfront project and see where costs are attributed and where value is added it may give a better understanding of the statement.

The first part of a project is the initial design and scope where the architect captures and translates the requirements of the customer in to technical solutions and business requirements. The value of an architect is very clear since they are the people who provide solutions to the questions raised by the business. They ensure that the requirements for the customer are met and what the customer asks for is what will be delivered. Those who work within software development may not agree with the final statement since sometimes features requested are not delivered or changes are made that affect part of the delivery. However in the majority of cases what the customer requires is delivered by the architect.

The project manager will then look at these requirements and sees which has the highest business value at the same time assessing the risk of adding the features before making the decision as to what to develop which will give the maximum value.

These technical solutions, risks and business value are then interpreted by the developers and turned into an actual project so their value can be measured in that they build the project. The developers also fix any problems and solve any issues that may occur either from the customer or the test team.

Marketing then can sell that product or may already have sold the product adding value to the company.

Some may say that testers must add value since they find problems before a product is released and as such stop a project costing more in terms of company reputation and actual monetary cost. This may appear true but testers do not actually provide solutions and fix the problems, this is the role of the architect and the developer.

So what does a tester do within a software project?

Testers look at the project being developed and raise questions and report observations as to what may not be working correctly. They look and report on areas which could cause the most risk and most expense to fix. They give priority to the main problems (defects) within the product.

However the people who fix these problems, act upon the observations and answer the tester’s questions are the ones who add value to a project.

This view is based upon a typical style of software development where all requirements are gathered upfront so what about agile style projects?

It is rather interesting that there are very differing viewpoints on this within the testing world, one side saying that if the testers are working directly with the developers and writing the test cases before the code is written then they are adding value to the project. The other side is saying they may write the test cases but they do not actually do any creative design work and as such do not add any real value to the project. My viewpoint is of the latter if the testers are not actually creating code that will lead to a function that will be used by the user in the live system then they are not adding any real value to the project.

The way in which testers could add actual value to the project when using agile is when they have close working arrangements with the actual customer since they can then suggest or inform the customer that if they did this a different way it would have more benefit. This would still not be intrinsic value unless the tester managed to get the customer to sign up for more work at an additional cost, therefore adding to the value of the project.

To conclude testers do not add intrinsic value to software development project but they prevent deprecation of the project value. If you have a project that a customer is willing to pay five million dollars for and you ship that project with x amount of high level problems. The customer then quite rightly demands the problems be fixed free of charge. Then the value of your project could go down from the initial five million to say four million, losing the company both real money and another value that cannot be measured: ‘your company reputation’.

It should be noted that there is the factor of costs of resource on projects and to keep things very simple I have excluded these other costs when making the above statement. Otherwise you would need to include the cost of equipment, building, utilities and so forth.

With testers on the project the majority of the high level problems would have been caught before being shipped to the customer and as such the value of the project would have been maintained.

Therefore testers do not add value but they certainly prevent value from being lost. It should be noted that no tester can guarantee one hundred percent bug free software. Even companies such as NASA who on the Space Shuttle program had a ratio of 10 testers to 1 developer 4 cannot give that guarantee as demonstrated by the tragic accidents that have occurred! The average ratio in the majority of projects is 1 tester to 3 developers5; therefore value will always be lost in some way from a project.

So remember those whose job it is to resource software development projects never skimp or cut costs by using less testing resources and think that you have saved yourself some money. You might just find that the 1 million dollar project you are working on becomes worthless due to lack of testing resource to ensure the value is maintained.

I will leave you with a view from James Bach, who is a software testing author and one of the main contributors in the software testing field.

‘James Bach suggests, testers help to defend the value that's already there, or help to identify ways in which value may be lacking. Testers raise questions and make observations; the people who make decisions based on those observations are the ones who add value. We help them do that, but we don't do it intrinsically on our own.’ (3)

References:

1 http://software-testing-zone.blogspot.com/2008/10/software-testing-add-value-to-project.html

2 http://www.developsense.com/2008/03/breaking-code.html

3 http://www.developsense.com/2008/10/while-back-i-wrote-post-on-breaking.html

4 http://codeidol.com/other/Software-Estimation/Estimating-Planning-Parameters/21.1-Estimating-Activity-Breakdown-on-a-Project/ (table 21-7)

5 http://www.infoq.com/news/2009/01/tester-to-developer-ratio

______________

Hopefully if I get myself sorted my next article will be about how we can learn to improve how we use exploratory testing from watching how children play and learn.

Testing Terminology – definition or context?

2010-01-07T01:18:00.000-08:00

Hello and Happy New Year

One of my New Year resolutions was to start using twitter more so that I could micro blog some of my thoughts.

One interesting topic that appeared to be going around was one of changing testing definitions and descriptions. This caused quite an animated debate amongst testing twitters.

So I thought I would use more that 140 characters and put across some of my thoughts and views on the subject of testing definitions.

I used to have the view that it was good to have a set of definitions for testing terms such as what is Acceptance testing, What is black box testing etc. The ISTQB has a document for this on their website (www.istqb.org/downloads/glossary-1.2.pdf)

The problem with this glossary is that it applies testing meaning with no context and that is the problem I find with trying to define testing terminology to do so requires context. If I wish to be somewhat controversial I should state that this is the problem I find with the ISEB and ISTQB examinations especially the ISEB foundation level which is based upon multiple choice answers. Some of the questions could in certain situations have multiple answers but ISEB only accept their definitions and do not allow the natural questioning skills of the tester to debate the answer.

For example let us look at the testing term ‘Acceptance Testing’

ISTQB define it as

Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system.

However if I would add the following context to Acceptance Testing:

The development team requires some form of automated Acceptance testing (Yes Michael I know if you are reading this it should be checking not testing - http://www.developsense.com/2009/08/testing-vs-checking.html) on the build machine before signaling that the build is suitable for release to the testing team

…would the above definition hold true?

A way to improve test coverage?

2009-12-10T03:56:00.001-08:00

So here is my first blog on the subject of software testing.

This article is taken from an article I published internally for the company I work for.

I was having a few conversations recently about the way we work within the company and how we appear to get fixated on our own areas and are not really aware of what our colleagues are doing in other areas. I am not saying we are an unsocial set of people rather the opposite that we do talk with each other and exchange pleasantries but rarely do we discuss our projects in detail with other teams.

At the end of the discussions I got thinking how we could improve our testing and at the same time get other project teams the opportunity to interact more,. I came up with an idea which I feel would be interesting to put into practice.

The idea is that one member of a test team, not necessary the test lead, organizes an informal 30 minute discussion to talk about a feature/area they are currently looking to test. They should spend five to ten minutes explaining what it is and five minutes should then be spent on describing the test ideas they have to test that feature area. The rest of the time should then be a general discussion with the group to see if there are any gaps in the tests or add test areas that the person chairing the discussion may have missed. During the discussion part someone should be recording details of what is being said, whiteboard post it notes, video recording, audio recording etc.

The crucial part of this informal chat is that it should involve people from outside the project for example if you are working on the project X you could invite someone from the Project Y and so forth. It does not need to be testers it can be developers and project managers, architects in fact anyone who you feel would like to attend.

One drawback of this idea is that I can see people saying they have not got the time to attend a session since they are too busy. However on every project you are in someway involved in reviews regardless if it is code, test or requirements hence my thinking of making sure the sessions are very short so that really they could be done over a coffee break. Think of the brown bag sessions but even shorter.

I think the benefits of using this could be quite large, not just from the perspective of what can be measured such as new test ideas uncovered and defects found but what cannot be measured. The things that can not be measured are items like people within the company from different projects working together and talking together building up product knowledge outside their normal domain, the building of stronger links between projects and the thought of being able to help fellow colleagues.

One thought to think of if you are approached and asked to attend one of these sessions is……

I may learn something new to today if I attend…..

This for your own personal development is always a good thing.

One thought that came up after I wrote the above article was that it might not just apply to testing but to any area within software development.

About time.

2009-12-10T03:44:00.000-08:00

I was looking around at all the blogs that testers have been running for quite awhile now and thought I should at least attempt to play catch up and start my own blog.

I am not sure how often I will update it but will ty to on a somewhat regular basis with my thoughts and ideas on testing.

Anyway welcome to all that have managed to find their way here I do hope over time to provide some useful information.