Are testers’ ethnographic researchers?

People who follow me on twitter or via this blog might be aware that I have a wide range of interests in areas outside my normal testing job. I like to research and learn different things, especially psychology, and see if it may benefit and improve my skills and approaches during my normal testing job. One area I have being looking at for awhile is the social science of ethnography. The approaches used when carrying out research appears to have many similarities to software testing and I feel we could benefit and maybe improve our testing skills by examining ethnography.

IMO there are two areas in which we can learn from ethnography:

• To improve our understanding of users and how they differ by using ethnographic methods

• Use ethnographic methods to test software in an exploratory way.

I should start by explaining what my understanding of ethnography is:

Wiki attempts to define it here:

http://en.wikipedia.org/wiki/Ethnography

The free dictionary attempts to give a definition here:

http://www.thefreedictionary.com/ethnography

A better definition can be found here:

http://www.brianhoey.com/General%20Site/general_defn-ethnography.htm

The problem with trying to describe and define ethnography is that it has wide and varied meanings.

To me it is a branch of the study of humanity (anthropology) in which the researcher actively gets involved and participates with the study group rather than just sitting back and observing. The reporting is doing using qualitative (words) measurements rather than rely on quantitative (numbers) measurements.

One of the key factors when approaching ethnographic research is to be aware that participation, rather than just observation, is one of the keys to the approach. Does this not sound familiar to testing, especially exploratory testing? Actively using the software under test to find out about its characteristics and behaviour are similar to a ethnographic researcher living within a community and participating with that community to learn about its beliefs and characteristics. There appears to be very close parallels between ethnographic research and exploratory testing. Wikipedia states:

One of the most common methods for collecting data in an ethnographic study is direct, first-hand observation of daily participation.

How similar is that to testing software?

Another approach within ethnography is the use of grounded theory to explain the results from the participation. This is when the data is used to provide theories about the data. This is different from grand theory in which the theory is defined without the use of real life examples and therefore has a danger of not fitting the actual data gathered afterwards (is this similar to scripted and exploratory, grand theory vs grounded theory?)

Grounded theory is a constantly evolving set of conclusions that can continue indefinitely based upon the changing data being obtained by the ethnographic researcher. One of the questions that are asked about ethnographic research is:

When does this process end?

One answer is: never! Clearly, the process described above could continue indefinitely. Grounded theory doesn't have a clearly demarcated point for ending a study. Essentially, the project ends when the researcher decides to quit. (http://www.socialresearchmethods.net/kb/qualapp.php)

How similar is this to testing?

When do we stop testing?

Many articles have been written on this subject and mainly we stop when we can learn nothing new, no time or ran out of money. See this article by Michael Bolton for more information

I feel that ethnographic research stops because of similar reasons.

One interesting section I saw within the wiki article was about the process of ethnographic research in which to aid the researcher areas were split and the research asked questions.

1. Substantive Contribution: "Does the piece contribute to our understanding of social-life?"
2. Aesthetic Merit: "Does this piece succeed aesthetically?"
3. Reflexivity: "How did the author come to write this text…Is there adequate self-awareness and self-exposure for the reader to make judgements about the point of view?"
4. Impact: "Does this affect me? Emotionally? Intellectually?" Does it move me?
5. Expresses a Reality: "Does it seem 'true'—a credible account of a cultural, social, individual, or communal sense of the 'real'?"

I thought about this and started to change the context to be about software testing:

1. Substantive Contribution: "Does the testing carried out contribute to our understanding of the software?"
2. Aesthetic Merit: "Does the software succeed aesthetically?" Is it suitable for the end user?
3. Reflexivity: "How did the author come to write this test…Is there adequate self-awareness and self-exposure for the reader to make judgements about the point of view?"
4. Impact: "Does this affect me? Emotionally? Intellectually?" Does it move me?
5. Expresses a Reality: "Does it seem 'true'—a credible account of a requirement'?"

By doing this I found I suddenly had a set of heuristics to measure against the software testing that has been carried out, yet again more similarities between the two crafts.

Another area in which ethnographic research can be useful to software testing is when you need to test software that has a lot of UI interactions. Using the methods of ethnography a tester could go visit the users and observe and participate in their daily routine to find out the common tasks carried out and what oddities are seen. The oddities are the things of greatest interest since these are the things that would not normally be planned for and without active participation with the users would normally not be uncovered until it is too late.

There are many studies being carried out to determine if ethnographic research should be used when designing software system, however my concern with this is that it appears to be stuck in the design up front way of working which is not a flexible iterative approach, in my view it is easier, quicker and cheaper to ensure that testers use ethnographic methods when testing to ensure the design is suitable for users or even better get the users involved earlier and observe them earlier.

The more I have delved into the study of ethnography the more and more I have seen similar patterns to software testing. This makes me aware that software testing is not solely a hard science but a craft that encompasses many disciplines outside of the typical number crunching and algorithm creating world of software development.

Within the testing profession we need to look outside of the box and find approaches, methods, structures that can improve the discipline. To ensure our craft grows we need to ensure we do not narrow out field of vision or thought.

Remember you’re a Tester

I want you to remember one word in the following list:

Bug
Insect
Ant
Dragon Fly
Ladybird
Crane Fly
Beetle
Bee
Wasp
Hornet
Cockroach
Earwigs
Termite
Grasshopper
Flea
Mosquito

My previous post was about debrief and how important it is to testing.

The problem I have come across during debrief has been trying to remember all the things that happened during the day or during the session(s). Maybe this is a, me getting older thing, and my memory is going.

One thing whilst I was reading recently about cognitive bias seemed to be a bias that could be helpful both during testing and during the debrief sessions. This was called the Von Restorff Effect and basically it was how our brains remember things that stand out.

http://changingminds.org/explanations/memory/von_restorff.htm

The above link uses an example of a list of words in which one word is in a different colour, our brains are more likely to remember the word that is a different colour and stands out.

You might be asking what connection does this have to testing.

Michael Bolton via twitter pointed me towards Adam White who has an interest in the Von Restorff effect. In his blog Adam states the following:

I use the Von Restorff effect in testing all the time. I frequently notice what doesn’t belong and it tends to be what I remember the most. http://www.adamkwhite.com/2007/09/30/using-heuristics-to-cook/

Part of our skills as a tester is noticing:

• When something does not appear to fit – we notice
• When something appears out of place – we notice
• When something appears not quite right – we notice.

Could it be that testers have a strong Von Restorff cognitive bias? Maybe this is the missing ‘thing’ that people say testers have. You can not describe it but you just know it is a skill you have.

Going back to this article…..

How can this help during debrief?

My thoughts on this are that to remember something that is important for the debrief when we are working within SBTM. We should make a note of what it is and ensure we highlight it in a different way to make it stand out and ensure that we remember it later. Maybe some people already do this (the use of the highlighter pen).

Maybe the excellent tool for recording sessions, Rapid Reporter by Shmuel Gershon be expanded. Can it have an option to highlight certain things to make them stand out. I know it can do rtf and bold but that is not enough for me. I need highlighting and colouring, plus an option to do freehand doodles.

Why doodles you may ask?

One of the side effects of the Von Restorff effects is that we remember words better if associated with a picture. If I need to remember a URL is not working I would doddle a chain with a link missing. Or an interface that is failing to communicate I could draw a face with a plaster over the mouth. Just little things that help me remember problems that occurred. By the way I am rubbish at drawing not on the same level as the cartoon tester.

To conclude this article I think as testers we already have a cognitive bias to remembering things that stand out but within testers it appears we notice these things a lot more, either via our continuing training or a natural skill we possess. We need to ensure that important things we need to remember for debriefs are made to stand out during our testing sessions to ensure we do not forget them.

Which word did you remember from the list at the beginning?

Was it termite?

The Feedback Loop

One of the critical elements of following the session based test management (http://www.satisfice.com/sbtm/) approach is the use of quick feedback. To achieve this it is suggested that a debrief should be done at the end of each session/day. Jon Bach (http://www.satisfice.com/articles/sbtm.pdf) suggest the use of PROOF

Past. What happened during the session?
Results. What was achieved during the session?
Obstacles. What got in the way of good testing?
Outlook. What still needs to be done?
Feelings. How does the tester feel about all this?

This approach is excellent for communicating what has happened during the testing session(s), however I keep hearing that people are not doing the debrief . There are many reasons why these are not being done, lack of time/resource or see no benefit are a few of the reasons given. This blog post is why it is important to carry out these debriefs and ensure they are done sooner rather than later.

I am looking at this from a psychology viewpoint to highlight the way our minds work and to keep reminding readers that software testing is a human sapient process and not an automated ticking of boxes process.

There are various studies that have indicated that the longer you take to act upon information the less you are able to recall that same information at a later date. During Eurostar 2010 Graham Freebur stated that unless you act upon information you had digested at the conference then within 72 hours that information would start to be lost and fade. The crucial part of this is that as humans we are fallible and lots of different psychological biases start to play with our minds so unless we can talk and pass on the information we have as soon as possible the more likely that the data we have will become clouded.

It is important that we debrief to someone to ensure that any error in our interpretation of the system under test can be corrected. The reasoning behind this is when we are testing a complex system we make assumptions as we test and the system may appear to confirm our assumptions and as such fuel what could be incorrect interpretations of the system. A computer system will never be able to inform you that your assumptions are wrong or right it could indicate a bias one way or another. The only way to repair errors in interpretations is to interact with a human being. This is the reasoning why debrief is very important so that any assumptions can be challenged and if necessary corrected.

As humans we are very good at being adaptive and changing our viewpoint and opinion when presented with new information but to do this effectively it needs to be a conversational setting, we are very bad at dealing with delayed feedback and the longer it is left the more likely we will keep our initial bias and interpretations.

The point of this rather short blog post is to explain why debrief after a testing session is important and that it needs to be done as soon as possible. Delays and excuses only cause more assumptions and incorrect information to appear to be the correct answer.

Make the time to debrief, plan for it and use it, it is crucial element of testing.

Autistic Software

When we start to test a system as testers we normally start with a lot of assumptions predicting how the software will act dependant on certain inputs. However how the software reacts will be interpreted differently depending on who is doing the testing. This is especially true when testing UIs or systems that depend on human interaction.

I recently read an * article by Danah Boyd in which they described a problem in the way in which software is developed without much thought being placed upon what the user needs or practices. This article was first published in 2004 and I still feel it is very relevant to software development today.
*(http://www.danah.org/papers/AutisticSocialSoftware.pdf)

More and more software is being developed which requires interactions with humans and of course as we as software testers know humans are fallible and unpredictable. The problem with developing software is that it is very simple and easy to do the complex mathematical formula stuff since this is what a computer is good at, in reality a glorified calculator. As testers finding problems in these logical areas should be simplistic. This is not to disrespect the skill of testing, but knowing the formula used and without any UIs it is a fairly easy task for a skilled tester to determine if a problem exists. IMO the problems start to appear when we develop software that requires interaction with humans. I am not aware of any research done in this field of software development but I wonder how much time is spent creating and fixing UI systems. If we look at the characteristics for a given piece of software we can see that it is very good with numbers but has poor social interaction skills. This is very similar to the characteristics of autism:

• "Socialising doesn't come naturally - we have to learn it."

People with autism often have difficulty recognising or understanding other people's emotions and feelings, and expressing their own, which can make it more difficult for them to fit in socially. They may:

• not understand the unwritten social rules which most of us pick up without thinking: they may stand too close to another person for example, or start an inappropriate subject of conversation
• appear to be insensitive because they have not recognised how someone else is feeling
• prefer to spend time alone rather than seeking out the company of other people
• not seek comfort from other people
• appear to behave 'strangely' or inappropriately, as it is not always easy for them to express feelings, emotions or needs.
• Difficulties with social interaction can mean that people with autism find it hard to form friendships: some may want to interact with other people and make friends, but may be unsure how to go about this.
(http://www.autismsussex.org.uk/training/WhatIsAutism/characteristics_of_autism)

So as testers what can we do to try and improve the poor human interaction skills of software? Are there any approaches that can help this? Another area I have been researching is Ethnography (http://en.wikipedia.org/wiki/Ethnography) and how similar this is to software testing. (More on this in a future blog)

One of the many things I liked in the article by Danah was how the software was being designed to encourage people to have different identities so that they could protect their identity. This is so much against human nature and Danah expressed their concerns with the following:

Why on earth should we encourage people to perform a mental disorder in the digital world?

Quite a statement!!!

I think from a testing perspective that we need to ensure that software being developed meets the practices and needs of the user rather than the technology needs of the software. We really should be trying to make the software more socialable and be able to interact with a variety of different human types. We should look to understand the user and what their needs are rather than force them to work in a particular way. We should look at what is most suitable for the user. If we can manage to do this then we can start to produce and release software that does not frustrate and annoy users. This maybe a Utopia belief but as testers I feel we are the ones that need to drive this way of thinking. So the next time you are testing a UI and it starts to frustrate you ask yourself why. Do not put up with autistic software, teach it to become more socially aware.

Sour Grapes

People who follow my blog may already know that I have an interest in psychology and how it can impact our thinking in relation to testing. Previously I have written articles on emotions , feelings and confirmation bias. I intend within this article to look at Cognitive Dissonance and how much of a help or a hindrance it can be for a tester.

There have been many studies on Cognitive Dissonance and if it really does exist however it would be best to describe what it is first. Wikipedia describes in it very simple terms:

Cognitive dissonance is an uncomfortable feeling caused by holding conflicting ideas simultaneously.

(http://en.wikipedia.org/wiki/Cognitive_dissonance)

The history of Cognitive Dissonance is accredited to Aesop’s tale of the Fox and the Grapes where the fox who wants to eat the grapes cannot reach them and as such to remedy the conflict that he really does want the grapes decides that the grapes are either not ripe or too sour, hence the title of this blog.

My thoughts on this conflict of beliefs or opinions and how we adjust to resolve the conflict is something that I feel could benefit testers especially when carrying out exploratory testing.

As testers I think it is important we recognise when we are experiencing Cognitive Dissonance and learn to ask questions before we make a decision to resolve the conflict. The problem with experiencing cognitive dissonance is that it is very easy to change our opinion and belief and suddenly we are drawn into the trap of confirmation bias.

As humans we do not like the feeling of conflict within our minds and we try to make a decision to resolve this conflict. Once we have made that decision we will try to justify the reasoning for making the choice we did.

For example if you had two areas to test and each area had the same level of important and you choose area A instead of B. You will now subconsciously favour choice A more than choice B. Now if someone comes up to you and says Area B is more important you have a conflict and your mind needs to give reasons as to why you choose A. You may make statements such as well at least Area A has been tested, it was important for me to test it. You will try to justify the reasoning for making what appears in your mind is the wrong choice.

When testing if we get two conflicting oracles and you need to make a decision on what is correct or the product owner needs to make a decision, cognitive dissonance could come into play. A decision could be made which is wrong. Only the problem now comes at a later date when you need to justify why you made that decision you will change your opinion or belief to make sure that the decision you made was the correct one.

It can become worse if you use a rating system to rate identical items. Imagine you are in a team and the team is given the task to rate features to be tested or developed. These ratings are then used to determine the order in which features for the product are developed. You then make a choice to test/develop your most highly rated feature, at a later date you get to rate the list again the item that you rated as next best has suddenly become of low importance even though it has the same value as the previous highly rated item. Making a choice after rating affects your value/belief/opinion of the item of same identical value and you start to score it lower to stop your uneasy feeling or cognitive dissonance. Project manager and team leads need to be aware of this since something of high importance could be downgraded by a team because of the issues of cognitive dissonance.

So how else can this affect testing?

I believe it can actually harm what we are testing. If the belief we have for the software we are testing in what it should be doing contradicts what it is doing and we adjust the conflict to justify what the software is doing there is a danger we could miss an important bug.

Further reading:
http://tip.psychology.org/festinge.html
http://www.colorado.edu/communication/meta-discourses/Theory/dissonance/
http://arstechnica.com/science/news/2010/12/this-is-your-brain-undergoing-cognitive-dissonance.ars

___________________________________________________________________________________

This is going to be my last post before the holiday season so I would like to say to all readers old and new have a happy holiday and look forward to writing again in the New Year.

Sorting the chaff from the wheat

One of the questions posed at Eurostar 2010 hot topics panel session (link) was:

What’s the most important skill for a tester?

Michael Bolton (Link) gave a witty reply of

Recognizing there is no most important skill.

Whilst I agree with Michael that testers need a wide variety of skills it got me thinking about skills that all testers need or should have. One of these is being able to deal with the vast amount of data that everyone has to deal with. How do we deal with?

I have recently been re-reading the classic HG Wells story “The Time Machine (Link) and started to think about the two communities with the story, the Eloi and the Morlocks and how society in general is getting so much information that it is starting to make us dumb, this IMO is a dangerous thing for testers.

Are we as a society becoming like the Eloi?

They have access to amazing technology that helps with all their needs however they come across in the book as dumb and lack curiosity. They see no need to be thinkers or philosophers. It appears technology is making things easier and easier for us.

When I was young (in the very old days) to find anything out I used to have to read a book. Since books were an expensive item I used to have a list of books I wanted for my birthday or Christmas and I used to visit the local library every week to sit and read and learn new fantastic things. I would get lost in a world of fantasy and knowledge; even then I had a thirst for learning which fortunately has never left me. It is such a shame that local libraries all over the world are shutting due to technology. (Do a Google search for News and Libraries and Closing)

Now information is available at the click of a button. We can find information on how an airplane works or the theory of relativity in an instance. Technology has made all this information easier to get, however IMO it has made us think less.

Do we still question all this information?

As testers we know we should be questioning everything, we learn to sort the chaff from the wheat as the title of this blog post implies. (Link). With such a wealth of information that is so easy to gather the skill is to be able to collate this information and remove the distractions. How easy as testers do we find this? I find it a natural thing I appear to do without thinking until I started to write this blog article. My concern is that with so much information do I end up throwing away something that is later proves to be vital or important.

Does anyone out there in the testing community have a method they use to help with this?

How do we not forget everything?

This then leads on to the topic of self learning – how do you select what to read and what not to read? How do you ensure you do not miss a really important article that has been blogged?

One approach I use for self learning is to use twitter and the software testing club (Link) within these communities’ people we talk about blogs they have read or recommend to read, the power of the crowd. This helps to reduce the amount of information I have to process. Another approach is to actually talk to people; humans are wired to be better at absorbing information via speech than from the written word, it is more likely to be remembered.

My other concern with all this information is our ability to remain focused, another important testing skill. With so much information to digest it is so easy to get into the habit of just scanning the information and not reading the whole article (I wonder how many people will get to this part of the article?). It is so easy to just start an article and get distracted by some other piece of information and not return to the original article. I sometimes think I should not add any hyperlinks to my articles and just add them to the end but I want to credit the people who inspire me or provide me with information as I write the article, it is one of those things which is important to me. A perfect example of this was the recently article Michael Bolton wrote about estimation (Link) which was a five part article, how many people read the whole of the article? It is so easy to skip or scan and miss an important point within an article and the same can be applied to testing. If we scan and miss something it could be that the thing we missed will cost us a lot of money.

My other concern is that we are becoming a society of 24x7 learners, we never switch off.

Are you one of these people?

My concern on this came from a conversation in which I stated I do hobby as a job, this scared me. I have a passion for testing and learning but am I not in danger of burning myself out or forgetting valuable knowledge unless I switch off?

How many others reading this blog switch off and pursue other interests outside of technology?

If you take nothing from this article please do switch off. I have hobbies that have nothing to do with computers. I enjoy being creative and take photographs, spending time outside at stupid o’clock catching sunsets and sunrises and landscapes. I enjoy growing things and spend time in my garden. I am fortunate to have a very large garden in which to grow and nurture things. I also have a family and I am a grand parent and spending time with my granddaughter is such a wonderful thing to do. We call her our little time waster – since time can go so quickly when you are engaged in playing. After all these hobbies it is surprising that I have time to do my job or to keep learning but I come back to my work more energized and ready to learn more.

Since starting to write this article I have found a couple of other blogs that mention the problems of attention span and remaining focused they can be found here:

http://cultureandcommunication.org/f09/tdm/sara-hardwick/attention-span-in-the-internet-age-information-overload-memory-and-teal-deers/

http://www.theatlantic.com/magazine/archive/2008/07/is-google-making-us-stupid/6868/
I do recommend people reading them.


PS I will leave it to your imagination who I think the Morlocks are :o)

The Human Element

I attended the Eurostar Testing Conference (http://www.eurostarconferences.com/conferences/2010/) in Copenhagen, Denmark this year and met a large group of very interesting people. A few highlights for me were:

Meeting the Cartoon tester (http://cartoontester.blogspot.com/) in person, a friendly unassuming guy with a quick sense of humour.

The other highlight was amount of ‘real life’ examples of exploratory testing and session based testing management. One of the best things I took away was from Carsten Feilberg’s talk on Session-Based Testing in Practice (http://carstenfeilberg.blogspot.com/) in which he reframed the wording of SBTM to Managing Testing Based upon Sessions. It was a why did we not think of that before!!!!

One of the keynotes was by Stewart Reid on “When Passion Obscures The Facts: The Case for Evidence-Based Testing” in which he looked at what testing could learn from Evidence Based Medicine (http://en.wikipedia.org/wiki/Evidence-based_medicine) . During the presentation I thought I could see many flaws in the argument he was trying to put together but could not quite work out what it was. One thing I have found out since and one point that Stewart did appear to miss was the work of the GRADE Working Group which is a newer system (and appearing to gain ground). The principles here are based upon Extrapolations (http://en.wikipedia.org/wiki/Extrapolation).

To quote from Wikipedia:

Extrapolations" are where data is used in a situation which has potentially clinically important differences than the original study situation. Thus, the quality of evidence to support a clinical decision is a combination of the quality of research data and the clinical 'directness' of the data.

Interestingly the data gather for extrapolation are more based upon human experience rather than just a set of numbers. Is it just me or is this like running a set of known tests then exploring afterwards? See my previous post on Hybrid testing (http://steveo1967.blogspot.com/2010/09/hybrid-testing.html)

So why have I called the title of this blog post “The Human Element”?
I was having a conversation with my wife(Tracy) after the conference since she is a retired theatre nurse and understand the medical arena very well and she came up with a wonderful phrase. It is all well and good having all these numbers and statistics but you cannot ignore the human element. She gave an example of this in which a nurse working in Intensive Care has a lot of machinery (with installed software) at her disposal however none of this equipment can tell her if the patient is feeling happy or sad or is uncomfortable.

Tracy said the problem is no machines have a soul they do not care how the patient is feeling, the machine could be saying everything is ok but the nurse and their compassion knows and understands how the patient is. I asked my wife to have a talk with Stewart and some other testers including Lynn Mckee (http://www.qualityperspectives.ca/)

This provided a wonderful insight to me in that we as testers forget that there are lots of people who we should be using as oracles for when we test a system we should not be forgetting about the human element.

During her conversation Stewart started to mention the use of statistics as evidence and for making healthcare decisions (Cochrane Library - http://www.thecochranelibrary.com/view/0/index.html?s_cid=citation) and Tracy said that to get to the point of making a decision still requires the GP to ask questions and to explore all possibilities. At the end of the day it is just statistics said Tracy and it does not help in a situation in which a perfectly healthy 20 year old is prescribed a drug for a problem and then dies due to an undetected heart problem. No amount of journals, evidence can account for this, since it is on a personal level between the patient and the medical expert.

The final conversation I remember Tracy having was with Lynn and a few other and it is very useful for testers when they come up against the problem of ‘It should do this.’
Tracy talked about Dr Spock and the book about the development of children (http://en.wikipedia.org/wiki/Benjamin_Spock) and that at certain age’s children should be doing this and that. This book causes major worries in parents when their child does not meet the timescales within the book for talking, sitting up, walking etc. Tracy then made a point which caused a great amount of laughter. “People seem to forget that babies have not read the book – they will develop at their own pace”

I found this a wonderful piece of insight, we seem to forget that everyone is different and if we apply this to software and the development of software we start to realise that every piece of software is different and that we need to explore the software and play with it to get the full potential out of the software.

To conclude this post I would like to say a big thank you to my wife Tracy for her encouragement and support in what I do and for giving us testers a lesson in remembering about the human element in what we do.