Ethnographic research feedback

Sometime ago I wrote an article about the relationship between ethnographic researchers and testers and how similar they are.  Recently Peter H-L (@Unlicensed2test) on twitter reminded me that I had also presented at the UNICOM  conference on using some aspects of ethnographic research to aid feedback when we are testing and from this I came up with a new mnemonic and a set of testing related social science questions.   I had thought that I had already posted this but it seems I had not.

What follows is taken from the talk I did.

*************

Within the article there was a section that dealt with questions that the researcher should be asking when studying the subject.  I changed this to make it relate to software testing and came up with the following:

• Substantive Contribution: "Does the testing carried out contribute to our understanding of the software?"
• Aesthetic Merit: "Does the software succeed aesthetically?" Is it suitable for the end user?
• Reflexivity: "How did the author come to write this test…Is there adequate self-awareness and self-exposure for the reader to make judgements about the point of view?"
• Impact: "Does this affect me? Emotionally? Intellectually?" Does it move me?
• Expresses a Reality: "Does it seem 'true'—a credible account of a requirement'?"

Lynne Mckee has been updating a list of testing mnemonics on her blog site  so I thought about this and came up with the following mnemonic:

R.A.I.S.E

From this I created a list of questions under each of these heading that can be used to aid feedback when you have been testing, ideally when you are following session based test management.


Use the following template to do a personal review of the testing that you carried out during the day.
Please try not to answer using yes and no, expand on your reasons for either it being yes or no.
This debrief/review is more about your views, opinions and feelings rather than the product you have been testing.
It should only take you 10 minutes to complete this feedback – try not to write essays.

_______________

Reflect

Personal reflection:

• Could you have done things better if so what? (Both from a personal and testing perspective)
• Have you learnt new things about the product under test (That are not documented)?
• Has your view of the product changed for better or for worse? Why has your view changed?

‘Epistemological reflexivity’ (What limits did we hit?)

• Did your defined tests limit the information you could find about the product?  (Did you need to explore new areas that you had not defined)
• Could your tests have been done differently? If yes how?
• Have you run the right tests?
• If you did things different what do you think you would have found out about the product?
• What assumptions have you uncovered to be true/false?
• Did the assumptions you make impede or help your testing?

Aesthetic:

• In your opinion is the product suitable for the end user?
• In your opinion is the product appealing at first look?
• In your opinion is the product confusing?
• In your opinion does the product flow?
• In your opinion are there any ugly areas?
• In your opinion does the product succeed aesthetically? Does it meet the image the customer is trying to portray?

Impact:

(this section is intended to be used to say how you 'feel' about the product, your first impressions, if you answer yes you should provide more details)

• Does this affect you?
• Emotionally?
• Intellectually?
• Does it move you?
• Does it cause you negative/positive feelings?
• Does it frustrate you?
• Does it annoy you?

Substantial:

• Have we covered a substantial amount of the key product areas?
• Has the testing contributed to your understanding of the product?
• Do you think you have a substantial understanding of the system and sub systems?
• Does your knowledge of the system have any substantial gaps?
• Could you easily explain the system to a first time user?

Expression:

• Does the product seem 'true'—a credible account of a requirement'?
• Does the product express what will happen in ‘real’ world?
• Does the reality of the product match the expectations of the product?
• Does the product express unexpected ways of working?

_______________

To make it easier I have create a MS word document with the questions in which you can download from Google docs here.

*************

An update or two.

I noticed that I have not written a blog article in awhile so I thought I would put together a short article on what I have been up to so that regular readers can be sure that I am still alive and well.

On the personal front we have had a few health scares over the past month hence my lack of tweeting or blogging.

On the work front I have been very busy and involved in a few different and exciting projects while continuing to look at different ways in which we can improve.

During this period I have been looking more and more into ethnographic research and its connection to testing. I find this area of social science fascinating and how much it appears to collate to testing. Since there does appear to be a connection to this I am current running a couple of case studies internally based upon methods from ethnographic research as mention by Richardson in their article for Qualitative Inquiry: Evaluating ethnography

The findings for this case study will be presented at the UNICOM Next Generation Testing conference 18/19 May 2011

If you cannot make this event I do intend to give a very basic/quick introduction to this approach the Software Testing Club meet up in Oxford on the 14th April 2011 This event will be used as a world premier for the approach I have been working on so that definitely makes it worth attending. Or the fact that Lisa Crispin and Rob Lambert will be there should tick everyone’s box.

Without giving away too much detail before the meet up here is a brief summary of the approach I have been investigating

• The concept is based upon questioning the tester as much as you question the product being tested.
• It is check-list that can be used on an individual basis and should take between 5 and 10 minutes. The idea is to look at what you are doing and checking it is the right thing and see if you missing anything.
• I will be giving away the check-list on the evening of the meet up. (wow a freebie)

Have I given away too much information, not enough or left you wanting more?

If you want to know more then I suggest you sign up to attend the meet-up or the UNICOM conference