Ethnographic research feedback

Sometime ago I wrote an article about the relationship between ethnographic researchers and testers and how similar they are.  Recently Peter H-L (@Unlicensed2test) on twitter reminded me that I had also presented at the UNICOM  conference on using some aspects of ethnographic research to aid feedback when we are testing and from this I came up with a new mnemonic and a set of testing related social science questions.   I had thought that I had already posted this but it seems I had not.

What follows is taken from the talk I did.

*************

Within the article there was a section that dealt with questions that the researcher should be asking when studying the subject.  I changed this to make it relate to software testing and came up with the following:

• Substantive Contribution: "Does the testing carried out contribute to our understanding of the software?"
• Aesthetic Merit: "Does the software succeed aesthetically?" Is it suitable for the end user?
• Reflexivity: "How did the author come to write this test…Is there adequate self-awareness and self-exposure for the reader to make judgements about the point of view?"
• Impact: "Does this affect me? Emotionally? Intellectually?" Does it move me?
• Expresses a Reality: "Does it seem 'true'—a credible account of a requirement'?"

Lynne Mckee has been updating a list of testing mnemonics on her blog site  so I thought about this and came up with the following mnemonic:

R.A.I.S.E

From this I created a list of questions under each of these heading that can be used to aid feedback when you have been testing, ideally when you are following session based test management.


Use the following template to do a personal review of the testing that you carried out during the day.
Please try not to answer using yes and no, expand on your reasons for either it being yes or no.
This debrief/review is more about your views, opinions and feelings rather than the product you have been testing.
It should only take you 10 minutes to complete this feedback – try not to write essays.

_______________

Reflect

Personal reflection:

• Could you have done things better if so what? (Both from a personal and testing perspective)
• Have you learnt new things about the product under test (That are not documented)?
• Has your view of the product changed for better or for worse? Why has your view changed?

‘Epistemological reflexivity’ (What limits did we hit?)

• Did your defined tests limit the information you could find about the product?  (Did you need to explore new areas that you had not defined)
• Could your tests have been done differently? If yes how?
• Have you run the right tests?
• If you did things different what do you think you would have found out about the product?
• What assumptions have you uncovered to be true/false?
• Did the assumptions you make impede or help your testing?

Aesthetic:

• In your opinion is the product suitable for the end user?
• In your opinion is the product appealing at first look?
• In your opinion is the product confusing?
• In your opinion does the product flow?
• In your opinion are there any ugly areas?
• In your opinion does the product succeed aesthetically? Does it meet the image the customer is trying to portray?

Impact:

(this section is intended to be used to say how you 'feel' about the product, your first impressions, if you answer yes you should provide more details)

• Does this affect you?
• Emotionally?
• Intellectually?
• Does it move you?
• Does it cause you negative/positive feelings?
• Does it frustrate you?
• Does it annoy you?

Substantial:

• Have we covered a substantial amount of the key product areas?
• Has the testing contributed to your understanding of the product?
• Do you think you have a substantial understanding of the system and sub systems?
• Does your knowledge of the system have any substantial gaps?
• Could you easily explain the system to a first time user?

Expression:

• Does the product seem 'true'—a credible account of a requirement'?
• Does the product express what will happen in ‘real’ world?
• Does the reality of the product match the expectations of the product?
• Does the product express unexpected ways of working?

_______________

To make it easier I have create a MS word document with the questions in which you can download from Google docs here.

*************