This article was first publish in Jan 2015 on the Ministry of Testing Website - http://www.ministryoftesting.com/2015/01/qualitative-research-coding-software-testing/

Qualitative Research Coding and Software Testing

Coding is used extensively in the field of social science for qualitative analysis.  Coding is not defined in the same ways as it is for software development.  The following quote by Saldaña provides a useful definition of coding within the social science world:

A code in qualitative inquiry is most often a word or short phrase that symbolically assigns a summative, salient, essence capturing, and/or evocative attribute for a portion of language-based or visual data  [1]

“By “language-based or visual data”, Saldaña means video, audio recordings or written notes.   Once the information has been gathered the researcher then assigns ‘a word or a short phrase’ to represent sections of the data. Then the researcher examines and categorises the codes to see what patterns and theories emerge.  These activities together are known as qualitative research coding.   For example if a researcher were studying the morals of teenagers, the researcher would record conversations and then use coding activities to see what patterns emerge.  Based upon these patterns the researcher would form theories about the morals of teenagers.

Testers using this type of coding may find it helps to label; organise and classify their testing observations.  It provides structure not only to the testers’ observations, but also to the process of observing.  Observation and gathering information are important aspects of testing.

The changing minds website describes qualitative coding as:

Coding is an important technique in qualitative research such as anthropology [2), ethnography [3] and other observer and participant-observer methods. [4]

Strauss stressed the importance of coding when carrying out qualitative analysis:

Any researcher who wishes to become proficient at doing qualitative analysis must learn to code well and easily. The excellence of the research rests in large part on the excellence of the coding. [5]

The same can be applied by testers when analysing their testing effort, there is a need as a tester to be able to ‘code well and easily’. Testers should examine the testing evidence that has been gathered and use coding to formulate flexible theories and ideas of the behaviour of the software.

There appears to be some comparisons between qualitative research coding and exploratory testing [6]; the table below describes some of these comparisons.

Social Science	Exploratory Testing
Evidence based upon immersion in the culture under investigation	Evidence based upon exploring the software under test
Theories formed based upon the evidence gathered during immersion	Theories formed based upon the evidence gathered during exploration
Theories adjusted and altered as new evidence is gathered	Theories adjusted and altered as new evidence is gathered
Further investigations take place to uncover evidence that may support or disprove current theories	Further exploration takes place to uncover evidence that supports or disproves current thinking about the behavior of the software under test
Coding and classifications of evidence to identify patterns	Coding of evidence from exploratory sessions to identify patterns and risks

Table of comparisons between social science research methods and exploratory testing

Testers can utilise social science coding to analyse the testing evidence gathered to help form theories about the behaviour of the software. These theories can be presented to stakeholders to provide valuable information and support decision-making.

When people first start to use social science coding they can find it complex and daunting.  To help simplify coding Chris Hann created the following diagram:

Coding pyramid [7]

The diagram shows the different levels of coding that social scientists go through to form and reform theories.

Testers can use social science coding not only for test execution, but also for other testing activities such as test planning, testing discussion and test reporting.

Coding in action

The following is an example of using level 1 coding for user stories.

User Story

As a user

I want to login in securely

So that my private information is kept private

A tester using coding uses the following codes for this user story. (Each code is separated by the | symbol)

Codes

Security | Login | Operations |Function

The tester then looks at more user stories and codes:

User Story

As a user

I want to record a currently playing live show

So that I can watch the show at a later time

Codes

Recording | Live TV | Device Remote | Operations | Data |Time | Platform | Functions

User Story

As a user

I want to playback a currently recorded show

So that I can watch the recorded show now

Codes

Recording | Playback | Device Remote | Operations |Time | Data | Structure | Interface

The examples above make use of the SFDPOT [8] heuristic.  This heuristic is a useful way of utilising social science coding to identify testing coverage gaps.   When coding it is acceptable to assign multiple codes to each piece of information or evidence gathered.

1. The process of coding can be defined in the following steps: (taken from [9])
2. Decide which types of coding are most relevant
3. Start coding
4. Create a start list of codes
5. Generate categories (pattern codes)
6. Test these categories against new data (start with contrasting data early on!)
7. Write about categories/pattern codes in a memo to explain their significance

The following is an example of a tester using these steps:

The testers’ initial testing effort showed that a particular API appeared to work correctly with the customer data set used.  The tester coded this as ‘API interface: customer data set ingests’.  The tester formed a theory that the API had been implemented and appeared to work for that data set. The tester then tested their theory using a different customer data set and the behaviour was inconsistent, this led to a change in the testers’ theory based upon the evidence gathered. This is level 2 and 3 on the coding pyramid diagram.

Memos and questioning

Testers when carrying out coding can use ethnographic research methods [10] by asking themselves the following questions.

• What is the software doing?
• What is the software trying to accomplish?
• How does the software accomplish this?
• Does the user understand what is being accomplished?
• What assumptions does the software make about the user?
• What surprises you about how the software is behaving?
• What do I see going on here? (To track your assumptions)
• What did I learn from the notes I have taken?
• What intrigued me? (To track your positionality [11] – where do you stand, what biases could be at play?)
• What disturbed me? (To track your tensions, beliefs, attitude)

These questions have been adapted based upon work by (Sunstein and Chiseri-Strater, 2007: 106) Field Working: Reading and Writing Research. [12]

Categorisation, level 4 on the coding pyramid, is the concept of tying together a number of observations and seeing if they have any common characteristics. These characteristics or connections in social science coding terms are called memoing.

Glaser defines memoing as:

(A memo is) the theorising write-up of ideas about codes and their relationships as they strike the analyst while coding… it can be a sentence, a paragraph or a few pages… it exhausts the analyst’s momentary ideation based on data with perhaps a little conceptual elaboration (Glaser, 1978: 83) [13]

The following gives a testing example of this:

After testing a User Interface, the tester defines the following codes from the gathered testing evidence:

UI inconsistent | error messages unclear | undermined feature behavior

The tester memos these codes as ‘unwarranted user behaviour.’ Ideally though, the tester should write more detailed memos than the example given.  The purpose of a memo is to write down your thoughts and reflections on the information you have gathered.

The more extensive the testers’ memos are the more they can form concrete theories.  This aspect of coding can prove to be valuable for testers, since the completed memos can be used as evidence of what the system they are testing is really doing.

Coding can be used when analysing the evidence gathered during testing; the tester can group together and code similar behaviours and observations to see if there are any patterns.  These patterns can be used by the tester to guide their testing effort and provide focus for areas that they may find useful to explore.

Ian Day in his book Qualitative Data Analysis – A user friendly guide for social scientists [14] makes the following statement about coding.

The term ‘coding’ has a rather mechanical overtone quite at odds with the conceptual tasks involved in categorising data. This arises from the association of coding with a consistent and complete set of rules governing the assignment of codes to data, thereby eliminating error and of course allowing recovery of the original data simply by reversing the process (i.e. decoding). Qualitative analysis, in contrast, requires the analyst to create or adapt concepts relevant to the data rather than to apply a set of pre-established rules.

Since coding is a conceptual approach, having pre-defined rules or processes before the tester has the evidence is not something that fits easily in to a qualitative research approach.   Codes are relevant to the actual evidence that has been gathered rather than based upon assumptions of what that evidence may be.

Summary

The use of qualitative coding in testing provides many benefits that testers can utilise across all testing activities, some of these benefits include:

• Pattern Recognition
• Theory Forming
• Critical Thinking
• System Thinking
• Testing Gap Analysis
• Meaningful reporting
• Evidence gathering

Learning about social science coding and applying it in practice can be a powerful tool in a testers testing skills portfolio.  Those who promote that testers should learn to code are correct; nevertheless they may want to explore alternative coding approaches.

For those who wish to learn more about the connection between social science and software testing I recommend the following articles and books as useful starting points:

• A complete beginner guide to qualitative analysis – Taylor and Graham R. Gibbs University of Huddersfield (web site)
• Analysis of Qualitative Data for Beginners – Demola Akinyoade – Department of Peace and Conflict Studies Afe Babalola University, 30th June 2005 (Web PDF)
• Software Testing as a Social Science – Cem Kaner – STEP 2008 (Slides)
• What Testers can learn from Social Sciences – Huib Schoots – TestBash 2.0 – 2013 (Video)
• What Testers can Learn from Social Sciences – Part 1 – Huib Schoots – 2013 (Series of Web Articles)
• Looking to Social Science for help with Metrics – Justin Rohrman – CAST 2014 (Video)

With special thanks and appreciation for their help and patience in making this a much more complete article than it was to begin with:

• Michael Bolton: http://www.developsense.com/ Twitter: @michaelbolton
• Huib Schoots: http://www.huibschoots.nl/wordpress/ Twiiter: @huibschoots
• Duncan Nisbit: http://www.duncannisbet.co.uk/ Twitter: @DuncNisbet

References

1. The Coding Manual for Qualitative Researchers – Saldaña, 2013 –http://www.amazon.com/The-Coding-Manual-Qualitative-Researchers/dp/1446247376
2. What is Anthropology – American Anthropological Association (website – Last accessed (Aug 2014) – http://www.aaanet.org/about/whatisanthropology.cfm
3. Brian A. Hoey. “A Simple Introduction to the Practice of Ethnography and Guide to Ethnographic Fieldnotes” Marshall University Digital Scholar (2014): 1-10. Available at:http://works.bepress.com/brian_hoey/12
4. Changing Minds – Ethnographic coding –http://changingminds.org/explanations/research/analysis/ethnographic_coding.htm
5. Qualitative Analysis for Social Scientists, 1987, p. 27 – Anselm L. Strauss –http://www.amazon.com/Qualitative-Analysis-Social-Scientists-Strauss/dp/0521338069
6. What is Exploratory Testing – James Bach –http://www.satisfice.com/articles/what_is_et.shtml
7. Techniques and Tips for Qualitative Researchers – Chris Hann –http://qrtips.com/faq/FAQ–code%20terms.htm
8. How Models Change – Michael Bolton –http://www.developsense.com/blog/2014/07/how-models-change/
9. 8 Qualitative codes and coding (2014) – Heather Fordhttp://www.slideshare.net/hfordsa/qualitative-codes-and-coding
10. Are testers’ ethnographic researchers? Stevenson (2011) –http://steveo1967.blogspot.com/2011/01/are-testers-ethnographic-researchers.html
11. What is positionality in practitioner research? – Dissertation Scholar (website Last accessed August 2014) http://dissertationscholar.blogspot.mx/2013/04/what-is-positionality-in-practitioner.html
12. Field working Reading and writing research – Sunstein and Chiseri-Strater (2007: 106) – http://www.amazon.com/FieldWorking-Reading-Writing-Research-Edition/dp/0312622759
13. Theoretical Sensitivity: Advances in the Methodology of Grounded Theory – Glaser (1978: 83) http://www.amazon.com/Theoretical-Sensitivity-Advances-Methodology-Grounded/dp/1884156010
14. Qualitative Data Analysis: A User Friendly Guide for Social Scientists – Ian Day (1993)http://www.amazon.com/Qualitative-Data-Analysis-Friendly-Scientists/dp/041505852
    X

Is testing a manufacturing process or a scientific approach?

Question

Does testing need to move away from manufacturing processes and more towards scientific approaches?

If we take a look at the world of manufacturing you can find many standards (ISO and others) for various everyday things.  However even within the same subject field there can be many competing standards.  For example look at the standards for mains electricity around the world.  Which standard do you use? 

http://en.wikipedia.org/wiki/Mains_electricity_by_country

What about light fittings in your country how many different types are available?

http://en.wikipedia.org/wiki/Edison_screw 

http://en.wikipedia.org/wiki/Bayonet_mount

In some cases these standards have been changed to meet local and cultural differences in others it has been a question of business and not wishing to pay a royalty fee. However in all these examples context plays a part in the adopting of a standard. Maybe I am being unfair and using examples that are not typical.  From the research I have done I find the same within the QC field, depending on what is being manufactured the processes and standard can and do change for similar products. 

The xkcd comic strip shows this very clearly.

(c) http://xkcd.com/927/

The problem I have within the domain of software testing is that when we apply manufacturing processes to software testing we are making many assumption. The biggest one being

 all software is the same and behaves the same 

which is not true. 


As such we require different approaches to deal with this which is in context with the software being developed rather than trying to make it fit to processes designed around the assumption that everything will be done exactly the same way.

My concern is with the new ISO software testing standard ISO/IEC 2911  which appears to be based on manufacturing process and practices.

The question I have is why? 

Software testing in my opinion is not a manufacturing driven process but more a scientific experimental approach in which the tester has questions and theories that they wish to prove or disprove.So why try and tie a thinking process down to a checklist style document driven process?  Is it because 'management' can have a false belief that it more easily to manage or am I being sceptical?

Looking at the field of science and research - yes I acknowledge there are still some processes but these seem to be based upon what has been done and experienced rather than up front unnecessary documents.  I see very little about processes more a case of techniques and approaches to use. 

http://en.wikipedia.org/wiki/Wikipedia:Scientific_standards

If you then add in other fields of science such as social and anthropology you have a wide range of approaches that lend themselves to testing. I see testers more as scientists, researchers or investigators looking at debunking their theories, biases and understanding and trying to learn more about what they are testing.

One approach I stumbled across was the following: http://www.respectproject.org/code/cstds.php?id and I think this could lead to another blog article there is so much useful information here for testers.



Quoting a couple of them:

• ensure factual accuracy and avoid misrepresentation, fabrication, suppression or misinterpretation of data
• critically question authorities and assumptions to make sure that the selection and formulation of research questions, and the conceptualisation or design of research undertakings, do not predetermine an outcome, and do not exclude unwanted findings from the outset 
• ensure methodology and findings are open for discussion and full peer review

I found this code of practice very similar to the ethics I try to employ when carrying our exploratory testing, as I have already stated I may come back to this with another article at a later date.

I remember in school during science lessons we would:

• Start from a theory
• We would then test that theory
• Write a conclusion based upon what we did 
• Report what we found.

In my mind this is like exploratory testing

• We have a charter and a mission
• We try to prove the mission right or wrong
• We write about our discoveries and what we learn
• We report what we find
• We check to see if what we find matches our original mission

We prove sufficient information for our peers to be able to replicate what we did and see if they come to the same conclusions or not.  We do this by treading similar ground but sometimes not exactly the same steps.

The similarities between exploratory testing and scientific research appear to be many.

• Formulate a theory
• Test the theory
• Explore ways to prove the theory is incorrect (Peer testing)
• Report your findings

So to conclude I feel that exploratory testing and testing in general has more to do with the sciences than it has to do with manufacturing.  The sooner we can move testing away from a manufacturing process centric way of working to a more natural scientific way of working the better the world of testing will be.  We really must stop testing being forced into a process in which people can tick a box and say yes we have a document for that and for that and that and of course that.   


Testing is a thinking activity not a document creation, ticking boxes checking process.