Test Execution Model - updated

This is a follow on to the post I recently did about the automation pyramid that Richard Bradshaw gave at MEWT.

Following on from this some in the testing Twitterverse asked if Richard and I would do a video for the new initiative created by Richard called Whiteboard testing.  Instead of one video we decided to do two.  The first looked at the history of the test automation pyramid and you can watch it here: A look at the test automation pyramid.  The second video was a chance for me to present my own model which I have called the test execution model and can be seen here: - John Stevenson Test Execution Model.


After the video was posted I had a message from Dan Ashby who wanted to discuss more with me about the  test execution model and what he felt was missing.  We arranged a chat and discussed the model and broke down a few assumptions we were both making.  The outcome from this chat was the need to add a few extra parts to complete the test execution cycle.  With thanks to Dan I would like to present the next generation of the model.

The changes that Dan suggested for the model are as follows:

• As you test you turn some of your tacit knowledge into explicit knowledge which then can be added to your checks and become known known information.  This is represented by the flow from the testing arrow to the checking arrow at the top of the diagram.
• As you create more checks these in turn become oracles which you can use as heuristics for your future tests.  This is represented by the flow from the checking arrow to the testing arrow.

This then gives a cycle of test execution were the testing feeds into the checking which in turn can feed back into the testing.  It should be noted that is model is for test execution which does not only apply to only testing the product deliverable.  Testing is carried throughout the development cycle and as such testing activities can occur during artitecture & design discussion and during coding. Testing activities can and should occur anywhere during the development and deployment of the product and this model should be used in that context with regards to test execution.

Testing Skills #7 - Reflection

What type of learning do you need to engage in today?

Do you need to learn facts?

Or

Do you need to analyse and critically think about what you need to learn or have learned?
Neither of these ways of learning are wrong and each has its place in your learning journey.  Knowing which to use and when is a great tool for a tester to have.

Learning facts is defined as a 'surface approach ' style of learning and writing and thinking critically about what your have learned can be defined as a 'deep approach'.

Reflection is important in software testing and you need to be able to communicate what you have found and uncovered when testing.  If you have no opportunities,  in your organization, to be able to communicate with others then you may find you have a lot on tacit knowledge but little in the way of explicit.  It is vital to set aside time for members of the team to discuss what new information they have uncovered or learned when testing.  This allows them to use reflective reasoning to turn tacit into explicit and gives the person providing the information a sense of purpose and achievement in what they have learned.

To improve your own self-reflection produce a reflective action plan. By doing this you are committing to your self what you intend to do and engage critical thinking skills in doing so. Make sure your plan has goals and deadlines that you can commit to.  Make this deadlines and goals public it encourages you to keep to them.  Using a personal Kanban board can help you achieve this.

"First, the deep and surface approaches are not personality traits or fixed learning styles.  Students adopt an approach which is related to their perceptions of the task."

Marton and Säljö - Student Approaches to Learning - University of Oxford

When you need to think deeply about what you have been studying then writing down you own thoughts and understanding is a good way for you to be able to see what you have remembered.  If you only need to learn information or facts then other models would be more suited.

Writing down you thoughts on what you have been attempting to learn enables you to better remember or apply what has been studied.  Reflection is about doing your own internal self-assessment  of what you have been learning.  This assessment enables you make what you know internally, tacit knowledge, and attempting to make it clear and detailed explicit knowledge.  To do this you need opportunities to talk to others verbally or by writing down what you have studied to invoke critical thinking.

"In general, writing appears to be suitable for tasks where the aim is fostering understanding, changing students' conceptions and developing their thinking skills, but less suitable if the goal is the simple accumulation of factual information"

Conceptualizing and Measuring Knowledge Change due to writing

Schumacher & Gradwohl Nash (1991)

One way in which can improve your learning is to use reflection:

"Reflection is an active process whereby the professional can gain an understanding of how historical, social, cultural and personal experiences have contributed to professional knowledge and practice "(Wilkinson, 1996).  

You learn from your experiences and to make this happen you need to engage in reflection.  If you think about what you are doing or what you have experienced you are engaging in self-reflection. This approach helps you to turn learning into knowledge and it is vital in improving your own knowledge and skills.  Reflection is really about looking back at a situation, thinking about it, critically and learning from the experience, then using that new knowledge to help in future similar situations.  This in many ways is similar to the approach you will taking when engaging in testing activities.

"The process of reflective writing leads to more than just a gain in your knowledge; it should also challenge the concepts and theories by which you make sense of knowledge. When you reflect on a situation, you do not simply see more, you see differently. This different way of viewing a situation is reflected in statements about a commitment to action. Action is the final stage of reflection. "

Learning through reflection(Word doc)?

Testing skills #4 - Note taking

 Why is note taking an important testing skill?

There are a variety of ways to capture the evidence of our testing but if are notes are not of suitable detail then the value of our testing can be diminished.  Taking notes enables us to improve our knowledge and reinforces our understanding of the product being tested. This is part of utilizing critical thinking skills which was discussed in the chapter on 'critical thinking'

Robert Lambert discusses the need to have good note taking skills when performing exploratory testing

"During a session a good exploratory tester will often narrate the process; all the time making notes, observations and documenting the journey through the product. This level of note taking allows the tester to recall cause and effect, questions to ask and clues to follow up in further sessions." 

Explaining Exploratory testing relies on good notes - Robert Lambert - 2013

Michael Bolton wrote the following about note taking when testing:

"One of the principal concerns of test managers and project managers with respect to exploratory testing is that it is fundamentally unaccountable or unmanageable. Yet police, doctors, pilots, lawyers and all kinds of skilled professions have learned to deal with problem of reporting unpredictable information in various forms by developing note-taking skills." 

An Exploratory Tester’s Notebook  - Michael Bolton - 2007

There are a variety of note taking methods which you as a tester can utilize.  This page has an example of a few of them. Note Taking Systems - Student Academic Services - Cal Poly

One method that I have found extremely useful especially when capturing information from conferences or for recording my findings when testing is the 'Cornell Method'

The Cornell method was developed by Dr Walter Pauk of Cornell University and is widely used by University students.  It is a very useful method to help you work out if you can remember what you have written.

First of all you need to create a layout for each page in your notebook as follows. Alternatively use this Cornell Method PDF generator.

The method has 5 steps.

1. Capture what is being said or what you observe in the note taking area
2. As soon as possible review your notes and capture key details in the Review(Cues) column, add any questions you may have thought of.
3. Cover up your notes only showing the review column and now try to summarize your thoughts based on the cues.  Provide answers to any of the questions you wrote in the review column.  Use the summary column at the bottom to summarize your understanding and learning.  If you are struggling with your summary it could indicate that your notes are not sufficient.
4. Ask yourself questions on the material both the cues and the notes. Think about how you apply this information to your work.  How does it fit with what you already know?
5. Spend some time reviewing your notes and summary every so often to reinforce your understandings.

It is crucial that as a tester you practice your note taking skills.  Poor note taking can lead to missed problems and hinder knowledge sharing with the team.  Your notes are what helps to turn your tacit knowledge into explicit knowledge. 

Tacit and Explicit Knowledge and Exploratory Testing

"We can know more that we can tell". - Michael Polanyi(1966)

Having finally read the excellent book Tacit and Explicit Knowledge by Harry Collins which I personally found to make a significant impact on my thinking of how we learn and record information (knowledge).  The book is not an easy book to read and it took me a few times of re-reading some sections to work out what the author may have meant.  

I will start by saying that this article is based upon my own interpretation of the book and the links I make between what the author writes and what I connect with regarding testing are entirely my own and could be flawed.

So what do we mean when we say tacit and explicit knowledge means?

Harry Collins in his book describes in great detail what he means by these terms and I could not find a clear definition that would be useful for this article.  So I used some of the research references I had used when reading the book.  One of the better ones I came across was the following website 

Explicit Knowledge: Knowledge that is codified and conveyed to others through dialog, demonstration, or media such as books, drawings, and documents.

Tacit Knowledge: Deeply personal experience, aptitudes, perceptions, insights, and know-how that are implied or indicated but not actually expressed — it resides in individuals & teams.

The site also has a great diagram showing visually the difference in the amount of knowledge we have in each type with explicit being quite small part of our total knowledge and tacit being the majority.

Diagram from http://www.cognitivedesignsolutions.com/KM/ExplicitTacit.htm

 In her study of organisational knowledge L Wah stated the following:

According to Buckman of Buckman Labs,, 90% of the knowledge in any organization is embedded and synthesized in peoples’ heads. Unleashing this knowledge- which is tacit in nature-presents a big challenge. (Wah, 1999b;)

We will revisit this later

Another article I found which tried to define the terms was the following:

http://www.hec.gov.pk/InsideHEC/Divisions/RND/HLIB/PublishingImages/Explicit%20Knowledge%20vs%20Tacit%20Knowledge.pdf

Explicit knowledge is formal and systematic. It can be easily communicated and shared. Typically, it has been documented. Articulated knowledge, expressed and recorded as words, numbers, codes, mathematical and scientific formulae, and musical notations. Explicit knowledge is easy to communicate, store, and distribute and is the knowledge found in books, on the web, and other visual and oral means.

Tacit knowledge on the other hand, is not so easily expressed. It is highly personal, hard to formalize and difficult to communicate to others. It may also be impossible to capture. The challenge is to identify which elements of tacit knowledge can be captured and made explicit—while accepting that some tacit knowledge just cannot be captured. For tacit knowledge that cannot be captured, the goal is to connect the possessors of tacit knowledge with the seekers of that knowledge

Another useful link that I came across was this one

Tacit Knowledge is highly personal and hard to formalize, making it difficult to communicate or share with others.

Explicit Knowledge is codified knowledge that can be transmitted in formal, systematic language.

With this in mind I started to look at how it applied to testing, immediately I started to see a relationship between exploratory and scripted testing. My research found that ‘The test eye people’ have talked about this here.   I highly recommend reading this since it goes much deeper into the kinds of tacit knowledge and how that relates to testing than I do in this article.

My thoughts then turned to how it relates to our everyday testing job and I found that the information we know and can write down works well for scripted testing.  This is what explicit knowledge implies, if we can document it (codify) it then we can script it.  However testing is about testing the information we do not know or cannot explain (the hidden stuff) to do this we have to use tacit knowledge (skills, experience, thinking) we need to experience it to be able to work it out.  This is what is meant by tacit knowledge – learn it by doing it rather than reading.

Harry Collins stated that all explicit knowledge comes from tacit and I agree with this.  Once we have done some exploratory testing and documented the tacit knowledge becomes explicit.  However this accounts for a very small portion of the whole wealth of our tacit knowledge and as we do exploratory testing we uncover more areas that will require our tacit knowledge skills to be used.

One interesting point that is that tacit knowledge can become explicit as we communicate and socialise more.  As our society evolves and our ability to put into words our thinking and thoughts, more of our tacit knowledge becomes explicit.  This appears to have happen over the lifetime of existence on the planet.  We started with primitive sketches and marks which involved into handwriting.  We appear to be going back to using sketching as a tool to make knowledge explicit .  Are we going back to our historic past to help turn tacit into explicit?  As history appears to show as we become better at documenting and writing down our thoughts and thinking, our ability to turn more of our tacit knowledge into explicit knowledge becomes easier.   

Ikujior Nonaka and Hirotaka Takeeuchi talk about how to turn our tacit knowledge into explicit knowledge in their book – The Knowledge Creating Company – How Japanese companies create the dynamics of Innovation.  They explain  the use of the “Knowledge conversation process and the knowledge spiral details of this can be found in this article. .  Their explanation of the knowledge conversation process diagram is reproduced below:

1. Tacit-to-tacit (socialisation) - individuals acquire knowledge from others through dialogue and observation

1. Tacit-to-explicit (externalisation) - the articulation of knowledge into tangible form through elicitation and documentation

1. Explicit-to-explicit (combination) - combining different forms of explicit knowledge, such as that in documents or databases

1. Explicit-to-tacit (internalisation) - such as learning by doing, where individuals internalise knowledge into their own mental models from documents.

Taken from: http://www.skyrme.com/kmbasics/kspiral.htm.

If most of our knowledge is tacit (90%) then it would make sense to ensure that software development processes were aligned to where the majority of our knowledge lies.  This would indicate that the majority of our testing effort should be spent on using the exploratory testing approach to increase our explicit knowledge.  Currently within the industry we seem to be spending a lot of our testing time utilising our explicit knowledge by means of requirements coverage, test automation checks, scripting without realising that we are only using 10% of our total knowledge capacity.  In any business utilising 10% of any resource would be seen as commercial suicide, yet this practise is being carried out across the majority of the software testing profession and supported by many testing organisations in the way they ‘certify’ their testers.  People need to be made more aware of the way in which human beings store and use their knowledge.  It is not by documentation but by doing, immersing, taking part, practising, experimenting.  It is time we started to recognise that testing is a tacit activity and requires testers to think both creativity and critically.

I feel that the way in which we use and share knowledge is going to become more important within the testing world and many others within the testing community are talking about the subject of tacit and explicit knowledge. I have already mentioned within the article a post by the test eye people ,  Michael Bolton posted an article about this subject recently  in turn which was  based upon a future prediction post Michael wrote in  2011.  Simon Morley talked about using peer conference to help turn tacit knowledge into explicit knowledge.  Marcus Gartner has written a great post about how automaton can help turn tacit into explicit and why exploratory testing is still vital. The Eurostar testing conference in Sweden this year has an underlying social science style feel to it especially with Harry Collins being one of the keynote speakers.  Jeff Lucas posted an article about the relationship between tacit and explicit knowledge and testing on the software testing club website.  I feel it may be prudent of anyone involved in testing to learn more about tacit and explicit knowledge and how you can use your understanding of this to help you add value to your team, project and organisation.

This is an exciting time within software development  and one in which testers with their ability and skill in turning tacit into explicit knowledge already have a lead and the way we utilise this expertise and ability will determine the success of the project or the organisation we are working within.