Test Execution Model - updated

This is a follow on to the post I recently did about the automation pyramid that Richard Bradshaw gave at MEWT.

Following on from this some in the testing Twitterverse asked if Richard and I would do a video for the new initiative created by Richard called Whiteboard testing.  Instead of one video we decided to do two.  The first looked at the history of the test automation pyramid and you can watch it here: A look at the test automation pyramid.  The second video was a chance for me to present my own model which I have called the test execution model and can be seen here: - John Stevenson Test Execution Model.


After the video was posted I had a message from Dan Ashby who wanted to discuss more with me about the  test execution model and what he felt was missing.  We arranged a chat and discussed the model and broke down a few assumptions we were both making.  The outcome from this chat was the need to add a few extra parts to complete the test execution cycle.  With thanks to Dan I would like to present the next generation of the model.

The changes that Dan suggested for the model are as follows:

• As you test you turn some of your tacit knowledge into explicit knowledge which then can be added to your checks and become known known information.  This is represented by the flow from the testing arrow to the checking arrow at the top of the diagram.
• As you create more checks these in turn become oracles which you can use as heuristics for your future tests.  This is represented by the flow from the checking arrow to the testing arrow.

This then gives a cycle of test execution were the testing feeds into the checking which in turn can feed back into the testing.  It should be noted that is model is for test execution which does not only apply to only testing the product deliverable.  Testing is carried throughout the development cycle and as such testing activities can occur during artitecture & design discussion and during coding. Testing activities can and should occur anywhere during the development and deployment of the product and this model should be used in that context with regards to test execution.

Testing Skills #9 - Differentiating between wants and needs

One common discussion that keeps cropping up from testers saying that their managers are telling them that they 'want' this and that they 'want' that.  Normally in connection with 100% test automation or some other new shiny testing discussion point. A crucial skill a tester can have is the ability to separate what some one wants and what they really need.

It is common for people to not see the huge difference between these two words but having the ability to do so can improve your testing skill set.

The difference between the two is easy to explain with the following scenario.

Imagine that you have decided that you to be fitter and to do so you decide you want to be able to swim ten miles in one session.

Now ten miles is a lot and since the average human swims at about  two mph that is going to take about five hours to complete. You may never get around to meeting this want but you may come close.

Now one day you decide that you want to go sailing off the coast of Britain, you sail out to ten miles from the coast when suddenly the boat is hit by a huge wave and destroys it.  You are ten miles from the coast and if you do not make it back within five hours you will die due to the exposure to the cold water.  You now need to swim 10 miles in a single session.

This is the difference between wants and needs.  What people want is not necessarily what they need.

As a tester you should try and uncover what people need when they say they want this or that.  In the test automation example  it is useful to probe deeper and ask what is the problem they are trying to resolve.  By doing this you can get to the needs rather than the wants.

When the manager says "They want to automate all the testing" delve deeper to understand the problem the manager is trying to solve.  Ask "Why do they want to try and automate all of the testing?", "Uncover the needs so that you can understand what is being asked.

** The story about swimming is not my own and I cannot recollect where I first heard this.  If any of the readers can let me know where it is from I will add the correct attribution. 

Testing Skills # 8 – Being a Skeptic

 “Science is the organized skepticism in the reliability of expert opinion.”  Richard Feynman

The reason for this article is based upon a recent twitter conversation regarding how someone sometimes avoids skepticism due to its negative nature.  I fully understand what they mean when skepticism is used in way to attack a person or their character (ad-hominem argument).  The purpose of this article is to provide information about how useful having a healthy dose of skepticism is for those involved in software testing.

Many people have a belief that the purpose of testing is to prove that the software works as expected.  This is really a fallacy since testing cannot prove that the software will always work the way you expect. There is always an element of doubt in proving that it works due to the nature of software and how it is used. This is the same as in the field of scientific research where someone comes up with a theory based upon their experiments and then their peers using a fair bit of skepticism try to prove the theory wrong.  This is the crux of any scientific based research method.  It is not about producing your theory it is about applying critical thinking to your theory as to why it can be wrong.   Testing software is similar in its approach it is very difficult to inform someone, who matters that the software is working. We can provide useful information about its behavior and what it actually doing.

As a tester you need to apply critical thinking to your testing and to the evidence you produce.   This is where being able to look at what you are doing and what information you find will a fair degree of skeptical thought.  The scientific method is a useful skeptic tool to apply to your testing and to any other information that people provide to you as ‘fact’.  One approach is the use of FiLCHers:

If you look at each of these headings you can see that it is about trying to find evidence where your theory and experiment could be wrong, acting as a skeptic.

This is a vital skill for testers to possess to ensure that their testing is unbiased and factually correct. If you feel the skeptic is lacking then look at ways you can improve it.

Here are some suggestions to help you become a better skeptic:

• Learn about logical arguments and fallacies
• This page has a substantial list of them - http://changingminds.org/disciplines/argument/fallacies/fallacies_alpha.htm
• This book is a wonderful illustrated book proving examples of fallacies  and bad arguments - http://www.amazon.co.uk/Illustrated-Book-Bad-Arguments/dp/1615192255/

• Practice using skepticism – apply it to your work and the work of others.
• Use James Bach’s’ , Huh, Really, So technique - http://itknowledgeexchange.techtarget.com/software-quality/james-bach-on-critical-thinking-huh-really-so/

• Use reflection to help improve your skeptic skills.
• See my previous post on reflection - http://www.steveo1967.blogspot.com/2015/10/testing-skills-7-reflection.html 

If you have some suggestions of your own to help others in their journey to being a great skeptic please let me know by using the comments section.

The Laws of Sport and Automation

I have had this idea in my mind and on my backlog for quite a while.  It was only after speaking at MEWT in Nottingham that I felt I really should get around to writing it. 

There are many debates in the software development world about ‘test automation’ and how we can ‘automate all of the testing’.  I am in the context of this article ignoring the difference between testing and checking, more details of this discussion can be found here - Testing and Checking Refined . However some of my ideas and concepts will touch on the difference between checking and testing. 

Many have put forward arguments about automating what we know and if we have defined requirements up front then it should be possible to automate these.  My counter to this is that in many sports there are well defined upfront requirements (laws) of how the game should be regulated.  For example the laws of football (Soccer to those outside of Europe) can be found online here: FIFA Laws of the Game.  If this is the case and these requirements are defined upfront then why do we not have automated referees? I asked this question on twitter and some of the responses gave reasons due to the psychical limitations, such as battery power unable to run and so forth.  My line of thought is on how these requirements can, and are, interpreted. 

Looking deeper in to the football laws of the game it can be seen there are many ambiguous statements which given the current state of AI, at the time of the publication of this article, I feel are impossible to automate. For example on page 39 it states the following as a reason for a player to be cautioned.

“unsporting behavior”

What does this mean?  Page 125 attempts to define this with a list of what constitutes unsporting behavior.  One of this in particular I found interesting, it is one based on human nature of trying to con or cheat

“attempts to deceive the referee by feigning injury or pretending to have been fouled (simulation)”

This I feel would be a common sense decision made by the referee. How could an automated system know if is fake or not?  Then again how would the ref know?  It it is a common sense decision, being made depending on a multitude of factors and contexts.

How about this one? 

“acts in a manner which shows a lack of respect for the game”

What would count as lack of respect?  A player who in the last second of the game lets in a goal that allows the opposition to win the title. The player shows human emotion and frustration, there is a fine line between emotion and respect or the lack of it?  

My issue with this automation debate is that at this time it is not possible to automate common sense and multiple contexts in the decision making process that a referee has to go though in their thinking process. 

For example a team is winning 20 – 0 a machine would continue to officiate the game in accordance to the strict letter of the law.  Whereas a human referee would allow some flexibility in the interpretation of the rules.   They will allow some aspects of empathy to be applied to the game.  Is it yet possible to automate empathy? 

James Christie made a valid point on twitter that the reason in the majority of sports they are called laws and not rules is that:

“rules are detailed and specific whilst laws can be based on vague principles, which require interpretation and judgment. “

This makes sense since most countries have courts where lawyers debate how the laws of the land can or should be interpreted.  Then a jury, judge or set of judges make a decision based upon the arguments presented. Another case of were the requirements are listed and known but given current AI limitations would be impossible to fully automate. Even though we know that human beings are flawed in the judgement that are made, would using an automated judgement machine be any less flawed, if at all possible to produce?

Returning back to the laws of sport and how ambiguous those laws are we can look at the laws of Rugby Union

Looking at the beginning of the laws on page 21 there is guidance on how the laws should be applied:

The Laws must be applied in such a way as to ensure that the Game is played according to the principles of play. The referee and touch judges can achieve this through fairness, consistency, sensitivity and, at the highest levels, management.”

How would you automate sensitivity in this context?

According to the Oxford English Dictionary this in this context is defined as:

“A persons feelings which might be easily offended or hurt”

Add into that equation “fairness”, we are now journeying down the automation rabbit hole.

Looking at the laws regarding fair play and the guidance that the document provides for foul play (Law 10) section m gives the following guidance.

“Acts contrary to good sportsmanship. A player must not do anything that is against the spirit of good sportsmanship in the playing enclosure”

What constitutes “the sprint of good sportsmanship”? How do you clarify between intentional and unintentional behavior?   Again I am uncertain if this kind of decision could be automated.

If we look at the laws of Rugby League we can see similar issues in how difficult it can be for the laws to be interpreted. Rugby league was one of the early adopters of video technology to help assist the referee in the game.  This is what Michael and James in their article would define as tool assisted testing.  In this case a video referee can review certain decisions via the use of video technology. 

Looking at the definition of a forward pass.

“is a throw towards the opponents’ dead ball line”

How do you define this in the context of a fast moving game?  Under the section 10 which offers some guidance there is a distinction between deliberate and accidental forward passes.  How do you make a distinction between these two actions? Also would an automated system be able to deal with factors such as the momentum of the player and the wind moving the ball.  Yes they could process information quicker than a human could but would it be right?

This is not to say that referees are not fallible and there are many instances in sport of them making mistakes; however people are aware of this and can accept that fact.  Would people be so willing to accept a machine making similar mistakes based upon our biases that machine are not fallible?

Many sports are implementing some level of automated systems which are used to aid the referees.  

• Tennis has been using Hawkeye since 2002 
• Football has started to implement goal-line technology
• Cricket uses the Umpire Decision Review System 

It is interesting to note that each of these automated systems have had some controversy regarding their accuracy and success especially with the cricket system.

To conclude when people discuss test automation and attempt to automate as much as possible there is a need to step back and think critically. Automation in software development has a place and is a useful tool to use, however, it should not be thought of as an alternative to testing as applied by a human being.  Even when you think you have the requirements nailed down they are words and as such are open to a multitude of interpretations.  Using a mixture of automation, tool assisted testing and human testing in a ratio that adds value to the quality of the product being delivered is a more thoughtful approach rather than the mantra of we can “automate all the testing effort.” Going forward we need to be thoughtful of what machines can do and what they cannot do.  This may change as technology progresses but as of the publication of this article there are big limitations in automation.

Testing Skills #7 - Reflection

What type of learning do you need to engage in today?

Do you need to learn facts?

Or

Do you need to analyse and critically think about what you need to learn or have learned?
Neither of these ways of learning are wrong and each has its place in your learning journey.  Knowing which to use and when is a great tool for a tester to have.

Learning facts is defined as a 'surface approach ' style of learning and writing and thinking critically about what your have learned can be defined as a 'deep approach'.

Reflection is important in software testing and you need to be able to communicate what you have found and uncovered when testing.  If you have no opportunities,  in your organization, to be able to communicate with others then you may find you have a lot on tacit knowledge but little in the way of explicit.  It is vital to set aside time for members of the team to discuss what new information they have uncovered or learned when testing.  This allows them to use reflective reasoning to turn tacit into explicit and gives the person providing the information a sense of purpose and achievement in what they have learned.

To improve your own self-reflection produce a reflective action plan. By doing this you are committing to your self what you intend to do and engage critical thinking skills in doing so. Make sure your plan has goals and deadlines that you can commit to.  Make this deadlines and goals public it encourages you to keep to them.  Using a personal Kanban board can help you achieve this.

"First, the deep and surface approaches are not personality traits or fixed learning styles.  Students adopt an approach which is related to their perceptions of the task."

Marton and Säljö - Student Approaches to Learning - University of Oxford

When you need to think deeply about what you have been studying then writing down you own thoughts and understanding is a good way for you to be able to see what you have remembered.  If you only need to learn information or facts then other models would be more suited.

Writing down you thoughts on what you have been attempting to learn enables you to better remember or apply what has been studied.  Reflection is about doing your own internal self-assessment  of what you have been learning.  This assessment enables you make what you know internally, tacit knowledge, and attempting to make it clear and detailed explicit knowledge.  To do this you need opportunities to talk to others verbally or by writing down what you have studied to invoke critical thinking.

"In general, writing appears to be suitable for tasks where the aim is fostering understanding, changing students' conceptions and developing their thinking skills, but less suitable if the goal is the simple accumulation of factual information"

Conceptualizing and Measuring Knowledge Change due to writing

Schumacher & Gradwohl Nash (1991)

One way in which can improve your learning is to use reflection:

"Reflection is an active process whereby the professional can gain an understanding of how historical, social, cultural and personal experiences have contributed to professional knowledge and practice "(Wilkinson, 1996).  

You learn from your experiences and to make this happen you need to engage in reflection.  If you think about what you are doing or what you have experienced you are engaging in self-reflection. This approach helps you to turn learning into knowledge and it is vital in improving your own knowledge and skills.  Reflection is really about looking back at a situation, thinking about it, critically and learning from the experience, then using that new knowledge to help in future similar situations.  This in many ways is similar to the approach you will taking when engaging in testing activities.

"The process of reflective writing leads to more than just a gain in your knowledge; it should also challenge the concepts and theories by which you make sense of knowledge. When you reflect on a situation, you do not simply see more, you see differently. This different way of viewing a situation is reflected in statements about a commitment to action. Action is the final stage of reflection. "

Learning through reflection(Word doc)?

MEWT4 Post #2 - A coaching model for model recognition - Ash Winter

Abstract:

As part of my role, I often coach testers through the early part of their career. In this context I have noted a pattern in the application and interpretation of models. They are generated internally through various stimuli (learning, influence of others, organizational culture) and then applied subconsciously for the most part, until there is sufficient external scrutiny to recognize them. To this end, I have created a model of questions to help testers to elevate their internal models to a conscious level and begin to articulate them.

To this end I hope to articulate at MEWT:

• Presentation of the model of questions to determine internal models in use, without introducing models explicitly.
• Use of Blooms Taxonomy to visualize a coachees modelling paradigm and the steps towards modelling consciously.
• Practical examples of using this model to assist early career consulting testers to cope with new client information saturation.

Slides for the talk by Ash can be download here - https://mewtblog.files.wordpress.com/2015/10/coaching-model-for-unrecognised-internal-models.pptx

____________________

The first speaker at Mewt was Ash Winter who talked about his experience of coaching and how coaches have their own internal models which still could be wrong.  Ash talked about the issue he and other coaches have experienced with using models and the risk that they can limit your thinking.  He had noticed that some coaches talk about models without really recognizing that they are using a particular model.  This appears to be especially true in the testing domain.

Ash presented a different coaching model based on Blooms taxonom  to provide a framework of asking questions of those you are coaching rather than providing answers.  Ash stated that we should, as coaches, “Build your model on pillars of questions, not answers.  You are coaching”

The levels of Blooms taxonomy can be seen here:

An in-depth look at Blooms taxonomy can be found here.

Ash displayed a different variant of this during his talk:

Ash stated that he felt that Blooms was good for learning and it was useful for coaching as well.  Since Blooms works on the basis that you work towards goals this also then applies to those who coach and utilize coaching models.   

Ash also stated that his model for coaches is for those who are experienced as coaches and who are involved in coaching those who are early in their career as a tester.  As with any other model Ash did point out that he felt this was a new coaching model which was still evolving and emergent and wanted input for the wider community?

During the discussion after Ash has spoken I highlighted that the Blooms taxonomy approach does have some flaws especially in a digital driven learning environment in which we are now situated. 

The hierarchical approach of Blooms does not encourage deep and meaningful learning aided by digital media.

The problem with taxonomies is their attempt to pin down the complexity of cognition in a list of simple categories. In practice, learning doesn’t fall into these neat divisions. It’s a much more complex and messier set of cognitive processes.http://donaldclarkplanb.blogspot.com/2006/09/bloom-goes-boom.html

Issues with Bloom taxonomy further reading:

• https://eppicinc.files.wordpress.com/2011/08/sugrue_bloom_critique_perfxprs.pdf
• http://www.corwin.com/upm-data/13602_Chapter_1_Marzano_Final_Pdf_2.pdf
•  http://www.learningspy.co.uk/myths/challenging-blooms-taxonomy/

There are alternative learning models which appear to overcome these flaws in Blooms and maybe mixing them together will provide a more robust model for Ash to work with.

For example:

“Heutagogy is the study of self-determined learning … It is also an attempt to challenge some ideas about teaching and learning that still prevail in teacher centred learning and the need for, as Bill Ford (1997) eloquently puts it ‘knowledge sharing’ rather than ‘knowledge hoarding’. In this respect heutagogy looks to the future in which knowing how to learn will be a fundamental skill given the pace of innovation and the changing structure of communities and workplaces.” https://heutagogycop.wordpress.com/history-of-heutagogy/

Or

“Connectivism is driven by the understanding that decisions are based on rapidly altering foundations. New information is continually being acquired. The ability to draw distinctions between important and unimportant information is vital. The ability to recognize when new information alters the landscape based on decisions made yesterday is also critical.” http://www.itdl.org/journal/jan_05/article01.htm

At the end of the talk by Ash the group felt they needed to go away and think more about the ideas Ash has discussed.

To finish I will leave you with a quote from Ash during the talk:

A lot of people do not know what models are sometimes they emerge during applied practice

Testing Skills #6 - Speaking the language of business

The following short article is based on a talk given by Keith Klain at both CAST and Testbash

• Talking to C Level Management about testing - https://www.youtube.com/watch?v=Xn-2iIs-VBA - CAST

• How to talk to your CIO if you really have to https://www.youtube.com/watch?v=CurOi7jKJ1M - Testbash

As testers we find it difficult to explain our value when we get given opportunities to talk to senior executives in companies.  We normally end up talking about the technicalities of testing or even worse talking down to them as if they do not understand how really important testing is.  The most important rule when talking to business people about testing is…

“Do not talk to them about testing”

You should instead try to make them feel comfortable in the knowledge that you as the person assigned the role of ensuring testing is done, have it covered.

Instead focus on how the testing approach you use is aligned to the business strategy of the company and how your role help the business be successful.  Talk to them in terms of business and how you align testing to the business.  What value does what you do add or prevent the business from losing money or customers.

Business people are focused on risk and trying to avoid risk that impacts the bottom line. When talking to executives instead of saying we should not be automating everything; talk instead about the risks associated with attempting to automate everything.  The business costs to maintain or the risk of not uncovering information that could cause loss in value to the business.  Talking about checking and testing can be useful to help people understand the value of what we do.  It is important to present a balanced view and explain the benefit of both and how using both can mitigate risk.

Look at providing examples to the executives of the bad things that we are going to do to the customer if we do not test properly.   Ask them how you can help with their decisions to help your clients and protect business value. 

If your company is listed on the stock market, do you read or watch your company financial statements?  This gives useful insights to their important values.  Learning the financial language of your company can be useful when talking.  With this information you can now tailor your discussion around the value your testing provides to the whole organization. Many tester focus on the value that they provide or that the testing provides, instead focus on defining the value of the whole team delivering the product. Explain how the testing is aligned to delivering as a team rather than focusing on testing. Look at the company annual report; it highlights their risks and issues.  Understand what that is and align your testing to this.  

Most importantly, be prepared.  If you know you are going to talk to these people understanding what is important to them, what motivates and drives them. Having this information can help build a relationship around their aspirations and make them feel you understand them and what their needs are.

As Keith states “Focus on the big stuff and work back from there.” that is what the executives are really interested in.

MEWT4 Post #1 - Sigh, It’s That Pyramid Again – Richard Bradshaw

This is the first in a series of posts I plan to write after attending the fourth MEWT peer conference in Nottingham on Saturday 10th October 2015.

Before I start I would like to say thank you all the organizers and for inviting me along and a BIG MASSIVE thank you to the AST for sponsoring the event,

Abstract: 

Earlier on in my career, I used to follow this pyramid, encouraging tests lower and lower down it. I was all over this model. When my understanding of automation began to improve, I started to struggle with the model more and more.

I want to explore why and discuss with the group, what could a new model look like?

___________________________

During the session Richard explained his thoughts about the test automation pyramid created by Mike Cohn in his book Succeeding with Agile and how the model has been misused and abused.

Richard talked about how the model has adapted and changed over the years from adding more layers..

...to being turned upside down and turned into an ice-cream cone.

Images taken from - http://watirmelon.com/2012/01/31/introducing-the-software-testing-ice-cream-cone/

Duncan Nisbet pointed out that this really is now an anti-pattern - http://c2.com/cgi/wiki?AntiPattern.  The original scope of the diagram by Mike was to demonstrate the need to have fast and quick feedback for your automation and as such focused the automation effort to the bottom of the pyramid. Where the feedback should be fast. The problem Richard has been experiencing is that this model does not show the testing effort or tools needed to get this fast feedback.  It also indicated that as you move up the pyramid less automation effort was needed or should be done.  The main issue for Richard was how the pyramid has been hi-jacked and used as examples of the priority of effort should be on automation rather than focus on the priority of both in given contexts.  

Richard presented an alternative model in which both testing and automation with the tools required could be shown on the ice-cream cone diagram.

With this diagram the sprinkles on the top were the tools and the flakes the skills.  He then in real time adjusted the model to say it would be better as a cross-sectional ice-cream cone with testing throughout the cone and the tools across all areas of the original pyramid.  Many attendees liked this representation of the model but some thought that it still encouraged the concept that you do less of certain testing activities as you move down the ice-cream cone.

At this stage I presented a model I had been using internally to show the testing and checking effort. 

Again people thought this indicated that we need to do less as we move up the pyramid and it went back to the original point being made by Richard that the pyramid should die.

After MEWT I thought about this problem and tweeted an alternative representation of the diagram. After a few comments and some feedback the diagram ended up as follows:

With this model the pyramid is removed. Each layer has the same value and importance in a testing context.  It shows that the further up the layers you go the focus should switch more from checking to testing and the lower down the focus should be on automating the known knowns. All of this is supported by tools and skills.  As a model it is not perfect and it can be wrong for given contexts, however for me it provides a useful starting point for conversations with those that matter.  It especially highlights that we cannot automation everything nor should we try to do so.

In summary the talk given by Richard was one of the many highlights of the day at MEWT and inspired me to look further into the test automation pyramid model and its failings.  I agree with Richard that this original model should die especially in the way it is often misused.  Richard provided some useful alternatives which could work and hopefully as a group we improved upon the original model.   Richard did clarify that his ice-cream cone model with sprinkles is not his final conclusion or his final model and he will be writing something more on this in the near future.  His blog can be found here - http://www.thefriendlytester.co.uk/.

Now it is over to you, please provide your feedback and comments on this alternative model.

Testing Skills #5 - Remote Experiential Learning

Many people work with teams which are globally distributed, this has some logistical issues, one being how to implement useful and practical training approaches.  One common approach used is C.B.T. (Computer Based Training).  This is where participants login in and listen to pre-prepared exercises and videos, sometimes with a test at the end.  Another approach is to arrange a video session with an online tutor where they go through the material and the participants can ask questions whilst listening to the tutor.  These are OK as learning tools, but it is difficult for the participants to apply the knowledge learnt to their daily role. 

There is an alternative distance learning approach that I experienced whilst attending an online workshop run by The Growing Agile team (Samantha Laing and Karen Greaves).  I have since this course created my own remote workshop using this approach with some success.  What follows is an introduction to this approach.  Hopefully you can take this and adapt it for your own teams.

The basic principles of this remote training approach is based upon the 4Cs as described in the book “Training from the back of the room” by Sharon Bowman. Each of your learning elements should include all elements of the 4Cs in each module.

For each module of the course I create a workbook which goes through each aspect of the 4Cs.

The first ‘C’ is Connect

Before you start teaching the students ask them what they already know about the topic.  Create activities they can do offline to find out how the topic is relevant to their current role or what they currently know about the topic.

The next ‘C’ is Concepts

This is the traditional learning part, where you can introduce and explain what the topic is about.  You can do this as either a series of written articles or pre-recorded videos.

The third ‘C’ is Concrete practice

Students apply the concepts in practice.  If you are running this remotely you can set up activities and exercises related to the concepts which the students should, ideally, apply to their own working domain.

The final ‘C’ is Conclusions

This is best to done as a small group, maybe as an online video call.  All the students get together and discuss what they have learnt.  This is a great way to reinforce the learning since each person should bring different examples of applying the learning to the discussion and provide a more context rich learning experience.

When you are looking to create any remote learning experiences it is worthwhile making sure that each of your training sessions covers all aspects of the 4Cs. An advantage this learning approach gives is that it requires only a couple of hours of learning from each participant.  They can do this at their own pace and then discuss their learning and how it applied to them during a weekly hour long video conference call with the others taking part in the course.  It is crucial to set your expectations of the participants and get them to give a commitment to spending some time doing the exercises before the video call.   

As an additional option when I ran my remote workshops I set up a closed wiki site so that the participants could have discussions and provide some information about what they have learnt.   Also with permission from the participants I recorded the video sessions  and uploaded them to the wiki so they could go back and watch them later.

Testing skills #4 - Note taking

 Why is note taking an important testing skill?

There are a variety of ways to capture the evidence of our testing but if are notes are not of suitable detail then the value of our testing can be diminished.  Taking notes enables us to improve our knowledge and reinforces our understanding of the product being tested. This is part of utilizing critical thinking skills which was discussed in the chapter on 'critical thinking'

Robert Lambert discusses the need to have good note taking skills when performing exploratory testing

"During a session a good exploratory tester will often narrate the process; all the time making notes, observations and documenting the journey through the product. This level of note taking allows the tester to recall cause and effect, questions to ask and clues to follow up in further sessions." 

Explaining Exploratory testing relies on good notes - Robert Lambert - 2013

Michael Bolton wrote the following about note taking when testing:

"One of the principal concerns of test managers and project managers with respect to exploratory testing is that it is fundamentally unaccountable or unmanageable. Yet police, doctors, pilots, lawyers and all kinds of skilled professions have learned to deal with problem of reporting unpredictable information in various forms by developing note-taking skills." 

An Exploratory Tester’s Notebook  - Michael Bolton - 2007

There are a variety of note taking methods which you as a tester can utilize.  This page has an example of a few of them. Note Taking Systems - Student Academic Services - Cal Poly

One method that I have found extremely useful especially when capturing information from conferences or for recording my findings when testing is the 'Cornell Method'

The Cornell method was developed by Dr Walter Pauk of Cornell University and is widely used by University students.  It is a very useful method to help you work out if you can remember what you have written.

First of all you need to create a layout for each page in your notebook as follows. Alternatively use this Cornell Method PDF generator.

The method has 5 steps.

1. Capture what is being said or what you observe in the note taking area
2. As soon as possible review your notes and capture key details in the Review(Cues) column, add any questions you may have thought of.
3. Cover up your notes only showing the review column and now try to summarize your thoughts based on the cues.  Provide answers to any of the questions you wrote in the review column.  Use the summary column at the bottom to summarize your understanding and learning.  If you are struggling with your summary it could indicate that your notes are not sufficient.
4. Ask yourself questions on the material both the cues and the notes. Think about how you apply this information to your work.  How does it fit with what you already know?
5. Spend some time reviewing your notes and summary every so often to reinforce your understandings.

It is crucial that as a tester you practice your note taking skills.  Poor note taking can lead to missed problems and hinder knowledge sharing with the team.  Your notes are what helps to turn your tacit knowledge into explicit knowledge. 

Testing skills #3 - Leading Teams

If you are leading a software development team it is a good idea to provide to provide an environment in which each member of the team feels safe, valued and important.  It takes a lot of skill to lead teams without resorting to command and control style of managing the team.  Providing the team a safe environment in which it is OK to experiment a little and be able to learn from failing is crucial to encourage creativity and innovation within the team.   When leading a team in this way you can act as the facilitator and ensure that the bad elements of teamwork do not start to impact the team dynamics.  You can be the sounding board, the oracle, the voice of reason, the mentor, the confidant, by doing this you encourage the team to flourish which leads to success.

As a leader you also need to give the team a clear direction of what you expect the team to deliver, this can change as time passes however, this should always be transparent and visible to the team.  Ask yourself as the leader what do you want from the team?  What is your ultimate goal and then explain that to the team along with your preferred priorities.  Then step back and watch the team self-form around the problem to provide you with a solution.

The principles behind the the agile manifesto has some useful tips for leading and working in teams.

"Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done."

"The most efficient and effective method of conveying information to and within a development team is face-to-face conversation."

"At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."

J Richard Hackman provides the following five tips for leading teams.

1. Teams must be real. People have to know who is on the team and who is not. It’s the leader’s job to make that clear.
2. Teams need a compelling direction. Members need to know, and agree on, what they’re supposed to be doing together. Unless a leader articulates a clear direction, there is a real risk that different members will pursue different agendas.
3. Teams need enabling structures. Teams that have poorly designed tasks, the wrong number or mix of members, or fuzzy and unenforced norms of conduct invariably get into trouble.
4. Teams need a supportive organization. The organizational context—including the reward system, the human resource system, and the information system—must facilitate teamwork.
5. Teams need expert coaching. Most executive coaches focus on individual performance, which does not significantly improve teamwork. Teams need coaching as a group in team processes—especially at the beginning, midpoint, and end of a team project.

Leading Teams: Setting the Stage for Great Performances - J. Richard Hackman - 2002

Testing skills #2 - Influencing by listening

Having recently attended an excellent internal workshop on the' artistry of influence'.  One of the key takeaways I got from the course was to be able to influence you need to STOP and LISTEN to what the person is saying and most importantly how they are saying it.

During the course I discovered that when communicating we have a preference for using one of three types of senses, sound, sight and touch.  Which when converted into communication terms become auditory (sound), kinesthetics (touch) and visual (sight).

Do they use visual words such as 'see, bright, focus', kinesthetic 'feel, handle, hard' or Auditory 'hear, ring a bell, sound right'.

Once you uncover their prefer sense for communicating you can try and match their preferred sense.  This is called pacing and its roots can be found in our tendency to form tribes and groups based around similar traits.  This enabled us to survive as a group, since we shared similar habits and vocabulary.  It also encouraged strong relationships within the groups.

It is also useful to established the tonality of the person you are listening to.  Are they talking quickly, loudly, slowly or quietly? Ask a friend to help you practice by matching the voice tonality and their preferred sense.  By listening to others and the way they talk you can start to establish a rapport and build a relationship in which they can be influenced on a subconscious level.

Be careful when influencing others that it does not become more than just trying to persuade others of your thoughts and ideas.  If people feel or find out they are being manipulated then you will loose their respect and destroy any possibility of a working relationship.  The key aspect when trying to influence people is to ensure that they know it will have a positive benefit for them, even if at the same time you will gain a positive benefit.  The person you are trying to influence has to see a benefit in it for them.

It is not about YOU it is about THEM,

There is a lot more involved in using listening as an influencing tool and this is only an introduction.

Let me know how you get on with this tip on influencing by listening.

Further reading:

• Real world applications of pacing and leading
• How to persuade anyone using pacing and leading
• What is NLP
• Influencing skills: a how-to guide, or, How to get what you want without making enemies

It meets the business requirements

Recently I came across an interesting software requirements issue whilst eating out at a UK restaurant chain.

They had one of those common deals you can come across in similar chains.  You choose a main meal from a limited, but tasty, menu.  You get unlimited drinks of either tea/coffee or fizzy drinks. (soda) and you can have unlimited access to the salad bar .  To end your meal you get an ice cream sundae.  All of this for the sum of £9.99!

I was with my wife and we both ordered a main meal and a drink.  Whilst our meal was being prepared we went and selected our bowl of salad from the salad bar.  We set about eating our salad, during this time the drinks arrived followed by our main meal.  We only just managed to finish our main meal and was feeling rather full by now.  So when the waitress came over and asked if there was anything else we both said no can we skip the ice cream sundae and just get the bill.

The table was cleared away and the bill arrived......

Can you guess how much the bill came to?

If you are thinking the same as my wife and I you would think the bill would be £19.98.   The actual bill came to £20.13.  I queried this with the waitress and she replied....

"Oh, that is because you did not have the ice cream sundae!"

Our reply was "WHAT", We have less but it will cost us more?

The waitress was very confused and went to see her line manager. Who came over and explained that what they will need to do is order us the ice cream sundaes and then go and let the kitchens know that there is no need to make it.  They did say we could take it with us!  Ice cream, in car for hours & warm day  not a good combination.

All of the orders were taken on a mobile phone app which fed back to the main system and the kitchens.  There appears to be no option to be able to mark an order as one of the deals. Each item is entered individually.  Once you have the four items it now knows which are part of the deal and works out the price.  Now the question to those reading this..... "Is this a defect?"

From the business requirements perspective this is, I guess, what they wanted from the software.

Since entering each item one at a time and then working out if it is part of the deal allows them to manage stock control and keep business costs under control.  It has one less step for the operator to carry out rather than if they had to press a button to indicate it was a deal,

From the end user experience perspective it is flawed.  It appears they did not expect customers to decline something that was in reality free. From the staff working there they had to override the system and implement manual processes to enable the correct price to be calculated. At the same time this will upset the stock control reporting that they have two less ice cream sundaes than they actually do.

When we look at software requirements and we start testing these requirements, it is important we look at the implementation from all perspectives.  This is especially true for systems that businesses use to help facilitate customer service.  Get this wrong and it can leave a bad impression on the business customers.

Testing skills - Abductive Reasoning

This is the first in a series of hopefully short articles which looks at  skills and techniques outside the traditional testing that can be useful to those who practice testing.

Future topics planned include:

• Influencing by listening
• Note Taking
• Leading teams
• Persuasion and how to sell
• Speaking the language of business 
• Remote teaching experiential style
• Going beyond the model

If you can think of any others that would be useful please leave a comment. 

I am making these public,  since by writing it down and making it public I am committing myself to do it.  That is your first tip in this article, if you want to commit to doing something which you keep putting off, writing it down and make it public.

Abductive Reasoning.

 Abduction came about from the work of Jo Reichert, in this work Reichertz came up with another cognitive logic process to describe discovery when the researcher encountered surprising findings in the data. He called this "a cognitive logic of discovery".  Before this there were two types of reasoning in common use, 'inductive' and 'deductive'.

• Inductive - Making generalized conclusions from specific observations
• Deductive - Proving or disproving a theory from observations (scientific method)

Abductive reasoning is an important process for those involved in testing.  The majority of time when we are testing we discover surprising behavior in the software.  This normally makes us rethink our theories of how the software works and as such we begin to re-evaluate our understanding of the software.  We create a new rule, or test idea to further investigate the surprising element of what we have just tested. This is key within grounded theory; our thoughts about the software and how it behaves change as we explore the software more.  How we report these surprises and the behavior of the software is crucial to the value that testers provide to a project.

"There are two strategies involved in abduction, both of which require creating the conditions in order for abductive reasoning to take place"* (Reichertz, 2007: 221). 

The first is a ‘self-induced emergency  situation’ (Reichertz, 2007: 221). This means that in the face of not knowing what to make of a surprising finding, rather than dwelling on the infinite number of  possibilities, the analyst puts pressure on themselves to act by committing to a single meaning.

The second strategy is completely antithetical to the first. It involves letting your mind wander without any specific goal in mind, or what Pierce (1931–1935), a key writer on abduction, called ‘musement’* (Reichertz, 2007: 221). "

Qualitative Research Methods in Psychology: From core to combined approaches - Nollaig Frost - 2011.

Reichertz makes the following observation about these two strategies.

"What these two quite antithetical strategies have in common is tricking the thinking patterns of the conscious mind in order to create ‘an attitude of preparedness to abandon old convictions and to seek new ones."
 The SAGE Handbook of Grounded Theory:(Sage Handbooks) - Antony Bryant, Kathy Charmaz  - 2010.

Testers need to be able to abandon their old convictions and seek out new ones.  This is especially important when we are testing software, since our biases and beliefs and previous experiences can influence our decision making. Using some of the methods described in this book can allow us to challenge our thinking about the software and engage in abductive reasoning.

One famous use of abductive reasoning is that used by the fictional detective Sherlock Holmes by Sir Arthur Conan Doyle.  Many people believe, wrongly, that Sherlock Holmes uses deductive reasoning to solve his cases, when in reality he used abductive reasoning.

"Holmes' method doesn't resemble deductive reasoning at all. Instead, it's much more similar to a form of reasoning known as "Abductive Reasoning"Debunking Sherlock Holmes Myths - Maiza Strange  May 2014

To summarise abductive reasoning is taking your best guess based upon your current knowledge, observations and experiments.  These pieces of information maybe incomplete but you use your cognitive reasoning processes to form a theory or conclusion.  For example:

"A medical diagnosis is an application of abductive reasoning: given this set of symptoms, what is the diagnosis that would best explain most of them? Likewise, when jurors hear evidence in a criminal case, they must consider whether the prosecution or the defense has the best explanation to cover all the points of evidence. While there may be no certainty about their verdict, since there may exist additional evidence that was not admitted in the case, they make their best guess based on what they know."Deductive, Inductive and Abductive Reasoning - Butte College.

Further reading on abductive reasoning:

• Abduction, Deduction and Induction in Qualitative Research - Jo Reichertz
• Abductive Thinking and Sensemaking: The Drivers of Design Synthesis - Jon Kolko
• Abductive Reasoning - Wikipedia - 2015

This article has been taken from the Testing and the Social Science chapter in my book - The psychology of Software #Testing.

Volunteers needed for a FREE remote testing workshop concept

I am looking for two or three people to act as guinea pigs for a remote testing workshop concept I am working on.

The workshop will be FREE but require a couple hours of your time each week for about seven weeks.  I am hoping to start this workshop be the end of August 2015.

• Ideally the participants will be fairly new to testing with less than two years experience. 
• Able to work within UK timezone (Early evenings)
• Willing to provide feedback about the workshop.

Why am I doing this?

I want to give back to the testing community, I understand it can be difficult for those new to testing to develop testing skills and techniques that they can apply directly to their work.  The concept of the workshop is that there will be series of self learning exercises and some challenges that they can apply to their testing in their current role.   Each week the group will get together using video conferencing to discuss what they learnt and share it with others in the group.

I currently have one volunteer and I am looking for 2 or 3 more, if you are interested then please reach out to me via twitter @steveo1967 and we can discuss in more detail.

**UPDATE - it is now full - please contact me via twitter if you are still interested and I can create a waiting list.**

Using Aspects of Lean Coffee to Drive a Retrospective

Those who have been following my blog may have noticed that I have not updated it as frequently as I normally would, or would like to.  There are been a variety of reasons for this, I am writing a book which is taking a lot of my spare writing time, I have not had much of interest to write about and finally over the past year my role within testing has changed to one of being a scrum master for agile teams.  It has been an exciting and challenging role which over time I may blog about.  It has been strange since people have commented that I appear to be a natural in this role and part of me thinks this could be due to my skills as a tester that helps in this role.  I am still trying to get involved in some testing and hopefully there will be opportunities for me to do this.

This post is about an experiment I attempted with a scrum team recently for the retrospective.  I am unsure if I obtained this approach from somewhere, or it was just an idea I came up with.  I have decided to share it  via my blog for others to see if they find it useful.

A common approach to retrospectives is to get the team together and discuss what they felt went well, what could they improve and to have some actions for the team.   The way I have run this in the past is we take each of these statements and get some feedback from each members individual perspective.   Some of the questions that can be used can be found here - http://www.benlinders.com/2013/which-questions-do-you-ask-in-retrospectives/

What I found was these retrospectives did not appear to be that engaging and after awhile became a little stale.  So I had an idea to change the dynamics by introducing a lean coffee (http://leancoffee.org/) format.  Lean Coffee is a structured but agenda less meeting, which consists of three steps:

(1) Set up a Personal Kanban

This basically means create a series of post-its to represent

• What we are currently discussing, "In Progress"
• What we have discussed "Done"
• What actions have comes from the discussions "Actions"

(2)What to Discuss

This can be any topic  that you wish to discuss, for the first retrospective I ran in this way I used the following:

• Shout outs
• What was good.
• What was Bad
• Improvement Ideas

The first four were placed upon the wall and then I gave the team members a set of post-its and pens to write down their thoughts for each of these titles. One post-it per comment

Shout Outs:

Who do you know who went beyond their normal day to day work to help the team.  This could be someone within the team or outside our team who helped support the team.

What was good

What do you feel went well in the sprint.

What made you feel good about what the team did,

What was bad

What do you feel went wrong in the sprint

What made you feel bad about something in the sprint

Improvement Ideas

How can we improve what we do?

What ideas do you have to make the team better?

(3) Vote and Talk

Once we had done this everyone was encouraged to go up and look what others had written and at the same time we decided to group similar comments together. After this we had some cookies, always bring cookies to a retrospective, I moved the In progress post-it over to the 'shout outs' column and people started to explain what they have written,  It was interesting that it was someone outside our team that people felt had added value to the team.  From this we as a team sent an internal gift and a recognition for their work.

Once we had finished with the 'shout outs' we moved this column to 'done' and move the 'in progress 'sticky to the next column  'What was good', we followed the same approach and everyone had opportunities to discuss what had been written.  At the same time any actions that came from the discussions was added to the Actions post-it.  This approach was then done for the rest of the columns 'What was bad' and 'Improvement ideas'.

The dynamics of the team during these discussions was far greater than at any of the other retrospectives and I felt as a scrum master it worked really well to encourage the different members of the team to participate.  Part of being a scrum master is to keep the various ceremonies you have for the team 'fresh' and 'interesting' and in this case it appeared to work well.  Will it have the same affect the next time, I am unsure however it is one more tool I can use to help the team. Let me know of your approaches to keep the team motivated and encouraged.  Also if you try this approach let me know how it works for you and your teams.

In Lean coffee you would normally VOTE on the topics you are interested in using dot voting,

"Each participant gets two votes. You can vote twice for the same thing or for two different topics. Simple put a dot on the sticky you are interested in. Tally the dots. "

Update: I have run this approach a couple of times since the first trial and  it appears to work very well.  I have tried to prevent it from being too routine and have changed the questions that are posted.  I have used titles such as "What was positive about the sprint" , "What was negative" "If we could improve on one aspect what would it be".  It is important to create varieity in the retrospective even if you are using the same approach.  It helps to keep the team interested and motivated.  Cakes and cookies help too!

What drives us? Extrinsic and intrinsic motivation.

After recently presenting a workshop at the Lets Test conference on Self-Learning one of the concepts that people found difficult to grasp was the difference between extrinsic and intrinsic motivation.  It may be due to the time constraints of the workshop or I did not explain clearly enough.   Therefore I decided to put together this article to give a little more detail about extrinsic and intrinsic motivational factors. 

One psychology aspect of motivation is to work out what motivation factors are intrinsic and which are extrinsic.  To begin with it is useful to define what is meant by extrinsic and intrinsic.

"Extrinsic motivation is ‘external’: people – in this case athletes – are driven to succeed by factors from outside i.e. money, prizes, acclaim, status, praise." 

"Intrinsic motivation comes from within i.e. an athlete driven by a need to succeed because they want to be the best and are not overly concerned by financial or ego boosts."

The Sports Mind - Extrinsic vs Intrinsic motivation

Many people and organizations mistakenly assume that people are motivated and driven by financial rewards and to some extent they are.  People do want to be financially rewarded for doing work. In the majority of cases money works as a motivation factor for people to get out of bed to go to work and do the normal everyday tasks.

As Kamenica points out:

"It is helpful to distinguish those tasks that people certainly do not want to do unless they are paid for them from those that people may or may not engage in.” 

Behavioral Economics and Psychology of Incentives -Emir Kamenica - 2012

However there are studies which show that rewarding someone with money for something they have a passion for can demotivate and make them less effective. 

“...tangible rewards tend to have a substantially negative effect on intrinsic motivation (…) Even when tangible rewards are offered as indicators of good performance, they typically decrease intrinsic motivation for interesting activities.” 

A meta-analytic review of experiments examining the effects of extrinsic rewardson intrinsic motivation. Deci EL1, Koestner R, Ryan RM

Since extrinsic rewards form only a small part of what motivates people it is important to find out what makes people 'tick'  How can you set an environment that encourages peoples passion and motivates them to be the best. Providing people with opportunities to pursue their passion be it time to study or learn can have a positive impact on a team as long as others in the team are given similar opportunities.  The psychology of motivation is complex and what may motivate someone may not motivate someone else.

If you find something that is of interest you and you want to become passionate about it or if someone on your team is showing a passion for a certain activity it is worth focusing on the intrinsic motivation rather than the extrinsic.  It is also important to be aware of the over-justification effect

"The catch-22 of extrinsic motivation. The over-justification effect occurs when someone naturally has a passion (intrinsic motivation) to see something through, but is offered a reward for its completion. Thus rendering them less effective. For instance, if an employee loves writing on your corporate blog but you decide to financially compensate them for each post. There is a chance they will find the writing less enjoyable. Since they have to be bribed into writing, then the task must not be worth doing for its own sake." 

12 Psychology Concepts for Improving Employee Motivation -Bradley Gauthier - August 17,2011

One way to inspire individuals is by using unexpected rewards. Unexpected rewards can inspire and motivate people; the key is to not expect a reward. For example if someone has done something that you feel was outstanding offering to take them for lunch and paying or giving a small gift of appreciation can go a long way to keep them motivated.  One approach that can be useful when showing your appreciation for someone is to say how much you appreciate their hard work rather than how clever they were.  This makes people value the effort more than anything else. You can use this kind of reward system to encourage the right behavior but it is important to realize that there is a thin line between unexpected and expected rewards.

“Yes, sometimes rewards do work, especially if people really don’t want to do something. But when tasks are inherently interesting to us rewards can damage our motivation by undermining our natural talent for self-regulation."

How rewards can backfire and reduce motivation -Psyblog

When thinking about the testing you are performing it is worthwhile investigating the motivating factors.  If the testing you are carrying out a scripted approach then your motivation could be linked to extrinsic rewards rather than intrinsic rewards.   It is worth asking yourself the following about the testing you are performing:

"Is the task at hand routine?  That is, does accomplishing it require following a prescribed set of rules to a specified end?" 

Daniel Pink -Drive

For these types of tasks extrinsic rewards can work.  As a tester you should question if testing is really this type of task? Read the following questions:

• -When you are performing testing activities what is it that drives you? 
• What gives you the most joy and value to yourself in the testing you are doing? 
• Is it the satisfaction you get internally from uncovering how the system is working or not working? 
• Is it the ability to be autonomous in your exploring of the software?
• Or is it something else that drives you to carry on with your investigations?

If you are nodding to any of these then maybe the testing you are performing is linked to your intrinsic motivation.   This is different from the feelings you may get if carrying out step by step test scripts.  

Daniel Pink sums up what intrinsic rewards means to the individual.

"It concerns itself less with the external rewards to which an activity leads and more with the inherent satisfaction of the activity itself."  

Daniel Pink -Drive 

As an added complication Carles Malet described three motivational forces in his article "Motivation from Maslow to PerezLopez".

• Extrinsic motivation: when individuals act prompted by an external reward (or punishment), such as wages or improvements in the labor conditions.
• Intrinsic motivation: linked to the satisfaction that individuals obtain when performing certain tasks. The intrinsic motivation is linked to the human need of learning.
• Transcendent motivation: when the action is directed towards satisfying needs of other human beings. The transcendent motivation is linked to human generosity and the inner call for serving other human beings. Parents will recognize transcendent motivation patterns in their acting with their children, and so will do senior supervisors when empowering employees and charting their career plans.

Adding the third motivation factor is an interesting one since it plays on our human nature to want to help and support others.  This as the example explains is apparent in our nurturing instinct where we get satisfaction for helping our offspring.  In the software testing industry I have seen many examples of this with people providing their time freely to support and help others to learn, rather than being inwards and looking for their own learning opportunities.   People depending on the context will apply different weighting to each of these motivational forces and being able to understand and know which has more significance to individuals and to yourself can help drive your and others passion.

Some of this material has been taken from the next chapter of my book – The Psychology of Software Testing – Building passion, due for publication July 2015.