Something Old, Something New

"Be careful. People like to be told what they already know. Remember that. They get uncomfortable when you tell them new things. New things . . . well, new things aren’t what they expect. They like to know that, say; a dog will bite a man. That is what dogs do. They don’t want to know that man bites a dog, because the world is not supposed to happen like that. In short, what people think they want is news, but what they really crave is olds . . . Not news but olds, telling people that what they think they already know is true." 

TERRY PRATCHETT The Truth: a Novel of Discworld

This passage from the Terry Pratchett novel "The Truth" resonates very much with me especially when it comes to world of testing.  I see a connection with how different people in our craft react with each other when it comes to new ideas, thoughts and innovations.  There are those who embrace change and want to know more.  There are then those who like time to think critically and use different forms of reasoning before deciding if it is good or not.  Then there are those who dismiss out of hand anything new, anything that makes them feel uncomfortable or goes against their current beliefs.

My concern and the reason for this blog post is how within testing we can become more creative and innovative.  As one of my previous post stated I think to be creative we need to think about finding problems than trying to solve them.  Continuing on the path of our focus being only to solve problems restricts our creative thinking.  At the same time we need to find ways to convince those who dismiss anything new or unexpected.  To do this a set of guidelines should be introduced to encourage creative thinking rather than discourage:

• Those who easily dismiss new ideas should not be quick to be negative, negative comments and views are one of the easiest ways to destroy creativity.

“The creative impulses of most people can be suffocated by negative criticism, cynical put-downs or dismissive remarks.” 

Ken Robinson – Out of our Minds

• Company leaders need to lead from the top and encourage new ideas and innovation, making the task of thinking (creative and divergent) as important as other everyday tasks. They need to give time to allow this to happen.

“it’s not enough to think differently. We also have to act differently” 

Abraham Lincoln (Taken from  Ken Robinson – Out of our Minds)

• There is a need to encourage people to try, and to see failure and mistakes as learning opportunities. We should stop blaming and encourage risk taking to enhance the opportunities for serendipity moments.

“We don’t teach people how to deal with failure and this is a fundamental oversight.” 

Ken Robinson – Out of our Minds

• We need to have more diversity within teams, encourage people who may have different views to yourself to work with you.  Employ someone who thinks differently to you.  This gives more chances that new ideas will be generated from these differences.

“Such people will provide a wider range of knowledge from which to extract information and build upon ideas.” 

Why diversity is the mother of  creativity - Jeffrey Baumgartner

• Spend more on training the people who work with you or for you.  Take an interest in their learning, encourage, mentor, and support their creative needs.

 “Creative teaching requires moving from a focus on imparting knowledge to knowledge acquisition, providing opportunities for the learner to engage in deep thought and productive action.” 

Susan Keller-Mathers, Encyclopedia of Giftedness, Creativity, and Talent

• It is not enough to come up with new ideas; creativity involves doing something and applying your ideas.

"Innovation is the process of putting new ideas into practice. Innovation is applied creativity." 

Ken Robinson – Out of our Minds

Some may find this a strange post and wonder what it has to do with testing.  I see testing as a very creative process especially when it is unburdened from too much process and stifled by procedures.  Exploratory testing lends itself very easily to the creative process and encourages the tester to think and discover new and exciting ideas.  We need to do more of this style of thinking so we become more engaged with our creative side.  A few people over the past few years have been saying that testing is dead, I would say that the non-thinking uncreative tester is going to die out and become extinct  (The checking robots as some may classify them).  We need to encourage and develop working environments in which  people can connect to their creative side and be allowed the freedom to explore new ideas and not be afraid of making affordable small mistake from which they can learn.

I will leave you with one of my favourite quotes from the American poet Jack Kerouac,  It is OK to be different and to challenge the status quo we need to encourage more crazy ones into testing.

“Here’s to the crazy ones. The misfits. The rebels. The trouble-makers. The round heads in the square holes. The ones who see things differently. They’re not fond of rules, and they have no respect for the status-quo. You can quote them, disagree with them, glorify, or vilify them. But the only thing you can’t do is ignore them. Because they change things. They push the human race forward. And while some may see them as the crazy ones, we see genius. Because the people who are crazy enough to think they can change the world, are the ones who do.” 

Jack Kerouac

Time to slow down.

This short blog post has been inspired by some of my reading over the holiday period including the following

• Future Minds: How the Digital Age is Changing Our Minds, Why This Matters and What We Can Do About It by Richard Watson 
• Future Shock - Alvin Toffler

We appear to be living in a constantly connected world where we are being bombarded with Terabytes of information each and every day and we could be approaching information overload and the dangers this bring.  Since we are receiving so much information our brains are taking the easy route and mainly accepting what it is been fed without questioning and you start believing things you would not normally accept.

 In A Mind of Its Own, Cordelia Fine makes the point that the brain’s default setting is to believe, largely because the brain is lazy and this is the easier, or more economical, position. However, when the brain is especially busy, it takes this to extremes and starts to believe things that it would ordinarily question or distrust.

Richard Watson in Future Minds states the following:

One of the consequences of rapid information transmission is that we increasingly fail to think properly about the validity of incoming or outgoing information; we are too busy and there is too much of it.

There is so much pressure on us to be doing stuff and looking up stuff and being always available no matter where we or what we are doing, there are very few places where you can escape to think.  The traditional places such as the pub (bar)  or the local park have been taken over by constant ringing and people with their faces bent over a tiny screen.

Alvin Toffler talks about Information Overload in his book Future Shock and talks about how we freeze when we get overloaded with information.  He talks about being overstimulated in war situations and how people will just shut down.  This appears to becoming more common with people turning into mobile phone zombies craving for their next information fix and ignoring all possibilities of serendipity moments from looking at the world around them.

We are using our memory less and less since we can now “Google” it, so we have less storage in our own heads to be radically creative and generate ideas.  Life is being run in the fast lane and we are in danger of doing less and less serious thinking.  We are being told we need a decision NOW, so we skim, scan or ignore and then make a (what could be a wrong) choice.

I feel very much spilt on this subject since I am a techno geek, I have a passion for gadgets and anything technological but at the same time I am starting to realise that I have less time to myself to do nothing and gather my thoughts and do some SLOW thinking.

Richard Watson suggests going for a walk or just starting out of the window.  How many people do you see today in the office day dreaming or just starting into space and really thinking?  What would happen if you did this in your office?  He suggests that we have a day in which we plan and do nothing and allow ourselves to be immersed in our own thoughts.

My own view is that we need to step back a little sometimes and slow down to allow our minds to think and to think deeply.  I have a concern that in the future there will not be any deep thinking and people will just be looking at what they see on the surface and believe that to be true.  We need to start carrying or using notebooks to capture our ideas especially when in a deep thinking moment.  We may have many ideas during the day, some great, some good and some bad but we need to start capturing this and help to provide situations which are conducive to the generation of ideas and deep thinking.

Richard gives the following helpful hint:

If it helps, create three physical notepads, files, or boxes marked “no,” “yes,” and “maybe” and place a note of your thinking in the appropriate one. 

This is a method I use for my ideas for a blog article I have currently about 400 ideas marked 1-3 with one being likely and 3 not likely.  I review these about once a month and change the rating of the chances that it will appear as an article.  This is important since my views and thoughts over time will change and something I felt was relevant at a certain time is no longer relevant.

We can also apply this to exploratory testing There is a still a strong contingent of people in the testing world who measure by number of tasks (test cases*) being completed is a useful way to measure progress and know when we are done testing.  However I feel we need to take a step back and slow down a little to allow ourselves sometime to think.

Exploratory testing is a very human thinking activity and it is easy to start measuring progress by number of missions/sessions completed.  Instead we should allow time to thinking deeply about what we are testing and what information are we uncovering since this could lead to moments of serendipity and that eureka moment.  So the next time you see a tester staring into nothing it may be that they are deeply thinking about what they are doing.  TE Lawerence is quoted as saying the following about people who daydream:

“Those who dream by night in the dusty recesses of their minds wake in the day to find that it was vanity: but the dreamers of the day are dangerous men, for they may act on their dreams with open eyes, to make it possible.”

(*This blog is not going to get into a metrics debate here others have done a much better job than myself see Michael Bolton links here)

So to conclude

• We need to get off the information highway sometimes and reduce the information we are receiving.  
• We need to slow down and allocate some time to thinking
• We need to do something different so we get more experiences that can help generate more serendipity ideas. 
• Do some gardening, take a bath, go for a walk in the countryside with no destination, lose yourself in your own mind.
• Take vacations(holidays) and remain unconnected from the office/work
• Do something you have a passion for and enjoy
• Do not be afraid of making mistakes with the ideas your generate this is valuable experiences

“I may not have gone where I intended to go, but I think I have ended up where I intended to be.” Douglas Adams

Problems and Creativity

I recently posted a tweet

When #testing should we be problem finders or problem solvers? Problem finders appears to think more creatively is this aligned to ET?

This started a few discussions with people tweeting that surely it is best as a tester to be both and that they see their role as being a problem finder and a problem solver.  Some took it that being a problem solver is better since you are looked on more positively.

Francesco @TesterSerendip
@steveo1967 @mpkhosla@JariLaakso @tonybruce77 hence better at finding bugs. Plus you are perceived as one with a positive mindset 

Other that they see themselves primary as problem solvers

Tony Bruce @tonybruce77
@steveo1967 I'm a problem solver, I approach the problem differently#testing

I then made the following tweet:

John Stevenson ‏@steveo1967
@mpkhosla @JariLaakso @TesterSerendip being in a problem finding mode may be better for a tester when testing since you may be > creative

My tweets came about after reading some books and articles on the following subjects

• Design thinking
• Divergent thinking
• Creating Opportunities
• Little Bets – Peter Sims 
• The Click Moment by Franz Johansson

This led to me researching the concept that our mind-set within the field of testing appears to mainly be focused on problem solving and that our default state when testing is be in a 'problem solving' mode.  Very little focus or attention seems to have been placed on using our creative skills and entering a mind-set of being a problem finder.  The more I researched into this the more I felt that in some testing situations especially when doing testing practically we are missing opportunities to think more creativity and we are limiting ourselves when we approach testing with the mind-set of trying to solve problems rather than find problems.

One piece of information I related to was from *James Bach or Michael Bolton in how we should approach session based test management.  When in a session we should be looking for problems, creating new tests and noting things of interest which could be bugs.  We should focus on the opportunities and come back to find out if it is a bug/issue (problem solving exercise) after the session as finished.  The principle of this is if you stop to try and solve the problem you could lose track and focus and your creative thinking will become disrupted.  Einstein is quoted as saying the following:

“We can't solve problems by using the same kind of thinking we used when we created them.”

Which IMO matches the thoughts I have on the issue of problem finding and solving.  Yes we need both but they require different thinking.

This is the key point I was trying to make with my statement whilst we are testing in the context of actually being in a session our mind set should be on problem finding.  Having a mind-set focused on problem finding according to some articles has been proven to be more creative.

“Anyone who is technically proficient can solve a problem that is already formulated: but it takes true originality to formulate a problem in the first place (Einstein and Infeld, 1938).”

“…persons who are likely to innovate tend to have personality traits that favour breaking rules and early experiences that make them want to do so. Divergent thinking, problem finding, and all the other factors that psychologists have studied are relevant in this context.”

 http://www.sagepub.com/upm-data/11443_01_Henry_Ch01.pdf

“Csikszentmihalyi and Getzels (1971) found that originality was high related to problem finding and discovery oirientation”

“Getzels believed that creativity investigators should turn their attention to and examine problem finding in addition to problem solving”

 http://books.google.co.uk/books?id=KDok5dKEhh0C&pg=PA15&lpg=PA15#v=onepage&q&f=false, 

“Many creative individuals have pointed out in their work that the formulation of a problem is more important than its solution and that real advances in science and in art tend to come when new questions are asked or old problems are viewed from a new angle . . . yet when measuring thinking processes, psychologists usually rely on problem solution, rather than problem formulation, as an index of creativity. . . They thus fail to deal with one of the most interesting characteristics of the creative process' namely, the ability to define the nature of the problem”

“We believe that this neglect of problem finding is a deficiency that is observable in the discourse on problem solving within technology education.”

http://scholar.lib.vt.edu/ejournals/JITE/v36n1/lewis.html

What I am not saying is that we do not need to be problem solvers; this is part of our role.  I think the subject matter of this article is similar back to a previous article I have written on which mood we should be in to test software  and which gives us more benefit.

In some situations it may be better to be in a problem solving mind set and in others it may be better to be in a problem finding mind set.  Both have their merits.  What I feel I am trying to get across that if you need to create ideas and be innovative you need to have a mind-set that is looking at being a problem finder rather than a problem solver.   However if have a solution to a problem you may want to flip this around.  Then with a problem finding mind-set examine the solution for problems.

To conclude I will repeat what was said earlier by Einstein and Infield

“Anyone who is technically proficient can solve a problem that is already formulated: but it takes true originality to formulate a problem in the first place (Einstein and Infeld, 1938).”

I feel if there is one thing that people take away from this article it is that statement.

Further reading

1. Out of our minds – Ken Robinson
2. Perspectives in Creativity -   By Irving A. Taylor, Jacob W.. Getzels
3. Wisdom: Its Nature, Origins, and Development – Robert Sternberg
4. Perspective: Problem Finding and the Multidisciplinary Mind -  Linda Austin
5. The Creative Vision: A Longitudinal Study of Problem Finding in -  J. W. Getzels, M. Csikszentmihalyi
6. What's Holding You Back?: 8 Critical Choices for Women's Success -  By Linda Austin, M.D

* If anyone can direct me to a reference where this came from and it was not just a conversation I have had then please let me know.

Ethnographic research feedback

Sometime ago I wrote an article about the relationship between ethnographic researchers and testers and how similar they are.  Recently Peter H-L (@Unlicensed2test) on twitter reminded me that I had also presented at the UNICOM  conference on using some aspects of ethnographic research to aid feedback when we are testing and from this I came up with a new mnemonic and a set of testing related social science questions.   I had thought that I had already posted this but it seems I had not.

What follows is taken from the talk I did.

*************

Within the article there was a section that dealt with questions that the researcher should be asking when studying the subject.  I changed this to make it relate to software testing and came up with the following:

• Substantive Contribution: "Does the testing carried out contribute to our understanding of the software?"
• Aesthetic Merit: "Does the software succeed aesthetically?" Is it suitable for the end user?
• Reflexivity: "How did the author come to write this test…Is there adequate self-awareness and self-exposure for the reader to make judgements about the point of view?"
• Impact: "Does this affect me? Emotionally? Intellectually?" Does it move me?
• Expresses a Reality: "Does it seem 'true'—a credible account of a requirement'?"

Lynne Mckee has been updating a list of testing mnemonics on her blog site  so I thought about this and came up with the following mnemonic:

R.A.I.S.E

From this I created a list of questions under each of these heading that can be used to aid feedback when you have been testing, ideally when you are following session based test management.


Use the following template to do a personal review of the testing that you carried out during the day.
Please try not to answer using yes and no, expand on your reasons for either it being yes or no.
This debrief/review is more about your views, opinions and feelings rather than the product you have been testing.
It should only take you 10 minutes to complete this feedback – try not to write essays.

_______________

Reflect

Personal reflection:

• Could you have done things better if so what? (Both from a personal and testing perspective)
• Have you learnt new things about the product under test (That are not documented)?
• Has your view of the product changed for better or for worse? Why has your view changed?

‘Epistemological reflexivity’ (What limits did we hit?)

• Did your defined tests limit the information you could find about the product?  (Did you need to explore new areas that you had not defined)
• Could your tests have been done differently? If yes how?
• Have you run the right tests?
• If you did things different what do you think you would have found out about the product?
• What assumptions have you uncovered to be true/false?
• Did the assumptions you make impede or help your testing?

Aesthetic:

• In your opinion is the product suitable for the end user?
• In your opinion is the product appealing at first look?
• In your opinion is the product confusing?
• In your opinion does the product flow?
• In your opinion are there any ugly areas?
• In your opinion does the product succeed aesthetically? Does it meet the image the customer is trying to portray?

Impact:

(this section is intended to be used to say how you 'feel' about the product, your first impressions, if you answer yes you should provide more details)

• Does this affect you?
• Emotionally?
• Intellectually?
• Does it move you?
• Does it cause you negative/positive feelings?
• Does it frustrate you?
• Does it annoy you?

Substantial:

• Have we covered a substantial amount of the key product areas?
• Has the testing contributed to your understanding of the product?
• Do you think you have a substantial understanding of the system and sub systems?
• Does your knowledge of the system have any substantial gaps?
• Could you easily explain the system to a first time user?

Expression:

• Does the product seem 'true'—a credible account of a requirement'?
• Does the product express what will happen in ‘real’ world?
• Does the reality of the product match the expectations of the product?
• Does the product express unexpected ways of working?

_______________

To make it easier I have create a MS word document with the questions in which you can download from Google docs here.

*************

Emergent Strategy

How often have you been asked why do we do exploratory testing rather than planned and predicted scripts.  Recently I have been reading some material on corporate planning strategies and how some become successful and other do not and looking at how this links into software development especially from a testing perspective. 

Given that software can be very dynamic and react in some unpredictable ways no matter how much planning we do.  It surprises us and more importantly it surprises the person who created it.  This goes against the commonly held notation that software is predictable since we planned with great care and detail what was going to be coded.  The problem comes is that we are human and may not act in a rational way and this is reflected in our creations.  

So what is the best way to do software testing?  

The purpose of testing is to learn things about it and the best way of doing this is to experience it and learn by do it.  This is best summed up by Nassim Nicholas Taleb

 We are better at doing than learning. Our capacity for knowledge is vastly inferior to our capacity for doing things – our ability to tinker, play, discover by accident.

http://www.fooledbyrandomness.com/notebook.htm

Within the corporate strategy world I have discovered that this appears  to have a  name  'emergent strategy'  (or realized strategy).  Looking more into this I found the following link http://planningskills.com/glossary/154.php

One of the most interesting part of the above link to me that I noticed was the following sentence:

Emergent strategy implies that an organization is learning what works in practice

Is this not similar to what we attempt to do when doing exploratory testing?  We try to learn about the product and what is working or not working by experiencing  it?

An interesting point made in the article is the following statement

Mixing the deliberate and the emergent strategies in some way will help the organization to control its course while encouraging the learning process.

This appears to link back to an  article I wrote about  hybrid testing and having a mixture of scripted checks and  exploratory tests.

So to go back to my first statement about the purpose of exploratory testing.  To me since we cannot predict everything that the software will do the only way to understand what it will do and to learn is to explore it and that is the purpose of exploratory testing.  To uncover information that may prove to be of value to someone who matters.

 I may need to look more into emergent strategy a little more.

Place your bets

Having recently completed reading the excellent book “The Click Moment” by  Franz Johansson (@Frans_Johansson).  I was amazed by how much of the material written in the book appears to relate to software testing.  The principles of the book are about creating opportunities in an unpredictable world and not about putting in hours and hours of practice.  The author explains that if there are fixed rules and these rules do not change too much then the 10000 hours rule of practice works  .  However the author points out we are living in a world in which the rules are always changing and unexpected (random) things can and do happen.

Some of the statements made in the book appear to have a correlation to the current state of software testing and the various “schools”.  (The scripted vs the exploratory debate).  The first thing that caused me to think was some of the comments on planning  and how this stifles opportunities for random events and for uncovering new and exciting things.  

For example Franz stated the following

…In fact, it might mean that the plan is outdated before you even start to execute it….”

I have often experienced this within companies that believe that we can plan upfront and know all we need to know to write scripts before we actually use the product.  I have seen test plans which when I start doing some testing are hopelessly out of date and then spend unnecessary time trying to retrofit what I am experiencing when testing with what the plan is saying.  Doing this makes me take my eye of the ball and miss chances to find out what could be important information.

Franz then makes a statement which could be taken directly from why we need to do exploratory testing.

“..As ironic as it may sound, it actually pays to schedule time to do something unscripted and unplanned. We need to leave enough room in our day to explore things that are not connected to our immediate goals. We need to free ourselves up to become aware of hidden opportunities and expose ourselves to significant click moments. Leave some flexibility in your schedule. Then, make sure you use the flexibility to explore something unrelated to what you are doing or to follow up on a curious idea you have been considering”

This offers so much potential for uncovering new and valuable information without the restrictions of following someone else’s thinking.  This way of testing in my world can lead to many serendipity moments.

So how can we help to make this happen in software testing?  Is there anything we can do to help create more of these moments of randomness?

Franz within the book gives 5 great tips which may encourage more serendipity.  I have listed the tips below and give a description of how this could apply to exploratory testing.

1. Place Many Bets 

Having a single exploratory testing mission which can consist of an infinite number of tests (bets) is surely much better than having a single scripted test in which you are only placing one bet.

2. Minimize the Size of the Bets

Instead of spending lots of time creating a test script based upon assumptions do the minimum required to do some actual testing and time box your testing sessions.

3. Take the Smallest Executable Step

Do the minimal amount of planning you can do to enable you to do some exploratory testing.  We need to stop thinking if we write detailed test scripts and plans before we really know anything that this will lead to us uncovering lots of information about the system.

4. Calculate Affordable Loss, Not ROI

We still believe that there is a justifiable, measurable cost in planning ahead and creating detailed test plans and scripts.  Which we then discover are outdated and very costly to maintain but we insist they are useful because someone else may use them in the future.  Instead look at creating lots of test ideals using test models and heuristics  which are cheap to create and if of no use can easily be discarded once we uncover more information when testing.  We should be looking at testing and its cost effectiveness from what can we afford to throw away if our assumptions are wrong.

5. Use Passion as Fuel

This is so important people with a passion for what they are doing are the drivers of opportunities.  This type of person is one where if they get stuck or falter they pick themselves back up, dust themselves off and look for ways around the problems.   These are innovators, the people who can radically change the market and improve what is already there.  There is a need to employ more of these passionate types of people in the world of software testing.  I am getting fed up of the 9-5 testers, the ones who have no desire to learn or improve themselves, the ones not reading this blog.

I do recommend that anyone involved in software development read this book it gives some great advice of how we can adapt what we do and how we think to improve our chances of delivering successful projects. 

PS No I am not being paid by Franz for writing this 

Testers should learn code

At the Eurostar conference in Amsterdam  Simon Stewart (@shs96c) presented a keynote on Selenium over the years

During the key note Simon made the following comment

"If you are testing the web you absolutely need to be able to code"

Now I am sure that out of context this could be taken in many ways and Rob Lambert has produced a very good discussion on this same subject here which takes on both sides.

I will add is that Simon did follow this up with the following line.

"If not become a specialist so you can add value"

This led to some interesting exchanges on Twitter (search for #esconfs) in which people came down on either side. I had concerns that people would only hear the first bit and this could cause barriers to some great people being able to be involved in testing just because they have no interest in coding or even wanting to learn to code.

IMO I am not sure about this statement and it caused much debate after the key note.  If you have an interest in learning code then do so otherwise do something that can add value. The discussions continued during lunch and the rest of this article is my own thoughts on this subject.

After talking to Simon afterwards it appears his message had got taken the wrong way.  He said it is helpful to code and that if all you do is test (check) scripts then you may not have a job.

My concern is forcing people to code if they have no interest could be a block from great people wanting to enter the world of software testing.

Dot Graham stated the following "Lose a good tester but gain a poor programmer".

I am not convinced that everyone needs to code, it can have its advantages but there is another perspective. If you do not understand the code you may test in a different non-confirmatory way.  You may be able to ask the difficult questions of why did you do it this way and made it complex?  You may not have a bias built up from your coding experiences and knowledge.  I think for some it can be useful but insisting on it is a very dangerous path to follow.

Some of the discussions that followed went along the lines that if testers refused to learn we should not employ them.  This is where I had a WTF moment.....  I have not said anything about testers not wishing to learn what I was saying was some people may not have an interest or a knack for coding or find it impossible for whatever reason to grasp.  However they make one hell of a tester and show a great thirst for learning new ways to exercise the software that are novel, unique and valued.  This was the second point that Simon was making and sadly appeared to have been missed.  As long as you can find ways to add value then you can be a tester.

I am afraid that a statement of this sort can be used as a filter to prevent people entering this great world of software testing.  There are many other things that IMO testers could learn about such as grounded theory, anthropology, social sciences, humanities, creative arts and the list goes on.  There are some great testers who have learnt these things and should we prevent them for working as software testers because they have no desire to learn code?

I will finish this on a positive note and that it was great to chat with Simon even if our views are slightly different and that he is a very thoughtful and  passionate person.  I look forward to meeting up again sometime  in the future and finding another topic to discuss.

PS Thanks to Rob Lambert for being the referee!!

(edited some of the grammar :o( )