An Open Letter to Professional Tester Magazine

Dear editors of professional tester magazine

It is not often that I get a strong enough reason to become political and reply to some editorial pieces within a published article.   It is against my better nature to enter into debates which one side attacks another side but unfortunately your recent articles have been on mind for a few days and I feel the need to reply to the following articles on your website.

• Book burners threaten (old) new testing standard
• PT’s independence is questioned

I feel the only way to express my disappointed is by use of social media, I 'had' a great deal of respect for your magazine and your in-depth reports and wide variety of articles, sadly that has diminished.

My concerns were raised upon the publication of the article about 'Book burners' in relation to the campaign instigated at the CAST conference to 'suspend' the publication of the ISO/IEC/IEEE 29119 software testing standards by means of a petition and enable some open debate on the validity of these standards too the software testing profession. NOTE the wording it says suspend it does not say as your article states 'suppress' which indicates that these people signing the petition wants to "forcibly put an end to." That is not the purpose of the petition. Its' purpose is to allow those who will be impacted by these standards a voice and  input.  That I feel is fair enough for any democratic process and hence why I proud to sign this petition.

Now let me get to the part that really got me annoyed.  The use of the term "Book burners".

• Did the author of this article understand what this term means?  
• Did they understand how much of an offensive term this is?
• Was this term deliberately chosen to ensure controversy?
• Was the author naive in choosing this term without understanding it meaning?

I have been attempting over the past couple of days to attempt to answers these questions and at one point I felt it was just me being over sensitive. Then the second article was published about 'PT independence is questioned' in which the author appears, to me, to take the higher moral ground and attack some within the testing community for making some assumptions about linkedin and comments being deleted, which they quickly retracted and apologized for once it was understood as a bug within linkedin.  This did not stop the author of the article stating that the comments made were deliberately false and meant to damage the reputation of the magazine.  I can understand how the magazine editors must have felt,  however this was an opportunity for the editors of the magazine to also apologize for their attack on some within the testing community by the use of the term 'book burners', sadly none was forthcoming.

To clarify the term 'book burner' is used against those who are attempting to suppress freedom of speech.

 Book burning can be emblematic of a harsh and oppressive regime which is seeking to censor or silence an aspect of a nation's culture - Wikipedia

Can you as authors of the magazine see the irony of using this term against those who are attempting to make something open and debatable rather than hidden behind closed doors?

Another ironic aspect is that those within the testing community who I engage with are amongst those who read a wide variety of books even those books that they may disagree with.  They are also writers, publishers and authors themselves to so use such a term is so derogatory and offensive to many of these people.

All I ask from this open letter is to acknowledge that using the term 'book burners' just may have been over the top, unjustified and offensive to those with the software testing profession who care about testing. 

To end this on a positive note I do intend to continue reading your magazine and will not encourage anyone to not read it or avoid it, since that serves no purpose.  I look forward to many more great articles from your magazine and hopefully some will be edgy and promote debate so all of those within the testing profession can be encouraged to learn from all the diverse views that this profession has. 

your sincerely

John Stevenson

A professional tester

If you care about the future of software testing please look at signing the petition to get ISO 29119 suspended and enabling a debate on its content by going here - http://www.ipetitions.com/petition/stop29119

Also have a look at the professional tester manifesto here - http://www.professionaltestersmanifesto.org/ and if you agree look at signing this too.

Experience Report - Small Steps - Agile 2014

I gave a  talk on 'Taking Small Steps in a Big Organisation (An experience report on implementing ET)'. at the Agile Alliance Conference in Orlando, Florida, USA.

The following link downloads the experience report I submitted with the talk.

Experience Report

It is in MS word format

A discussion on do testers really need to code.

Recently there have been a few posts talking about testers learning to code.

• Rob Lambert: - Why Testers Really Should learn to Code
• Fiona Charles - Why I believe its good for testers to learn to code
• Anand Ramdeo - Do Software Testers need to learn coding
• Maaret Pyhäjärvi  - Learning to Code - Just Learning

I find these discussions interesting and at the same time very worrying.  I have seen many posts that say testers ‘should’ or ‘need’ to code as shown from some of the examples above, excluding the one from Maaret. To make it clear none of the posts dictate that testers MUST code, but they do appear to give an impression that for testers to make themselves employable they sure should learn how to code.

My temptation was to reply to each of these posts as comments but in the end felt that would not be a great platform to get across my concerns and views on this topic, a topic that keeps coming around again and again.  I am not against testers learning to code but feel that going down this path could have a significant impact on the testing profession in the future.

What do people mean when we say "Learn to Code."?

To the majority of people learning to code means learning a programming language and being able to write an application/script in that language. I can see,  to some, in certain situations where that can be very useful, however do we really need to learn coding or would just having an understanding of the syntax of code be just as useful?

What does this mean when I say syntax?

The Oxford dictionary defines Syntax as:

The arrangement of words and phrases to create well-formed sentences in a language:

Or

The structure of statements in a computer language.

http://www.oxforddictionaries.com/us/definition/english/syntax

E.g.: the syntax of Java or Python.

Having an understanding of the structure of a programming language does not necessary mean you can code an application in that language but you should be able to read and follow lines of code and have a fair idea of what it means and what it does.  The first key point in this article is that we need to move away from creating generic testers who all code to ones that have a fair grasp of the syntax of languages.  This way we will not excluding, what could be, great testers, by turning them away from testing by insisting they learn coding, which is something they may have little or no interest in.  An English major for example could make a wonderful tester by understanding the subtle written words given by a customer to the project team and providing a multitude of creative testing ideas.  Do we really want to exclude these types of people from the testing profession?

This leads on to my next concern, if we insist that all testers need to learn to code we miss out on building diverse teams with many skills that compliment and sometimes conflict each other.  This enables much deeper critical thinking processes and ideas to emerge.  It is a concern that if we insist on testers all learning to code or need to code, then becoming an entry barrier into testing then we could lose a great deal of diversity among software development teams. We instead get a set of ‘cookie cutter’ clones who all code, think and work in similar ways.  This will diminish the testing talent pool, as those who would love to become a tester get set aside for an ex developer who can code but now wants to try their hands at testing. This is not to say that ex developers do not make great testers, we just to need to welcome people with a diverse range of skills and expertise into this great career rather than limit it to those with a coding background or skill set.  Also what do we mean by coding?  I touched on this earlier, within software development coding normally indicates being able to program.  However in social science and especially ethnographic research coding has a different meaning.

Coding is an important technique in qualitative research such as anthropology, ethnography and other observer and participant-observer methods.

http://changingminds.org/explanations/research/analysis/ethnographic_coding.htm

If we insist that testers need to learn code, how about we twist this around and say that testers need to learn social science methods and use that in our testing activities.  To quote the title of the article by Rob Lambert:

“Why testers should really learn to code”

If the code Rob was referring to above was the social science technique and he stated in his article that all testers ‘should really’ learn to do social science coding.  How much debate would that have caused?  If others insisted that every tester should really have this skill and that without it you have a far less chance of being employed as a tester, how would you react? Does a tester having this analytical skill make them any less important to a development team than someone who can do programming? I am unsure if this is the case, since the cognitive thinking skills needed to do programming and to do testing are different as mentioned extremely well in the article by Maaret.

One argument put forward is that testers who can code can create their own tools to do tasks such as data generation, performance, load or stress. I do not see a problem with this but I would add that there are many FREE tools available that may or could do the job without the need to re-invent the wheel. For example for data generation I use Excel  and PerlClip both of which suit my needs.  I have a tool inventory that for the majority of my tool requirements perfectly meets my needs.  I have seen many cases of people writing tools or scripts for the sake of doing so instead of researching or asking advice on tools that are already available.  This is not to say there are situations where creating or writing a tool could be more useful and time saving.  I have in the past done this and my programming skills are not great and I would rate as rather mediocre.

We have to be careful about the barriers we put in place for those who show an interest in having a career in testing.  Sure they need to have certain characteristics such as:

•  A willingness to learn.
• Self-improvement
•  Critical and creative skills
• Plus many more.

Classifying this to one particular skill such as programming rather than a characteristic, such as analytical thinking or logical thinking in my opinion is very narrow minded.

One other aspect that I have not seen mentioned in any of the articles is the issue of bias caused by thinking in a similar way.  If there is a continuing trend to insist that testers learn to code then there could be a convergence towards similar thinking for development and testing.  This may lead to bias in the approach to testing where the effort and focus is on the logical way in which the code works rather than an outward view of how the user may, or may not interact with the application.  Humans are not logical and will not follow the path you expect them to do so.  This is what I feel distinguishes good from great testers, the ability to think critically and apply creative ideas in a way that exposes flaws or issues within the system under test.  If we  continue to insist that testers should really code we could end up with a set of people developing software in which everyone thinks in a similar way and there is no movement from the well-trodden path.

If people have no interest in learning about coding and instead have a passion for testing and how to improve testing then we need to be careful about the message we are sending out from this profession.  I would rather have people working with me who are passionate, driven to learn and improve themselves and others than an excellent coder who want to do just a 9 to 5 job and no more.

I will finish this post with the following quote:

“We are not supposed to all be the same, feel the same, think the same, and believe the same. The key to continued expansion of our Universe lies in diversity, not in conformity and coercion. Conventionality is the death of creation.” Anthon St. Maarten - Divine Living: The Essential Guide To Your True Destiny

EDIT : Corrected some typos.

First four chapters completed

As some of you may be aware I have been spending some time writing a book (http://www.steveo1967.blogspot.com.es/2013/12/writing-book.html) about psychology and software testing.  It has been a huge learning experience and at times I have questioned myself on if this is something that people really want and more importantly willing to pay for.

I have kept pushing back the publishing date due to a number of reasons.  One of the main reasons has been do I publish early, even though the book is not complete at a discount price.  Since I am using LeanPub and can do this and get feedback on what i have done already.  Would people still be willing to buy a book that is not complete?  Having spoken to a few people there have been some mixed reactions.  Some people like the idea that they can get an unfinished book at a cheap price and then get the updates for free.  Others would rather wait and pay for a completed book.  So this dilemma has been running around my head for awhile now and even though I have about 30% of the book complete I was still reluctant to publish.

So it now comes to the reason for this post.

I have decided to go ahead and publish the first four chapters of the book.

These chapters include the following topics:

• Chapter 1 Software Testing Biases
• Chapter 2 An Emotional Rollercoaster
• Chapter 3 The Fallacy of Intuition
• Chapter 4 Quality Preference or Perception

I have another chapter on learning and training almost complete and once it has gone through the editing process it will be added to the published book.

As more of the book is completed I will be increasing the price by a dollar or two once a new chapter has been added.

Some of the topics (chapters) I plan to add in the future include:

• Heuristics Good and Bad
• Teamwork positive and negative
• Building Passion
• Automation: Machines and Humans
• The Importance of critical thinking
• System Thinking
• Being Creative

Some of these are, at the moment, a dump of my thoughts and ideas, others are nearly completed chapters.

I do hope you enjoy the book and please provide me with feedback and I may give you a mention in the book..

https://leanpub.com/thepsychologyofsoftwaretesting

Breaking down the Agile Testing Quadrants (with an exploratory testing context)

Recently there have been a few articles discussing the Agile Test Quadrants so I thought I would post some of my ideas and thoughts.

The purpose of this article is to look at how Exploratory Testing (ET) fits into the agile development model?  It investigates ways in which to utilize ET across all phases of development.  Its focus is on the ‘Agile Test Quadrants’ and how test engineers fit into these quadrants to support the team and test the product as early as possible.

What are the Agile Testing Quadrants?

First of all some of the readers may not be too familiar with what the Agile Testing Quadrants are.  The original concept came from Brian Marick and looked like figure 1.

Figure 1:  Brain Marick Agile Testing Quadrants

This concept was expanded upon by Lisa Crispin and Janet Gregory in their book Agile Testing – A practical guide, this is shown in figure 2 and to some is the most familiar agile testing quadrant.diagram.

Figure 2: Lisa Crispin and Janet Gregory Agile Testing Quadrants

.

Some observant readers may have noticed that I have adapted the diagram a little and added exploratory Testing across all quadrants..

Some interesting aspects of this diagram are that each quadrant is numbered and the people think is that they assume this is the order in which testing should be done in agile development projects.   These numbers do not represent an order or a timeline and there is some overlap within the quadrants with other quadrants.   To help simply it is easier to think that the quadrants are not mutually exclusive and that they can be covered in any order depending on the context on the agile team you are working with.

Within each of these quadrants some form of exploratory testing can be implemented even though the clouds within the diagram indicate tools or automated.  There could be value in doing some manual exploration.  The rest of this article focuses on each of the quadrants and even though they will follow the numbering system again it is important to repeat this does not mean this the logical order you MUST follow.

Quadrant 1 (BL)

Figure 3: Quadrant 1 (BL)

Traditionally this is or has been seen as a developer task.  Therefore how can tester get involved at this level?  There are many opportunities for testers to support the team within this quadrant by doing some of the following:

• Explore the documentation and ask questions about what would be suitable in a unit test suite. Documentation can be defined as system model diagrams, emails, conversations, database schemas anything in which you can apply critical thinking skills to determine what may be of value to automate as a unit test.
• Create the actual unit tests in a suitable framework for that project. For example JUnit uses very simple assertion statements that most testers should be familiar with (if not then that maybe a skill to learn)
• When creating the unit tests explore opportunities to ensure common boundary/ error conditions are covered to ensure there is no need to duplicate this manually.

Quadrant 2 (TL)

Figure 4: Quadrant 2 (TL)

Traditionally that has been known as ‘system’ testing and where in the past testers first saw and interacted with the system under test.  This is an area where testers seem to be comfortable and can see where their role and value is for the team.  As we move forward with being agile there is more that the tester can do in this quadrant to support the team.

 

One vital task that testers should be involved with is ensuring that there is a Continuous Integration (CI) or Continuous Delivery system in place.  Without this is it difficult to move away from only checking the system since the team will be wasting effort on manually building and performing regression checks.

 

If there is CI in place then the testers can help support the automation effort by helping to create feature files in plain language these can be done using the cucumber framework. Whilst creating the feature files and before they are automated it is normal to step through the automation checks and during this time it is possible to do some exploration to uncover new potential automation checks or areas to explore in the future.  I have written an article on this which can be found here.  When doing this walk-through the tester can be looking for undesirable behavior within the use cases or user stories.  Also it is great opportunity to involve other members of the team to share knowledge and skills by pairing up with others in the team when doing the exploratory testing walk-through.

Quadrant 3 (TR)

Figure 5: Quadrant 3 (TR)

Traditionally this has been known as End to End Testing and within the model of the software development life-cycle (SDLC) comes at the end of the development of the code phase.  Since agile is about delivering working code with each and every check-in, the testers can be involved sooner in doing some exploratory testing.  It is expected that during a sprint some scenarios and features will be completed and available for testing.  The testers role involves using exploratory testing to ensure that the end to end system functions as expected.  Testers can support the team in prioritizing the tasks within the sprint to enable some parts of the end to end system to be testers as soon as possible. 

 

Testers should also look to use or create automation tools that help support manual exploratory testing.  If there are no tools available then this can become a task for the sprint or even added to the backlog.

Quadrant 4 (BR)

Figure 6: Quadrant 4 (BR)

Traditionally this quadrant is seen as non-functional testing, where load, stress and performance testing is carried out.  Normally this has been done in environments that are representative of the real system.  Testers can get involved in this at an earlier stage and find these non-functional testing issues earlier during an agile sprint.  Some of the ways that a tester can support the team in this quadrant is:

 

• Exploring data variants and design models for performance, load and stress testing.  Normally when carrying out these types of tests the data being used is happy data.  Testers can explore stressing or loading the system with cases where the data is a mixture of valid and invalid values.  Experiment and discover how changing the data variant impacts performance and load.  
• Explore ways to interact with the system which may compromise the security and integrity of the system. Even if our systems are being designed for business to business use there are opportunities to explore holes in the system in which scrupulous people (both internal and external) can exploit.  Just because a system is designed to not validate from outside the normal input entry points it does not mean it should not.  Testers can provide valuable information on how the system can be interacted with outside the design that may impact the stability or privacy of the system.
• Use the ‘Test Eye’ software testing characteristics cheat sheet  to explore ‘illities’.  This gets the tester to ask questions such as “How is the testability of the system?  What is the usability of the system? What is the reliability of the system?

Conclusion

This article is intended to provide an overview of how test engineers can provide value for each of the agile testing quadrant by implementing an exploratory testing approach.  The key points that this article makes are:

 

• Exploratory testing can be utilized across all Agile Testing Quadrants.
• Early test engineer involvement is crucial for successful delivery of quality systems. 
• The testing role is about supporting the team and the business rather than a fixed role within a fixed stage of testing.
• Early use of exploratory testing will find issues sooner and prevent later fixes and associated costs in time and money.
• To succeed there needs to be a continuous integration/delivery system in place.

Epilogue

Some observant readers may have noticed the lettering for each quadrant heading (BL/TL/TR/BL).  This change from numbers to letters was a concept that a tester in the testing community called Duncan Nisbet came up with to move away from the problem of the numbers being used to indicate a timeline or an order in which to do things. One of the reasons for publishing this article is the inspiration I got from reading the series of articles by Duncan. He has a series of articles on dissecting the agile testing quadrants which was the inspiration for some parts of this article.  Figure 7 is from his article and you can see how it has evolved from the one defined by Lisa Crispin and Janet Gregory in their book.

So thank you Duncan for inspiring to complete this work in progress.

Figure 7: Duncan Nisbet - Dissecting the Agile Quadrants

Maybe they should have thought about that.

Recently I was working in the USA with some of our teams, during this time I unfortunately ended up having to visit a medical centre due to getting a chest infection.  This article is my experience of this visit from a testing perspective.

Some background context, I am a resident of the United Kingdom; my permanent address is in the UK format of:

Line 1 – the addressee's name
Line 2 – building number and street name
Line 3 – locality name, if required
Line 4 – POST TOWN, please print in capitals
Line 5 – POSTCODE, please print in capitals, in full and on a separate line

Courtesy of http://www.royalmail.com/personal/help-and-support/How-do-I-address-my-mail-correctly

I arrived at the medical centre, approached the receptionist and explained my need to see a doctor.  The receptionist asked if I had visited the medical centre before.  I replied that this was my first visit and the receptionist stated that I would need to fill out some registration details.  They handed me an Ipad which has some medical registration software installed.  I sat down and proceeded to enter the details the software was asking.  Standard information was asked for such as name, DOB (in the American format of mm/dd/yy), previous conditions.  After about ten minutes of filling in personal information the system then asked for my home address, with each line being a separate screen.  I entered the details for my home address in the UK, including country and then I came across a problem.  (Surprise surprise) It asked me to enter a ZIPCODE.  I proceeded to enter my POSTCODE and the system reported this was not a valid ZIPCODE and would not let me proceed.  There was no option to bypass this and the system did not seem to be able to deal with international visitors. So after wasting fifteen minutes entering my details on a system the receptionist gave me a pen and a printed form.  This form did not have anything for stating if you were an international visitor; it still only had an option for a ZIPCODE. I completed the form and handed it back.

I eventually saw a doctor, got the required medicine and went to pay for the treatment.  This is where the fun started!  I needed an invoice to be able to claim back the costs via my medical insurance.  The receptionist could not enter my details on to the billing system without a valid US ZIPCODE!  After an hour of attempting many ways to get around this I gave them my office address and was able to get a copy of the invoice.  So in total what should be have been an hour visit to the medical center ended up being a three hour visit.

We are living in a global world where it is more and more common for people to be working away from their normal residency it would make sense for software to be able to deal with international visitors.  Maybe whoever tested this system should have thought about this and tested that people from outside the USA can register on the system. 

So the next time you are testing a system in which address details are being requested maybe think about what are the chances that someone who is not from that country, in which the system is being used, may need to register.  If there is a possibility that this could happen then test for that possibility.  

Using Exploratory Testing to Drive Automation

Traditionally automation and exploratory testing have been seen as independent tasks that work in different ways and normally do not appear to be ideal partners. This article suggests that there is value in utilizing the work done when automating to add some exploratory testing effort.

In most cases there is some form of documentation or communication which indicates what the system being designed will do. This can be in the form of requirements, design specification, email, design meetings and so forth. An assumption has been made here that testers are involved in the design phase, if they are not for your project then they should be.

Within the context in which I work these documents/communications are analyzed for automated cucumber scenarios and exploratory charters. For the automation scenarios these are created using the Gherkin format:

Feature: Password management
Scenario: Forgot password
Given a user with email "someuser@example.com" exists
When I ask for a password reset
Then an email with a password reset link should be sent

Exploratory charters are created in the Exploratory Charter format as described by Elisabeth Hendrickson in the book Explore It!:

Explore (target)
With (resources)
To discover (information)

Where:

·                  Target: Where are you exploring

·                  Resources: What resources will you bring with you

·                  Information: What kind of information are you hoping to find?

Once the exploratory testing charters and cucumber scenarios have been initially defined the Test Engineer manually steps through the cucumber scenario to determine the pre-requite components needed to complete the scenario from an automation perspective, at the same time adding details to the Gherkin feature file. This may involve building the environment and integrating the components. It is at this point that some level of exploratory testing can be performed to gain more value from this manual effort. This manual effort can be recorded on a wiki or some other similar tool. The suggestion for using a wiki is the ease of access and use for cross regional teams. It is not mandated for teams to us a wiki but from experience it appears to be the best way to share information.

This will require a mindset shift for the test engineer instead of only focusing on the automation scenario they should be looking for future opportunities to test either automated or manually and these should be fed back to the team and added as backlog items. One other benefit is that as you explore the automation scenario you may uncover issues or defects earlier which are always a good benefit for the project.

Once the step definitions have been coded and implemented for the automation scenario they should feed into whatever automation reporting system you have selected. This should report scenarios pass or fail. It is expected that defects should not be found at this stage and if scenarios fails it is indication that your expectations of what the system does has changed and should be investigated before any possible defects are raised. This is a simple approach to close the link between test automation and exploratory testing and leverage the skills of the test engineers to their full value.

It should be noted that there will still need to be more effort to ensure that the exploratory charters are run manually and recording using the current approaches for your projects.  This then become a continuous cycle as more scenarios are discovered for automation then more exploratory testing can be done and more information about the system can be uncovered.  This approach makes exploratory testing and automation partners rather than as independent entities.  It hopefully reduces the checking vs testing debate and focuses on utilizing the skills of testers to deliver a product that customers will enjoy using.

The diagram below is a simple visually representation of this concept.