Friday 16 May 2014

Breaking down the Agile Testing Quadrants (with an exploratory testing context)

Recently there have been a few articles discussing the Agile Test Quadrants so I thought I would post some of my ideas and thoughts.

The purpose of this article is to look at how Exploratory Testing (ET) fits into the agile development model?  It investigates ways in which to utilize ET across all phases of development.  Its focus is on the ‘Agile Test Quadrants’ and how test engineers fit into these quadrants to support the team and test the product as early as possible.

What are the Agile Testing Quadrants?
First of all some of the readers may not be too familiar with what the Agile Testing Quadrants are.  The original concept came from Brian Marick and looked like figure 1.

Figure 1:  Brain Marick Agile Testing Quadrants


This concept was expanded upon by Lisa Crispin and Janet Gregory in their book Agile Testing – A practical guide, this is shown in figure 2 and to some is the most familiar agile testing quadrant.diagram.



Figure 2: Lisa Crispin and Janet Gregory Agile Testing Quadrants
.
Some observant readers may have noticed that I have adapted the diagram a little and added exploratory Testing across all quadrants..

Some interesting aspects of this diagram are that each quadrant is numbered and the people think is that they assume this is the order in which testing should be done in agile development projects.   These numbers do not represent an order or a timeline and there is some overlap within the quadrants with other quadrants.   To help simply it is easier to think that the quadrants are not mutually exclusive and that they can be covered in any order depending on the context on the agile team you are working with.

Within each of these quadrants some form of exploratory testing can be implemented even though the clouds within the diagram indicate tools or automated.  There could be value in doing some manual exploration.  The rest of this article focuses on each of the quadrants and even though they will follow the numbering system again it is important to repeat this does not mean this the logical order you MUST follow.

Quadrant 1 (BL)



Figure 3: Quadrant 1 (BL)

Traditionally this is or has been seen as a developer task.  Therefore how can tester get involved at this level?  There are many opportunities for testers to support the team within this quadrant by doing some of the following:
  • Explore the documentation and ask questions about what would be suitable in a unit test suite. Documentation can be defined as system model diagrams, emails, conversations, database schemas anything in which you can apply critical thinking skills to determine what may be of value to automate as a unit test.
  • Create the actual unit tests in a suitable framework for that project. For example JUnit uses very simple assertion statements that most testers should be familiar with (if not then that maybe a skill to learn)
  • When creating the unit tests explore opportunities to ensure common boundary/ error conditions are covered to ensure there is no need to duplicate this manually.

Quadrant 2 (TL)



Figure 4: Quadrant 2 (TL)


Traditionally that has been known as ‘system’ testing and where in the past testers first saw and interacted with the system under test.  This is an area where testers seem to be comfortable and can see where their role and value is for the team.  As we move forward with being agile there is more that the tester can do in this quadrant to support the team.
 
One vital task that testers should be involved with is ensuring that there is a Continuous Integration (CI) or Continuous Delivery system in place.  Without this is it difficult to move away from only checking the system since the team will be wasting effort on manually building and performing regression checks.
 
If there is CI in place then the testers can help support the automation effort by helping to create feature files in plain language these can be done using the cucumber framework. Whilst creating the feature files and before they are automated it is normal to step through the automation checks and during this time it is possible to do some exploration to uncover new potential automation checks or areas to explore in the future.  I have written an article on this which can be found here.  When doing this walk-through the tester can be looking for undesirable behavior within the use cases or user stories.  Also it is great opportunity to involve other members of the team to share knowledge and skills by pairing up with others in the team when doing the exploratory testing walk-through.

Quadrant 3 (TR)



Figure 5: Quadrant 3 (TR)


Traditionally this has been known as End to End Testing and within the model of the software development life-cycle (SDLC) comes at the end of the development of the code phase.  Since agile is about delivering working code with each and every check-in, the testers can be involved sooner in doing some exploratory testing.  It is expected that during a sprint some scenarios and features will be completed and available for testing.  The testers role involves using exploratory testing to ensure that the end to end system functions as expected.  Testers can support the team in prioritizing the tasks within the sprint to enable some parts of the end to end system to be testers as soon as possible. 
 
Testers should also look to use or create automation tools that help support manual exploratory testing.  If there are no tools available then this can become a task for the sprint or even added to the backlog.

Quadrant 4 (BR)


Figure 6: Quadrant 4 (BR)

Traditionally this quadrant is seen as non-functional testing, where load, stress and performance testing is carried out.  Normally this has been done in environments that are representative of the real system.  Testers can get involved in this at an earlier stage and find these non-functional testing issues earlier during an agile sprint.  Some of the ways that a tester can support the team in this quadrant is:
 
  • Exploring data variants and design models for performance, load and stress testing.  Normally when carrying out these types of tests the data being used is happy data.  Testers can explore stressing or loading the system with cases where the data is a mixture of valid and invalid values.  Experiment and discover how changing the data variant impacts performance and load.  
  • Explore ways to interact with the system which may compromise the security and integrity of the system. Even if our systems are being designed for business to business use there are opportunities to explore holes in the system in which scrupulous people (both internal and external) can exploit.  Just because a system is designed to not validate from outside the normal input entry points it does not mean it should not.  Testers can provide valuable information on how the system can be interacted with outside the design that may impact the stability or privacy of the system.
  • Use the ‘Test Eye’ software testing characteristics cheat sheet  to explore ‘illities’.  This gets the tester to ask questions such as “How is the testability of the system?  What is the usability of the system? What is the reliability of the system?

Conclusion

This article is intended to provide an overview of how test engineers can provide value for each of the agile testing quadrant by implementing an exploratory testing approach.  The key points that this article makes are:
 
  • Exploratory testing can be utilized across all Agile Testing Quadrants.
  • Early test engineer involvement is crucial for successful delivery of quality systems. 
  • The testing role is about supporting the team and the business rather than a fixed role within a fixed stage of testing.
  • Early use of exploratory testing will find issues sooner and prevent later fixes and associated costs in time and money.
  • To succeed there needs to be a continuous integration/delivery system in place.

Epilogue

Some observant readers may have noticed the lettering for each quadrant heading (BL/TL/TR/BL).  This change from numbers to letters was a concept that a tester in the testing community called Duncan Nisbet came up with to move away from the problem of the numbers being used to indicate a timeline or an order in which to do things. One of the reasons for publishing this article is the inspiration I got from reading the series of articles by Duncan. He has a series of articles on dissecting the agile testing quadrants which was the inspiration for some parts of this article.  Figure 7 is from his article and you can see how it has evolved from the one defined by Lisa Crispin and Janet Gregory in their book.

So thank you Duncan for inspiring to complete this work in progress.


Figure 7: Duncan Nisbet - Dissecting the Agile Quadrants

1 comment:

  1. Hey Steve,

    Firstly, thanks for the kind words in the Epilogue - I’m glad I could help get your work finished :-)

    I like how you frame the post in relation to Exploratory Testing - ET occurring throughout each of the quads is a very important point to remember!

    I do have a couple of thoughts/questions though…

    Labels:

    You provide some great context around the different examples in each of the quadrants, but I don’t see any reference to why those examples are in a particular quad(e.g Scenarios in TR quad Q3)

    Is that a deliberate exclusion?

    I only ask because you have left the labels in the cropped images, making me think they offer some value.

    What do the think about the labels as they stand in Lisa & Janet’s version of the quadrant?

    Tools & Processes:

    You have specifically mentioned a couple of tools / processes in the post - namely Cucumber, Junit, Sprints. I feel like adding the disclaimer “other tools & processes are available” for those of us not using the tools & processes mentioned in the post :-)

    Good work providing an example of how a Tester can provide value at the unit test level (boundary / error conditions) - that is the trickiest conversation I have when explaining my approach to testing to Programmers (These aren’t the tests you’re looking for….)

    In the conclusion:

    “The testing role is about supporting the team and the business rather than a fixed role within a fixed stage of testing."

    When I read that statement, at the end of walking through the model which talks about "business facing" & "supporting the team", I see testing as residing solely in the TL quad 2, which is the opposite of what the rest of the post was saying.

    I’m guessing that’s not what you meant?

    Finally:

    “To succeed there needs to be a continuous integration/delivery system in place.”

    Do you believe that? Can you think of another scenario which would disprove the statement above?

    Once again, nice post John, thanks for sharing. I always enjoy dissecting a model in ways I’ve not done before. It’s great to see how other folks see the world!

    As always, I’m open to a Skype chat as well.

    Duncs

    ReplyDelete