The Size of the Automation Box

I have previously written an article about doing too much automation and have recently been involved in successfully implementing automation and exploratory testing in agile environments.  During discussions about how to implement the automation framework a work colleague made the following statement in regards to defining the automation boundaries:

“Defining the size of the black box is critical”.  

 It made me think about the importance of knowing where your automation boundaries are and how critical it is for successful deployment of an automation framework.   One point to keep in mind when you try and implement an automation framework is the principal of K.I.S.S (‘Keep it simple stupid’)

I have seen (and been involved in) several failed automation implementations and in retrospect the main reason for failure has been trying to automate everything that you can possibly think to automate. I discussed this problem in the article too much automation and the importance of thinking critically about what to automate. Time and time again people go rushing ahead to try and automate everything they can without thinking about the boundaries of what their automation framework should be covering. 

 So what is the solution? The rest of this article will offer some guidelines that may help you in your quest of implementing a successful automation framework

The first thing to look at when trying to implement automation is the context of the testing you are carrying out or attempting to automate.  Are you involved in sub system testing, component testing, end to end testing, customer driven testing or any other type of testing?  It is important to look at this and found how what do you want the automation to do for you.  

• At Customer level you may want to automate the common customer scenarios.
• At a system level you may want to validate the input and outputs to a specific set of components and stub out the E2E system.

So defining your context of the purpose of your testing from an automation viewpoint is important.  The reason for this is that links to the second point, defining the size, or scope, of your automation for the project.  Most automation implementations fail to take into account the limits of what you intend to cover with the automation.  As my colleague put it 

The size of this will change depending on the context of the type of testing that you are carrying out.  For your automation implementation to be successful it is important that you define upfront your boundaries of what the automation is expected to cover.  Automation should be about what you input into the system and what you expect as an output everything in-between this is your black box .  So knowing what your testing context is should help define the size and limits of your black box knowing this information before you start to implement a test automation framework should greatly improve the chances of your automation implementation being successful.

Example of defining black box at a Component level of testing

    

Shaded Gray Area = Black box

For this example the boundaries would be

• Create automated test inputs for Input A in relation to defined requirements and expectations
• Automate expected outputs Output 1 and Output 2 based upon requirements and your expectations
• What happens inside of Component X is of no interest since this is outside the boundaries of what we are automating

Another example of defining black box at a Component level of testing

    

• Create automated test inputs for Input B in relation to defined requirements and expectations
• Automate expected outputs Output 1, Output 2, Output 3 and Output 4 based upon requirements and your expectations
• What happens inside of Component Y is of no interest since this is outside the boundaries of what we are automating

Example of defining black box at a Product level of testing

    

• Create automated test inputs for Input A and Input B in relation to defined requirements and expectations
• Automate expected outputs Output 1, Output 2, Output 3 and Output 4 based upon requirements and your expectations
• What happens inside of Component Y is of no interest since this is outside the boundaries of what we are automating

I am aware that this appears to be common sense but time and time again people fail to take on board the context of what they expect the automation to be covering.

So next time you are involved in implementing an automation framework remember to define the size of your black box before you start your implementation that way you know the boundaries of what your automation will cover and check for you.

The ‘Art’ of Software Testing

I was recently in Manchester, England when I came across the Manchester Art Gallery  and since I had some time spare I decided to visit and have a look around.  I have an appreciation for certain styles of art especially artists such as William Blake  and Constable.  During my visit I had a moment of epiphany.  Looking around the different collections, this appeared to be set out in a structured style, with different styles collated together.  Apart from an odd example of the famous “Flower Thrower" artwork by Banksy being placed in the seventeenth century art collection area.  I wondered if this was a deliberate action to cause debate.

What struck me when looking around was the fact that even though there were many similar painting techniques and methods being applied there was no standard size for any of the painting on display.  I looked around and could not find two paintings that appeared to have the same dimensions.  I even started to think if there was a common ratio being used and the so called golden ratio. Looking around quickly I could see some aspects of ratios being used but to my eyes it appeared that even though the artists used similar approaches and techniques they were ‘free’ to use these methods as a guide to producing their masterpieces.

This made me think about the debates in the field of software testing and how we should be taking on board engineering processes and practices.  If this is the case and we try to rein the imagination how are we supposed to hit upon moments of serendipity and be creative? I agree there needs to be structure and some discipline in the software testing world Session based testing takes account of this.

We have common methods and techniques that we can use in software testing but how we apply this should surely be driven by the context of the project.  I believe we need to stop or resist processes and best practices that hinder or suppress innovation and prevent those moments of enlightenment in which deeply hidden ‘bugs’ are discovered. Following pre planned and pre written test steps by rote can be useful but if we all follow the same path how many wonderful things are we going to miss.  I liken it to following a map or a GPS system and not looking around you at the amazing landscape that you are not noticing or appreciating.  In our field of software testing we must allow testers to explore, discover and sometimes find by accident.  We need to stop forcing processes, best practices and standards upon something which is uniquely human the skill of innovation and discovery.

The title of this article is homage to one of the first books I read about software testing by Glenford Myers – The Art of Software Testing

Tis the Season for Conferencing

It will soon be the start of the main software testing conference season and there are many people who will not be able to attend for lots of reasons.  So if you cannot attend in person then why not use social media to follow what is happening at the conference.  The best way I have found to do this is to use twitter and a hash tag designated for the conference.  I personally use twitter a great deal when attending conferences even going so far to automate my tweets when presenting at Lets test.   So I have gathered together as many hash tags I can think of for upcoming testing conferences so you can be virtually there.

If there are any I have missed please add them as comments and I will add them to this article giving credit of course.

For those who are attending a testing conference can I ask that you use twitter to keep others informed about the conference and some of the key points being said it helps the whole of the testing community.

For those organising a testing conference please make sure you have a hash tag for your conference and make it widely known.  There are some conferences organisers that do not have this in place and it is shame since it is way of drawing attention to your conference and for sharing knowledge beyond the confines of a physical location. It is also good to keep it the same each year instead of adding the year on that way it can be used all the time over a year and keep conversations going.

The following is a list of hash tags for upcoming testing conferences for which I could locate a hash tag for.

• #CAST2013 - CAST –  - 26-28th August 2013 - Madison, WI, USA

• #SIGIST - SIGIST Conference - 12th September 2013 - London, UK (@Stephen_J_Hill)

• #STARWEST – STARWEST -   29 Sept – 4 Oct 2013 Anaheim, California, USA

• #STPCon – Software Testing Professional - 21-24 Oct 2013 Phoenix, Az, USA

• #UKTMF  UK Test Management Forum - 30th October 2013, London, UK (@Stephen_J_Hill)

• #agiletd – Agile Testing Days - 28-31 October 2013 – Berlin, Germany

• #esconfs – Eurostar Test Conference - 4-7 November 2013 – Gothenburg, Sweden

• #bscadc – Better Software Conference East - 10-15th Nov 2013- Boston, MA

• #testbash – Test Bash - / 28th March 2014 – Brighton, UK

• #letstest – Lets Test Conference -  May 26-29th May 2014 – Runo , Sweden

• #bscadc – Better Software Conference west - 1-6 June, Las Vegas, NV, USA

• #letstest – Lets Test OZ - 15-17th September 2014, Gold Coast Australia

• #shefftest– Sheffield Test Gathering - TBD - Sheffield, UK (@TNRidgeback)

Also there are testing events being organised that do not require you to pay.

• Software Testing Club Meet ups
• Weekend Testing

If there is not one near you, why not organise one?

For those attending a testing conference I would recommending reading the excellent guide to attending a conference written by Rob Lambert (@Rob_Lambert)

Stop Doing Too Much Automation

When researching my article on testing careers for the Testing Planet a thought stuck me about the amount of people who indicated that ‘Test’ Automation was one of the main learning goals of many of the respondents.  This made me think a little about how our craft appears to be going down a path that automation is the magic bullet that can be used to resolve all the issue we have in testing.

I have had the idea to write this article floating around in my head for a while now and the final push was when I saw the article by Alan Page (Tooth of the Angry Weasel) - Last Word on the A Word  in which he said much of what I was thinking. So how can I expand on what I feel is a great article by Alan?

The part of the article that I found the most interesting was the following:

“..In fact, one of the touted benefits of automation is repeatability – but no user executes the same tasks over and over the exact same way, so writing a bunch of automated tasks to do the same is often silly.”

This is similar to what I want to write about in this article.  I see time and time again dashboards and metrics being shared around stating that by running this automated ‘test’ 1 million times we have saved a tester running it manually 1 million times and therefore if the ‘test’ took 1 hour and 1 minute to run manually and 1 minute to run automated it means we have saved 1 million hours of testing.  This is so tempting and to a business who speaks in value and in this context this mean costs.  Saving 1 million hours of testing by automating is a significant amount of cost saving and this is the kind of thing that business likes to see a tangible measure that shows ROI (Return on Investment) for doing ‘test’ automation.  Worryingly this is how some companies sell their ‘test’ automation tools.

If we step back for a minute and go back and read the statement by Alan.  The thing that most people who state we should automate all testing talk about the repeatability factor.  Now let us really think about this.  When you run a test script manually you do more than what is written down in the script.  You think both critically and creatively , you observe things far from the beaten track of where the script was telling you to go.  Computer see in assertions, true or false, black or white, 0 or 1, they cannot see what they are not told to see.  Even with the advances in artificial intelligence it is very difficult for automation systems to ‘check’ more than they have been told to do.  To really test and test well you need a human being with the ability to think and observe.   Going back to our million times example.  If we ran the same test a million times on a piece of code that has not changed the chances of find NEW issues or problems remains very slim however running this manually with a different person running it each time our chances of finding issues or problems increases.  I am aware our costs also increase and there is a point of diminishing returns.  James Lyndsay has talked about this on his blog in which he discusses the importance of diversity   The article also has a very clever visual aid to demonstrate why diversity is important and as a side effect it helps to highlight the points of diminishing return. This is the area that the business needs to focus on rather than how many times you have run a test.

My other concern point is the use of metrics in automation to indicate how many of your tests have you or could you automate.  How many of you have been asked this question?  The problem I see with this is what people mean by “How many of your tests?”  What is this question based upon?  Is it...

• all the tests that you know about now?
• all possible tests you could run?
• all tests you plan to run?
• all your priority one tests?

The issue is that this is a number that will constantly change as you explore and test the system and learn more. Therefore if you start reporting it as a metric especially as a percentage it soon becomes a non-valuable measure which costs more to collect and collate than any benefit it may try to imply.  I like to use the follow example as an extreme view.

Manager:  Can you provide me a % of all the possible tests that you could run for system X that you could automate.

Me:  Are you sure you mean all possible tests?

Manager: Yes

Me: Ok, easy it is 0%

Manager:  ?????

Most people are aware that testing can have an infinite amount of tests even for the most simple of systems so any number divided by infinity will be close to zero, therefore the answer that was provided in the example scenario above.  Others could argue that we only care about of what we have planned to do how much of that can be automated, or only the high priority stuff and that is OK, to a point, but be careful about measuring this percentage since it can and will vary up or down and this can cause confusion.  As we test we find new stuff and as we find new stuff our number of things to test increases.

My final worry with ‘test’ automation is the amount of ‘test’ automation we are doing (hence the title of this article) I seen cases where people automate for the sake of automation since that is what they have been told to do.  This links in with the previous statement about measuring tests that can be automated.   There need to be some intelligence when deciding what to automate and more importantly what not to automate. The problem is when we are being measured by the number of ‘tests’ that we can automate human nature will start to act in a way that makes us look good against what we are being measured. There are major problems with this and people stop thinking about what would be the best automation solution and concentrate on trying to automate as much as they can regardless of costs. 

What ! You did not realise that automation has a cost?  One of the common problems I see when people sell ‘test’ automation is they conveniently or otherwise forget to include the hidden cost of automation.  We always see the figures of the amount of testing time (and money) being saved by running this set of ‘tests’ each time.  What does not get reported and very rarely measured is the amount of time maintaining and analysing the rests from ‘test’ automation.  This is important since this is time that a tester could be doing some testing and finding new information rather than confirming our existing expectations.  This appears to be missing whenever I hear people talking of ‘test’ automation in a positive way.  What I see is a race to automate all that can be automated regardless of the cost to maintain. 

If you are looking at implementing test automation you seriously need to think about what the purpose of the automation is.  I would suggest you do ‘just enough’ automation to give you confidence that it appear to work in the way your customer expects.  This level of automation then frees up your testers to do some actual testing or creating automation tools that can aid testing.  You need to stop doing too much automation and look at ways you can make your ‘test’ automation effective and efficient without it being a bloated, cumbersome, hard to maintain monstrosity (Does that describe some peoples current automation system?)  Also automation is mainly code so should be treated the same as code and be regularly reviewed and re-factored to reduce duplication and waste.

I am not against automation at all and in my daily job I encourage and support people to use automation to help them to do excellent testing. I feel it plays a vital role as a tool to SUPPORT testing it should NOT be sold on the premise that that it can replace testing or thinking testers.

Some observant readers may wonder why I write ‘test’ in this way when mentioning ‘test’ automation.  My reasons for this can be found in the article by James Bach on testing vs. checking refined.

Small Steps in a Big Organisation

I recently did a talk about taking "Small Steps in A Large Organisation" at the Israel SIGIST in Tel Aviv - http://www.istqb.org/newsevents/events/istqb-conference-network/sigist-israel-conference-2013.html

I  have had an article based upon the talk published on the Eurostar blog page
https://conference.eurostarsoftwaretesting.com/2013/taking-small-steps-in-a-big-organization/ to go along with the presentation.

Once the details of the recording from the conference are made available I will update this article.

Measuring Test Coverage

This article follows on from my previous article on Why we need to explore and looks at when we simplify the constructs of software development to expectations and deliverables how measuring test coverage becomes a difficult task.  I should acknowledge that the model I use here is extremely simplified and is only to used to aid clarification. There are many more factors that are involved especially within the expectations section as Michael Bolton quite rightly commented about in the previous article.

If we go back to our original diagram (Many thanks to James Lyndsay) which shows our expectations and our deliverable and the it where they meet is where our expectation are met by the deliverable.

At a simple level we could then make the following reasonable deduction (at a simplified level)

We can express all our known expectations as 100% and therefore for measurement purposes say that x % of our expectations have been met by the deliverable and y % have not been met.  This gives us a simple metric to measure how much of our expectations have been met.  This seems very clear and could to some people be a compelling measurement to use within testing.    The following diagram gives a visual reference to this.

This is only half the story since on the other side the part where we need to do some exploring and experimentation.  This is the stuff in the deliverable that we do not know or expect.  This is the bread and butter of our testing effort.  The problem is since we do not know what is in this area or how big or small it is (I will return to that point later).  We are now in a measurement discomfort zone, how do we measure what we do not know?  The following diagram shows  a visual representation of this.

This measurement problem is also compounded by the fact that as you explore and discover more about the deliverable you tacit knowledge can become more explicit and your expectations start to grow.  So you end up in the following situation:

Now your expectation percentage is 100%+ and as you explore more continuously growing. So your % of meeting and not meeting your expectations becomes a misleading and somewhat pointless metric.

I was ask if there is anything that could be done to increase the area where the expectations are met by the deliverable and this lead to me adding another diagram as shown below.

**Still not to scale

Since testing in theory could be an infinite activity, how much testing we do before we stop is determined by many factors, Michael Bolton has a superb list as an article here. 

In summary the amount we know and expect from a piece of software is extremely small in comparison to what we do not know about the software (deliverable) hence my first post in the article series on the need to explore the system to find out the useful information.  We need to be careful when using metrics to measure progress of testing especially when that measurement appears easy to gather.

Further Reading on Metrics and Testing.

• But how many test cases?
• Test Cases Counting reflections
• Response on how many test cases
• Software Engineering Metrics: What Do They Measure and How Do We Know?
• The Darker Side of Metrics
• What counting measurements in testing
• Measurement Issues and Software Testing
• The Impossibility of Complete Testing
• Why 100% test pass rates are bad
• Test Estimation and the Art of Negotiation
• End to End Software Development Traceability with Mind Maps
• Reporting dashboard example
• Meaningful Measures
• Meaningful Metrics - Michael Bolton
• Three Kinds of Measurement (And Two Ways to Use Them)
• Issues About Metrics About Bugs
• Got you Covered.
• Cover or Discover
• A Map by any other name

Why we need to explore

This article looks at why within testing we need to carry out exploratory testing.

It is based upon the following diagram that James Lyndsay  drew during a recent diversity workshop.

When we talk about expectations we mean something that is written upfront , or known about before delivery of the product.  For example requirements or specification documentation.  There are other types of expectation that depend on the person such as experiences, domain knowledge or business knowledge.

The deliverable is the product being delivered and is normally the software under test.

When the product is delivered sometimes the expectations are met by deliverable and this is the area shown on the diagram where the ovals overlap and other times they are not.  Where they do overlap we can say that the deliverable meets the expectations in this area.  This could be seen as what we would with the testing world call verifying requirements, user stories, stuff/information that is known or that we know about.

**Note the area where expectation meets deliverable is not to scale or representative of a real world situation it is for clarification purposes only

The expectations to the left of the overlap are where our expectations are not met by the deliverable and these could be defined as bugs, defects, issues or a misunderstanding of what was expected.

The stuff to the right of the overlap is stuff in the deliverable that you did not expect and is the stuff you need to use exploratory testing to discover what could be vital and important information to someone who matters.  This is the stuff you do not know about, nor could you have known about until you started to do some testing.

Outside the ovals is an area in which you did not have any expectations nor is in the deliverable and as such is not important with regards to testing for the purpose of this article.

The following diagram is a visual representation of what I wrote above.

With this as a model we can now start to think about what James Bach and Michael Bolton discussed with reference to the terms checking and testing.  I see that for the stuff we know about (the expectations) we should look at covering this with machine checking (automation). This does not mean we should automate ALL expectations we need to base our automation on many factors cost, priority, risk, etc.  (I plan a future article on not doing too much automation). If we do this we should have some confidence that our deliverable meets our expectations.  This then allows testers to start to do some 'testing' and uncover the useful information that exists in the deliverable that no one knows about until some testing is carried out.

This is why IMO the only way we can test is not by following scripts (even though they can be useful to aid some elements of meeting our expectations) but by exploring the deliverable and finding out what it is actually doing that we did not expect it to do.

My next article will extend on this model and revisit measuring test coverage.