The Size of the Automation Box

I have previously written an article about doing too much automation and have recently been involved in successfully implementing automation and exploratory testing in agile environments.  During discussions about how to implement the automation framework a work colleague made the following statement in regards to defining the automation boundaries:

“Defining the size of the black box is critical”.  

 It made me think about the importance of knowing where your automation boundaries are and how critical it is for successful deployment of an automation framework.   One point to keep in mind when you try and implement an automation framework is the principal of K.I.S.S (‘Keep it simple stupid’)

I have seen (and been involved in) several failed automation implementations and in retrospect the main reason for failure has been trying to automate everything that you can possibly think to automate. I discussed this problem in the article too much automation and the importance of thinking critically about what to automate. Time and time again people go rushing ahead to try and automate everything they can without thinking about the boundaries of what their automation framework should be covering. 

 So what is the solution? The rest of this article will offer some guidelines that may help you in your quest of implementing a successful automation framework

The first thing to look at when trying to implement automation is the context of the testing you are carrying out or attempting to automate.  Are you involved in sub system testing, component testing, end to end testing, customer driven testing or any other type of testing?  It is important to look at this and found how what do you want the automation to do for you.  

• At Customer level you may want to automate the common customer scenarios.
• At a system level you may want to validate the input and outputs to a specific set of components and stub out the E2E system.

So defining your context of the purpose of your testing from an automation viewpoint is important.  The reason for this is that links to the second point, defining the size, or scope, of your automation for the project.  Most automation implementations fail to take into account the limits of what you intend to cover with the automation.  As my colleague put it 

The size of this will change depending on the context of the type of testing that you are carrying out.  For your automation implementation to be successful it is important that you define upfront your boundaries of what the automation is expected to cover.  Automation should be about what you input into the system and what you expect as an output everything in-between this is your black box .  So knowing what your testing context is should help define the size and limits of your black box knowing this information before you start to implement a test automation framework should greatly improve the chances of your automation implementation being successful.

Example of defining black box at a Component level of testing

    

Shaded Gray Area = Black box

For this example the boundaries would be

• Create automated test inputs for Input A in relation to defined requirements and expectations
• Automate expected outputs Output 1 and Output 2 based upon requirements and your expectations
• What happens inside of Component X is of no interest since this is outside the boundaries of what we are automating

Another example of defining black box at a Component level of testing

    

• Create automated test inputs for Input B in relation to defined requirements and expectations
• Automate expected outputs Output 1, Output 2, Output 3 and Output 4 based upon requirements and your expectations
• What happens inside of Component Y is of no interest since this is outside the boundaries of what we are automating

Example of defining black box at a Product level of testing

    

• Create automated test inputs for Input A and Input B in relation to defined requirements and expectations
• Automate expected outputs Output 1, Output 2, Output 3 and Output 4 based upon requirements and your expectations
• What happens inside of Component Y is of no interest since this is outside the boundaries of what we are automating

I am aware that this appears to be common sense but time and time again people fail to take on board the context of what they expect the automation to be covering.

So next time you are involved in implementing an automation framework remember to define the size of your black box before you start your implementation that way you know the boundaries of what your automation will cover and check for you.