This is the first in a series of posts I plan to write after attending the fourth MEWT peer conference in Nottingham on Saturday 10th October 2015.
Before I start I would like to say thank you all the organizers and for inviting me along and a BIG MASSIVE thank you to the AST for sponsoring the event,
Abstract:
Earlier on in my career, I used to follow this pyramid, encouraging tests lower and lower down it. I was all over this model. When my understanding of automation began to improve, I started to struggle with the model more and more.
I want to explore why and discuss with the group, what could a new model look like?
Before I start I would like to say thank you all the organizers and for inviting me along and a BIG MASSIVE thank you to the AST for sponsoring the event,
Abstract:
Earlier on in my career, I used to follow this pyramid, encouraging tests lower and lower down it. I was all over this model. When my understanding of automation began to improve, I started to struggle with the model more and more.
I want to explore why and discuss with the group, what could a new model look like?
___________________________
During the session Richard explained his thoughts about the test automation pyramid created by Mike Cohn in his book Succeeding with Agile and how the model has been misused and abused.
Richard talked about how the model has adapted and changed over the years from adding more layers..
...to being turned upside down and turned into an ice-cream cone.
Images taken from - http://watirmelon.com/2012/01/31/introducing-the-software-testing-ice-cream-cone/
Duncan Nisbet pointed out that this really is now an anti-pattern -
http://c2.com/cgi/wiki?AntiPattern. The original scope of the diagram by Mike was
to demonstrate the need to have fast and quick feedback for your automation and
as such focused the automation effort to the bottom of the pyramid. Where the
feedback should be fast. The problem Richard has been experiencing is that this
model does not show the testing effort or tools needed to get this fast
feedback. It also indicated that as you
move up the pyramid less automation effort was needed or should be done. The main issue for Richard was how the
pyramid has been hi-jacked and used as examples of the priority of effort
should be on automation rather than focus on the priority of both in given
contexts.
Richard presented an alternative model in which both testing and automation with the tools required could be shown on the ice-cream cone diagram.
With this diagram the sprinkles on the top were the tools and the flakes the skills. He then in real time adjusted the model to say it would be better as a cross-sectional ice-cream cone with testing throughout the cone and the tools across all areas of the original pyramid. Many attendees liked this representation of the model but some thought that it still encouraged the concept that you do less of certain testing activities as you move down the ice-cream cone.
At this stage I presented a model I had been using internally to show the testing and checking effort.
Again people thought this indicated that we need to do less as we move up the pyramid and it went back to the original point being made by Richard that the pyramid should die.
After MEWT I thought about this problem and tweeted an alternative representation of the diagram. After a few comments and some feedback the diagram ended up as follows:
With this model the pyramid is removed. Each layer has the same value and importance in a testing context. It shows that the further up the layers you go the focus should switch more from checking to testing and the lower down the focus should be on automating the known knowns. All of this is supported by tools and skills. As a model it is not perfect and it can be wrong for given contexts, however for me it provides a useful starting point for conversations with those that matter. It especially highlights that we cannot automation everything nor should we try to do so.
In summary the talk given by Richard was one of the many highlights of the day at MEWT and inspired me to look further into the test automation pyramid model and its failings. I agree with Richard that this original model should die especially in the way it is often misused. Richard provided some useful alternatives which could work and hopefully as a group we improved upon the original model. Richard did clarify that his ice-cream cone model with sprinkles is not his final conclusion or his final model and he will be writing something more on this in the near future. His blog can be found here - http://www.thefriendlytester.co.uk/.
Now it is over to you, please provide your feedback and comments on this alternative model.
Now it is over to you, please provide your feedback and comments on this alternative model.
Firstly, great post on an interesting subject. Keep them coming, please!
ReplyDeleteShould this original model die, though? If you actually read the original description of the model, its about effort and cost of automating the checks of an application at certain levels, not how to structure your entire testing strategy. Unfortunately, many have taken the model to mean "all things, to all men". Nowhere was it ever said that this would be the silver bullet to solving testing problems.
However, I can see why its necessary to debate the subject. I've been using this model pretty much since I read about it when the book came out. I thought it expressed clearly an approach to some of the key automation problems I was, and still do, have. Although maybe a generalisation, experience tells me that high level UI checks are very costly to produce and maintain, unit and api checks, as long as they are created at the same time as the code is created are relatively cheap in terms of time and maintenance. But thats just about the checks I like to have in place to reduce the risk of critical functionality being shipped in a broken state.
Things seems to have got complicated with this model when someone put a cloud on top of the pyramid and called it manual testing. Thats when it became more about an entire test strategy. And I hold my hand up. I am guilty of using this cloud to convey all the other testing "stuff" that needs to happen. People just didn't seem to get that the pyramid was talking about one aspect of how we reduce risk of feature failure in application.
So, in terms of teaching us about the time and cost of automating your checks, it still holds true.
Here is generally what I strive to achieve automation wise on projects, and this is usually the case with or without a UI. I can't vouch much for embedded systems as its been years since I have done any.
The amount of checks increases down the stack, although the data layer tends to have fewer checks in my experience as the time and cost effort seems to increase again.
-- UI -- Stubbed user journey testing
-- UI -- A few critical path e2e type tests running on a full stack
-- UI -- Unit tests (JS or whatever language your presentation layer is created in)
-- Back End (Services/APIs/etc) -- Stubbed acceptance checks
-- Back End (Services/APIs/etc) -- Integrated contract checks (3rd parties, database connections, etc)
-- Back End (Services/APIs/etc) -- Unit checks
-- Data Layer -- Checks on store procedures, schema changes, migrations, etc
-- Configuration -- Checks on key components being configured in software and target deployment environment (this is something I have only just started doing, but my does it help!)
Some of these tests are run per build, some only once certain stages of the build pipeline have completed.
Unless experience gives me some different insight, I'll keep this model in mind when thinking about tackling automation checking on a project. What I may not do is use it so carelessly, as its quite obvious that 5 years on, many people really do believe it to mean something entirely different.
As for general test models, I'm really keen to see how this whole discussion develops (and participate in it as well!). So far I really like your test execution model, its simple, and does not seem to convey any doubt over what it is - only time and use will tell, though!
Thank you for your comments Pete it certainly adds to the discussion. One other reason I have with the current automation pyramid is part of what you pointed out in your reasoning why it is useful.
Delete"..its about effort and cost of automating the checks of an application at certain levels.."
Yes it can be used to say the amount of checks done at these levels have less costs, however I feel it does not represent the hidden 'testing' effort and costs. To create these checks requires testing effort which therefore requires testing cost.
At the same time people have used the model to say we need to have huge numbers at the bottom layers without thought and produce a automation maintenance nightmare.
Also the pyramid may not work well for some contexts in which the focus may be more of user experiences and therefore require more effort in UI.
This is not to say your thoughts about the model are incorrect I feel that since it was first published a lot in the software development world has changed. Which is one of the points you mentioned.
Hi John, You make a really good point about the hidden testing effort in making these checks, and certainly in environments where the automation of checks is uncoupled from the actual development, we can underestimate massively the effort required to put those checks in place. However, if we couple the automation of these checks to the time of development of the features we are writing checks for, then its no longer hidden testing effort. It becomes more about the general development effort of delivering that feature, and therefore not hidden.
ReplyDeleteGenerally, as with any model, and I guess this is something discussed at MEWT, a model such as the automation pyramid is created to help someone think about what they are doing, not to prescribe a way of doing something. Its probably really important when we promote these ideas/models that we give examples and reasoning for why and how a model could be used.
I don't actually visibly use the pyramid anymore, but prefer to model a desired delivery process for an organisation which takes into consideration things like the pyramid, how we test, how we deploy etc to provide a more holistic idea of what a team needs to be doing to provide confidence internally and externally. I'll try to blog that soon to give you a better idea of what I mean.
Read this posting with growing excitement because where I work we "had" a view of the pyramid which I always wanted to invert, but I never knew why I had wished to make it into icecream. It's about not mixing the whole effort of manual and automation with the different types of quality we get from each. Beautifully explained and clearly separated from an ugly argument that was growing in my mind until now in a similar fashion to the top-down programming versus old-school bottom-up building.
ReplyDeleteHi John!
ReplyDeleteI really like your model and I liked your whiteboard session video on it too!
I have a query on it:
I may have picked this up wrong, but it seems that you are implying that the further up the layers you go, the more unknown unknowns there are (that need to be explored)?
My niggle is that I think that there are unknown unknowns at every layer, that we should investigate through testing. E.g. when a developer is writing unit tests, I've seen tons of situations personally that the developer is thinking more about constructing the code rather than about the variables in the types of errors to handle (how many times have you heard "I didn't think of that" when talking about things like this).
I think the real problem is that testers don't tend to get involved at these lower levels of your model. Not that there are less unknown unknowns...
(if that makes sense).
If we look at testing, checking and development as activities, then it's easier to see that the distinction is between information surrounding explicit expectations and a lack of information (the unknowns and unknown unknowns) that need investigation.
I guess I'm trying to say that we need these testing and checking activities throughout all layers of the model.
Keen to hear your thoughts on this!! :)
Dan