Is Product Knowledge essential for effective testing?

I might not be blogging or being online as much as I have been this is due to family life, those close to me know what I mean but here is a new article. I do have lots of ideas and thoughts it is difficult finding time to put them together. I will be at Eurostar in Manchester this year and I hope lots of you will be attending.

It has been awhile since I posted a new blog article so here is a new one.

I recently read an excellent article by Paul Gerrard about all testing being exploratory http://gerrardconsulting.com/index.php?q=node/588 and thought it was so good I posted within our company intranet. I got an unexpected reply which made me think about testing and the skills that are required to be an effective tester. This reply is the reasoning behind this blog post.

The reply I got was as follows:

The interesting challenge to the basic idea is that the tester needs to have knowledge , and very good knowledge of how the system is supposed to work. Only with that knowledge in place is it then really possible to 'intuitively' carry out testing that will be good exploratory testing. Without that deep knowledge it turns into 'random' testing, which, while it has its place in a test approach, I’m not sure it could form the bedrock of a test plan.

The challenge then becomes how to get that knowledge to the tester/test team. I can see how for long term projects/products, the tester becomes truly expert in his Component Under Test, but for new things, or new people to that test team, the ramp up time and 'completeness' of such an approach is questionable and a bit difficult to scale.

For sure exploratory has a part to play - but hard to see how its 'all'.

This made me think about product knowledge and is it really essential for effective testing. So I posted the question on twitter:

Interesting discussion about needing 2 have product knowledge 2 do good exploratory #testing and without this becomes random testing. (1/2)

I have my views on this but would love 2 have #testing community opinions, views, counter views on this. Might do blog post on this (2/2)

I got some replies to this very quickly (as I would expect from such a dynamic community) – sorry if the time order appears a little wrong – I wish twitter would let me do this easier than a cut and paste job

Radionotme

@steveo1967 before gaining experience and understanding I'd have agreed that my testing was more random than exploratory.

@steveo1967 I've recently been exposed to Microsoft AX and found that with experience my exploratory testing is becoming more fruitful.

@mgaertne

@steveo1967 @QualityFrog Well, I would argue that if I know how the software behaves, I don't need to test at all, do I? :)

@steveo1967 @QualityFrog "what I believe it should do" is not knowing to me. :)

@QualityFrog

@steveo1967 @mgaertne product behaviour can be observed in testing. It is implementation, not requirement.

@can_test

@steveo1967 if you know how to do Exploratory Testing well then no prior product knowledge is required. Someone on the team needs it tho!

@steveo1967 IMO that is a very good post on ET. I see nothing there that says prior product knowledge required. (cc @paul_gerrard)

Which were very interesting since they were opposing views about the statement made…

@Radionotme stated that he found product knowledge useful to prevent random testing while mgaetne and qualityfrog stated that it was not necessary and that you could learn about the system whilst exploring.

I countered this with:

@steveo1967

@QualityFrog @mgaertne the counter claim made is that knowing how the product behaves is essential to test quicker and save time

By this time @Michaelbolton had joined the debate

@michaelbolton

@steveo1967 @mgaertne @qualityfrog Whether knowledge or belief, where do you obtain it? From /testing/ what you know.

steveo1967 It takes exploration to develop a decent strategy, tactics, and checks. To me,

@steveo1967 One can develop product knowledge, learning through exploratory #testing. Running scripts helps to suppress that learning.

Trying to keep up with all the threads I replied:

@steveo1967

@mgaertne @QualityFrog not really you would still test that what you believe it should do it actually does do.

@michaelbolton they are not saying ET is not useful they are saying it is more effective when you have product knowledge. Do you agree?

@Radionotme interesting experience with some product knowledge making your #testing more efficient would like 2 know more.

@Radionotme was it random because u had no structure? SBTM for example helps to structure ET or is it ur ET skills improved?

@mgaertne @QualityFrog v true but the view expressed states have knowledge of the product and how u expect it to behave is essential

More people started to enter the debate:

@can_test

@steveo1967 last thought: is Jazz music random? To the untrained ear, perhaps it is. To the experienced, you see it takes great skill.

@Veretax

@can_test @steveo1967 What is what we perceive as random, really isn't random at all, just we lack sufficeent perception to see its order?

@steveo1967

@can_test if someone on the team needs product knowledge why can this not be the tester?

@can_test

@steveo1967 I didn't say it _couldn't_ be the tester. I said it doesn't _have_ to be the tester. Financial Svcs jobs are bad about this.

@steveo1967

@can_test very well put about random being undisciplined that is my point ET without discipline is random

@can_test

@steveo1967 that's what it sounds like to me anyway. "Effectiveness" is meaningful in a certain context. Don't blame the tools.

@steveo1967 I think it's about Trust, or lack thereof. That is, I will trust your ET if you are a product expert, otherwise no, its random

@steveo1967 experience in anything increases your testing efficiency. ET usually looks random to those who don't understand it.

@steveo1967 IMO, good testing is about changing your perspective on the system. That's harder when you are the SME too.

@steveo1967 what does "random testing" mean to them? I may have many hypotheses I want to test that are off the main path. Is that random?

@steveo1967 the statement I disagree with is that Exploratory Testing requires prior industry/product knowledge. That's not true.

@javandervlis

@michaelbolton @steveo1967 @mgaertne @qualityfrog You either belief there’s milk in the fridge or you don’t be…

@michaelbolton

@steveo1967 In addition, exploration implies that someone intends to discover something. Knowledge can never be known to be complete.

@steveo1967 Biases can't be eradicated, but they can be recognized, controlled, and managed in a number of ways. It starts with awareness.

@WadeWachs

@michaelbolton @steveo1967 I am currently the 'fresh eyes'. Old and new eyes both find bugs, though they can be different bugs. 1/?

@Steveo1967

@WadeWachs @michaelbolton very good point. My concern with knowing product is having bias expectations of behavior and not seeing problems

@michaelbolton

@steveo1967 More effective /for what/? More product knowledge /vs. more what-else/? Heuristic: fresh eyes find failure. #testing

So from this lively debate what can we conclude?

Some people think that you do not need to have any product/domain knowledge to be able to carry out exploratory testing since one of the principals of ET is that you learn about the system as you test. Other people say that you would be just doing random testing if you had no knowledge of the product since your expectations of how it should work can guide your testing.

My own personal view is very much in the middle it has been known for me to say that I can test any product without any prior knowledge (domain or otherwise of the product), however the important word missing there is ‘effective’ How effective is my testing without domain knowledge? Does it suddenly become hit and miss and as stated by @radionotme more random. I am currently working on a product which is very niche without understanding how certain packets are formed and transmitted you could spend a lot of time testing unnecessary stuff (there is a counter to this that no testing is unnecessary – in that it exercises the system in nonstandard ways – true but doing too much of this soon makes it less effective)

So to conclude I do not think there is an obvious answer to this. In some cases I feel domain/product knowledge could be essential to make the testing efficient. l it does not mean that a competent tester could not learn this domain knowledge very quickly and start to be effective and efficient at testing the product. However it needs to be recognised that when someone joins a team and comes without domain knowledge there will be some ramp up time for them to become familiar with the domain. In my opinion this where exploratory testing comes into its own, as an approach to use for someone to get on board with a system and learn about the system it is the most effective way especially if you can afford to do paired exploratory testing.

Domain/product knowledge is not essential to do effective testing but it can certainly help.