Testing Qualifications - Certification revisited

Let me start this article by saying I am not against testing qualification at all and they can be a useful resource. Note I use the word qualification; certification is getting to become a bit of a loaded word. Richard Bradshaw talked about his experience here.  I tend to agree with what Richard is saying, as a starting point ISQTB (Or whatever name they wish to use) is a good idea and concept.  For those who have never been involved in testing and wish to know more about ‘some’ of the techniques and methods and a little about the history of testing then it can be a useful building block.  My concern is that it is not being sold in this way and that it becomes a filter to be used by companies and employment agencies.  I have talked about this previously here.   Danny Dainton also talked about his views on ISQTB from the new tester perspective here 

The reason why I have revisited this discussion is due to the recent debate on twitter about test certification and especially ISQTB.  Part of the discussion was the way in which numbers appear to have been used in an advertisement for companies to ensure their testing teams get the ISQTB certification.

Advert can be seen here. Googling a little more provided me with this information 

The bit that concerns me with this sort of misleading propaganda is this:

Various studies estimate the cost of a post-production software defect in the range of $4,000 - $5,000.[1] If ISTQB Software Tester Certification can help a software tester to eliminate just one post-production defect in his or her career, the return on investment for an ISTQB exam could exceed 1500%. With the Volume Purchase Program, that ROI could exceed 2000%.

Where are these figures obtained from?  It is a fallacy that with the way software development works today that fixing a defect later costs so much more.  Details of the cost of defect fallacy can be found here and ANYONE working in software development please read this great  book – The leprechauns of software engineering

This then lead to a talk that Rex Black was doing at STPCON in which the phrase:

“…greatly lowers the cost of post-release defects.”

Can clearly be seen

This is what concerns me a great deal, everything within the ISQTB world appears to be focused on profit and how to maximize the most amount of money with the least amount of effort.  Now someone may correct me if I am wrong on that one.  I do hope that I am and that the people who provide this training do really care about the testing profession and the people paying their hard earned money is of something that is worthwhile to them.  The problem is when profit is involved in learning activities the needs and the best interests of the student are normally lower down the list than the ROI (sic) for the training company shareholders.

I am not against testing qualifications I am against the way in which they are being sold and used within the profession.  I do not like the use of multiple choice exams in which someone can learn by rote and then pass and not know anything really valuable about testing.

We need a system of learning in which we can learn the basics, practice them and be assessed on our thinking and reasoning. The problem is that this is too difficult to do en masse since it eats into profit hence my concerns about profit before learning.

There are other testing training opportunities such as the Rapid software testing course by James Bach and Michael Bolton is another alternative.

Or  black box software testing course created by Cem Kaner is a step in the right direction.

How come we do not hear more about these?

Is the lobbying and scaremongering of the ISQTB too big?  I really hope not there are many passionate testers in the world and we have a moral and ethical obligation to provide the correct training and learning opportunities for these people.  We need to stop using false data and information to scare companies and managers into making people attend these courses

We need to be truthful to both those attending such courses and those paying for these courses.   Maybe there should be a disclaimer on the ISQTB website?

• This foundation exam will teach you about some methods and techniques of testing however it will NOT be able to prove the testing competencies or abilities of those doing the examination.
• It will be able to tell you that the person doing the exam has a good ability to remember stuff or got very lucky when selecting multi-choice answers at random.

I think I would be more comfortable with ISQTB if they provided alternatives and did not sell the exam as a way to be competent tester.

Off track slightly – when I decided to become a rugby league coach many years ago I had to do the following

• exam
• a practical assessment with the trainer
• 3 assessments in the field with an assessor
• keep a training diary for a year, of new stuff I learnt or stuff I implemented 

Only after that year could I class myself as a competent rugby coach.   Hmmm that is an idea maybe ISQTB could do that kind of assessment and training?

We are all responsible for our own training and learning and there are many ways in which we can learn. The problem is I see time and time again many people who call themselves testers and have done no learning or training specifically about testing since sitting the ISQTB foundation course.  This is what really needs to change in our craft.  We need to have a passion for what we enjoy.

I will finish this article with something I tweeted during the discussion and something I really do believe in

Testing is about asking questions and using critical and creative thinking.  It cannot be measured with a simple pass/fail

Professional Qualifications and Bodies

I saw an interesting tweet from James Bach (@jamesmarcusbach) the other day:

@testingclub What counts as certification? What's a "professional qualification?" Why is schooling confused with education?

Which was in reply to seeing the following post from the software testing club (@testingclub) about a survey of testers?

@jamesmarcusbach you may be interested in the Education for Testers survey results http://www.thetestingplanet.com/2011/07/infographic-education-for-testers/


Whilst the data within the survey may be of interest to some people what really got me thinking were the questions James was asking and within this blog article I am going to attempt and answer some of them from my perspective. It does not necessary mean that my view is correct and I encourage people to debate and correct points that I make, however it is important to remember that the context of this, it is my own personal view of the testing world.

One of the key points that James states to the testing community is that testing is context driven, I feel the answer to these questions are also dependant on context and as such the answers to the questions are context driven.

The first question I intend to try and answer is “What’s a professional qualification?”

The context I am using to answer this is within the UK and Europe where they appear to be very well defined.

Professional qualifications in the UK are generally awarded by professional bodies in line with their charters. These qualifications are subject to the European directives on professional qualifications. Most, but not all, professional qualifications are 'Chartered' qualifications, and follow on from having done a degree (or equivalent qualification).
(http://www.wordiq.com/definition/British_professional_qualifications)

However the important point to note here is the word ‘generally’ to me this does not mean all professional qualification are awarded by professional bodies.

So ‘generally’ professional qualifications are awarded by professional bodies – but what are professional bodies? How do you become a professional body? It appears that it is simple to set up a professional body, all you need to do is:

Get a group of people interested in the same subject

Produce a charter which describes your aims and ethos

Have regular meetings

One interesting point that is made about profession qualifications and bodies that I found was:

Membership of a professional body does not necessarily mean that a person possesses qualifications in the subject area, or that they are legally able to practice their profession.

Some professional bodies can be cartel in which anyone who is not a member cannot practice legally in that domain. Examples of this are within the field of Medicine doctors need to register with the BMA and nurses with the RCN to be able to practice.

So professional qualification in this context indicates that you are proficient in your field and some professional bodies only allow you to practice if you continue to keep up to date with current practices and methods and publish new findings for your peers to review. Without doing this you lose your right to practice. IMO this is the direction tester should be going in. We need to be continuing to learn, read articles, publish articles and enter into debates about the course we take.

ISEB and the other certification schemes are ok as a starting point but it is not the end of learning. We need to adapt these schemes so that they are not static and become out-dated as they currently are. The problem comes that for the people who run these schemes to do this would not make it cost effective and as such it is not in their interest to change. This goes against the reasoning for having these ‘professional qualifications’ the bodies that are saying they represent us on a professional level are not adhering to two KEY parts of being a professional body.

• Protecting its fellow professionals

• Looking after public interest by maintaining and enforcing standards of training and ethics

Without this happening I have little confidence in the current testing ‘professional qualifications’

Moving on to James question about confusing schooling and education

I find this interesting since seen both sides of the education system (formal schools) having been to school up to the age of 18 and from working within an education system. I think I see what James is getting at. Formal education worked and did not work for me, due to my circumstances up to a certain age I was away from school more than I was there by my own choice I just did not go. Once I did settle into going to school regularly I found it offered me some fantastic grounding in key subject skills (maths, science, history, English) – I really struggled with English and still do according to my wife! It also gave me social skills in being able to share, communicate, listening to others, letting others have their view point which may not agree with mine. I feel lucky in the schools I attended, they may not have been the highest achieving schools but they taught great life skills they I am always thankful for. (Pity think more about league tables than the students). So how does this differ from schooling? The confusion I think comes from the fact that most definitions of schooling see schooling as part of being at school and formal education.

http://www.yourdictionary.com/schooling
http://www.merriam-webster.com/dictionary/schooling

I find this definition worrying since I see schooling as something slightly different. It can mean the education you get at school. However what about ‘home’ schooling, self-schooling? In which you embark on a different style of learning which is not institutional.

The other context here could be that James could be referring to the differing schools of testing. This does not sit right with me and I do have a problem with having different ‘schools’ of testing. I see testing as one big thing not lots of different fragmented schools. Since each school has some strong views and ideas that the others do not agree with we end up in heated debates in which no one side wishes to back down. I am not sure how that helps the testing profession, debates are ok, but constant fighting is not good and at some point a middle ground should be found even if it does not sit easy with all sides. Sometimes it is better for the good of the all rather than for the good of the individual.

My thoughts on these different schools and professional bodies etc. is that maybe just maybe all sides should come together and look at forming a learned society.

What is a learned society?

A learned society is an organization that exists to promote an academic discipline or group of disciplines

http://en.wikipedia.org/wiki/Learned_society

I think this would be a wonderful way forward and maybe the software testing club could be (form) the society? I am not sure nor have I investigated what would be needed but it looks like that they do some of it already publication of articles etc. I would be most interested in what the people at the software testing club think of this and what the general community feels within all of the different schools.

Finally to answer the last question by James:

What counts as certification?

There are many definitions of certification the main being one in which an organization recognise individual/company etc. that meet certain criteria. These criteria could be passing exams, years of experience, publication of articles and so on.

However this really does not ask the question that James asked. Within the survey it shows how many people hold a certification. However as correctly noted by James it does not say which certification. I would have expected this to be much higher. I have many certificates, PAT testing, rugby coach and first aid. None of these are really relevant for my day to day job of testing so I still no sense in the results as they are displayed. However even if it said testing certification which would it mean? ISEB? AST? Etc etc. This one question really stumped me since I could not find an answer that sat easy with me. If I write regular testing article (blog, magazine) and publish should I be certified? If I get my work colleagues to write a report on how competent I am at testing would make me certified? I really do not have an answer for this one and as James did on twitter open up this question to the community.

So the challenge is set:

In your opinion:

What counts as certification?

A Competent Tester

You can teach a student a lesson for a day; but if you can teach him to learn by creating curiosity, he will continue the learning process as long as he lives. ~Clay P. Bedford

I started writing this blog article in draft about a month ago to have a little rant about how certification does not make you a competent tester based upon my experiences and frustrations whilst trying to recruit testers. I was prompted by a post by Rob Lambert to revisit the article and try to complete it.

Recently I have been trying (and I emphasis the word trying, it has been really trying and challenging to find the right people) to recruit testers to work with me on some quite technical projects. None of the projects have any real UI interfaces nor are they web based applications, this is real hardcore technical testing at a binary level.I became very frustrated and even got to the stage that I felt my standards were too high.

I lot of the CV and candidates that were interviewed all stated that they had ISEB or ISTQB certification and from that I assumed they would have a basic grasp of boundary and edge cases. How wrong was I!!! To make it worse I asked about their views on any of the current new approaches and techniques in testing, even having to prompt to what do you think of context driven testing and all I got back in return was a blank look. I asked what articles or books have you read recently about software testing or any book that you could relate to software testing and again all I got was blank looks and shrugs of shoulders.

I am not against certification in any shape or form and I do not have a view if people try to make money out of it that is not the issue. The issue I have is that in some (most) cases these schemes are being sold on the basis that once you have completed them you are now a ‘skilled’ tester and know everything there is to know about testing.

PASSING A MULTI-CHOICE EXAM DOES NOT MAKE YOU COMPETENT!!!!!

Rob Lambert in his article explains in great depth what makes testers ‘skilled’ and I have to agree with him. Those who read my blog know I have a major interest in the social sciences and psychology and how this can collate to testing. More importantly how it can help make you a better tester.

So if you want to become competent at testing you have to read more, interact more with the testing community, become self-learning. My ethos is always to keep on learning and never stop doing so.

Taking any of the certification courses can be, for someone new in the world of testing, a good STARTING point, grounding in SOME of the techniques and skills. These courses will NOT teach you about how testing fits into Agile and about exploratory testing. Nor will they teach you how to test and make you think ‘outside the box’ Only getting involved within the testing community will do that. I good start is the Software Testing Club – read some of the articles and blogs there, subscribe to the excellent RSS feed. I am not being paid by the Software Testing Club to say this I am promoting it because again it is a good starting point for those would wish to learn more about testing and want to improve so they can become competent testers.

I will finish with a quote about learning.

It's what you learn after you know it all that counts. ~Attributed to Harry S Truman

The Certification Filter

I had a few moments to myself the other day and decided to have a bit of fun and do some research about what agencies required when you apply for testing roles. I was surprised (or not) by the number of roles that stated “will be ISEB certificated”

How can they get it so wrong? The ISEB is now defunct it should be ISTQB (sic) so all those who hold an ISTQB qualification need not apply………

Anyway back to the point I am trying to make.

I did some quick research and found that within the last seven days of the 122 testing roles listed I would be excluded from 28 of them since they stated …will be ISEB certified. If you add in the ones that add ‘would prefer candidates with ISEB certification. You get to over half of the roles advertised that I could not apply for.

@Mheusser made the following tweet @steveo1967 - certification might just get you the job you don't want to have.

That maybe be true but I got thinking and maybe it is not the company using the agency that are mandating this but the agency adding its own filter. The company could be missing out on some great testers because of the agency filtering system.

The following are some imaginary conversations with an agency (Sadly I am sure some people may have already experienced this…)

************************************************************************************

Me: Hi I am interested in being put forward for the role of chief tester as per the advert you posted this morning on your website.

Agency: Sure, have you already sent us your CV, Resume?

Me: Yes

Agency: OK, what is your name, let us see if we can find your CV

Me: My name is Timmy Tester

Agency: Wow cool name sure matches your profession

Me: Yes (Sigh) I get that all the time..

Agency: OK Got your details – just looking at them now, wow 20 years in software testing, that is impressive, I see you have worked at some very well known companies. Oh wait a minute…. You do not list that you hold the ISEB qualification.

Me: No I do not.

Agency: I am sorry Timmy I cannot put forward your application – we only accept people who are ISEB qualified.

Me: Why? I have more than 20 years in software testing.

Agency: Yes I can see that, however the ISEB qualification means that you know how to test and is mandatory for any roles you apply for with us. Sorry but I cannot put you forward for this. I suggest you go and sit the exam and get back to us. Goodbye.

************************************************************************************

Me: Hi I am interested in being put forward for the role of chief tester as per the advert you posted this morning on your website.

Agency: Sure, have you already sent us your CV, Resume?

Me: Yes

Agency: OK, what is your name, let us see if we can find your CV

Me: My name is Ivor Certificate

Agency: OK Got your details – just looking at them now, you have been in testing for 8 years now, I see you also hold the ISEB qualification. That means you must know a lot about testing.

Me: No I do not.

Agency: Sorry, you said you do not?

Me: Yes I did say, no I do not know anything about testing

Agency: Then how come you have a ISEB Certification?

Me: Because I noticed that if I had this I could apply for any testing job.

Agency: So you must know something about testing?

Me: Well it is an interesting story. I paid to do the multi choice exam, sat down and completed it in 10 minutes by just randomly answering the questions. No one checks if I have any competence at testing, by luck I managed to get the pass mark required to get the certificate. Hence I am now classed as someone who must know about testing and how to test.

Agency: But it says on your CV that you have been testing for the last 8 years.

Me: Yes I have but I just go in and let others, who are not certified do all the work while I just copy what they have done and claim credit for it. So are you going to put me forward for the role?

Agency: Yes I will you meet the selection criteria, so I cannot see any problems in putting you forward.

Me: Thank You.

************************************************************************************

This may seem like a silly situation but I am sure it could happen in reality. I am not against qualification and people trying to improve themselves but when those qualifications are then used as a filter to exclude people from applying from jobs, it makes me see red.

There are some good examples of proving your ability as a good tester. You can talk to previous companies that you have worked for. They could interview you and talk to you about testing and your thoughts, problem solving ability. However this would take too much time.

I have not as yet found any roles that have stipulated that they require that you attended a Rapid Software Testing Course or The Black Box Software Testing course through the Association for Software Testing. Why is this so? Is it that these types of courses do not have the huge budget to promote themselves? Or that they try to be non profit making?

What is the solution to all of this?

I think within the testing community we need to start educating agencies and companies about how to sort out good from bad testers, how we go about this I am not sure.

Should we have a dedicated website that we can direct agencies to, to explain about certification? I feel this could be a good start and would need someone with far better web skills than myself to get running and also it would need to be unbiased as possible.

Then mail shot the CEOs at each agency when we hit this problem directing them to the website.

Do we try to do presentations at employment agency conferences?

I feel there is a need to educate agencies and companies that are looking to employ testers and give advice on how to spot the good from bad candidates but they need to get rid of the certification filter.

Another thought I had would it be a good idea to have a vetting service for testers and agencies? There could be a one stop service for agencies to verify testers, their abilities and obtain a list of people who would vouch for them.

I tried to think at the weekend if this could work or not. I have a lot of concerns about it being misused and ‘gamed’ by people who have a moral compass that is slightly off balance. How would it be funded? Would it become a monster of its own making? How would testers be vetted and vouched for? Would testers be vetted and vouched based upon their online presence? For example I am sure I could ask a few people online to vouch that I am a good tester however none of these people have worked with me and seen me carry out testing. I could be just saying the right thing at the right time to impress people – how would anyone know unless they have worked with me?

This really brings us back to the beginning of the article, agencies and companies need someway to vet testers and get some guarantee that they know about testing (regardless of which school of thought they follow). So using the certification method is an ideal way to sort out candidates quickly no matter how flawed the certification may be. Do I just bite the bullet and sit the exam if I do not want to be excluded for any testing job? Does anyone have any other methods that agencies or companies can use when they are looking for skilled testers?

Bangalore Testers meet-up (15-May-2010)

I thought I would write a brief blog about a meet up of testers whilst I was in Bangalore, India.

It started with me sending a tweet out asking if any testers in Bangalore were interested in meeting up at the weekend. After a short while Santhosh Tuppard (@santhoshst ) responded by putting an invite on his blog site (http://tuppad.com/blog/2010/05/13/bangalore-testers-meetup-–-may-15th-2010/) then all of a sudden there was a flurry of activity on twitter as people asked about timing and where and when.

So on the Saturday I made my way down to the forum and met up with seven testers from Bangalore.

Pradeep Soundararajan (@testedtested - http://testertested.blogspot.com/)

Ajay Balamurugadas – (@ajay184f - http://www.enjoytesting.blogspot.com/)

Dhananjay Kumar (@dhantester)

Santhosh Tuppad (@santhosht - http://tuppad.com/blog/)

@nitinpurswani

+ two others – if you can remember who they add please add their details as a comment.

We had an agenda or some points to talk about:

Should we record our emotions when testing?

Exploratory Testing – good and bad points – how can we solve the bad parts?

Measuring Testing quality – How?

If we kept to these themes was a different matter.

After some brief introductions everyone was keen to know about me. After explaining I had been in IT for over 24 years which was older than some attending!!! Pradeep pointed out I did not look that old!!!!

We then started to discuss some testing issues and the following is what I remember of the conversations that took place. I am sure some of the others who attended will correct any details I have got wrong :o)

One testing issue we looked at is when teams are being managed remotely and managers at not in the same location as the testing team. Pradeep says this becomes a problem because managers want a breakdown of the testing activity all the time and the testing team spend a great deal of time answering questions from managers about what they are testing.

My thoughts on this was that anything that is actually stopping a tester from testing should be recorded and reported so at the end of each test phase/cycle when reporting back to management if a lot of time is being spent reporting test activities to managers rather than testing then this can be clearly seen and a ‘good’ manager will try to rectify it. One idea suggested was that the managers should become more hands on and take part in the testing activities. The exploratory testing approach makes this very easy to do.

There was a lot of interest in reporting what is stopping testing rather than any thing else – this lead on to a discussion of measuring testing and testing quality. This is always a difficult question and I am not sure we fully answered it in our discussions. I mentioned that when testing should we record our feelings and emotions about what we are testing. I suggested that we use a happy, sad face system so when we feed sad about testing we have a sad face when we feel angry an angry face and so on, It would then be very easy to see a trend for a area under test. If there were lots of angry faces then someone will spot and trend and start asking questions of the testers about why they feel this way. Pradeep pointed out that developers may try to influence the testers to always make happy faces. I stated that testers should be still independent of developers and that we should support and help developers but we would not tell them how to write an algorithm so why should they tell us how to test?

On the subject of measuring testing the discussion revolved around looking for trends rather than looking at numbers. The numbers could still be wrong but that is where talking helps to understand what the numbers and trends are saying.

Another point was made about using Tech support people as a resource for testers. I posted this on twitter as a question and got the following responses: (may not be in time/date order)

testingqa: @steveo1967 Tech support people hold some of the skills, but I wouldn't immediately say it implies they will be a good tester too #testing

testingqa @QualityFrog @michaelbolton My 1st role in IT was Tech Support role,those I worked alongside were slack & passed along cases for me to solve

testingqa @QualityFrog @michaelbolton So my faith in Tech Support staff isn't great ;) But I do agree that a good tech support person can hold an... appropriate background useful for testing, but even then Tech Support doesn't require an eye for detail often.

qualityfrog @michaelbolton @testingqa I've seen good ppl move fr tech supt 2 testing & vice-versa. there r similarities in foundation in what makes good

qualityfrog @michaelbolton @testingqa And, sadly, some in tech support are about as astute at faking support as some testers are in faking testing.

michaelbolton @testingqa Not automatic, but the skills and experience of support greatly overlap those for #testing. Great foundation, I'd argue.

michaelbolton @QualityFrog @testingqa I took *good* tech support person as read. Your pessimistic interpretation is also valid, alas.

So even here there are two sides to the argument but it appears that ‘good’ tech support could be valuable resource to testing. Maybe I am a little biased on this since my background is technical support and I believe that if you have a good grounding of how users think and technical knowledge as well (preferably at a coding level) then you have the right attributes to look at becoming a tester.

It was interesting that whilst talking to the group and I was aware I was doing a lot of the talking how much interest was shown in the subjects we were discussing, more importantly how much passion all the people at the meeting had for testing. I could see that in everyone’s faces a passion for learning and understanding. Key attributes of ‘excellent’ testers. I twittered about meeting the future teachers of software testing and I still believe that these people hold the key to the next 10 – 20 years of software testing.

The next subject that came up was one of certification – I explained that on the way to the office the other day I saw a sign that said “Learn software testing in two weeks”, everyone in the group laughed. We talked about the fact that software testing is not like programming in which once you have learnt the foundations of the language you then have the building blocks to create code. Software testing is not computer science but a mixture of many types of science including psychology and philosophy. It is not an exact science and there is plenty of scope to get it very wrong which we have all seen. The question of certification seems to be a very interesting one considering the latest blog posts coming from James Bach and Stewart Reid. (http://www.satisfice.com/blog/archives/457, http://www.eurostarconferences.com/conferences/session-details.aspx?sessionId=209)

I do not understand the need why fellow professionals need to attack each others ideas, it is getting very political. This is just my own personal viewpoint and nothing to do with the meet up. I believe in the need to debate and discuss ideas and opinions but when it appears that someone is personally attacking someone it then appears like a personality problem and distracts from a meaningful debate on the issues – I really want to attend Eurostar now just to hear the debates.

This nicely leads us to the subject of certification that was brought up during the meet up. We discussed certification and all of us agreed that it appeared to be a money making scheme that gave no benefit to experienced testers. The concern of the group was that agencies would now demand that testers have this certification and people will take the exam worried that if they did not they would not get a job in testing. The group felt this was very wrong, to be pressurised in taking an exam because there is no other measurable way to prove you are an ‘excellent’ testers. As a group we came up with some ideas that may be a way forward and needs everyone in the testing community to push forward.

• All testers should have an online presence
• They should be involved in writing blogs.
• Be actively involved in testing discussion (software testing club, twitter)
• Should try to meet fellow testers a couple of times a year at testing meet up – the internet allows this very easily as this meet up has proved

Once this is done when you apply for a job and the agency asks for your certification – point to your active online presence in testing ask them to talk to peers who have met you and who can vouch for you. If we ALL did this then the need to pay organisations to prove you can test goes away. Let us as a testing community certify each other.

So with this last thought I did a little experiment on the people in the meet up involving the calculator experiment as taught by Michael Bolton – if you have experienced this from Michael then you will know what I mean if not then you need to find someone who knows because I am not going to spoil it by explaining on here. Thank you again Michael for giving me a useful way to demonstrate a key point.

So all too quickly the meet up had to end. Did we all learn something? I hope so. I know I did and I had a wonderful time and left feeling encouraged and motivated. I still think I talked far too much and for that I apologise hopefully if there is a next time I will encourage others to speak even more – you have been warned……