Bangalore Testers meet-up (15-May-2010)

I thought I would write a brief blog about a meet up of testers whilst I was in Bangalore, India.

It started with me sending a tweet out asking if any testers in Bangalore were interested in meeting up at the weekend. After a short while Santhosh Tuppard (@santhoshst ) responded by putting an invite on his blog site (http://tuppad.com/blog/2010/05/13/bangalore-testers-meetup-–-may-15th-2010/) then all of a sudden there was a flurry of activity on twitter as people asked about timing and where and when.

So on the Saturday I made my way down to the forum and met up with seven testers from Bangalore.

Pradeep Soundararajan (@testedtested - http://testertested.blogspot.com/)

Ajay Balamurugadas – (@ajay184f - http://www.enjoytesting.blogspot.com/)

Dhananjay Kumar (@dhantester)

Santhosh Tuppad (@santhosht - http://tuppad.com/blog/)

@nitinpurswani

+ two others – if you can remember who they add please add their details as a comment.

We had an agenda or some points to talk about:

Should we record our emotions when testing?

Exploratory Testing – good and bad points – how can we solve the bad parts?

Measuring Testing quality – How?

If we kept to these themes was a different matter.

After some brief introductions everyone was keen to know about me. After explaining I had been in IT for over 24 years which was older than some attending!!! Pradeep pointed out I did not look that old!!!!

We then started to discuss some testing issues and the following is what I remember of the conversations that took place. I am sure some of the others who attended will correct any details I have got wrong :o)

One testing issue we looked at is when teams are being managed remotely and managers at not in the same location as the testing team. Pradeep says this becomes a problem because managers want a breakdown of the testing activity all the time and the testing team spend a great deal of time answering questions from managers about what they are testing.

My thoughts on this was that anything that is actually stopping a tester from testing should be recorded and reported so at the end of each test phase/cycle when reporting back to management if a lot of time is being spent reporting test activities to managers rather than testing then this can be clearly seen and a ‘good’ manager will try to rectify it. One idea suggested was that the managers should become more hands on and take part in the testing activities. The exploratory testing approach makes this very easy to do.

There was a lot of interest in reporting what is stopping testing rather than any thing else – this lead on to a discussion of measuring testing and testing quality. This is always a difficult question and I am not sure we fully answered it in our discussions. I mentioned that when testing should we record our feelings and emotions about what we are testing. I suggested that we use a happy, sad face system so when we feed sad about testing we have a sad face when we feel angry an angry face and so on, It would then be very easy to see a trend for a area under test. If there were lots of angry faces then someone will spot and trend and start asking questions of the testers about why they feel this way. Pradeep pointed out that developers may try to influence the testers to always make happy faces. I stated that testers should be still independent of developers and that we should support and help developers but we would not tell them how to write an algorithm so why should they tell us how to test?

On the subject of measuring testing the discussion revolved around looking for trends rather than looking at numbers. The numbers could still be wrong but that is where talking helps to understand what the numbers and trends are saying.

Another point was made about using Tech support people as a resource for testers. I posted this on twitter as a question and got the following responses: (may not be in time/date order)

testingqa: @steveo1967 Tech support people hold some of the skills, but I wouldn't immediately say it implies they will be a good tester too #testing

testingqa @QualityFrog @michaelbolton My 1st role in IT was Tech Support role,those I worked alongside were slack & passed along cases for me to solve

testingqa @QualityFrog @michaelbolton So my faith in Tech Support staff isn't great ;) But I do agree that a good tech support person can hold an... appropriate background useful for testing, but even then Tech Support doesn't require an eye for detail often.

qualityfrog @michaelbolton @testingqa I've seen good ppl move fr tech supt 2 testing & vice-versa. there r similarities in foundation in what makes good

qualityfrog @michaelbolton @testingqa And, sadly, some in tech support are about as astute at faking support as some testers are in faking testing.

michaelbolton @testingqa Not automatic, but the skills and experience of support greatly overlap those for #testing. Great foundation, I'd argue.

michaelbolton @QualityFrog @testingqa I took *good* tech support person as read. Your pessimistic interpretation is also valid, alas.

So even here there are two sides to the argument but it appears that ‘good’ tech support could be valuable resource to testing. Maybe I am a little biased on this since my background is technical support and I believe that if you have a good grounding of how users think and technical knowledge as well (preferably at a coding level) then you have the right attributes to look at becoming a tester.

It was interesting that whilst talking to the group and I was aware I was doing a lot of the talking how much interest was shown in the subjects we were discussing, more importantly how much passion all the people at the meeting had for testing. I could see that in everyone’s faces a passion for learning and understanding. Key attributes of ‘excellent’ testers. I twittered about meeting the future teachers of software testing and I still believe that these people hold the key to the next 10 – 20 years of software testing.

The next subject that came up was one of certification – I explained that on the way to the office the other day I saw a sign that said “Learn software testing in two weeks”, everyone in the group laughed. We talked about the fact that software testing is not like programming in which once you have learnt the foundations of the language you then have the building blocks to create code. Software testing is not computer science but a mixture of many types of science including psychology and philosophy. It is not an exact science and there is plenty of scope to get it very wrong which we have all seen. The question of certification seems to be a very interesting one considering the latest blog posts coming from James Bach and Stewart Reid. (http://www.satisfice.com/blog/archives/457, http://www.eurostarconferences.com/conferences/session-details.aspx?sessionId=209)

I do not understand the need why fellow professionals need to attack each others ideas, it is getting very political. This is just my own personal viewpoint and nothing to do with the meet up. I believe in the need to debate and discuss ideas and opinions but when it appears that someone is personally attacking someone it then appears like a personality problem and distracts from a meaningful debate on the issues – I really want to attend Eurostar now just to hear the debates.

This nicely leads us to the subject of certification that was brought up during the meet up. We discussed certification and all of us agreed that it appeared to be a money making scheme that gave no benefit to experienced testers. The concern of the group was that agencies would now demand that testers have this certification and people will take the exam worried that if they did not they would not get a job in testing. The group felt this was very wrong, to be pressurised in taking an exam because there is no other measurable way to prove you are an ‘excellent’ testers. As a group we came up with some ideas that may be a way forward and needs everyone in the testing community to push forward.

• All testers should have an online presence
• They should be involved in writing blogs.
• Be actively involved in testing discussion (software testing club, twitter)
• Should try to meet fellow testers a couple of times a year at testing meet up – the internet allows this very easily as this meet up has proved

Once this is done when you apply for a job and the agency asks for your certification – point to your active online presence in testing ask them to talk to peers who have met you and who can vouch for you. If we ALL did this then the need to pay organisations to prove you can test goes away. Let us as a testing community certify each other.

So with this last thought I did a little experiment on the people in the meet up involving the calculator experiment as taught by Michael Bolton – if you have experienced this from Michael then you will know what I mean if not then you need to find someone who knows because I am not going to spoil it by explaining on here. Thank you again Michael for giving me a useful way to demonstrate a key point.

So all too quickly the meet up had to end. Did we all learn something? I hope so. I know I did and I had a wonderful time and left feeling encouraged and motivated. I still think I talked far too much and for that I apologise hopefully if there is a next time I will encourage others to speak even more – you have been warned……

The Online Testing Community

I have worked in the field of IT for a long time and seen many changes but none have been as significant as the one that has happened online. The rise in the number of fellow professionals now blogging and tweeting (or should that be twittering) is amazing. The online testing community is increasing more and more as testers take the plunge and start to write and debate about testing online.

We have online testing clubs: http://www.softwaretestingclub.com/

magazines: (http://blog.softwaretestingclub.com/magazine/)

testing knowledge exchange : (http://testing.stackexchange.com/)

Organizations trying to get more of the testing community together online: (http://blogs.stpcollaborative.com/stpcollab/)

I would count myself as a late starter in this revolution only being brave enough to start twittering and blogging late last year. Why do I say brave enough? I am sure I am not the only one who feels that there are so many peers who we read about online and who we, deep down, admire. We may feel that we can never be good enough to write articles about testing or that what we write will be dismissed by the community. In my case I am well aware that my grammar is not the best in the world but all I am doing is writing down what I am thinking and I hope it comes across in a good way. There can never be enough of us online talking about our own opinions and valuable experiences. I had a fear that people would not be interested in what I had to say or worse still would think what I had to say is silly and I would feel rejected and humiliated. Surprising the testing community has not been like that.

I have had some wonderful debates and discussions on testing issues. I have been coached online by Michael Bolton (@michaelbolton) on using Exploratory Testing, in which he gave up his own time for free and is wonderful patience person who really makes me think differently. I have been introduced to such great testing thinkers as Rob Lambert (@Rob_Lambert) who has been a leading character in organizing the online testing community. I have had some great comments on my blog about subjects that I find interesting. I have found it a great outlet for my thoughts and ideas on testing that I once kept to myself, afraid that what I was thinking would not be of interest to anyone. I have also found it to be a wonderful resource for information and ideas about testing and how much people are eager to help.

I would like to say to anyone who is reading this and does not have a presence online to just go for it, start a blog, start a discussion on twitter or join the software testing club and start a debate. I have found it has given me a new lease of professional life, it has made me more aware that whatever problems I come across other are coming across the same problems. It has encouraged me to start writing about my experiences of testing and that I may have some useful information that others want to hear or read about. It has encouraged me to come out of my shell and talk to people about my passion for testing and that cannot be a bad thing……

Where do all the ‘old’ (experienced) software engineers go?

I was having a conversation with a colleague the other day whilst in the kitchen area making a coffee. When they stated the following:

“You never see many engineers over the age of 40 working in a development environment, actually writing code or testing”

I thought this was a interesting statement and it made me think. I have been in the IT business for a long time and yes I am over 40 and I am still actively involved in cutting edge software development projects. However when really thinking about the statement, how many more people did I know or have known who work in software development as developers and testers are still actively involved?

We had a discussion about it and came up with some reasons why.

• They move up further in the company (VP, CEO, etc) and take a less active role
• They switch careers becoming technical architects etc.
• They give up working in IT

One other point was made

“Software engineering is a young person’s career”

Quite a controversial statement!!!!

However could this be true?

Another colleague mentioned a point that as we get older we lose our mental ability or cognitive processing. However this article seems to debunk this: http://www.healthandage.com/html/min/afar/content/other6_1.htm

The article does state that we do lose our attentional ability and processing speed – key elements for software engineers.

What did interest in the article was the following:


“In general, memory tasks that are complex and require manipulating a lot of new information quickly become more difficult with age. Facts, names, and events that are not often accessed may become more difficult to retrieve from memory. However, knowledge that has been accumulated over a lifetime, which is repeatedly accessed and expanded, is generally retained. Well-practiced skills and abilities remain intact. And vocabulary usually continues to increase throughout life.”

So we may become slightly slower mentally as we get older but we retain our well practiced skills and abilities.

One important point made in the article which is related to my discussions on the telling of stories and is a key skill of an excellent tester: “….vocabulary usually continues to increase throughout life”

So my question and the real point of this article is:

“Do companies make a conscious or unconscious decision to remove older software engineers?”

It would be a shame if this is happening since currently I feel like I am in my prime. I am still discovering new and wonderful things about software development each and every day. I still have the same passion for my chosen career as I did when I first started with the added advantage that I have years of experience to fall back on as well.

I would love to hear from other people who, once they look around their respective companies, notice the same trend. Or from anyone who has any more theories on this.

Measuring with Stories

In my experience one of the most difficult tasks I have is to try and change the way we measure testing. When I first started out in the profession of testing there was very little thought given to how we should report the quality of the testing that has been carried out. The most common way was to record the number of test cases and report how many passed and failed.

Then the managers wanted more information such as:

• The number of defects raised
• The number of defects fixed
  
• The severity of the defects.

Does this sound familiar to anyone?

Sadly even today when testing as a profession has started to mature, managers still measure the quality of testing by using these figures and metrics.

I read the following article the other day: Defect Detection By Developers

The article talks about how developers can discover more defects in the same amount of time frame as a tester.

I always have problems when I see articles like this.

How is the statement:

'The number of defects detected by the developer is of the same order as detected by a test engineer in the same time frame'

Quantifiable or Qualitative?

There is no mention of the type of defect and the measure of risk of the project if the defect was not found. What happens if say 90% of the defects found using this method were purely cosmetic? Would this indicate this method is better than using a skilled tester during the same time frame? The skilled tester may find less defects during the same timescale but they may (and normally do) find the difficult to detect defects. Or having the tester and the developer work together during the development phase using continuous build to check as they go?

The approach discussed is one which many companies should already be using

In my experience one of the most difficult tasks I have is to try and change the way we measure testing. When I first started out in the profession of testing there was very little thought given to how we should report the quality of the testing that has been carried out. The most common way was to record the number of test cases and report how many passed and failed.

Then the managers wanted more information such as:

• Code reviews
• Peer reviews
• Documentation reviews (What happens if the project is a prototype project in which no documentation exists? - Maybe BDD could cover this?)
• Unit tests etc.

The article misses vital approaches such as continuous integration? How many defects are trapped during this method?

I think the article has some valid points however I strongly object to the statement that a developer can detect the same quality of defects as a skilled tester can in the same period of time.

I do not like 'measure by defects' to prove quality - quality is proven by the telling of a story to indicate that the product is of the right quality for its purpose.

This leads me nicely into the title of this blog.

Why does management insist on measuring the quality of testing using numbers?

Are managers measured using numbers?

In my role as a test manager I have never, in my experience, been appraised solely using numbers nor as far as I believe has any senior stake holders been measured solely using numbers. When I talk to my superiors about my management goals I try to tell stories and avoid using numbers. For example if I said to my line manager ‘I have met 70% of my targets’ and left it at that. What information would my line manager get? Would that be good or bad? Compare that to some tester reporting to their manager that 70% of their testers have passed. What information can you get from that?

I understand these are extremes but unfortunately these extremes are common when it comes to measuring the quality of testing.

So if I spoke to my line manager and said that over the last six months I have been involved in addressing x, y and z, which are the most important tasks, I still have problems with a and b and I have not had the time to deal with T and S. I have enjoyed the challenge of x y and z and feel I have achieved the best I can in these areas, a and b are ongoing and I am sure that I can complete them soon as long as I get support for b. My estimates where too ambitious for me to even start T and S but these were not too important so I gave them a low priority for completion.

Does this give you a better understanding of what has been happening? Do you notice there is no use of numbers in the above? I feel it gives far more information.

So if we now do the above again and convert it to a story about testing:

For the last six months I have been involved in testing areas x, y and z, which are the most important areas of the product. I still have problems with test areas a and b which appear not to be working as expected and I have raised defects for these issues which are reported below. I have not had the time to test areas T and S. Test areas x y and z are of sufficient quality for release. Currently test areas a and b are not of sufficient quality but once the defects in b are fixed I am confident that the quality will improve. Due to the time taken to address the issues with a and b my estimates where too ambitious for me to even start test areas T and S but since these were not too high priority the product can work without them as long as it is documented.

How does this sound? If you as a manager read this would you have a better understanding of the quality of testing?

Management is normally measured by the telling of a story, so why not measure the quality of testing by the telling of a good story?

I was on twitter the other day I noticed an article by BJ Rollison talking about a similar talking in how we measure the quality of testing. Meaningful Measures

There was an interesting line he came up with:

'At one time I naively believed that there was a core set of metrics that all teams should be collecting all the time that we could put into a ‘dashboard’ and compare across teams. In retrospect that was really a bone-headed notion. Identifying these measures is not easy, and there is no cookie-cutter approach. Each project team needs to decide on their specific goals that may increase customer value or impact business costs. Testers should ask themselves, “why are we measuring this?” “What actions will be taken as a result of these measures?” And, “if there is no actionable objective associated with this measure, then why am I spending time measuring this?'

Some very valid points, testers should be asking why are we measuring this, is it quantifiable? Will it help lay people understand the quality of the product?

When we measure the quality of testing it should be clear, concise and in a language that anyone can understand. Blinding people with numbers does not aid clarity. Trying to measure different teams on different projects with the same metrics (code coverage, defect counts, test case pass/fail) does not indicate one team is better than the other all it does is help management pretend that they understand the quality of testing that took place.

With this article I am not saying we should abandon all numerical metrics to measure the quality of software testing but we need to look more at the story behind the numbers since these can give you far more information on the quality of testing.

Some other useful articles on Software Metrics:
meaningful-metrics by Michael Bolton
Metrics by Kaner and Bond

Does manual testing really lose its value as Companies encouraging more and more automation?

A colleague the other day asked me for my views on the automation/manual debate and asked the following question:

Does manual testing really lose its value as Companies encouraging more and more automation?

I thought this is a very interesting question and decided to blog my response.

I should start by saying I am not split into either camp on the automation vs. manual testing debate. I can see the benefit of both depending on circumstances you are in. If you have an old legacy project in which you are 100% sure nothing will change then automation could be the answer. If you involved in a system in which changes could be made and you want to CHECK that the functionality or the business rule that is in place is still giving the exact same result with no deviation then automation could work.

My thoughts on the question that I was asked (and the title of this article) are completely the opposite. Automation requires no sapience or thinking to be executed, so once the automated checks have been written (which does require sapience) you can run and forget.

The problem is that the world in which we work in is all about changing, adapting and making things better (normally) especially in an agile environment where change is embraced.

I should state that my definition of manual testing is not of following pre scripted tests but of being a test explorer, searching in all the nooks and crannies, trying to discover new and intriguing things about the software. If it is pre scripted then automate it, do not waste good tester intelligence and skill on running a check list, your testers deserve better.

So I would ask the following questions on any company who want to encourage more automation at the expense of manual testing.

• Is it cost effective to write lots of automated checks compared to carrying out manual exploratory testing?
• Which method would in the same time period give the most test coverage?
• Which would be the most easy to adapt to major changes?
• Which would uncover the most problems or issues?

I feel there is some value in using automation at unit level and build level, continuous integration with acceptance checks is a useful tool for the software tester since it lets them have a early look at changes with some confidence that what they have been given will at least have a chance of working. Sure beats the good old days of rejecting x releases in day because of a typo in an install script or a missing dll from the build.

You hopefully can see that I am not against using automation; however there appears to be a view in the testing world that automation can replace manual testing or make its value less. This view does worry me since if the corporate suits think they can get more value and better quality using automation then the message about the art software testing is not being broadcast well enough.

Manual and automation can co-exist very well together; however:

• manual testing can exist without automation
  
• but automation cannot exist without manual testing.

There is an interesting podcast with Jon Bach and Michael Bolton with their viewpoints on the difference between checking and testing available here:

http://www.quardev.com/blog/2010-02-02-1123487836

Exploratory Testing and Scrums

I was having a conversation with Michael Bolton (www.developsense.com) and we started to discuss how ET can be incorporated effectively within agile scrums. After a few minutes of chatting Michael suggested that I blog my thoughts on this since it appeared to be a good idea.

So here we have a short article on how we could use ET within a scrum.

The structure of a scrum in my experience is

What you have done?
What you intend to do?
What is stopping you?

In most cases a scrum normally turns into a technical debate – this may not always be the case but it is something that I have experienced. My thoughts are why not use the time in the scrum to the benefit of testers?

Instead of just saying what you have done, tell the story of what charters you have followed and what you have found interesting. Talk about the test ideas you followed and what other test ideas (test coverage?) you have thought about while testing.

The crucial difference is the next part:

Instead of just saying what charters or testing ideas you intend to carry out for the next days work why not start a discussion on what test areas they feel you could be missing – this would involve the programmer, the product owner and any others in attendance at the scrum. Imagine the benefit in using the experience and knowledge of other people on the project to help with your test coverage.

I am sure people could say why not just go and talk to them individually for their ideas? My view is that people are more than likely to bounce ideas off each other and trigger responses in others when they are together as a team rather than as individuals.

I have attempted to do this in my own scrums with mixed success. Sometimes people within the team have pointed out huge areas of coverage that the test team had not even thought about whilst in other cases there has been a muted silence. I feel for this to work depends on the dynamics of the team and how they interact.

The next stage for this experiment would be to try and take the user stories during the planning sessions and open a discussion on test coverage/ideas. I really would like to try to use something like HICCUPPS (http://www.developsense.com/articles/2005-01-TestingWithoutAMap.pdf ) or SFDPOT (http://www.developsense.com/articles/2005-10-ElementalModels.pdf) with all the development team to see if we as a team could improve test coverage.

I know I promised an article on my experience of being coached by Michael Bolton on Exploratory Testing and my discussions with Jon Bach (http://jonbox.wordpress.com/) on managing ET. However I thought I should blog this while it is still in my mind. The article will happen I promise. I just want to wait until I have finished the coaching sessions.

Plus shrinik – I have not forgotten that I will blog an article measuring management with a story. – I hope to put this together sometime in March.

Child’s Play

This article is based upon some thoughts that I have had over the past year while watching my granddaughter playing and learning new things, she will be two years old in April. It is amazing how quickly children of that age learn to do tasks without being taught such as walking, the beginnings of communication and how to play and explore.

I am an exploratory tester and my thinking is how we can as tester harness what children do naturally? Some may say that they do exploratory testing and they feel it is natural. If that is the case why do so many testers have difficulty adapting to exploratory testing and keep falling back to scripted testing?

Peter (unlicensed testers) asks a lot of questions about children and learning in his blog article here: http://007unlicensedtotest.blogspot.com/2009/11/what-do-you-get-if-you-cross-7-month.html

Watching my granddaughter the other day I observed that she was trying to put her trousers on. At first she managed to get both legs into the same leg hole, she noticed that this did not feel right so started again but this time she tried to put both legs into one of the small leg holes and found that this did not work either. After this she then managed to get the trousers the correct way around and one leg in each hole but did not pull them up and ten tried to walk and fell over.

What can we learn from this?

We can see that she tried following different options and observing the results, she then thought she had completed the task but found that it was not really complete. So if we convert this to software testing we can see that she is using heuristics to determine how to do the testing, the trial and error approach. She is using her emotions and feelings that something is not correct and she is doing a lot of noticing, which is something that every good exploratory tester should be doing. There are also examples of mentally noting future areas to test, the fact that when she tried to walk she fell over. The next time she tried to put on her trousers she did manage to pull them all the way up before setting off to walk.

There are many other examples of trial and error that children appear to do when playing. If something does not feel right they will suddenly change the approach to the problem or in some cases they just give up.

What can we learn from children playing? I have observed that this exploring behaviour appears to start diminishing once children start to attend full time school. Why is this so?

Is it because schools start to impose on children their own ethos and standards and re-model children to not take risk?

One of the main elements of testing IMO is the taking of risks.

We all do the ‘Let us try this to see what happens’, ‘Let us try something else and see what happens.’ Many corporations are risk adverse and as such when testers are brought in they have to provide a return on investment (a hot topic on twitter at the moment) so they are less likely to follow a risky approach. Some may argue that exploratory testing is not risky and I would intend to agree with them. However the business world does not seem to afford the time to be able to cope with let us try this and see what happens and then let us try that and see what happens. They require order and structure and no risk.

Children on the other hand when learning and playing do not take risk into account they try and if it does not work try something else and if that does not work they continue trying until they get a result they are happy with. They remain focused on the task at hand but appear to be able to solve problems without the fear of failure.

So what happens as we get older? Why do we lose this ability to explore and learn without fear of failure? It appears to be a natural human instinct that somehow is gradually removed as we get older.

Is it to do with the education system and how they remove the risk factor and make everyone fear failure and taking risks? Do we become institutionalize in to conforming to the known path, to stop asking the probing questions and to stop playing? I am not sure I have the answer to these questions however I am sure as testers there are some valuable lessons we could learn from children and how they explore, learn and play. (Simultaneous learning, test design and test execution)

I wonder if James would not mind if we change the definition of exploratory testing to learn, explore and play?

I think everyone who wants to learn more about exploratory testing should take some lessons from children. Do not be afraid to explore, if you make a mistake learn from the mistake to improve the next time and have fun. Testing should be about having fun and enjoyment it should not be a chore. If it becomes a chore get a different job……

Does anyone have any interesting games for testers? If so I would love to hear from you.
___________________________________


My next blog should be on my experiences of being coached by Michael Bolton and Jon Bach on using and managing Exploratory Testing.