Where do all the ‘old’ (experienced) software engineers go?

I was having a conversation with a colleague the other day whilst in the kitchen area making a coffee. When they stated the following:

“You never see many engineers over the age of 40 working in a development environment, actually writing code or testing”

I thought this was a interesting statement and it made me think. I have been in the IT business for a long time and yes I am over 40 and I am still actively involved in cutting edge software development projects. However when really thinking about the statement, how many more people did I know or have known who work in software development as developers and testers are still actively involved?

We had a discussion about it and came up with some reasons why.

• They move up further in the company (VP, CEO, etc) and take a less active role
• They switch careers becoming technical architects etc.
• They give up working in IT

One other point was made

“Software engineering is a young person’s career”

Quite a controversial statement!!!!

However could this be true?

Another colleague mentioned a point that as we get older we lose our mental ability or cognitive processing. However this article seems to debunk this: http://www.healthandage.com/html/min/afar/content/other6_1.htm

The article does state that we do lose our attentional ability and processing speed – key elements for software engineers.

What did interest in the article was the following:


“In general, memory tasks that are complex and require manipulating a lot of new information quickly become more difficult with age. Facts, names, and events that are not often accessed may become more difficult to retrieve from memory. However, knowledge that has been accumulated over a lifetime, which is repeatedly accessed and expanded, is generally retained. Well-practiced skills and abilities remain intact. And vocabulary usually continues to increase throughout life.”

So we may become slightly slower mentally as we get older but we retain our well practiced skills and abilities.

One important point made in the article which is related to my discussions on the telling of stories and is a key skill of an excellent tester: “….vocabulary usually continues to increase throughout life”

So my question and the real point of this article is:

“Do companies make a conscious or unconscious decision to remove older software engineers?”

It would be a shame if this is happening since currently I feel like I am in my prime. I am still discovering new and wonderful things about software development each and every day. I still have the same passion for my chosen career as I did when I first started with the added advantage that I have years of experience to fall back on as well.

I would love to hear from other people who, once they look around their respective companies, notice the same trend. Or from anyone who has any more theories on this.

Measuring with Stories

In my experience one of the most difficult tasks I have is to try and change the way we measure testing. When I first started out in the profession of testing there was very little thought given to how we should report the quality of the testing that has been carried out. The most common way was to record the number of test cases and report how many passed and failed.

Then the managers wanted more information such as:

• The number of defects raised
• The number of defects fixed
  
• The severity of the defects.

Does this sound familiar to anyone?

Sadly even today when testing as a profession has started to mature, managers still measure the quality of testing by using these figures and metrics.

I read the following article the other day: Defect Detection By Developers

The article talks about how developers can discover more defects in the same amount of time frame as a tester.

I always have problems when I see articles like this.

How is the statement:

'The number of defects detected by the developer is of the same order as detected by a test engineer in the same time frame'

Quantifiable or Qualitative?

There is no mention of the type of defect and the measure of risk of the project if the defect was not found. What happens if say 90% of the defects found using this method were purely cosmetic? Would this indicate this method is better than using a skilled tester during the same time frame? The skilled tester may find less defects during the same timescale but they may (and normally do) find the difficult to detect defects. Or having the tester and the developer work together during the development phase using continuous build to check as they go?

The approach discussed is one which many companies should already be using

In my experience one of the most difficult tasks I have is to try and change the way we measure testing. When I first started out in the profession of testing there was very little thought given to how we should report the quality of the testing that has been carried out. The most common way was to record the number of test cases and report how many passed and failed.

Then the managers wanted more information such as:

• Code reviews
• Peer reviews
• Documentation reviews (What happens if the project is a prototype project in which no documentation exists? - Maybe BDD could cover this?)
• Unit tests etc.

The article misses vital approaches such as continuous integration? How many defects are trapped during this method?

I think the article has some valid points however I strongly object to the statement that a developer can detect the same quality of defects as a skilled tester can in the same period of time.

I do not like 'measure by defects' to prove quality - quality is proven by the telling of a story to indicate that the product is of the right quality for its purpose.

This leads me nicely into the title of this blog.

Why does management insist on measuring the quality of testing using numbers?

Are managers measured using numbers?

In my role as a test manager I have never, in my experience, been appraised solely using numbers nor as far as I believe has any senior stake holders been measured solely using numbers. When I talk to my superiors about my management goals I try to tell stories and avoid using numbers. For example if I said to my line manager ‘I have met 70% of my targets’ and left it at that. What information would my line manager get? Would that be good or bad? Compare that to some tester reporting to their manager that 70% of their testers have passed. What information can you get from that?

I understand these are extremes but unfortunately these extremes are common when it comes to measuring the quality of testing.

So if I spoke to my line manager and said that over the last six months I have been involved in addressing x, y and z, which are the most important tasks, I still have problems with a and b and I have not had the time to deal with T and S. I have enjoyed the challenge of x y and z and feel I have achieved the best I can in these areas, a and b are ongoing and I am sure that I can complete them soon as long as I get support for b. My estimates where too ambitious for me to even start T and S but these were not too important so I gave them a low priority for completion.

Does this give you a better understanding of what has been happening? Do you notice there is no use of numbers in the above? I feel it gives far more information.

So if we now do the above again and convert it to a story about testing:

For the last six months I have been involved in testing areas x, y and z, which are the most important areas of the product. I still have problems with test areas a and b which appear not to be working as expected and I have raised defects for these issues which are reported below. I have not had the time to test areas T and S. Test areas x y and z are of sufficient quality for release. Currently test areas a and b are not of sufficient quality but once the defects in b are fixed I am confident that the quality will improve. Due to the time taken to address the issues with a and b my estimates where too ambitious for me to even start test areas T and S but since these were not too high priority the product can work without them as long as it is documented.

How does this sound? If you as a manager read this would you have a better understanding of the quality of testing?

Management is normally measured by the telling of a story, so why not measure the quality of testing by the telling of a good story?

I was on twitter the other day I noticed an article by BJ Rollison talking about a similar talking in how we measure the quality of testing. Meaningful Measures

There was an interesting line he came up with:

'At one time I naively believed that there was a core set of metrics that all teams should be collecting all the time that we could put into a ‘dashboard’ and compare across teams. In retrospect that was really a bone-headed notion. Identifying these measures is not easy, and there is no cookie-cutter approach. Each project team needs to decide on their specific goals that may increase customer value or impact business costs. Testers should ask themselves, “why are we measuring this?” “What actions will be taken as a result of these measures?” And, “if there is no actionable objective associated with this measure, then why am I spending time measuring this?'

Some very valid points, testers should be asking why are we measuring this, is it quantifiable? Will it help lay people understand the quality of the product?

When we measure the quality of testing it should be clear, concise and in a language that anyone can understand. Blinding people with numbers does not aid clarity. Trying to measure different teams on different projects with the same metrics (code coverage, defect counts, test case pass/fail) does not indicate one team is better than the other all it does is help management pretend that they understand the quality of testing that took place.

With this article I am not saying we should abandon all numerical metrics to measure the quality of software testing but we need to look more at the story behind the numbers since these can give you far more information on the quality of testing.

Some other useful articles on Software Metrics:
meaningful-metrics by Michael Bolton
Metrics by Kaner and Bond

Does manual testing really lose its value as Companies encouraging more and more automation?

A colleague the other day asked me for my views on the automation/manual debate and asked the following question:

Does manual testing really lose its value as Companies encouraging more and more automation?

I thought this is a very interesting question and decided to blog my response.

I should start by saying I am not split into either camp on the automation vs. manual testing debate. I can see the benefit of both depending on circumstances you are in. If you have an old legacy project in which you are 100% sure nothing will change then automation could be the answer. If you involved in a system in which changes could be made and you want to CHECK that the functionality or the business rule that is in place is still giving the exact same result with no deviation then automation could work.

My thoughts on the question that I was asked (and the title of this article) are completely the opposite. Automation requires no sapience or thinking to be executed, so once the automated checks have been written (which does require sapience) you can run and forget.

The problem is that the world in which we work in is all about changing, adapting and making things better (normally) especially in an agile environment where change is embraced.

I should state that my definition of manual testing is not of following pre scripted tests but of being a test explorer, searching in all the nooks and crannies, trying to discover new and intriguing things about the software. If it is pre scripted then automate it, do not waste good tester intelligence and skill on running a check list, your testers deserve better.

So I would ask the following questions on any company who want to encourage more automation at the expense of manual testing.

• Is it cost effective to write lots of automated checks compared to carrying out manual exploratory testing?
• Which method would in the same time period give the most test coverage?
• Which would be the most easy to adapt to major changes?
• Which would uncover the most problems or issues?

I feel there is some value in using automation at unit level and build level, continuous integration with acceptance checks is a useful tool for the software tester since it lets them have a early look at changes with some confidence that what they have been given will at least have a chance of working. Sure beats the good old days of rejecting x releases in day because of a typo in an install script or a missing dll from the build.

You hopefully can see that I am not against using automation; however there appears to be a view in the testing world that automation can replace manual testing or make its value less. This view does worry me since if the corporate suits think they can get more value and better quality using automation then the message about the art software testing is not being broadcast well enough.

Manual and automation can co-exist very well together; however:

• manual testing can exist without automation
  
• but automation cannot exist without manual testing.

There is an interesting podcast with Jon Bach and Michael Bolton with their viewpoints on the difference between checking and testing available here:

http://www.quardev.com/blog/2010-02-02-1123487836

Exploratory Testing and Scrums

I was having a conversation with Michael Bolton (www.developsense.com) and we started to discuss how ET can be incorporated effectively within agile scrums. After a few minutes of chatting Michael suggested that I blog my thoughts on this since it appeared to be a good idea.

So here we have a short article on how we could use ET within a scrum.

The structure of a scrum in my experience is

What you have done?
What you intend to do?
What is stopping you?

In most cases a scrum normally turns into a technical debate – this may not always be the case but it is something that I have experienced. My thoughts are why not use the time in the scrum to the benefit of testers?

Instead of just saying what you have done, tell the story of what charters you have followed and what you have found interesting. Talk about the test ideas you followed and what other test ideas (test coverage?) you have thought about while testing.

The crucial difference is the next part:

Instead of just saying what charters or testing ideas you intend to carry out for the next days work why not start a discussion on what test areas they feel you could be missing – this would involve the programmer, the product owner and any others in attendance at the scrum. Imagine the benefit in using the experience and knowledge of other people on the project to help with your test coverage.

I am sure people could say why not just go and talk to them individually for their ideas? My view is that people are more than likely to bounce ideas off each other and trigger responses in others when they are together as a team rather than as individuals.

I have attempted to do this in my own scrums with mixed success. Sometimes people within the team have pointed out huge areas of coverage that the test team had not even thought about whilst in other cases there has been a muted silence. I feel for this to work depends on the dynamics of the team and how they interact.

The next stage for this experiment would be to try and take the user stories during the planning sessions and open a discussion on test coverage/ideas. I really would like to try to use something like HICCUPPS (http://www.developsense.com/articles/2005-01-TestingWithoutAMap.pdf ) or SFDPOT (http://www.developsense.com/articles/2005-10-ElementalModels.pdf) with all the development team to see if we as a team could improve test coverage.

I know I promised an article on my experience of being coached by Michael Bolton on Exploratory Testing and my discussions with Jon Bach (http://jonbox.wordpress.com/) on managing ET. However I thought I should blog this while it is still in my mind. The article will happen I promise. I just want to wait until I have finished the coaching sessions.

Plus shrinik – I have not forgotten that I will blog an article measuring management with a story. – I hope to put this together sometime in March.

Child’s Play

This article is based upon some thoughts that I have had over the past year while watching my granddaughter playing and learning new things, she will be two years old in April. It is amazing how quickly children of that age learn to do tasks without being taught such as walking, the beginnings of communication and how to play and explore.

I am an exploratory tester and my thinking is how we can as tester harness what children do naturally? Some may say that they do exploratory testing and they feel it is natural. If that is the case why do so many testers have difficulty adapting to exploratory testing and keep falling back to scripted testing?

Peter (unlicensed testers) asks a lot of questions about children and learning in his blog article here: http://007unlicensedtotest.blogspot.com/2009/11/what-do-you-get-if-you-cross-7-month.html

Watching my granddaughter the other day I observed that she was trying to put her trousers on. At first she managed to get both legs into the same leg hole, she noticed that this did not feel right so started again but this time she tried to put both legs into one of the small leg holes and found that this did not work either. After this she then managed to get the trousers the correct way around and one leg in each hole but did not pull them up and ten tried to walk and fell over.

What can we learn from this?

We can see that she tried following different options and observing the results, she then thought she had completed the task but found that it was not really complete. So if we convert this to software testing we can see that she is using heuristics to determine how to do the testing, the trial and error approach. She is using her emotions and feelings that something is not correct and she is doing a lot of noticing, which is something that every good exploratory tester should be doing. There are also examples of mentally noting future areas to test, the fact that when she tried to walk she fell over. The next time she tried to put on her trousers she did manage to pull them all the way up before setting off to walk.

There are many other examples of trial and error that children appear to do when playing. If something does not feel right they will suddenly change the approach to the problem or in some cases they just give up.

What can we learn from children playing? I have observed that this exploring behaviour appears to start diminishing once children start to attend full time school. Why is this so?

Is it because schools start to impose on children their own ethos and standards and re-model children to not take risk?

One of the main elements of testing IMO is the taking of risks.

We all do the ‘Let us try this to see what happens’, ‘Let us try something else and see what happens.’ Many corporations are risk adverse and as such when testers are brought in they have to provide a return on investment (a hot topic on twitter at the moment) so they are less likely to follow a risky approach. Some may argue that exploratory testing is not risky and I would intend to agree with them. However the business world does not seem to afford the time to be able to cope with let us try this and see what happens and then let us try that and see what happens. They require order and structure and no risk.

Children on the other hand when learning and playing do not take risk into account they try and if it does not work try something else and if that does not work they continue trying until they get a result they are happy with. They remain focused on the task at hand but appear to be able to solve problems without the fear of failure.

So what happens as we get older? Why do we lose this ability to explore and learn without fear of failure? It appears to be a natural human instinct that somehow is gradually removed as we get older.

Is it to do with the education system and how they remove the risk factor and make everyone fear failure and taking risks? Do we become institutionalize in to conforming to the known path, to stop asking the probing questions and to stop playing? I am not sure I have the answer to these questions however I am sure as testers there are some valuable lessons we could learn from children and how they explore, learn and play. (Simultaneous learning, test design and test execution)

I wonder if James would not mind if we change the definition of exploratory testing to learn, explore and play?

I think everyone who wants to learn more about exploratory testing should take some lessons from children. Do not be afraid to explore, if you make a mistake learn from the mistake to improve the next time and have fun. Testing should be about having fun and enjoyment it should not be a chore. If it becomes a chore get a different job……

Does anyone have any interesting games for testers? If so I would love to hear from you.
___________________________________


My next blog should be on my experiences of being coached by Michael Bolton and Jon Bach on using and managing Exploratory Testing.

Why Testers do not add Value

This is an article I published internally last year when there was a debate going on about the value of testers within the testing community so I thought would re-publish it here for people to have a read.

___________________

This may seem like a strange statement coming from a tester; however the purpose of this article is an attempt at justifying the need for testers on a project. Financial people might use traditional cost/benefit comparison models when recruiting people for a team or project which may not necessarily be applicable when resourcing testers.

What made me think about writing this article was something I happened to read on the Software Testing Zone website by Debasis Pradhan 1 and from Michael Bolton at the Developsense website .2

There are two comments within these articles that appear to come from a developer’s point of view about the role of testers within software projects.

The first is from Debasis blog:

Software Testing is a worthless process. It does not add any value to the project!

And one from Michael’s blog

But testers don't add value to a project;

These comments made me think about the way testers are perceived on projects and whether or not they really do add any value to a project. This led to researching how accurate these statements were and the production of this article.

The problem I have found when carrying out the research for this article is that the term ‘value’ can have different meanings depending on the context in which it is used.

If I were to ask you what is the value of a gift a loved one gave you who is no longer here?

What value would you put on that item?

In money terms it might not be worth that much but in emotional or sentimental terms, to you, it could be priceless. It is very important when reading this to use the correct context for the meaning of value. For the purpose of this article value will be measured in terms of monetary value otherwise your emotional viewpoint may get in the way of what is being said.

So the definition of value for this article will be:

Value implies monetary value.

If we take a look at a typical upfront project and see where costs are attributed and where value is added it may give a better understanding of the statement.

The first part of a project is the initial design and scope where the architect captures and translates the requirements of the customer in to technical solutions and business requirements. The value of an architect is very clear since they are the people who provide solutions to the questions raised by the business. They ensure that the requirements for the customer are met and what the customer asks for is what will be delivered. Those who work within software development may not agree with the final statement since sometimes features requested are not delivered or changes are made that affect part of the delivery. However in the majority of cases what the customer requires is delivered by the architect.

The project manager will then look at these requirements and sees which has the highest business value at the same time assessing the risk of adding the features before making the decision as to what to develop which will give the maximum value.

These technical solutions, risks and business value are then interpreted by the developers and turned into an actual project so their value can be measured in that they build the project. The developers also fix any problems and solve any issues that may occur either from the customer or the test team.

Marketing then can sell that product or may already have sold the product adding value to the company.

Some may say that testers must add value since they find problems before a product is released and as such stop a project costing more in terms of company reputation and actual monetary cost. This may appear true but testers do not actually provide solutions and fix the problems, this is the role of the architect and the developer.

So what does a tester do within a software project?

Testers look at the project being developed and raise questions and report observations as to what may not be working correctly. They look and report on areas which could cause the most risk and most expense to fix. They give priority to the main problems (defects) within the product.

However the people who fix these problems, act upon the observations and answer the tester’s questions are the ones who add value to a project.

This view is based upon a typical style of software development where all requirements are gathered upfront so what about agile style projects?

It is rather interesting that there are very differing viewpoints on this within the testing world, one side saying that if the testers are working directly with the developers and writing the test cases before the code is written then they are adding value to the project. The other side is saying they may write the test cases but they do not actually do any creative design work and as such do not add any real value to the project. My viewpoint is of the latter if the testers are not actually creating code that will lead to a function that will be used by the user in the live system then they are not adding any real value to the project.

The way in which testers could add actual value to the project when using agile is when they have close working arrangements with the actual customer since they can then suggest or inform the customer that if they did this a different way it would have more benefit. This would still not be intrinsic value unless the tester managed to get the customer to sign up for more work at an additional cost, therefore adding to the value of the project.

To conclude testers do not add intrinsic value to software development project but they prevent deprecation of the project value. If you have a project that a customer is willing to pay five million dollars for and you ship that project with x amount of high level problems. The customer then quite rightly demands the problems be fixed free of charge. Then the value of your project could go down from the initial five million to say four million, losing the company both real money and another value that cannot be measured: ‘your company reputation’.

It should be noted that there is the factor of costs of resource on projects and to keep things very simple I have excluded these other costs when making the above statement. Otherwise you would need to include the cost of equipment, building, utilities and so forth.

With testers on the project the majority of the high level problems would have been caught before being shipped to the customer and as such the value of the project would have been maintained.

Therefore testers do not add value but they certainly prevent value from being lost. It should be noted that no tester can guarantee one hundred percent bug free software. Even companies such as NASA who on the Space Shuttle program had a ratio of 10 testers to 1 developer 4 cannot give that guarantee as demonstrated by the tragic accidents that have occurred! The average ratio in the majority of projects is 1 tester to 3 developers5; therefore value will always be lost in some way from a project.

So remember those whose job it is to resource software development projects never skimp or cut costs by using less testing resources and think that you have saved yourself some money. You might just find that the 1 million dollar project you are working on becomes worthless due to lack of testing resource to ensure the value is maintained.

I will leave you with a view from James Bach, who is a software testing author and one of the main contributors in the software testing field.

‘James Bach suggests, testers help to defend the value that's already there, or help to identify ways in which value may be lacking. Testers raise questions and make observations; the people who make decisions based on those observations are the ones who add value. We help them do that, but we don't do it intrinsically on our own.’ (3)

References:

1 http://software-testing-zone.blogspot.com/2008/10/software-testing-add-value-to-project.html

2 http://www.developsense.com/2008/03/breaking-code.html

3 http://www.developsense.com/2008/10/while-back-i-wrote-post-on-breaking.html

4 http://codeidol.com/other/Software-Estimation/Estimating-Planning-Parameters/21.1-Estimating-Activity-Breakdown-on-a-Project/ (table 21-7)

5 http://www.infoq.com/news/2009/01/tester-to-developer-ratio

______________

Hopefully if I get myself sorted my next article will be about how we can learn to improve how we use exploratory testing from watching how children play and learn.

Testing Terminology – definition or context?

Hello and Happy New Year

One of my New Year resolutions was to start using twitter more so that I could micro blog some of my thoughts.

One interesting topic that appeared to be going around was one of changing testing definitions and descriptions. This caused quite an animated debate amongst testing twitters.

So I thought I would use more that 140 characters and put across some of my thoughts and views on the subject of testing definitions.

I used to have the view that it was good to have a set of definitions for testing terms such as what is Acceptance testing, What is black box testing etc. The ISTQB has a document for this on their website (www.istqb.org/downloads/glossary-1.2.pdf)

The problem with this glossary is that it applies testing meaning with no context and that is the problem I find with trying to define testing terminology to do so requires context. If I wish to be somewhat controversial I should state that this is the problem I find with the ISEB and ISTQB examinations especially the ISEB foundation level which is based upon multiple choice answers. Some of the questions could in certain situations have multiple answers but ISEB only accept their definitions and do not allow the natural questioning skills of the tester to debate the answer.

For example let us look at the testing term ‘Acceptance Testing’

ISTQB define it as

Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system.

However if I would add the following context to Acceptance Testing:

The development team requires some form of automated Acceptance testing (Yes Michael I know if you are reading this it should be checking not testing - http://www.developsense.com/2009/08/testing-vs-checking.html) on the build machine before signaling that the build is suitable for release to the testing team


…would the above definition hold true?