The context of this article is about software test estimation, but some of these secrets can be seen in other types of estimating as well. This list is not intended to be complete, but rather the things I see most often in trying to estimate software testing efforts.
Secret 1 – Your Estimate is Probably Wrong
Take comfort in this. Too many people work and work to get “just right” estimates. The problems are:
1) As a tester, your estimates often depend on other estimates, such as the time needed for development, people available, etc. These numbers are often inaccurate, so that means yours will be also.
On one project, I was asked to come up with some quick estimates on about 15 projects for a given business. To help in this effort, I was sent a set of 15 documents, each of which contained estimates for one of the projects to be estimated. My approach was to apply a factor of 25% of the total development effort. Now this factor could be considered low, since the actual percentages can range between 20 and 50%.
However, there was one thing that made my test effort factor practically irrelevant. Many of the prior estimates had “TBD” on the design and code efforts. This gap in data rendered my assumptions to the level of sheer guesswork. I had a bigger problem, though. My client still wanted estimates. What's a consultant to do?
In this case, I made some assumptions with a warning that the estimates could be way off. I estimated the effort to design the project equal to that of gathering and documenting requirements. I also made the effort to code the projects the same as the effort to design the project. Yes, this gave the same number for person-weeks for requirements, design and coding. Keep in mind I had no resource numbers at this point, only estimated weeks of effort. I documented my assumptions and also the low confidence I had in the estimate. I assured my client the estimate would change once we had numbers for design and coding. Plus, we would need resource estimates before we could estimate costs.
About a week later, I got the missing data. Not surprisingly, the newly supplied estimates for design and coding were different from the ones I assumed. So, I used the new numbers and my estimates changed. I have yet to know how accurate any of the estimates are at this point.
(By the way, as things turned out, none of the projects were ever started!)
2) The Project Scope Will Probably Change.
There's not much you can do about this. Scope change happens for a variety of reasons, some foreseen and some unforeseen.
You never know when new requirements will emerge. New opportunities can appear as well.
Last year, I had a fence installed at my home. When we first measured the dimension of the area to be fenced, I used the area that I had assumed was my property. Since there is not a house on the other side of us, I could not use the adjoining property as a basis. After reading the plat map, I learned that our property is actually about ten feet narrower than I had believed. For years, I have been watering, fertilizing and mowing about 1,000 square feet of property that did not belong to me!
This new information changed the scope and therefore, the estimate and final costs of the project.
Software projects are more complex than home improvements, and this makes the challenge even greater. Our assumptions get challenged and often proven wrong. Whether it is the scope of requirements or the number of resources needed, scope can change either up or down.
3) You Don't Know What You Don't Know.
And...you won't find that out until it's too late. This is one of the biggest project risks and impacts much more than estimates. There's not much you can do about this one either. Just keep an open mind, keep questioning decisions and information throughout the project. Realize that you don't know everything and what you do know will change.
Here are a couple of prime examples of project “surprises” :
- People will fudge numbers to make themselves look better. Realize this happens and always question information, whether it be bug counts, estimates or the time spent on testing a feature.
- Deliverables will be late. In fact, they will be later than you think they will. Some things will be early, but more often than not, your work will be delayed by delays experienced by others.
Secret 2 – Your Estimate Needs to be Wrong – In the Right Direction
Why do I say something so outlandish? You only know how close your estimate is when the work is done. When my focus is on the accuracy of the estimate, I am at risk of conforming the work to the estimate, not what is in the best interest of the project.
When you truly understand your estimate is just that – an estimate, you can be free to adjust it. At least, you can adjust it for your own purposes if your management is not accepting of reality.
So what is the “right” direction? It's high.
There's an interesting debate among project managers as to whether or not it is ethical to “pad” estimates for the purpose of dealing with future unknown needs. Some believe you must have pads and others believe it is unethical to give an estimate you know is higher than you think is accurate.
I fall into the camp of overestimating. However, I don't call this extra a “pad”. Instead, I call it a “reserve” that is the safety net should extra time, money and/or people be needed. Even Scotty never gave Captain Kirk the real estimate!
I don't see this as deception. I see it as contingency planning and a necessary part of any estimate. In my experience, it's better to come in “under” than “over.”
It's funny that the people I know to be the best estimators have been doing it a long time and their estimates seem very high at first. I think these people have learned the lesson of having reserves in the estimate.
Secret 3 – The Recipient of the Estimate Already Has a Number in Mind
This is why I say that estimation of anything is an inherently human activity. There's a lot of feeling involved and not much logic at times.
Understand that your customer has an expectation already in mind. If your estimate goes too far above or below the customer's early expectation, silent alarms go off in their head.
In the case of my fence project, I had a number in mind. It wasn't a low price and it wasn't too far off from the estimate I received. When I did get the estimate, at least I didn't have sticker shock. Had my expectation been a lot lower that the estimate, I would have probably looked for a cheaper option.
Once I had an experience in which I gave an estimate that was so much lower than a competitor's and I almost didn't get the work! The prospective client felt that because of the lower price I must have been leaving something out. Thankfully, they asked me to explain why my quote was so much lower and I explained it was because I had a faster and more effective way of doing the job.
Secret 4 - Your Customer Assumes Your Estimate is Too High
This is one of the biggest hindrances to good estimation. Let's say that you do your best job of estimation with no reserves included. You present these to your management or customer and they take one look and immediately reduce it by one-half because they believe you must have inflated it. After all, the last person they got an estimate from did so. Therefore, the assumption is that everyone who estimates has excess in the numbers.
What does this do? It forces people to pad their estimates because they know management will assume they have been inflated. Therefore, it is a self-fulfilling prophecy.
Secret 5 – Your Estimates are Incomplete
Most people estimate time and cost. How many people do you know that also estimate the quality of the product? Not many I would guess.
You have probably heard the old adage, “cost, schedule, quality – pick two.”
While it's true that it's very hard to deliver all three aspects, it's still possible. High quality will probably cost more and may take longer than lower quality – at least in the short term. In the long run, high quality costs less because of lower maintenance.
In software development and testing, we often fail to consider the quality levels in a given estimate. For example, how much would a test cost that found a few defects and resulted in a buggy product versus one that found many defects (or found the right ones) and resulted in a high-quality product? More often than not we estimate how long testing will take, how many people we need, and how much testing will cost. A lot fewer people estimate how many defects they find at the various levels of criticality.
Conclusion
As you go forth and estimate, keep these little secrets in mind and you will have less stress over your estimates. If you look at how people respond to your estimates, you will see many of the secrets in action. These are things you can't control. However, you can understand that the dynamics exist and be prepared to keep expectations reasonable.