Defects are Good

"What in the world is he thinking?" you may be asking. After all, we all know that software defects are things that cause applications to not work correctly, so how can a defect be good?

I first saw this idea expressed by a client¹ as a slogan that was used throughout their organization. As I continued to think about the statement, "Defects are good," it caused me to consider how the many perceptions of defects affect how QA and testing professionals deal with software quality in general.

The point is that a defect found early, or before a customer finds it, has value. To the producer, the value lies in a higher quality product that will have a better reputation, or, when the defect is found early the cost to fix it is substantially less than if found later in the project. To the customer, the value of a defect found by the producer is that the customer will not encounter the problems caused by the defect.

Of course, if processes, people and tools were perfect, perhaps we could achieve "zero defects." However, defects are a fact of life with the resources we work with at present. Another given is that defects can cause serious problems in a project if they are not managed properly. Defect correction consumes resources and can even cause new defects.

This is another, "dark" side to defects that supports why a better view of defects are needed. When defects are seen as bad, management has a tendency to punish the people who create them. Even Dr. Deming said that people create defects and get paid for doing so. However, Dr. Deming also saw the greater value in the prevention of defects. When people are punished for defects, they can do all kinds of creative things to shift the blame. This includes hiding them until a point in time when blame can no longer be established, or until the person responsible leaves the project or company. Some testers have even been known to "hoard" defect information until the end of a project so that a last-minute "discovery" can be seen as a heroic act. Developers have also been known to revel in their fire-fighting ability as opposed to building fault-tolerant software. These developers are often rewarded as the people who burn the midnight oil, while the people who build reliable software often go unnoticed in the background where things don't go wrong.

What Can We Learn From Defects?

There are many things we can learn from defects, but many people never get past the basic purpose of problem resolution. Here are some of the things that can turn these lumps of coal into diamonds:

1. Prevention of Future Problems

If adequate information is obtained and defects are analyzed from a preventative standpoint, this is where the greatest value of a defect is seen. This kind of analysis requires that the defect's point of origin can be identified and tracked.

Preventive analysis also requires that a correctable process is in place, otherwise there is no vehicle to convey how to prevent the problem next time. This type of defect analysis also requires that defects be categorized in ways that can be meaningful for future improvement. Examples of categories would include:

• Computational defects
• Data validation defects
• Usability defects
• Logical decision defects
• Requirements defects
• Design defects
• Presentation defects
  
  

2. Tracking of Relative Improvement

When an organization works on a series of projects, a natural question is knowing whether or not the team is improving in terms of the overall quality of the projects they complete. Information gained from defect analysis can help quantify relative improvement.

Defect metrics are often seen as a noble but elusive thing to track. This is due to the fact that defects must be relative to something to yield meaningful information. The myth is that the only relative measurement must be size. Although expressing defects per line of code or function point will give a defect density, there are other ways to relate defects that do not require a sizing measure.

For example, how about defects per defects? That's right - if you count the number of defects you find in your testing (producer view) and divide those by the number of defects found over the defined life of the product (producer and customer views), you arrive at a percentage called "defect detection percentage (DDP)."

As an example, if I find 85 defects in all phases of my testing and in the next 12 months (defined life of product) my customers find 15 additional defects, then the DDP is 85%.

With this information, I can look at past projects and compare DDP levels and should be able to see improvement or regression. I can also compare my metrics with those of other organizations to see how I am doing with others in my industry.

3. Project Status

When defects are tracked throughout a project, they can give developers, testers, management and customers a quantitative idea of when the product is ready to release. Unless, that is, management has already decided on a date that will be met regardless of defects.

As an example, perhaps there are five days left before the project deadline and the testers have found 200 defects to date, 10 of which are severe and will require several more days to correct. This kind of information should be of great value to management as it is a key piece of information to judge the readiness of the application.

In addition, the location of defects can reveal opportunities to focus resources. In some cases, a phased implementation may be a contingency if an area of an application is of non-critical nature and high defects exist in that area.

In my experience, when this type of information is ignored by management in favor of meeting a deadline, a high-risk situation exists. The result of proceeding with implementation in the face of high unresolved defects is a toll taken in rework, lost customers and high inefficiency.

4. Defect Prediction and Estimation

Being able to predict defect numbers and trends is something that usually takes a good historical basis of defects in your own organization. This is because nobody else creates defects the same as in your organization. It may take years to accumulate enough defect data to start predicting trends, which is why this use of defect information is usually seen in organizations that do a lot of measurement.

If you are not at this point, don't feel bad. For my money, this is icing on the cake. You can get 80% or more of the value of defects from the other three categories. Plus, people have been known to abuse this particular application of defect analysis by withholding defect discoveries until they fit the prediction model and make the tester(s) appear to be a hero - "The prediction model indicated we would find one more defect in the final week of testing and here it is!"

Summary

If we look at defects in the light of the previous four purposes, we can see how something as frustrating and annoying as defects can actually have a good purpose. I am certainly not proposing a lax attitude toward defects. They cost money, delay projects, and even place lives at risk in some cases. However, since each defect does cost money and time, let's learn the most we can about defects so we can prevent them or at least find them earlier in the software development process.

1. Capt. Kirk Phillips and his team at Randolph, AFB. They have since started using the term "Defects-R-Good." They say the change is because the developers said "Defects are Good" had too many characters. Those darn developers keep changing the requirements!