Achieving Software Quality Using Defect Filters

The concept of defect filters is a powerful one that many others have written about in the past. Michael Fagan is credited by many as a pioneer in the area of software inspections as an effective method to perform quality control early in a software project. Others, such as Vern Crandall and Alka Jarvis have also written on the topic of filters in their book, Inroads to Software Quality.

My experience is that many of the defects that people normally find in testing or even worse, in production, could be more easily and cheaply found early in a project by using filters.

I discussed the value of defects and the importance of learning from the information we can gain from them in my article (Defects are Good). Filters give us a place to gather this information while reducing the cost of finding them. With an effective series of filters, you can virtually eliminate some forms of testing from the end of a project.

What is a Defect Filter?

Vern Crandall has an excellent illustration for software testing. Let's say that we want to brew a pot of coffee. When the pot is finished brewing, we notice that there are grounds in the coffee. Yuck! So, the task becomes to get rid of the grounds.

We could get a pair of tweezers and reach into the dark coffee and squeeze the tweezers, hoping that we would find a ground. Each ground we find would represent a successful effort. This effort might work, but would not be considered effective, in fact, it would look silly to most of us.

Another approach would be to run the coffee through a filter to remove the grounds. It's much faster and not as painful on the fingers!

In testing, each ground represents a defect. The tweezers approach is how many people approach testing. They're not sure where the defects are, so testers perform a series of tests. Some tests will pass, others will fail. The ones that fail are like finding a ground with the tweezers - a successful test!

The filter represents a method to screen defects. In a software project, a filter could be an inspection, review or walkthrough. If one filter doesn't catch all the grounds (defects), then keep adding filters until the grounds no longer appear. Ideally, the filtration process would start early in the process to achieve the lowest cost of defect removal.

The High Cost of Late Defect Discovery and Repair

People have been measuring software defect information for over 30 years. One of the most significant things we have learned is that the longer a defect stays in a product, the more costly and risky it is to remove.

Although numbers vary by project and environment, the costs to fix defects average to what has come to be known as the "1:10:100" rule. This rule says that a defect that costs $1 to fix in requirements or design costs $10 to fix in a traditional test phase and $100 to fix after the product goes into production (live) use.

There are various reasons for defect removal costs to increase as the project progresses. These include:

- The ripple effect of changes throughout the system or application

- Repeating past tasks in the project (re-design, re-coding, new or changed documentation, etc.)

- Notifying people on the project and users of changes

Regardless of the reasons, late defect removal is costly and time consuming. It is also a known fact that changes at the end of a project add risk. If all of these reasons aren't enough, consider the fact that many times the options to fix a defect may be greatly reduced toward the end of a project.

All of these reasons should support the need to find defects as early as possible in a project. In fact, the emphasis should be on defect prevention, but that requires defect measurement and analysis, along with defined processes that can be refined. So, for the purpose of this article, we will focus on early defect detection as maturity level 2 and leave the higher levels of test maturity for other future articles on this topic.

How Do Filters Work?

You might think of a filter as a gateway or check before going on to successive stages of a project. in the figure below, an example is shown of a filter between the process of defining requirements and performing application design. Such a filter would ensure that the requirements to be used by application designers met pre-defined levels of quality.

The filters could consist of checklists, reviews, inspections, or any other method that would check for the presence of defects.

How Are Filters Implemented?

While people recognize the need and benefits of filters, it's hard sometimes to get others in the organization to see the same need. Reasons vary, but many times the people creating the defects aren't impacted enough to be motivated to change.

The implementation of filters is a process decision and should be driven by the senior management of the organization. This means that management must take an active role in designing and introducing filters to the organization. Unless this happens, people will not see the importance of adhering to a process that uses filters.

My experience is that filters are best introduced gradually into an organization. The implementation effort can be determined by scope and intensity.

Scope of Implementation

One approach to implementing filters is to pilot the process on a small project where the results can be measured. In addition, the lessons learned on a small, lesser risk project, are less risky than on a mission-critical project.

Intensity of Implementation

Since walkthroughs are less formal than reviews or inspections, they are a natural entry point for implementing filters. This is especially true for first-time implementations or when strong resistance is involved. A word of caution is to be aware that since walkthroughs are less intense than inspections, the defect discovery rate will be lower

Training is Essential

People must be trained in the filtering process and interpersonal facilitation skills that are critical to the success of inspection-based techniques. If you can't or don't want to design and conduct your own training, Rice Consulting Services has a course on Implementing Defect Filters.

Many filter implementation efforts have failed due to damaged relationships. This usually occurs because the reviewers have not been trained in how to give interactive criticism. In addition, it is easy to drift into critiquing the producer instead of the product.

Common Inhibitors

In implementing defect filters, you may likely hear one or more of the following objections. Don't be argumentative, just be prepared to answer them with facts.

"We don't have time to perform filtering tasks"

Actually, experience from companies show that although they may appear to take more time, filters actually reduce overall project time by as much as 15%! Plus, the lower defect rates will reduce costs and increase the quality of the release.1

You can always bring up the point that there is always time to perform rework!

"People can't get along well enough"

Filtering activities such as walkthroughs and inspections are certainly affected by cultural issues. There are some options:

- Initially focus on non-interpersonal filters, such as checklists.

- Initially involve people with interpersonal skills in review sessions.

- Conduct training to increase interpersonal skills.

Make filters a part of new and challenging projects. If someone wants to be on the "A" team, they will need to follow the processes management have in place.

"We have no processes."

Okay, that's a big problem. The solution is to define simple and repeatable processes that can accommodate the filtering concept. The key words here are simple and repeatable. Once again, management needs to reinforce the importance of processes or else people will ignore them.

Conclusion

Defect filters are simple and effective ways to dramatically improve the quality of software. Not only is overall quality improved, but project time is reduced. Filters can be implemented gradually and there are several good resources you can use. The keys to success are to have a process in place and to stay aware of interpersonal issues.