How Much is Software Quality Worth?

As a consultant and trainer I am often asked for the hard numbers of software quality, such as "How much does an average defect cost to fix?" or "What is the average cost of downtime for a major e-commerce web site?"

Something that often gets lost in the discussion is the perspective of these types of metrics. The figures will vary based on factors such as technology, market, people, process, tools, etc. and it is sometimes hard to make sense of them in the real world.

What's more troubling and confusing are the senior management teams that disregard any kind of supporting evidence of the costs of quality and throw all risks to the wind. These people don't care how much it will cost to fix problems. They don't care about the efficiency of an operation or the morale of the employees. Heck, they don't even care about the poor customer who tries to do something using the product and can't because of a defect. (Internally at the company these are not called "defects", but rather "issues," "incidents," "trouble reports," etc. Defects have too much of a negative connotation and we must be politically correct, after all.) When management takes the risk to be first instead of right, the stakes can be huge. Companies have been lost by taking these kinds of risks.

We can talk averages and percentages all day, but let's bring the discussion to a case in point. A recent story caught my eye about the case concerning the State of Mississippi vs. American Management Systems, Inc. In this case, the State of Mississippi alleged that AMS failed to deliver an automated tax system on schedule. According to Washington Technology magazine,

"After a Mississippi jury Aug. 23 ordered AMS to pay $474.5 million in damages for breaching its contract with the state's Tax Commission, the Fairfax, Va.-based company had 10 days to post a bond that analysts estimated would have approached $660 million.

On Aug. 28, AMS and its attorneys settled the case, agreeing to pay the state $185 million over 13 years. AMS will pay about $23.5 million, and the company's two insurance carriers will cover the rest, a company official said.

AMS Chairman and Chief Executive Officer Paul Brands told investors shortly after the verdict that the bond comprised 125 percent of the amount of the award, a 15 percent penalty and 8 percent interest through an appeals process he expected to take two years."

The contract was initially signed on December 13, 1993 and started out as a three-year $11.2 million contract.

"As for the actual damages, the state claimed $18 million for staff support to the project, $32 million to hire a new contractor and $184 million for lost savings because the state continued to use less efficient methods of compiling and reporting tax payments, collections and refunds."

"The state claimed that, despite three attempts in 64 months, AMS had not even launched one of the 36 tax applications promised. AMS, however, said it was on schedule according to the terms of the most recent contract amendment, and was scheduled to go live within 10 days of the termination."

 So, in five years AMS failed to launch an application, yet was still "on schedule." Hmmm, sounds like some contract re-definition to me. What is important to see is:

"If AMS had appealed the verdict and lost, this would not necessarily have bankrupted the company, but it "would have been very expensive and would have hung over the company, not for just a quarter, but for many years," said Bill Loomis, managing director of the technology research group Legg Mason Wood Walker Inc., Baltimore.

As evidence of investors' concerns over the jury award that Brands called "of unprecedented magnitude," AMS' stock plummeted 31 percent to $15 from $21.81 the day the verdict was announced. The stock rose $4.50 to $21.31 the day the settlement was announced. AMS issued a statement saying it will take an after-tax charge of about $23.5 million, or 56 cents per diluted share, in its fiscal third quarter to cover its costs relative to the settlement."

Of course, that's what insurance is for, right?

"The balance of the $185 million fine will be paid by AMS' insurers, said Mark Andrews, executive vice president and general manager of the company's state and local government group. "We aren't going to have any more cash outlays beyond what is in the [statement]," he said."

 However, there is a bright side for the company,

"Business analysts expect no permanent damage to the company's strong reputation in the state and local marketplace and anticipate no adverse affects on its future business prospects arising from the Mississippi incident.

"It may be a short-term deterrent to other states doing business with them, but I think they'll get past that," Meagher said. "In the long term, they'll be fine. They've been a player in this industry for a long time."

The state of Mississippi got an apology from AMS, which said that on the project, AMS "fell short of the high standards of performance that the company has set for itself." However, AMS also told its shareholders that the settlement "expressly provides that AMS was not at fault and did not breach its contract with the state."

So, AMS is $23.5 million poorer, the insurance company is out $185 million, and the state of Mississippi gets $30 million to try the whole thing over again. One of the most amazing parts of this story is found in the last paragraph of the article,

"In a statement announcing the settlement, the Mississippi tax commission and department of IT services said that "contractors with integrity have nothing to fear" from the verdict and settlement. Further, they said that Mississippi is willing to work with any vendor, including AMS."

Sounds to me like the main people to profit from this case are the attorneys!

I would be willing to bet that this project had all of the classic marks of project risks, including poor contractor management, increasing scope, and technology changes. In fact, from 1993 to 1999, technology went from client/server to web-based!

My point in all of this is that software defects and poor project management carry a big price tag. However, in this case, the guilty go free and live to go on to other projects. In other cases, the guilty go into bankruptcy, which was a distinct possibility in this case as well.

The next time your company thinks about rolling the dice with project decisions, remember that some risks really do materialize!

This story validates my answer to the question people often ask me, "Why does software have so many bugs?" My answer is simple, "Because people keep buying it that way."

Quotes taken from Washington Technology Magazine, September 11, 2000