Managing Expectations in Testing Dirty Systems

One of the most important activities in a software project is to manage people's expectations. Even in projects that are well-managed, people are prone to bring varying and sometimes unrealistic expectations to the table.

The thing that often distinguishes well-managed projects from those that aren't is how the project manager is able to effectively deal with a variety of expectations and guide the project to a successful completion. There is a fine art to managing expectations and still meeting user needs. For a more detailed discussion of this topic in general, we recommend Naomi Karten's book, Managing Expectations, from Dorset House.

William E. Perry and I discussed the Expectation Gap in our previous book, Surviving the Top Ten Challenges of Software Testing, but feel that the concept bears repeating here.

The expectation gap is the difference between what the user expects and what the user gets. Ideally, the gap should be zero. Realistically, it is a challenge to keep the gap small.

The best way to keep the expectation gap small is to involve users and customers from the outset of the project. This can be done in a variety of ways, including reviews, prototyping and walkthroughs.

In testing dirty systems, expectation management becomes important because of the following characteristics of dirty systems:

• Testing is often seen and practiced as an end-project activity.
• Certain people tend to influence project decisions to a greater degree than others.
• People tend to override processes.
• Early project assumptions and requirements are seldom documented in writing.

We will deal with each of these dirty system characteristics in more detail, along with solution strategies to help you overcome some of the problems associated with them.

Testing is often seen and practiced as an end-project activity.

When testing is performed as an end-project activity, there is a lot of risk accumulated at a very critical point in time. Problems, which have not been seen or realized throughout the project, become evident. Sometimes, the resolution of these problems are very difficult due to the ripple effect seen when fixing one problem requires changes to many other areas of the system.

Big-bang testing is risky because options to fix problems are reduced at the end of the project. This is often seen in the case of performance problems where the options to a) tune the application software, b) tune the database and c) upgrade to more powerful hardware. The problems or reduced options appear in losses of time and money. The project is essentially held hostage to whichever solution must be delivered to make the project a success. People who sponsor projects do not expect to have months of work depend on the success or failure of one aspect of the project, which may become apparent at the last-minute and which may be a very expensive solution.

End-project testing is also the time when end-user see the application for the first time. If the application looks good to them, then life is good. However, if the users feel that the application is not what they expected, then the project is at risk of user non-acceptance. Some IT managers have the attitude, "If the users don't like the new system, then that's their problem." The reality is that if the users don't like the system, they will be less efficient in their work. Although there are cases where the users are on different political agendas, there are also valid reasons for non-acceptance. These situations must be considered objectively to avoid a last-minute project failure. These types of last-minute expectation problems can be avoided by involving the users and customers throughout the project.

Solution Strategies

• Perform walkthroughs and reviews of project deliverables, such as requirements documents and prototypes.
• Involve users in early testing and system testing.
• Perform performance and usability testing early in the development and maintenance cycles.

Certain people tend to influence project decisions to a greater degree than others.

A casual observation of most organizations will reveal the strong personalities. To hear these people describe their function in the organization, you would think the organization revolved around them only. These are the people who tend to influence project decisions to their benefit, with little regard to the big picture.

In testing dirty systems, these highly influential people may try to exert pressure on the test team leader to test in a way that will prove the person's point to management. Dirty systems are more susceptible to this kind of influence because of a lack of process and standards for test execution and reporting.

Solution Strategies

• Define objective criteria to manage test activities
• Define objective test reporting criteria
• Define objective test reporting standards

People tend to override processes.

In dirty systems, people tend to bypass processes (if processes exist) to achieve an objective more quickly, or to achieve an objective outside of the scope of the project or to meet an expectation. This is a major reason why dirty systems become that way!

These shortcuts almost always leave something undocumented, undone, or added to the project, which will adversely affect the overall quality of the project. Over a period of time these shortcuts accumulate into a mass of undocumented features and additional scope that leaves people wondering "How did this happen?"

In reality, processes are in place for a reason. The process adds consistency, control and repeatability to the work being done - all the things that are missing in dirty systems! To have a process defined but not in use is to virtually not have a process at all.

The reason people tend to bypass steps in the process due to human nature. Some processes get complicated to follow and it's easier to take the path of least resistance. A true quality assurance group is one that manages quality and does not perform the actual testing of a product. The value of QA is that it assures compliance to processes. If defects are to be prevented, the job of QA becomes to fix the process to prevent defects. This does not happen if there is no process or there is no one to monitor compliance to the process.

Solution Strategies

• Establish a QA function to assure compliance to processes.
• Emphasize the importance of process to management and staff.
• Get management to make the message of the importance of following processes.

Early project assumptions and requirements are seldom documented in writing.

If user requirements are not documented in writing, do they still exist? The answer is "yes, they exist in the expectations of everyone on the project." It's a sure thing that each person's expectations will differ in some way because they view the project from their own perspective. For a customer service person, the expectation is to be able to serve the customer more efficiently. For the technical support person, the expectation is a system easy to maintain. The fact is that these assumptions and expectations exist on every project. When the initial project objectives, assumptions and requirements are not defined in writing, there is no common starting point, so you already have an expectation gap from the start! Therefore, you are assured of an expectation gap at the end of the project as well.

Since the lack of documented requirements is a foundational symptom of dirty systems, testers in these environments must be aware that they will be battling the expectation gap throughout the project. These issues will become apparent especially as users and customers become involved in testing.

Process

From the process perspective, the main process to be performed is that of requirements management. Like CM, there is a growing body of knowledge in requirements management. These processes for requirements management can be expressed at a high level as:

1. Define the problem to be solved.

2. Involve the correct people

3. Interview stakeholders

4. Define the requirements as a high level

5. Review the high level requirements

6. Define requirements at a detailed level

7. Review the detailed requirements

8. Finalize and publish requirements

QC

The following questions can be the basis of a checklist for requirements management:

1.  Is there a requirements management process in place?

2.  Is there a standard for the type and amount of information to be documented for requirements?

3.  Are standards or guidelines in place to prioritize requirements?

4.  Have requirement categories been defined and published?

5.  Is there a specific person responsible for requirements administration?

6.  Have people been trained in how to gather, verify and manage requirements?

7.  Is there a contact person in the user and development groups responsible for developing and reviewing requirements?

8.  Is there a tool in place to document and manage requirements?

People

Project Manager- Oversees the requirements gathering effort at a management level. For example, the project manager would ensure that the requirements do not creep outside of the intended scope of the project.

Systems Analyst - Works with the users to elicit and define user requirements. The systems analyst is the person who takes a variety of needs, ideas and technical options and creates a workable technical and business solution. The systems analyst must have a blend of skills, which include working with users, understanding business needs, and understanding technical issues and solutions. Many times, the systems analyst it is the bridge between users and the development staff.

Users - Provides input to the analysts during the interview process. the users are an essential group to the requirements gathering process, as they provide the critical information needed to meet real world needs. It is important to understand that to each individual user will have a slightly different perspective on what is needed in solving a problem or delivering a new application. In gathering user input, it's necessary to talk to a variety of users representing a variety of levels, such as senior management, middle management, and " in the trenches" workers.

Customers - Provides input to the analysts during the interview process. The customers are the people who are sponsoring a project and may or may not be the actual users of the delivered solution. Therefore, the customers' needs and agendas may likely differ from those of users. It may take a variety of creative techniques to obtain input from users and customers, such as focus group sessions, facilitated group interviews, questionnaires, and surveys.

QA - A true QA group facilitates the requirements gathering process, although they may not be directly responsible for documenting the obtained requirements. One of the main responsibilities of the QA group is to ensure that the requirements gathering process is being followed correctly. An example of this would be for a QA analyst to be part of the requirements interview process and to follow a checklist of requirements attributes that should be present in the requirements that are to being gathered.

Test Team Leader - The test team leader can also play a leadership role in gathering requirements. The test team leader is very familiar with what is needed in a general nature in terms of requirements and can help keep the requirements gathering and documentation process on track. The test team leader is a good judge of scope and feasibility while requirements are being defined.

Testers - Since testers work with requirements as much as developers, they know better than anyone else the level of detail and scope that is needed for workable requirements. Therefore, testers are an ideal group of people to be involved in writing requirements. When testers are involved in the requirements gathering process, they see gaps and inconsistencies very early in the process. In addition, testers also get an idea very early of what the test objectives will be.

Tools

The primary tools for this topic are requirements management tools. Representative examples of these tools are listed in the tools section of this chapter, but include:

• Requisite Pro by Rational Software
• Caliber RM by Borland
• DOORS by Telelogic

Conclusion

Projects fail for a variety of reasons, but a leading cause is missed expectations on the part of users and customers. It is possible with the tools and techniques available to IT professionals to manage expectations effectively. RCS has a new course on requirements management, which is a key activity for managing expectations.