Testing Middleware and Firmware

Every software application in use today relies on some level of middleware and firmware to deliver the functionality the user eventually sees. This places middleware and firmware in a very important, yet often overlooked, layer of architecture. When approaching the test of middleware and firmware, people sometimes struggle to find effective strategies and techniques. The purpose of this article is to outline a testing strategy and to point you to further resources.

Definitions

Middleware

Middleware is the layer that resides between the hardware layer and the application layer to provide services such as database management. From the CNET glossary, middleware is defined as:

"This software manages the communication between a client program and a database. For example, a Web server connected to a database can be considered middleware--the Web server sits between the client program (a Web browser) and a database. The middleware allows the database to be changed without necessarily affecting the client, and vice versa."

The Software Engineering Institute (www.sei.cmu.edu) defines middleware as:

"Middleware is connectivity software that consists of a set of enabling services that allow multiple processes running on one or more machines to interact across a network. Middleware is essential to migrating mainframe applications to client/server applications and to providing for communication across heterogeneous platforms."

Middleware can handle transaction processing (TP) for distributed applications, Remote Procedure Calls (RPCs) for extending application logic across a network, Object Request Brokers (ORBs) which allow for the distribution of application objects across heterogeneous networks, and Message-Oriented Middleware, which allows the passing of messages between objects and applications across a network.

Firmware

Firmware can take on many forms, but is basically the embedded software that is stored in hardware memory to control hardware functions. Firmware controls PCs, printers, and every other electronic device (remember Y2K?).

The Issues

As you can see, middleware carries quite a responsibility in delivering quality information. In some recent projects, I have had the opportunity to delve into the testing issues surrounding middleware. There are significant testing differences required for middleware and firmware, however, most testing literature assumes the ability to test through a user interface. While this article might not have a broad audience, I often tell people that the information they think is not applicable today may become very important tomorrow.

Some of the issues surrounding both middleware and firmware are:

• Complex processing due to extremely high numbers of interface combinations
• High criticality in the operations performed
• The difficulty in testing from an external perspective
• The importance of developer testing at the unit level
• The uniqueness of the types of test tools needed
• The difficulty in building a test environment.
• An emphasis on structural testing
• An object-oriented focus

All of the above concerns shape an overall test strategy that is:

• Performed early
• Performed by developers first
• Leveraged by structural test tools
• Realized by using test harnesses and stubs
• More object-oriented than procedural
• Involves testing large numbers of interfaces

We will deal with each of these points in the test strategy for middleware and firmware.

A Test Strategy for Middleware and Firmware

We have defined middleware and firmware and understand that they are different, yet have many characteristics in common when it comes to testing. The discussion of test strategy for these types of software will include both middleware and firmware, and can be extended to test any software which is not accessed by a user interface.

Early Testing

Early testing will multiply the testing effectiveness of any software application, regardless of technology. However, in the world of middleware and firmware early testing is most critical because finding defects at later stages carries a higher penalty of rework. This is due to the extent of integration with hardware and other software.

The problem with early testing in this environment is that with so many integration dependencies, how does someone create test harnesses and stubs that allow for an accurate test? Manually, the job is possible, but can be overwhelming when there are many interfaces involved. If you are developing in a language that has tool support for structural test case design and testing, you may find that the job can be very easy. Specifically, for C++ and Java, Parasoft (www.parasoft.com) has a great toolset to design and perform structural tests, with a feature to automatically create a test harness and test stubs. Similar tools are available from International Software Automation (ISA) www.softwareautomation.com.

Developer Testing

Developer testing is essential to avoid high rework costs. To the testers, the software is a black box. Only the developers have the view and access to the code to test all conditions. In addition, not only are functional cases at stake, but also the structural tests for memory boundary violations and memory leaks.

My experience is that developers can test software if the have a good process to follow, standards to show what is expected of them in terms of testing, and a way to hold developers accountable for the quality of their work. Management must also be making the message loud and clear that testing is part of the job and that quality is a shared responsibility between developers, testers, QA, and management.

An Object-oriented View of Testing

In the object-oriented view of testing, tests are isolated at a smaller scope, yet can have high complexity due to the interfaces with other objects. The object-oriented view of testing must be able to deal with classes, methods, and attributes and to validate those at a high level of coverage.

In Shel Siegel's book, "Object-Oriented Software Testing," he describes the Hierarchical approach to O-O testing.

"The hierarchical approach is at the heart of the object-oriented testing system. This test approach uses and builds upon several well-understood testing techniques, tying them together into a comprehensive testing system. The hierarchical approach leverages the fact that "everything is a system." It defines and applies testing standards for several levels of software component: objects, classes, foundation components, and systems. The hierarchical approach designates as SAFE those components that meet the testing standards for that kind of component.

Once you designate a component as SAFE, you can integrate it with other SAFE components to produce the next-level component. In turn, you test this component to the level of safety associated with the component level it represents. SAFE is always a relative state. It depends entirely on the standards you choose to enforce, your application, your attitude toward risk, and the specific risks and risk management practices you adopt in your project. The hierarchical approach provides guidelines for minimum safety; you decide what is right for you."

Figure 2 - The Hierarchical Test Approach

Much more could be said about testing middleware and firmware. This article is intended to provide a strategy for testing and a starting point to start developing detailed techniques. My best advice to you is to read the books on Object-oriented testing by Shel Siegel and Robert Binder, investigate the available test tools (new ones are emerging all the time) and work at the developer level to reinforce the criticality of testing.