Using Test Cycles for Data-Driven Testing

There are a variety of testing techniques available that can be easily adapted and applied to testing data-driven projects. One of those techniques that greatly helps in planning, coordinating and tracking testing is the test cycle technique. In this technique, testing is organized and performed in cycles that can be defined to simulate specific dates. This article presents an overview of the test cycle concept, the benefits of using test cycles and an example of how test cycles can facilitate data-driven testing.

What is the Test Cycle Concept?

First, let's define a test cycle as any defined period of testing. A test cycle could simulate a day, a week, a month, or no time period at all. The ability to simulate a given period of time, however, is what makes test cycles an ideal technique for date-sensitive testing.

Exactly what happens during a test cycle depend on the technology involved. For example, in a traditional legacy mainframe environment a test cycle usually consists of three parts: online data entry, batch processing, and the verification of batch results (Figure 1). In an environment which does not contain batch processing, the test cycle consists of interactive processing only.

Figure 1 - A Traditional Test Cycle

For each test cycle, a simulated processing time period can be defined. That is why test cycles are an ideal way to plan and organize data-driven testing. One test cycle can be set for 12/15/1999,

another cycle defined as 12/31/1999, another at 1/1/2000 and so on. The test environment date for each test cycle will need to be set using a date simulation tool.

Figure 2 - The Test Cycle Matrix

The number of test cycles required for a test will depend on the amount of simulated time to be spanned during the test. For example, if you are simply testing the action that will occur on a certain date, you will only need a few cycles -probably the days surrounding the boundary or threshold date. However, if you are going to perform a more extensive test, such as a business process that lasts over a week, you will need to define test cycles that allow a longer span of testing.

For example, if you are testing a 30-day cancellation period across the year-end, you might have one cycle defined as 12/15/1999 and another at 1/15/2000. You would also want other test cycles defined at 2/28/2000 and 2/29/2000 to test leap year processing. With the test cycle approach and a date simulation tool and a data aging tool, you can define cycles as far in the future as you like. So, for testing leap year processing, you could also have cycles for 2/28/2004 and 2/29/2004 (Figure 2).

Within each test cycle, one or more tests are defined to be performed. In some test cycles, it may be desirable to define no tests depending on the cases being tested. The tests may be defined using test scripts and/or test cases.

The Process of Defining and Using Test Cycles

So far, we've discussed the concepts of using test cycles. Let's look at the nuts and bolts of planning a data-driven testing using test cycles.

Step 1 - Make sure you have the right tools

You will need a date simulation tool to easily change your test environment dates. You will also need a data aging tool to advance the dates in the test data and keep the relationships in sync.

Step 2 - Define the dates you will need to simulate

These simulated system dates will depend upon the extent of your testing -- namely, the levels of date accuracy you need to validate. There are four basic categories of date correctness to consider:

• No value for current date will cause any interruption in operation.No mater what the date is in the future, the system will work correctly.

• Date-based functionality must behave consistently for dates prior to, during and after year 2000. All functions using dates as a basis should be correct. This includes calculations in the 19th, 20th, and 21st centuries, and calculations that span those centuries.

• In all interfaces and data storage, the century in any date must be specified either explicitly or by unambiguous algorithms or inferencing rules. Either the century must be explicitly shown in the date (e.g., as a four-position field, or by using a century indicator) or by using a logic routine to interpret the date based on a window of time.

• Leap years must be identified and processed correctly. If your system processes data from early in the 20th century, you need to be able to distinguish the year 1900 from 2000 for leap year purposes.

Your specific system dates will depend on your applications, business and technology.

Step 3 - Build a Test Cycle Matrix

Spreadsheets are great tools for this. You need to leave a least the first two columns blank, but then define the test cycles along the top of the spreadsheet (Figure 2).

Step 4 - Define the test cases or business cases to be placed on the matrix

Test cases and business cases are those entities you intend to test. These cases will go through one or more cycles of testing and will execute several test scripts or test scenarios. This approach to testing is what give the test cycle concept so much power. You get to simulate not only the effect of the century rollover, but you also simulate how people and things are actually processed through your systems from beginning to end. This is in contrast to simply testing one program at a time in a standalone fashion.

Some examples of test/business cases would be a policyholder, a customer, a patient, a taxpayer, etc. Each of these entities would then have attributes that would make it unique. For example, if you are testing policyholders, you might have one policyholder with a deductible of $500 and another with a $1,000 deductible (Figure 3). The number of test/business cases you will test will depend upon how detailed you need the test to be and how much test coverage you need relative to the risk involved.

Figure 3 - The Test Cycle Matrix With Business Cases

Step 5 - Define the order of testing for each test/business case and place the tests in the appropriate cell on the spreadsheet.

Each cell can contain a reference to a test or tests that are to be performed for a particular test/business case in a particular test cycle (Figure 4). You might opt to skip a cycle or two for some cases and double up or have several tests in other cycles. Once again, this is an example of how test cycles help you simulate the real world. Just like your live production databases were not instantly created in your business, the test data entered into the system cycle by cycle will continuously build. Keep in mind, however, that every test/business case added to the test will be one more item to maintain throughout the test.

Figure 4 - The Test Cycle Matrix With Tests Defined

Step 6 - Define the tests in detail.

For every test indicated on the test cycle matrix, a detailed description of the test will be needed for documentation both before and after the test. The details should include controls (when will the test start and stop, etc.), input, expected output, and the procedure to be followed in performing the test. An ideal way to document these aspects of a test for interactive software is to use a test script (Figure 5). You must determine how much detail is reasonable, given the amount of time you have left for testing.

Figure 5 - Sample Test Script

Step 7 - Put it all together.

After using this method for many years, I have developed what seems to be a fairly smooth procedure for organizing a major test based on the test cycle concept. First, you'll need a manila folder for each test/business case you have defined. There will be a folder for each row on the matrix (spreadsheet). Label each folder with a business case ID number. This should also correspond to the ID on the matrix. Next, place everything you will need for the business case in the folder. This will include test data and test scripts or test procedures (Figure 6).

Figure 6 - Test Folder

A way to simplify things and to find the right test information quickly is to place a cover sheet (Figure 7) on the outside of the folder that shows the test cycles, the test scripts/procedures performed in each test cycle, and a signoff column to be initialed by the person who tested the business case.

The final piece is to get as many cardboard bankers' boxes as you have test cycles. If you have only a few folders per cycle, you can get by with using dividers in one or two boxes. You will need a way to start out the test with each set of folders separated by test cycle. Place the folders in the boxes by test cycle and in business case ID order. Each test cycle should contain a certain number of folders.

Step 8 - Execute the test.

You will start the test by setting the system date with the date simulator to the first test cycle date. If a bed of test data will be used from the start, you will need to make sure the dates in the test data are correct.

Figure 7 - Folder Cover Sheet

Starting with the folders in the cycle one box, perform the tests in each folder for cycle one only. During the test, you might create documentation you would like to save, such as screen prints or reports. These can simply be placed in the folder, unless the volume is large. In this regard, the test is self-documenting. When the test is complete, initial the folder and place it in the next cycle in which it will be used. If batch processing is part of the test cycle or test procedure, then the folder will go back into the same test cycle division from which it was retrieved. After batch processing is complete, then the folder can be pulled, evaluated, and moved on to the next cycle division in which it will be used (Figure 8).

This process continues until the folder is finished and placed in a "done" box. Eventually, all of the business case folders will be filed in the "done" box in business case ID order. A year or two from now, if anyone needs to know exactly what was tested, it is a simple matter to locate and retrieve the test documentation.

Figure 8 - Test Execution Using Test Cycles and Folders

Step 9 - Evaluate and Track the Test

As the test is performed, you will evaluate the results and determine if the test passed or failed in that particular cycle. There are two effective and easy ways to keep track of test progress manually. One way is to use the outside cover sheet of the folder to indicate pass/fail. The other is to highlight each cell in the matrix as the test is completed and passed. It is good to use both methods.

The Key Benefits of Using Test Cycles

While it is true that going to the trouble of designing test cycles and business cases is extra work, there are some very good benefits that you achieve with no other test methods. These benefits are especially important for data-driven testing.

Benefit #1 - The ability to simulate a business case from point A to point Z in your processing. Most other test methods focus on one process or software module at a time, but never have a way to effectively string them together for "end to end testing" of a system or systems.

Benefit #2 - The ability to plan and coordinate the march of time for a test. For data-driven testing, we know that time must be advanced, but the problem is how to keep not only the test data, but the test environment and test cases in sync. The test cycle concept allows you to do this with ease.

Benefit #3 - A safety net in case the test environment gets corrupted. A common situation that occurs in testing is when the test itself destroys data or updates data files with incorrect information. It is also not uncommon for other people to delete or to restore over test files. The common response to this situation is to simply restore from the last backup, but how do you know what was tested since the last backup? In most test processes you don't know exactly what was done, but with test cycles, you do know. In fact, the backup process is fairly straightforward. You take image backups of the test environment before and after online input. If batch processing is part of your test, the backup taken after online processing will also suffice for the batch backup (Figure 9).

Figure 9 - Test Cycles and Backups

Conclusion

In testing, the reliability of the test depends on the rigor of the test. Also, the rigor of the test depends on the relative risk, both business and technical. While some might look at the work involved in planning a test using test cycles as being excessive, others will testify that this kind of effort is required on some projects and systems to validate their operation through multiple simulated dates. The extent of test planning and execution always depends on the scope of coverage and risk. The question is, are you willing to bet your business or systems operation on anything less than the right test method for the job?