Why I'm More Positive About Exploratory Testing

In the past I have not given a lot of attention to exploratory testing in my writing, speaking and teaching. I have favored the more defined testing techniques. However, as I continue to train and test, I see things that make me more amenable toward exploratory testing as a valuable testing technique.

Testing Dirty Systems

As I have been researching and writing on the topic of testing dirty systems (to be published in the future by Dorset House), I see more and more that initial testing of those systems is an exploratory process. In many cases of dirty systems, requirements are not known, expertise may be missing and the quality of the application unknown.

I relate the process of software discovery to that of being a "software archeologist." This implies a method and rigor, but also an exploration of the unknown to find objects of interest and value. To perform this kind of testing, exploratory testing is a great match.

The Lost Art of Validation

We have been very successful about drilling into people that test design starts with good requirements. After all, if you can't describe something, how can you test it or build it, right?

Unfortunately, we have made the message too well. When requirements or other documentation is lacking, people are at a loss many times as to how to test something.

Validation is testing something to see if it meets the users' needs, regardless of what has been specified. This is a valuable form of testing because no requirement is perfect. However, ask a group of 20 testers what validation is and I'll bet that less than five get it right.

To regain this lost art, testers need to be able to test from a user's perspective at times. This requires critical thinking and creativity. Exploratory testing is a good way to bring out both of these attributes in a person.

Stay on the Path!

Before I write what follows, let me state for the record that I still like defined tests for the cases where they make sense. For example, when you need high repeatability, exact procedures and when you want to leverage your testing by having one person design the tests and many people perform them.

The problem is that I have seen far too many testers become test script or test case executers instead of good inspectors. In some organizations, testers are told specifically just to stick to the script. In my opinion, this view of testing is too restrictive and inflexible to find the more subtle defects. People need the freedom to follow their intuition.

Bruce Lee and Testing – Agility and Speed vs. Traditional Rigid Forms

I know you're thinking, "What does Bruce Lee have to do with testing?" I was thinking about agile software development and testing and the recent history of the martial arts came to mind. Most people in the martial arts and those who are just spectators agree that Bruce Lee was a phenomenal martial artist. However, what got Lee in trouble was that he challenged the traditional ways of the masters. In most martial arts schools today, the traditional ways are still the primary ways taught and practiced. The more powerful and flexible ways were the techniques refined and taught by Lee. His untimely death is what has kept his Jeet Kune Do methods from becoming more prominent although he was able to write books and teach a core of students who have continued teaching his methods. Personally, if I were ever in a situation where I had to defend myself, I would opt for the speed and power over traditional forms.

In testing, we can continue to do testing in strictly traditional ways, or we can learn to be more flexible and flow in the moment. I think there is a need for both approaches in testing. In fact, if we don't do exploratory testing, we are probably missing defects. When teaching validation methods, I always make the distinction between the "paper world" and the "real world." I think most of us have experienced the gap between those worlds.

Exploratory Testing is Not Necessarily Random

If you talk with people who have refined their experience with exploratory testing, you will likely find they do not consider their tests to be random or ad-hoc. Many exploratory testers keep records of what they tested for repeatability.

In the past, a major concern of mine was that if you don't have planned and repeatable tests, your investment is lost due to lack of repeatability. However, I see the value of exploratory testing in situations where you don't understand the software, don't trust the documentation, or feel that planned tests may be too weak. I've seen situations where people have performed thousands of planned test cases, then send the application to a user who wanders off the normal trail a little and causes the application to fail. These situations make me question how much weight we should attach to our extensive test suites.

The Pesticide Paradox

In his classic book, Software Testing Techniques, Boris Beizer describes the effect we see in pesticides, antibiotics, and yes, software.

Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.1

Essentially, what has worked well in the past will eventually prove to be ineffective as "new" defects (many which may have been undiscovered because they have been masked by other defects) are discovered. Exploratory testing is a technique that can be used to find defects that past ways of testing, or past tests may reveal.

Success Relies on the Tester

One of my basic testing principles is to let the testing process drive the effort. A process gives repeatability, a way to teach how to test, and something to improve. You may have several testers at varying levels of experience and skills, but a common process helps people test at about the same level. With a good process, you shouldn't have to be concerned with individual skills or other factors to the extent that tests will be performed inconsistently or incorrectly. This could be seen as an engineering approach, or in some cases, a clerical approach. That's fine until you have no process, or even worse, you have a process but nobody understands or uses it (a broken process).

Exploratory testing places a great deal of weight on the skills, experience and intuition of the individual tester. This could be seen as a craftsman approach. Of course, we always would like to have a team full of such people, but that may not be reality, either.

I still take a balanced view of this aspect of exploratory testing. I would like to see at least a simple process performed by trained and motivated people, using the right tools. I think it's fine for people to use their intuition and knowledge to question the tests and the process itself. However, if the process is working as it should be, there are reasons for doing things as the process indicates and any deviations or changes to the process should be well understood and carefully considered.

And Yet, I Still Have Some Issues With Exploratory Testing

While thinking about exploratory testing, something was bothering me and I wasn't really sure of what it was until I heard a session at EuroStar 2003 in Amsterdam. The speaker was comparing exploratory testing to gold mining, which has both engineering and exploratory approaches. The speaker had experience in gold mining from an industrial engineering approach. He also told about the methods used by indigenous people to mine gold by digging with shovels and prospecting, based on what they had found in the area previously. One approach uses highly specialized tools, such as maps and specialized mining tools, while the other approach uses intuition and basic tools, such as shovels.

The question occurred to me, "How would the exploratory miners know the quality or authenticity of the minerals they find?" As the saying goes, "all the glitters is not gold." There's not only "gold in them thar hills", but there's fool's gold there as well. Of course, even a beginning rock hound can tell the difference between fool's gold and real gold, but there are varying degrees of real gold that are certified by an assayer.

So, there it was. It seems to me that one of the issues that I am still working through is how a tester knows when something is or isn't a defect. With pre-defined specifications and requirements, the criteria are clear, and we hope it is correct. Of course, software developers and testing do not always get requirements in any form or condition. This leads us to the need for validation. Validation requires that we look to the users or domain experts to tell us what is right or not, regardless of what the specifications and requirements indicate. In exploratory testing, the testers have to be good enough to know or be keen enough to ask the right questions of the right people to draw the right conclusions.

Conclusion

I give a lot of credit to James Bach and his work on this topic in helping me see the value of exploratory testing as a valid and needed method. James and I spent some time at that same EuroStar conference discussing our views on testing and I left with a newfound respect for exploratory testing (I've always respected James), even though I'm known as a "scripting guy" or a "process guy."

Exploratory testing is not weak, random or ineffective. However, the power of exploratory testing rests with the individual tester. This means that a tester performing an exploratory test needs to have a keen sense of intuition and the ability to assimilate, analyze and make reasoned judgments. Shouldn't we all strive for these abilities?

Behavior theorist Abraham Maslov has been quoted as saying "When your only tool is a hammer, every problem looks like a nail." My challenge to testers is to add a lot of tools, skills and techniques to their toolkit so their response to problems can be agile and powerful. There are times will you will need to use scripted and pre-designed tests. There are times when you will want to explore an application to learn how to break it. Whether you prefer pre-planned tests or not, I think we can all benefit from more exploration in testing.

1. Beizer, Boris. Software Testing Techniques, 2nd. Edition