Introduction

How many of the usability problems identified in a heuristic evaluation are true usability problems?

Several years ago, I published an article suggesting that many of the "problems" identified by heuristic evaluators were not problems at all (Bailey, Allan and Raiello, 1992). Even so, many of us have continued to waste time and go to the expense of fixing many usability problems that were not problems. Recently, three research papers were published that provided some insights into the validity of heuristic evaluations (Catani and Biers, 1998; Rooden, et.al., 1999; Stanton and Stevenage, 1998). The articles discussed usability testing in three totally different domains with very similar results.

In all three of these papers it was possible to determine (a) what heuristic evaluators thought the usability problems would be, and (b) compare their responses with the problems that users actually had.

The results showed that 36% (about 1/3) of identified problems were actually usability problems (hits). Some of these problems were serious and some were trivial. The heuristic evaluators missed identifying about 21% of the problems that users had. Finally, about 43% of the "problems" that were identified by the heuristic evaluators did not turn out to be problems at all (these are usually referred to as "false alarms").

In other words, of those identified as usability problems 46% were hits and 54% were false alarms (and about 20% were missed altogether). If we round off the numbers, we could conclude that when a heuristic evaluation is conducted about half of the problems identified will be true problems and about half will not be problems. More specifically, for every true usability problem identified, there will be a little over one false alarm (1.2), and there will be about one-half of one missed (.6). If this analysis true, heuristic evaluators tend to identify more false alarms and miss more problems than they have true hits.

I believe that the best way to initially find true usability problems in a website is to use research-based heuristics. This obviously requires that the evaluators understand the usability research. (For a more complete discussion on this topic see my May, 1999 newsletter.) All heuristic evaluations should be followed-up with well-designed and professionally executed performance tests.