Testing: "What, Me Worry?" (continued)

The "Disease of Knowledge" Justice Douglas, it turned out, suffered defects in reasoning and emotions from his stroke, although his speech appeared normal. Luckily, other people could easily see that he denied his paralysis, thus bringing questions about his other decisions. Imagine if there were no paralysis for him to deny. He might have appeared entirely normal, yet still remain afflicted with agnosognosia – the "disease of knowledge." Diagnosis would be doubly difficult. (Such cases occur.) Alfred E. Newman must be one of those patients, looking normal, but blissfully lacking insight into any risk he incurs. It won't be easy to tell Al the truth. This sounds ominously like cryptodesign at its deadliest. We call it "cryptotestiness" – a grouchy refusal to spend money to test usability early in the life cycle – a false economy.

Figure 1. Here's two instances in which users became test subjects after product release. Clearly, "What, Me Worry?" attitudes governed the development process. Software for printing out notebook tabs suffered competitively. We found out why with a simple test. Remember, it costs 60-100 times as much to make changes after product release compared to changes during project definition. Early testing pays off.

CRYPTOTESTING SNEAK ATTACK Even worse than cryptotestiness, we have a sneak attack from the "junk food" of testing – user preferences. This empty calories approach to testing consists of asking users what they think about a screen. Managers think they get the benefits of testing. But they fail to realize that users simply don't have knowledge about important detailed design issues such as we covered previously in this column (see previous articles).

User Preferences Ergonomics researchers have studied the dichotomy of user preference versus user performance, often finding that users don't know what provides the best performance. For example, Dr. Robert Bailey, our Chief Scientist at Human Factors International, published results indicating users failed to discriminate between three levels of consistency in interface design. They had no preference. However, they performed better on the most consistent interface. Therefore, had the designer adopted user preference to guide the design (junk testing), the outcome could easily have been an inferior interface.

Even worse, clearly stated user preference can contradict performance results. Researchers Mack and Lang found that users preferred using a stylus and mouse for precision pointing. However, these methods created far more errors than performance with a keyboard-command input method. Tognazinni conducted a study in which subjects thought that using cursor keys was faster than a mouse to replace text in a word processing task. However, subjects using a mouse performed twice as fast as the cursor key condition! In both cases, designs adopting user preference would have resulted in a far less efficient product (see Figure 2).

Figure 2. User preference often fails to create detailed designs with the best performance. Design requires specialized knowledge. However, users can give you feedback on how well your design supported their workflow and functional requirements, areas in which they have a lot of experience.

Soul Testing In cryptotesting, designers mistakenly accepted user preference as the only feedback on the success of the total design. The antidote requires that you abstain from such automatic responses. Instead, use soul. It turns out that user performance gives a better answer for many design issues. However, sometimes user preference also provides good feedback when done correctly. This refinement in response constitutes soul testing. Let's sort out the issues and show the special cases where user preference becomes useful.