UI Design Newsletter – April, 2002

Bailey, G. (1993), Iterative methodology and designer training in human-computer interface design, INTERCHI '93, 198-205.

Tan, W., Liu, D., Bishu, R. R., Muralidhar, A. and Meyer, J. (2001), Design improvements through user testing, Proceedings of the Human Factors and Ergonomics Society 45th Annual Meeting, 1181-1185.

Almost ten years ago, Skip Bailey (Bailey, 1993) published the results of his doctorial dissertation, where he had tracked user performance through a series of 3 or 4 iterations. Each of his eight test participants created a software system, and then made improvements to the system over the next several months. Each person made changes to their system after watching (on videotape) three participants individually try to use their system to perform a series of tasks. Over the course of his study, he reported many usability improvements, including an overall 26% improvement in performance time.

This was one of the first reported studies to show convincing evidence that "iterative design" methodologies were valid. If we have usability improvements with each iteration, is this evidence that "usability testing" works? I believe the answer is "Yes." We have many studies showing that each iteration does help to improve the usability of a system—even if the improvements are only modest ones (which is usually the case).

Wei-Siong Tan, Dahai Liu and R. Bishu from the University of Nebraska in Lincoln and A. Muralidhar and J. Meyer from Ajenda Interactive Media in Chicago reported a study that provides another estimate of the percentage improvement in speed with each iteration.

They had 15 representative users complete 5 scenarios while using a preliminary version of a commercial Web site. The scenarios were created to represent tasks that were typical of those encountered in real life, and to reflect the major functionality of the site. The performance test identified a total of 49 usability problems. They attempted to eliminate all the problems "by applying human factors knowledge." After doing their best to eliminate the problems, they then conducted a retest by having 20 representative users complete the same 5 scenarios.

The changes to the Web site resulted in 28% faster average completion times for the 5 scenarios. However, about 45% of the problems "found and fixed" as a result of the first test, showed up in the retest. In addition, they identified 9 new usability problems that were created when fixing previously identified problems. Overall, they reported only a 37% reduction in the number of usability problems.

Using their results, I figured that if they had the same level of improvement with each iteration, it would take at least six tests and 105 participants to detect and correct 95% of their usability problems.

The iteration-related questions that have not yet been answered are:

1. How many iterations does it take to create a Web site that meets the usability goals of the organization? and

2. Which of our many usability testing methods, including heuristic evaluations, expert reviews, cognitive walkthroughs, think-aloud evaluations, performance tests, are most powerful. That is, which methods will lead to the most usability improvements in the shortest period of time?

As Bob points out there have been some public studies of usability engineering. There have also been many private studies. I almost always see at least a 25% improvement in speed. This is USEFUL, as the 25% improvement seems to be a conservative objective in business cases.

Bob is concerned that we do not know how many iterations it should take to reach an optimal design. Of course it takes an infinite number to reach "truly usable." But there are obviously diminishing returns. As a rule of thumb, keep testing and iterating until you either reach your objectives, or the improvements from your test cycles get small.

It is becoming obvious from the literature that there are MANY dimensions of usability. Many facets of the design to test. There are too many issues to hope for perfection. But this is not an excuse to just give up and slop something together. Instead, do due diligence in your design process. Get trained. Follow a systematic, user-centered process. Include cycles of testing, early and often. Apply the established literature and principles in the field. You may not be "truly usable," but you are likely to beat the competition.