|
Bailey, G. (1993), Iterative methodology and designer training in human-computer
interface design, INTERCHI '93, 198-205.
Tan, W., Liu, D., Bishu, R. R., Muralidhar, A. and Meyer, J. (2001),
Design improvements through user testing, Proceedings
of the Human Factors and Ergonomics Society 45th Annual Meeting,
1181-1185.
|
Almost ten years ago, Skip Bailey (Bailey, 1993) published the results
of his doctorial dissertation, where he had tracked user performance through
a series of 3 or 4 iterations. Each of his eight test participants created
a software system, and then made improvements to the system over the next
several months. Each person made changes to their system after watching
(on videotape) three participants individually try to use their system
to perform a series of tasks. Over the course of his study, he reported
many usability improvements, including an overall 26% improvement in performance
time.
This was one of the first reported studies to show convincing evidence
that "iterative design" methodologies were valid. If we have
usability improvements with each iteration, is this evidence that "usability
testing" works? I believe the answer is "Yes." We have
many studies showing that each iteration does help to improve the usability
of a system—even if the improvements are only modest ones (which
is usually the case).
Wei-Siong Tan, Dahai Liu and R. Bishu from the University of Nebraska
in Lincoln and A. Muralidhar and J. Meyer from Ajenda Interactive Media
in Chicago reported a study that provides another estimate of the percentage
improvement in speed with each iteration.
They had 15 representative users complete 5 scenarios while using a preliminary
version of a commercial Web site. The scenarios were created to represent
tasks that were typical of those encountered in real life, and to reflect
the major functionality of the site. The performance test identified a
total of 49 usability problems. They attempted to eliminate all the problems
"by applying human factors knowledge." After doing their best
to eliminate the problems, they then conducted a retest by having 20 representative
users complete the same 5 scenarios.
The changes to the Web site resulted in 28% faster average completion
times for the 5 scenarios. However, about 45% of the problems "found
and fixed" as a result of the first test, showed up in the retest.
In addition, they identified 9 new usability problems that were created
when fixing previously identified problems. Overall, they reported only
a 37% reduction in the number of usability problems.
Using their results, I figured that if they had the same level of improvement
with each iteration, it would take at least six tests and 105 participants
to detect and correct 95% of their usability problems.
The iteration-related questions that have not yet been answered are:
1. How many iterations does it take to create a Web site that meets the
usability goals of the organization? and
2. Which of our many usability testing methods, including heuristic evaluations,
expert reviews, cognitive walkthroughs, think-aloud evaluations, performance
tests, are most powerful. That is, which methods will lead to the most
usability improvements in the shortest period of time?
|
|
|
As Bob points out there have been some public studies of usability engineering.
There have also been many private studies. I almost always see at least
a 25% improvement in speed. This is USEFUL, as the 25% improvement seems
to be a conservative objective in business cases.
Bob is concerned that we do not know how many iterations it should take
to reach an optimal design. Of course it takes an infinite number to reach
"truly usable." But there are obviously diminishing returns.
As a rule of thumb, keep testing and iterating until you either reach
your objectives, or the improvements from your test cycles get small.
It is becoming obvious from the literature that there are MANY dimensions
of usability. Many facets of the design to test. There are too many issues
to hope for perfection. But this is not an excuse to just give up and
slop something together. Instead, do due diligence in your design process.
Get trained. Follow a systematic, user-centered process. Include cycles
of testing, early and often. Apply the established literature and principles
in the field. You may not be "truly usable," but you are likely
to beat the competition.
|
