|
Eric Schaffer, Ph.D., CPE,
Founder and CEO of HFI
|
If you are launching an in-house usability team, NEVER make your first
project a big usability testing program. All that will do is highlight
the awful mistakes the developers are making, before they have any way
to fix them. You will embarrass the developers, they will hunt you down,
and your group will be gone. Instead, give your developers training, standards,
and consultative support. Start serious testing next year. Then usability
tests become a way to fine-tune the design and highlight success.
|
|
Bob Bailey, Ph.D., Chief Scientist for HFI, continues last month's discussion
on the effectiveness of heuristic evaluations.
|
I received many emails and had many discussions after last month's article
where I suggested that heuristic evaluations have limited usefulness in
the design of systems.
Here is my main problem with most of the heuristic evaluations that I
have seen over the past several years. There are some influential people
in the user experience community who have convinced many others in the
community, and most people outside the community (managers, system designers,
programmers, etc.) that heuristic evaluations (and other low-level evaluation
techniques) are good enough on their own for achieving acceptable levels
of human performance. I do not believe that this is true!
Jakob Nielsen and (very few) others studied heuristic evaluations for
a few years in the early 1990s. Since then, he has summarized many of
his findings in his books. He always assumed that heuristic evaluations
were a valid way to identify usability problems. He never (not once) tried
to find out how good they really were when compared against actual usability
problems. Nevertheless, through his large number of publications, he has
convinced just about everybody that they are a cheap way to build effective
systems "at a discount"!
A good evaluation of human-computer interactions is very difficult to
do, requires considerable expertise, and in many cases the final payoff
can be very low. As I mentioned last month, heuristic evaluators can end
up missing some serious problems, and causing designers to go to the expense
of making many changes that make no performance or preference differences
at all. In fact, some of the new design changes most likely will introduce
new usability problems.
I believe that the research is clear on this, but unfortunately, it is
not consistent with trying to design and develop websites quickly in a
real-world environment. This poses a serious problem for usability specialists.
Computer professionals believe that they are getting much more from the
typical heuristic evaluation than they really are. This is important because
it is only after they see the fallacy of relying so heavily on heuristic
evaluations that they will change the way they approach usability in systems.
We cannot move forward, we cannot create more usable systems, as long
as most designers and managers believe that heuristic evaluations are
providing more information than they really are.
I suspect that the limitations with heuristic evaluations are understood
by few, and for this reason they will continue to be used almost exclusively
as the main test for system usability. Unfortunately, it is like debugging
a program by looking through the program code, rather than running the
code on a computer. Looking for coding problems is fast, and works sometimes,
but none of us would rely on it as the major means for finding coding
errors. But that is exactly what we do with heuristic evaluations.
|
|
The Pragmatic Ergonomist weighs in on heuristic evaluations.
|
We have a furor of opinion on heuristic evaluations. During my 25 years
of user interface experience, I have seen dozens of attempts to solve
usability with a "simple 10 step" process. It cannot be done.
There is no substitute for a systematic design process that includes contextual
inquiry with representative users, task design, interface design by experts,
and several cycles of usability testing. It is hard work, but the only
way to succeed.
I agree that the traditional heuristic evaluations have been sold as
an alternative to good design practice. It is no surprise that this very
quick and inexpensive process has so little value. But these low-level
heuristic evaluations are different than the rigorous "Expert Reviews"
done by the usability professionals at HFI (and by other top usability
experts).
When we do an Expert Review at HFI it can take as long as two weeks,
and can cost up to $40,000. We have a systematic process of walking through
each website (similar to Bob's "algorithmic evaluations"). [Note:
Bob will discuss algorithmic evaluations in a future newsletter.] We have
hundreds of pages of specific research-based design requirements that
we consider. Our experts can identify some usability issues that cannot
be found even with the best Performance Tests. So a heuristic evaluation
may be of limited value, but a professional Expert Review can be a very
cost-effective evaluation technique. When we begin with new systems, I
almost always recommend that an Expert Review be conducted first, all
issues resolved, and only then do we conduct Performance Tests (why use
the more expensive Performance Tests to find problems that we can find
by an Expert Review?).
|
