Ask Eric: Questions & Answers

November 15, 2007 – submitted by Christopher Kelley of NC, USA

Eric's response: Actually we do know the correct process and methods that work in different situations. When Susan and I wrote the Schaffer Weinschenk Method™ we not only defined a series of methods in depth, but we also defined the criteria for selecting different methods. So a good methodology will answer much of this type of question.

Beyond selecting the basic methods needed, there are many decisions related to cost (e.g., with a geographically dispersed population, the remote testing methods are often indicated). There are also decisions related to culture (e.g., many of the basic American usability methods have to be modified to work in Asia). Of course you also must decide on many issues of detailed methodological design, such as locations and numbers of participants to test.

So first you need a good methodology, which answers most questions. Then you need an experienced expert to refine the details of the methods.

August 16, 2007 – submitted by Jordi Lalonde of Andorra

Question: Well, here's my problem: I just don't know how to approach this. The thing is that I've coded a Web portal along with a back office tool to upload all the content you want, both from scratch. But now I need to test my back office's usability. But the main problem is that most of the tasks that I should ask to the user require some previous knowledge of the back office functioning. So I don't think I can ask the user to do all the tasks I want without getting him to understand how the back office works. It would be a waste of time and the test results would be terrible. But on the other hand if I told the user how the back office works then it wouldn't make much sense to do a usability test, right?

Well, thanks for your patience and for your time!

Eric's response: First of all let me voice a major concern about process. It sounds very much like you are about to start doing "usability" with the system coded. This is really, really, really, really, really, really bad.

First of all usability is NOT just usability testing. You should be following a user-centered process from the beginning. You need to understand the user, their needs, their taskflow, their environment (etc) before you conceptualize the system and certainly also when structuring the interface. You should have standards and follow usability principles in the detailed design of the interface.

In terms of testing, you should be testing early and often. The kind of final testing you are describing is almost certain to show lots of problems that you will not have time or budget to fix.

Finally, when you look at the test process, you will need some training in how to do it successfully. Usability testing primarily is useful for measurement of self-evidency. This is important. But is NOT an indicator of efficiency if you have long-term, high-volume users. In this case there are modeling techniques that let us tell if the interface will work efficiently for test users.

July 11, 2007 – submitted by Sudha R. of India

Eric's response: By "schema evaluation" I assume you mean testing the structure to ensure that people can understand the high level organization of the software. This is VERY important. But there is more needed. One key part is the testing of the detailed pages. It is easy to have a nice structure that is useless because of confusion with wording, layout, color, highlighting, control selection, and other detailed design issues. There are lots of other issues to test of course. Is the software useful? How is the brand manifest?

So if I only had one test to run, I would run a test of the high level structure. But that is like saying if I could only debug one page of a website I would debug the home page. Not exactly a professional viewpoint.

July 1, 2007 – submitted by Ashish Anand of India

Question: Recently I got into this debate of how many users are good enough for usability testing. What I know from recent research is that this is purely dependent on the requirement but in any case we just can't stick to the number 5 as said by Jacob N. This also came in a recent paper from HFI Also I read that 3 to 4 are good enough to get results, but is this actually based on research or just a thought by someone?

Eric's response: You can certainly get results with 3-4 users being tested. You can get results with ONE user. The question is if those results are sufficient for to assess usability or guide design.

As you reduce the number of users tested, you increase the chance you will miss problems. You also increase the chance you will falsely identify problems due to the random nature of your selected participant(s).

You will have an even harder time making decisions. Let's say that 3 of your 5 participant prefer one design over another. You might decide to go with that design. But you might as well just flip a coin, because the "3 of 5" is pretty close to a random guess.

It is even WORSE if you want to estimate things like the percentage of people who would use a feature, buy a product, or follow a pathway in the interface. If you have 3 out of 5 participants say that they value a feature, does that really mean that 60% would value it? Actually, you have almost no idea.

You may want to look at some of our newsletters on this topic:
Why "how many users" is just the wrong question
Enough is enough... but five probably isn't
How reliable is usability performance testing?

March 12, 2007 – submitted by Linda Loracco of Ontario, Canada

Question: Hi Eric, What a great resource, wish I'd found it sooner. If I'm running a focus group (in my case on research hospital intranet site) should I have a laptop set up for each member of the focus group? Do you have any suggestions on questions for an intranet focus group study?

Would you advise giving participants something to do prior to the meeting (a survey, a task to measure functionality)? Any advice is appreciated

Eric's response: The critical question here is "Why are you running a focus group?" There is almost no place in the usability field for focus groups. A focus group is good only for generating lots of ideas, with most of them bad. So we use them for new product ideation (and then work hard to sift out the few good ideas). But when focus groups are used for evaluation of a site they provide VERY poor data. There are "group think" effects where basically the loudest person's opinion dominates. They also don't reveal at all what the real behavior would be. You are far better off with a talk-aloud usability test. This will show what a person will really encounter and give you very useful insights.