Ask Eric: Questions & Answers

December 10, 2002 – submitted by Cátia B. of Lisbon, Portugal

Question: I'm doing some research for my degree in ergonomics and I'm searching for expert systems that have been developed to help usability professionals in the evaluation of interfaces. Do you know of any? Where can I find this kind of information?

Thanks for the help!

Eric's response:There is a long history of work in this area. There are indeed a number of facilities available. A good example is "Bobby" which is a good facility that checks Web pages for compliance with accessibility issues. The ability to analyze a page directly for ergonomic issues has never been terribly successful. There are specialty tools that can help check various issues. For example AnyBrower.com will check for browser compatibility. Colorfield Insight will check for usability by color blind users. But currently, there is no facility that comes close to the quality of a trained reviewer following a systematic process.

October 1, 2002 – submitted by Nancy Neira of USA

Question: The project I am working on is requiring an in-house validation for a new mechanical enhancement. I need to write up a test scenario, then a satisfaction questionnaire. How many questions do I really need because of the few users that will be present (Users: 4) to justify my data?

Eric's response: There has been much discussion of the number of subjects needed for usability testing. Various positions have been taken. But 4 is too few. You cannot get a range of users. You cannot get reliable results. I would think 12 would be the minimum. Without enough subjects, no number of questions will 'justify' your data.

With at least a minimum number of subjects of the right type you can begin to look seriously at the results. With subjective rating questionnaires you do not need more questions to 'justify' the results. You can have a single question. The value of additional questions is primarily to dissect out the ASPECTS of the subjective experience. For example, I might ask overall satisfaction. But if it is poor, what does this mean? So to know more I might ask about perceived speed, aesthetic appeal, and simplicity. Now if they do not like it I could find that the main problem is that the design is ugly. I can then use graphic arts staff to improve it. The other reason for asking more questions is to make sure that your results are not an artifact of wording confusion. So you might ask if it is aesthetically pleasing in several different ways. You can then combine the aesthetic items to make a single assessment. This will tend to improve the reliability of the questionnaire. Generally, I would provide about 20 subjective rating questions to get a good understanding.

June 1, 2002 – submitted by Valerie of USA

Question: Which usability testing software do you recommend?

Eric's response: In fact HFI does not use any event monitoring package. We find such packages of little use for practical testing used in design work. They indeed can capture every keystroke and mouse movement. But then who can pay for the detailed analysis of all that data? We do simple video records and have the human factors specialist keep a log of issues and insights.

In the area of remote testing we have simply used WebEx or the Question Mark exam software for simple tests and surveys. We have also used NetRaker though as yet I remain unconvinced that this and other remote testing packages are fully ready to integrate into the design process. Right now they seem appropriate for specialized situations (as when the users are in many different countries and therefore hard to test). In any case we are tracking the remote testing software closely and hope to see it designed and practical for mainstream usability work.

May 1, 2002 – submitted by Nishant Singh of Fairfield, IA

Question: Eric, with software development teams moving towards agile development, how do usability testers keep up with this? Are we moving towards agile design and usability testing? It seems that usability testing is getting overlooked in agile development method. What is the best way of usability testing for agile development projects?

Eric's response: Nishant, usability testing is essential to agile development. But we need to move toward more dynamic testing methods. The long formal simulation testing at the end of the process is less our focus. Instead, use more dynamic methods. Do quick studies that take a couple of days. Use remote testing. Change the design after every few tests if appropriate.

The one danger of the agile manifesto is that it tends to rely on collaboration with customers. This sounds wonderful, but does not work. Customers are not designers. The do a terrible job of creating user interface designs. In addition, the customers selected for collaboration are almost never representative; and even if they are representative they quickly learn too much. They learn the design team's viewpoint and models. They will understand and like a design that will not actually work well for the users.

So make usability testing a routine practice. Do repeated quick tests. Test early in the design process. Use current methods that are fast and efficient. Highlight for the developers each time the test reveals a problem that would have been missed.