Ask Eric: Questions & Answers

November 8, 2004 – submitted by Leona of Malaysia

Question: I would like to know more about usability testing, especially how to conduct a usability test for a Web site. Thank you.

Eric's response: User-centered design takes much more then just a usability test. Usability testing is the way that we 1) fine-tune our design and 2) verify that the design is working. We have two types of testing that correspond to these two objectives. Formative testing helps us fine tune designs. It's a dentist having you bite down on carbon paper and grind your teeth to see where the teeth must be smoothed out. Like this, we have users try to use the site and see where they get stuck. In summative testing we have criteria for success and we test the user to see if these benchmarks can be reached. For example: we know a competitive site allows users to sign up for an account in 3 minutes. We test to ensure that our new site is more efficient.

September 29, 2004 – submitted by John of Greece

Question: What factors influence how quickly Web pages are retrieved from the Internet?

Eric's response: There are a whole set of factors that affect the bandwidth. This includes the type of connection (dial-up, various broadband choices, etc.) There are also factors related to the positioning of the server and its level of connection to the Internet backbone. Then there is a major factor of the page weight (number of bits in the page design).

Interestingly, the actually page display time is not as important as we initially thought. The way that the page displays is important (if you can get the user working, say by displaying test first, then the fact that the download is continuing is not that important). In addition, a useful and engaging page will be SEEN as faster, even if the actual download is slow.

June 18, 2004 – submitted by Julie Christensen of Green Bay, WI

Question: We would like to measure how much time savings there is before our portal implementation and after. Is there any industry average on how long one mouse click takes? We thought if we could measure how many clicks were saved with the portal we could put a time/dollar figure to that.

Eric's response: Indeed you can measure the average time for a mouse lick or keystroke. However this is a small fraction of the time spent at a portal. The primary time is spent scanning and reading. So measuring the physical movements won't be very interesting. If you want to make a comparison to show the change, develop a 'basket' of representative tasks (e.g., "find new employee's phone number", "find HR site"). Then run a usability test contrasting the original situation with the portal. I think this will be most meaningful.

May 14, 2004 – submitted by Audrey Klarzynska of United Kingdom

Question: I am a Corporate Data Manager and would welcome your views on the following. I recently requested that a supplier currently developing a COTS prototype for the organization provide feedback on the level of compliance against the corporate policy and standards. One of the policies to be looked at was the Usability Policy. In the comments, I stated (perhaps unadvisedly) that implementing a COTS solution puts the organization at a disadvantage regarding usability as a truly usable system can only be achieved by designing it to meet the needs of the specific user group(s) involved, i.e. bespoke. And that using COTS means that a compromise must be reached between the solution offered and the users' perception of how information should be presented. This was stated in the context that the design of the COTS product could not be customized to any large extent. I would welcome your views on this as the project manager has taken umbrage and said that COTS are "more usable than bespoke".

Eric's response: Audrey, I am sorry that your supplier took offense. However, in general you ARE correct. Without question the wider the target population and taskflow, the harder it is to make a solution that is ideal for all users. Since I think it is safe to say that COTS packages must accommodate a greater diversity of users, taskflows, and environments, they have a much harder time being usable. On the other hand, if a COTS development company invests enough in usability work their result can be BETTER then a company doing a customized job without the benefit of a mature usability operation. So rather then depend on generalizations it is best to evaluate the supplier based on the quality of their process and product. Check carefully how well the product meets your needs.

March 26, 2004 – submitted by Jeff White of Charlotte, NC

Question: My company is finally redesigning their corporate Web site. They are striving to be more "customer-centric". Toward this end, they plan to hold focus groups, and focus groups alone to determine our user needs, and from that determine feature sets and functionality requirements for the site. The plan is to hold the focus groups and then let internal line of business reps determine the feature set for the new site.

I am a big proponent of User-Centered Design. I am thrilled that our executives are finally striving to be customer focused, but I am petrified of relying of focus groups alone, and then letting internal staff run with the data.

I know focus groups often produce misleading, biased results and I foresee them causing some big problems in the long run unless we adopt a more robust UCD process that includes other research methods such as contextual inquiry, task analysis, user profiles, etc.

What advice can you give me, or what resources could you point me to that will help me convince my bosses that we need a larger variety of research techniques to accurately get a picture of our target audiences?

Eric's response: Well it is common today and a good thing that executives are understanding that usability is mission critical. However, as your case illustrates beautifully; this understanding of criticality does NOT translate to an immediate understanding of the HOW of user-centered design. Their focus group approach is virtually certain to fail. It is pathetic. What is WORSE, you may find that they decide that user-centric design is of limited value when they find that a focus group does not translate to a great Web site. (Sigh).

The very best thing you can do at this moment is get the executives educated. Don't expect them to listen to you from within the organization. They will only listen to ultra-credible outsiders. My book "Institutionalization of Usability: A Step-by-Step Guide" is a good bet. It gives a clear outline of the requirements for mature usability work. It also specifically addresses the focus group problem. Also consider getting them in touch with a usability consultant who can guide them to a proper understanding and plan.

March 9, 2004 – submitted by David Wasilewski of Manchester, UK

Question: Are there any standardized tests/research that suggests at what point that a college or university student's handwriting speed will start to adversely effect their exam performance?

I am asking this from an angle of testing for exam concessions e.g. extra time.

Eric's response: In the USA students with significant handwriting impairment get extra time in some tests. You might check the rules used. However, I think they are quite rough and probably have little scientific basis. I suspect they amount to a 'note from a doctor' approach.

February 16, 2004 – submitted by Dana Griffith of Columbus, OH

Question: Hello to all at HFI. I am searching for research on what, if any, effect privacy policy statements have on users of online account management sites. Do people avoid using sites that don't have them? Do they consider sites that do have them more worthy of trust? I would appreciate any insight you can offer. Thanks.

Eric's response: This is an interesting case where many people SAY that it is important, but almost NO ONE actually looks at it.

January 24, 2004 – submitted by Umesh Persad of Cambridge, UK

Question: I would like to find out what the latest research has to say about the number of users, and type of users necessary for testing products for older users. Do we include users with all types of ailments e.g. visual, tactile, cognitive and try to optimize the product for all types? If the effects of aging are gradual, then the users chosen never remain the same, so how can we be sure that the population of older users is being represented accurately in our sample?

Eric's response: Well actually you are raising two separate issues. In testing for aging populations it makes sense to test five to ten older users. There is MUCH controversy over exactly what is considered 'old'. This is best determined by your actual expected population. If you will indeed HAVE a significant user set over 80, then test users over 80.

Specific ailments are different. They are not entirely age related (though there is a correlation). You can certainly find young users with almost every possible impairment you might find in older users. In general we manage this not so much by testing, but by designing with those limitations in mind.