Ask Eric: Questions & Answers

December 1, 2006 – submitted by Christina Chang of St. Louis, MO

Question: We're in the process of choosing a portable usability testing software. I know Morae is a good tool but need to have other alternatives to compare with and submit the proposal to management. Do you know of other software that's comparable to Morae or any open source software?

Eric's response: Actually the best cheap alternative is to just run the test with video and not use specialized software at all. Even the video is really just for record in case there is a question about what happened during the test. Other than that you can test with nothing but paper notes and get fine results. The main practical advantage of software is easy editing of video clips. If you need this you can also get Camtasia, or use a scan converter and mixer direct into a video recorder. Then you can use commercial editing software to pull together clips.

October 11, 2006 – submitted by Ronald Verschueren of Rotterdam, Netherlands

Question: Good morning Eric.

What's your opinion about think aloud? We hear and read more and more critical notes related to the once much appraised think aloud protocol. According to critics, it does not only disturb time measurements (which is of course logical and not new to all of us) but also would disturb the clickflow. In other words: according to critics, think aloud leads to other - not the right - results compared to just issuing tasks, observing and not applying think aloud.

When to use think aloud and when not? Would you use think aloud in eye tracking research? Would you use eye tracking time measurement and think aloud separated in one research? What's your opinion?

Eric's response: Think aloud is wonderful. We use think aloud regularly for formative usability testing. It allows us to quickly understand where and why people get stuck.

It is not useful for summative testing where you can not interfere with the task performance and experience with such an obtrusive method.

It is also not good when you have very complex tasks (like landing a plane on instruments) or where absorption in the behavior is essential (like some testing of computer games). In these cases we use retrospective testing, where we show the participant a tape of their behavior and they narrate what is happening. But retrospective testing takes TWICE as long as normal talk aloud. So it is only used when we have no other choice.

September 5, 2006 – submitted by H. Jabbar of Iraq

Question: How do you measure the emotional response of users? Can you give me an equation? I don't prefer a questionnaire-driven measurement, do you have a stand alone formula? Your comments please.

Eric's response: Well there are really only three ways to measure the emotional response of a user.

1. You can ask them
So this is the questionnaire. There are good ones and bad ones. But it is the primary method we use. The Microsoft game testers just have one scale... "How much fun is this?". We have in-person, remote, online survey, popups, etc., etc.

2. You can check their physiological responses
There are some physiological correlates of emotional response. You can look at things like pupil dilation. This does show arousal level. But it gives little indication of what the arousal is really about. So it is used infrequently.

3. You can see what they do
Finally, you can see what they do. Watch if they ACTUALLY buy the product. This gives a nice objective measurement. If you really care if they buy the product that is good. But it does not really tell us exactly what the dynamics are that make someone want to buy it.

May 18, 2006 – submitted by Ravindra Papineni of San Antonio, TX

Question: What is your take on current trends using "eye tracking" tools and software-based (MORAE) usability testing? I understand they are only a part of the equation, but do you think companies will rely more and more on these type of new tools to bypass traditional usability testing (to save money)?

Eric's response: Well let me answer in terms of the different methods. First, eye tracking is impressive. But it has very little applicability to practical design. It is more a research tool. If someone recommends eye tracking for normal design work it usually means they are trying to impress, but don't know what they are doing. Actually, in China I see this all the time where labs are set up with eye trackers just to make it seem impressive. I wish they spent as much on staff that knew how to test.

There are two types of remote testing: moderated and unmoderated. The moderated testing is VERY promising as a way to get good data and reduce costs. We run remote tests all the time from our offices in India for example. This works very well on every type of usability test. The unmoderated testing is less valuable I think. It is OK as a summative test, to measure if people can do a task. But it does not tell you much about WHY a problem is happening. The unmoderated testing is more like clickstream analysis.

Finally, tools for support of in-person testing are reasonably useful. There are some packages that are total overkill for practical design. They are mostly for research labs, or perhaps some military grade applications. But a nice simple package for capturing the test and editing highlights is very useful.