Why usability testing can not certify usability

Many people ask "Doesn't a usability test perform the same function as this certification program?"

The answer is "NO".

Actually, we spent a couple of decades thinking about how we could use a usability test to certify something as usable. You can't do it.

Imagine for a minute that you're trying to certify wheat. If you certify wheat based on its nutritional content, you can put it in a machine, it grinds it up, it tests the nutritional content, you read the results and say, "Okay, it meets the criteria, it's certified." No problem.

But what if you want to certify the taste of the wheat? You can't put that in the machine. You have to have people try it. No matter what you do with the wheat, some people will like it, some people won't.

So, what's the level of taste that you consider to be usable, how many people do you have to test with to find out whether it's 79% usable (which is not certifiable) versus 80% usable (which is certifiable). It's impossible to find a point. Does 80% of people mean it's usable? Then, I have to test with thousands and thousands of people to tell the difference between 79% and 80%.

Here is another major problem. When you run a usability test, how long to you run a test for? Maybe one hour. But in a one hour test we are measuring initial usage. What if we have a product that's going to be used for months or years? Now, we don't really even care about the first hour. So you might be certifying with completely incorrect criteria.

Usability tests are not a valid way of telling if things are actually usable. What you need is completely different approach. It took us some time to figure this out, but certification means that you have trained, certified people, following a process within an organization, that has the right kind of capabilities and tools. It means they do the process, and document that they do the process, and that certified people validate good designs. Then we do an audit to make sure that's true. Part of what HFI will do in this process is ensure that people, when put under the pressure of their organization, won't certify things that shouldn't be certified.