Usability Reflections: Quantitive Metrics
This week’s assignment was to select a single Quantitative Usability Metric and evaluate its benefits and limitations.
Definitions
WikiBooks provides the following definitions of quantitative and qualitative data in the abstract (emphasis mine):
Quantitative data is a numerical measurement expressed not by means of a natural language description, but rather in terms of numbers. 2
Qualitative data is a categorical measurement expressed not in terms of numbers, but rather by means of a natural language description. 2
This article posted at UserFocus does a great job of explaining it in the context of usability testing.
Time on Task: A Quantitative Measurement
The quantitative metric I chose to examine was time on task. This is a good example of a metric that has high potential value and can also be very misleading if taken out of context.
The Good: Can help pinpoint areas in an interface that need help.
For example, the business goal for a particular page may be to have the user complete some process as fast and as accurate as possible. Numbers showing very lengthy times on task might indicate a problem with the fundamental concepts of a particular step in a process or some other usability shortfall.
The Bad: Risks drawing false conclusions and might not indicate whether business goals are achieved
Consider the following examples:
- A user who speeds through an online shopping workflow has the lowest time on task. Did they take notice of any of the special offers or products the retailer wanted them to?
- A user who spends 2 minutes carefully overlooking every single facet of a page before proceeding to the next has the highest time on task. Were they confused by the workflow? Maybe they are notorious procrastinators, preoccupied with other thoughts, plain-old indecisive or some combination of all of those things?
- A user brute-forces their way through the entire process, clicking on random things until they arrived at the finish line, has an average time on task. The videogamer lexicon has had a pejorative name for this type of behavior for years: The Button-Masher. Does this score even count?
The Takeaway
Quantitative metrics can be misleading if not put in the proper context. For example, in my client space, an application that has 5,000 users could be considered a huge success. Meanwhile, on the Internet at large, an application with 10,000 users is probably going out of business. Of course, an app’s number of users says nothing about its usability, but this demonstrates my point that just looking at numbers alone without any kind of background can paint a very different picture.