Usability Reflections: Designing a Remote Usability Test

What is Loop11?

Loop11 is a service that lets you design, execute and analyze remote usability tests.

How does it work?

I snuck a peek at the source code for the actual test pages, and it seems to work as a proxy—the page under test is returned from their servers, resource URLs are rewritten from relative to absolute and page URLs are rewritten to use their server as a gateway. This way, they can intercept page navigations and plot the path a user takes through your site for any given task on your test. Pretty scary slick, no?

What does it look like?

Loop11 places a toolbar at the top of each page to act as the moderator. The toolbar lets the user know their progress through the test, what task they are on, what the task requires and lets them announce completion or abandonment of that task.

Building the Test

A Loop11 test is an arbitrarily-ordered queue of two basic constructs: Task and Question. You can add things to the queue, reorder, edit or remove them at any time.

Tasks

A task can be described as follows:

Starting from URL X, do activity Y and if you end up on URL Z[], it’s a success (anything else is considered a failure)

X is the particular page on the site you are trying to test.

Y is the scenario and instructions, which will be displayed at the top of the window while the user is attempting a task.

Z[] can be a set of as many URLs as you’d like—useful if there are more than one path to completing the task successfully. Any navigation outside of one of the URLs in Z[] counts as a failure.

Questions

Loop11’s Questions come in a huge variety of styles in presentation and in response. I barely scratched the surface, using only the following:

• Single choice (radio buttons)
• Multiple choice (checkboxes)
• Likert Scales
• Open-ended (free text)

The Takeaway

Once I learned their UI, I was able to build the test via the composition of a flexible set of components. At any point in the test design, I was able to preview the individual component I was working on or the entire test as a whole.

Technical issues

I did run into some minor technical problems due to the proxying Loop11 does. Google Maps doesn’t like hostnames using API keys that don’t match and will complain about it very noisily (in the form of alert boxes on every page).

Thankfully, since I didn’t base any of my tasks on the map interactions, it was more of an annoyance than a show stopper.

Addendum: Tasks & Questions

The site under test was Weather Underground.

Tasks & Verification Questions

1. You are planning a hiking/camping trip at Shenandoah National Park in Virginia for next week Monday-Wednesday. Will you experience any inclement weather (e.g., thunderstorms, snow, heavy winds, etc)?
2. Which of the following types of weather are predicted to occur on your camping trip?

3. How easy was it to find the information you needed to answer that question?

4. You want to visit Denali National Park in Alaska but don’t want to freeze to death. Which months have average lows above 20°F?

5. Which months have an average low of higher than 20°F?

6. How easy was it to find the average low temperature?

7. Can you find the page on the website that has data on hurricanes & tropical cyclones?

8. How easy was it to find the link to the Hurricanes & Tropical Cyclones page?

9. You are doing research on Hurricane Sandy. Can you find a map showing the path the storm travelled, from the time it formed to the time it made landfall?

10. How easy was it to find the link to the map of the path Hurricane Sandy took?

11. You want to track Tropical Cyclone Nathan, an active storm. Can you find the map showing TC Nathan’s predicted path?

12. Which country is the storm over now?
13. Which country is the storm predicted to be over on Tuesday 24-March?

Follow-up Questions

1. Were any of the tasks hard to understand?
2. Did you run into any technical problems when trying to do the tasks?
3. What is your opinion on the amount of information shown on the Forecast page?
4. How likely are you to use Weather Underground in the future to Forecasts?
5. What is your opinion on the amount of information shown on the Weather History page?
6. How likely are you to use Weather Underground in the future to view Historical Weather Data?
7. What is your opinion on the amount of information shown on the Hurricane & Tropical Cyclones page?
8. How likely are you to use Weather Underground in the future to track and research Hurricanes and Tropical Cyclones?
9. Please rate your satisfaction with the following pages (measured in Likert scale):
10. Weather Forecast Page
11. Historical Records Page
12. Hurricane and Tropical Cyclone Page