How-to Guide: Usability Testing — Part 1

Words by UX Research Team Graphics by Ajj Morales
Words by UX Research Team Graphics by Ajj Morales

Yes, you can run Usability Tests too.

Our UX Research team at On-Off Group wanted to create a guide that could encourage and help people to start Usability Testing their products— wherever they might be in the development journey. Depending on the breadth of your research, Usability Testing can be a daunting task. But truth be told, you don’t need a whole team to run tests and your product doesn’t have to be 100% complete before testing either.

By following this guide, you’ll be able to start testing right away and gain the ability to incorporate user feedback into your design as early as possible. In Part 1 of the series we will discuss the basics of Usability Tests, how to prepare a Research Plan, and digital tools to use for Remote Usability Testing.

What are Usability Tests?

Usability Testing is a method of measuring the usability of a website, app, or other digital products. ‘Usability’ is defined as the extent to which a product can be used to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use (ISO 9241-11).

The methodology revolves around observing real users as they attempt to complete tasks of the product. The goal is to reveal areas of confusion and uncover opportunities to improve the overall User Experience (UX).

Uncover Problems in the design
Uncover Problems in the design
Get Insights from your users
Get Insights from your users
Discover Opportunities to improve
Discover Opportunities to improve

Why is Usability so important⁣?

Users can feel frustrated when they find a product difficult to use. Poor usability can eventually lead to the loss of a product’s competitive advantage or worse, low customer retention. Think about it: there are countless other websites and apps available out there that might provide a better and more hassle-free user experience. This is where UX Researchers & Designers find common ground with the C-Suite and Business Managers.

Usability goals are business goals. Websites that are hard to use frustrate customers, forfeit revenue, and erode brands. — Forrester Research

This is why Usability Testing is crucial to product development. When you deeply understand the needs and pain points of your user as they use your product, you can deliver a better experience. This research method raises the questions: What kind of User Experience do you want to give your customer? How are you willing to improve your UX to keep their business?

When to Run Usability Tests?

In an ideal world, the correct answer to "When should you conduct a usability test?" would simply be "early and often.” But that's not always possible so testing during these two key moments below will do. Every project's development is unique but if you conduct tests during the moments we’ve described, you can be sure that you're creating designs that are validated by your user.

audit usability.png
  1. When it’s time to evaluate.
  2. Once you've made some design decisions and created a wireframe or prototype, you've transitioned to an evaluative phase.
  3. When it’s time to optimize.
  4. After launch or redesign, there's always something that can be improved, and if you want to continue optimizing your design — or if you want to start developing new features — it's important to continue running usability tests.

How Many Participants Do I Need to Test?

At a minimum, the baseline for testing is 5 users. This is because testing 5 individuals can help reveal at least 85% of the problems in an application. Once you cross the threshold at 5 and begin to add more test participants, the number issues that you’ll uncover reduces. This is because you’ll start to hear the same feedback from the different users.

With the magic number of 5, you’ll still need to consider your different personas or user segments. This is dependent on how many personas you have for your target market — for example, you have a site that will be used by both children and parents. Then the two groups of users will have sufficiently different behavior so it becomes necessary to test with people from both groups.

In testing multiple groups of users, you don't need to include as many members of each group as you would in a single test of a single group of users. The overlap between observations will ensure a better outcome from testing a smaller number of people in each group.  

What do you need to do Usability Testing?


To conduct a Usability Test, you will need:

  1. Test Participant
  2. A realistic user of the product being tested
  3. Product to be Tested
  4. Can be prototype or a finished product
  5. Test Facilitator
  6. Guides the participant through the testing process

How does Usability Testing work?

The Test Facilitator administers the tasks to be done and observes the Participant as they interact with the product and accomplishes the tasks. The Test Facilitator does this based on a prepared research guide that gauges the Participant’s experience. The Test Participant is expected to “Think Aloud” while accomplishing the task, providing feedback on the product. This is done, as much as possible, without the Facilitator’s direct instruction so that the Participant’s experience can be observed without interference.

How to Run Your Own Usability Test.

Running a Usability Test might seem daunting at first but with adequate preparation, you and your team can accomplish a successful test. Breaking it down in stages will help you prepare what’s needed before the test so you can implement the test, analyze your data, and synthesize actionable steps with ease. It begins with preparing a Research Plan. This research plan will establish the scope of work to be done. It will help to keep the team aligned when implementing the research.

Stage 1: Preparing your Research Plan

Step 1— Identify Business + Research Questions

Research Plan and discussion guide icon.png

Start writing your Plan by forming a Business Question.

For example, “What new features can we add to our app to entice our younger market to get insurance?”

This business question will prompt you to explore more questions such as:

  • What are our user’s motivations, successes, and frustrations in life?
  • What do they look for in a product?
  • What are their spending habits?

These inquiries are now your Research Questions. Research Questions should provide you enough information to answer your big Business Question. By having well-defined questions, the User Research is given direction and a goal.

Step 2 — Set Success Metrics

clipboard logo.png

Measure if the results of the Usability Tests are successful by setting Research Objectives.

This can be directly related to your Research Questions or geared towards your overall Business Question. For Usability Tests, we evaluate the effectiveness, efficiency, and ease of use of products. This is based on the ISO 9241-11 definition of usability:  ‘extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use’.

Each of these factors is gauged to determine if the product can be considered usable.

  • Effectiveness - “Can they use it?”

80% of the participants are able to finish applying for life insurance online.

  • Efficiency - “How fast were they able to do it?”
  • Participants can complete their online application within 30 minutes.
  • Satisfaction - “Are they happy with it?”
  • 85% of the participants rated the online application experience a score above 7.

Step 3— Determine Tasks to Test

tasks to test.png

Identify Red Routes, or the key tasks or activities that your users want to carry out with your product.

Red Routes can help your team:

  • Prioritize user needs along with your business objective. From here, your team can decide which features to test or build first.
  • Manage the continuous growth of a product and avoid unnecessary features.
  • Build and optimize product features that deliver the most value to your customers and drive key metrics.

Here is a ready-to-use template of the Red Route Matrix that you can build on for your product. The top right is your most frequently and widely used feature, while the bottom left is your least used and more narrowly tailored feature.

In developing your product, you can use this Matrix to prioritize which features to build and test. You don’t need a fully working application to test your product. You should, however, make sure that what you are testing has been created according to the red route priority.

Here is a ready-to-use template of the Red Route Matrix.

Step 4— Create Task Scenarios

task scenario.png

Giving tasks scenarios to your participants provides the context needed to complete the task.

Once you’ve decided on your tasks, you can write your task scenarios. These scenarios are meant to paint a picture, not guide the user on what to do. Here’s an example of a task scenario for getting an insurance product:

Task Scenario:

As a first-jobber, you were very excited when you received your 13th month pay. You figured that the adult thing to do would be to invest on something long term, like life insurance. You heard from your friend that you can apply for life insurance online through Insurance Company X.

Task: Get a basic package for life insurance online

Step 5— Write a Discussion Guide

discussion guide.png

Standardize each usability test by writing a discussion guide to follow.

Once you’ve finished identifying the scope of the research, you can now write your discussion guide. Your discussion guide is a script you’d follow when you conduct your tests. This helps ensure that the tests are standardized— you need to be able get all your participants to answer the same questions and accomplish the same tasks. This guide will:

  • Prepare you for the flow of the conversation with the questions you intend to ask
  • Align your team and observers, if there are any.
  • Describe the tasks you have designed for the usability test.

Tip: Follow the sequence of questions and tasks but remember that a research guide is linear. Participants may take things in a different direction so adopting flexibility is key.

We recommend using Steve Krug’s Research Test Script as a guide for building your discussion guide. This example follows this structure:

  1. Ask Pre-Interview Questions.
  2. These questions are geared towards uncovering more about the user and their context of use.
  3. Give the tasks to be performed by the participant during the Test.
  4. Ask post-interview questions and any follow-up questions.
  5. Have them to fill-out the System Usability Scale (SUS) questionnaire.

Stage 1: Checklist of Tasks

  1. Identify Business + Research Questions
  2. ☐ Start writing your Plan by forming a Business Question.
  3. Set Success Metrics
  4. ☐ Measure if the results of the Usability Tests are successful by setting Research Objectives.
  5. Determine Tasks to Test
  6. ☐ Identify Red Routes, or the key tasks or activities that your users want to carry out with your product.
  7. Create Task Scenarios
  8. ☐ Give Tasks Scenarios to your participants to provide the context needed to complete the task.
  9. Write a Discussion Guide
  10. ☐ Standardize each Usability Test by writing a Discussion Guide to follow.

Tools to Use for Remote Usability Tests


Once you have the bulk of your Research Guide done, you have to determine what tools to use for your Usability Test. Although In-person Usability Testing is the preferred method, we have since adjusted our tools since the start of quarantine. Here are two tools you can use for Remote Usability Testing.

Basic Level: Get into Hangouts with your participant.

Photo c/o TechRadar
Photo c/o TechRadar

Google Hangouts is a platform that allows users to converse, present documents, and share device screens to users in the meeting. You may schedule meetings and send customized invitations to participants. These are accessible through email.

How it works: Have the participant open the prototype on their desktop and ask them to share their screen while they interact with the prototype


Advanced Level: Try your hand at Lookback

Lookback is a usability testing platform that assists in carrying out remote moderated and unmoderated tests through different devices. It allows test moderators to invite participants to a session, where they can share their device screens. Doing this allows both parties to see the participant’s journey and interactions with the product.

How it works: With microphones and cameras enabled, Lookback automatically records each session and uploads these to their cloud. This allows test moderators the opportunity to review the usability test after the session.

Important Note: Your participants may not be familiar with the aforementioned platforms, so it is best to brief them in an email about the tools they’d need to have before you start your scheduled session. In addition to this, let them know beforehand that they would need to have their microphones and cameras turned on for the session.

For optimal preparation, you may opt to allocate time for you and the participant to test your equipment prior to the test. This is so both parties are able to gauge whether internet connection and equipment are functional and ready to run a usability test.