Usability testing is an integral part of user experience design. It helps product teams understand how their target audience interacts with their product and what problems they face during an interaction. Most usability testing methods fall into one of two categories: moderated and unmoderated usability testing.
In this article, we will explore both types of testing and share practical recommendations on how to conduct each of them.
Moderated usability testing
Moderated usability testing is a usability testing technique that requires the active participation of a moderator (a real person who will facilitate the testing). The moderator will have solid experience in the field of usability testing and user research and they will work directly with the test participant, guiding them through the testing process.
Moderated testing can be done either in-person (in a lab or work environment, where a moderator can observe user interactions with a product), or remotely (via a video-conference call, during which a moderator asks test participants to share their screens).
Moderated usability testing in the lab. Image credit projekt202.When to conduct moderated usability testing
Moderated testing is better suited for collecting qualitative feedback (times when it helps to find the answer to “Why?” questions, such as “Why did participants interact with a product in a certain way?” or “Why is a certain problem happening?”). For that very reason, it’s recommended to conduct moderated testing in the earlier stages of design – when a team is working on a product concept and wants to explore various solutions. Moderated usability testing helps the product team understand the mental model of a user and use this information to inform their product design.
Pros of moderated testing
- Moderated testing gives you more control. This testing allows guidance and support. The facilitator can get a test participant back on track if they notice that the participant has misunderstood or is confused. Facilitators can also ask clarifying questions to better understand user behavior.
- This form of testing results in participants that are more engaged. Moderators can have a natural conversation with test participants, which helps establish trust and makes test participants more motivated to complete tasks.
Cons of moderated testing
- It is often expensive. Moderated testing requires you to hire relevant test participants, experienced moderators, and for you to find a place for the testing.
- It also requires preparation. Moderated testing needs upfront planning – you’ll need source the necessary equipment and coordinate the time and date with your test participants.
Best practices for moderated testing
Define goals and plan the scope of testing
Knowing what parts of the product you want to test isn’t enough. You need to define key goals you want to achieve by testing your product with users and selecting tasks according to this goal. For example, if your goal is to improve the usability of a specific flow, it’s better to avoid vague goals (such as, “Our goal is to streamline the signup process.”). Be more specific on what you want to achieve (say, “Our goal is to reduce the number of dropouts during the sign up from 60% to 30%”). These specifics will help you to define the standards for your product and measure whether your existing product meets those standards.
Clearly define criteria for test participants
Test participants should represent your target audience. And for that very reason, you need to know what kind of people you want to hire. Make sure those you hire are in the right demographic and demonstrate the right behavioral traits you’re seeking. You can use the information from your user personas to define the criteria for test participants.
Follow the classic structure of moderated usability testing
Setting up a usability test involves carefully creating a scenario. The classic structure for a moderated usability test (a proven way of testing a product) includes the following steps:
- Pre-session questions. Asking questions about the participant’s lifestyle, background, and/or general behavior. Pre-session questions will help identify typical habits and other relevant personal information about your target audience.
- Actual tasks. In this step, the test participant performs a list of tasks using the product. All tasks should be selected according to the goal of the testing.
- Post-session questions. This involves asking test participants general questions about the experience and collecting their thoughts about a product. Post-session questions allow test participants to provide their opinions and ask questions about the product, as well.
As soon as you create a structure with actual tasks, run a pilot test with one or two of the test participants to ensure that this structure works well for them.
Select the right fidelity for testing
As a tool, moderated usability testing can be run during the product design process. In this case, the testing is conducted using a prototype, not a real product. While it’s possible to use low-fidelity and high-fidelity prototypes for moderated testing, it’s always better to test with hi-fi prototypes because such testing will give you a more realistic idea of how users perceive your design. When test participants see a prototype that looks like a finished product, they will evaluate it accordingly. On the other hand, testing with low-fidelity prototypes is valuable in the early stages of product design when you want to collect opinions about your design from real users.
Ask test participants to think out loud during the testing
When it comes to completing actual tasks using a product, it’s recommended to ask test participants to talk through what they’re doing as they’re doing it. When test participants think out loud as they complete tasks, it helps moderators to understand the rationale behind every decision, which reveals their motivations/perceptions.
Record the interaction on video
Both a user’s reactions and their screens should be recorded during testing. The video can be used during the results analysis phase to draw a conclusion from the research. It’s recommended that you share the video with the entire product team to make them aware of any problems users face when they interact with the product.
Take notes during the testing
A lot of things might happen during moderated testing, and it’s relatively easy to forget something. Thus, it’s recommended that you write down any interesting observations and facts that you can later use during the test results analysis.
Analyze results and present them in a report
It’s not enough to run a moderated test. You need to invest time in analyzing the results of testing. The report you create after the testing should contain the most important insights and help the product team and stakeholders prioritize design decisions.
Here are some recommendations for creating a report:
- Avoid UX jargon. Since you will show your report to various types of readers, you need to ensure that they won’t need to decode the meaning of any information provided in the report. Thus, it’s better to avoid industry-specific terms like “interaction cost” or “attention span” and replace them with clear synonyms.
- Use visuals to aid the information provided in a report. Humans are visual creatures, and it’s easier for us to comprehend information when it’s’ provided in visual format.
- Have a clear summary of the most important insights from the testing. This is especially important for long reports as people tend to skip over long documents. A quick summary will help readers get a sense of the information provided in a report at a glance.
Unmoderated usability testing
Unmoderated usability testing is completed by test participants in their own environment without a moderator present. This technique is usually used to test specific parts of a product (specific interaction scenarios), rather than providing an overall review of the user journey. Unmoderated testing is typically done using special tools for automated testing. The chart below visualizes the difference between testing tools in terms of the data they help to collect.
When to conduct unmoderated usability testing
Generally, unmoderated usability testing is better suited for collecting quantitative feedback (providing answers to the “How many?” questions). For example, this testing method is valuable when you need to identify how common certain problems or behaviors are among test participants.
Pros of unmoderated testing
- Unmoderated testing is less expensive and quicker than moderated testing. Unmoderated tests don’t require a reserved location (users can complete them in their own environment) or a predetermined time (users can complete tests on their own time). This makes unmoderated testing a good option for projects that have tight time constraints or limited budgets.
- There is less chance for bias. Unmoderated testing does not suffer from the observer-expectancy effect (which states that people being observed act differently than when they’re alone). Since there is no moderator, test participants interact more naturally with a product.
Cons of unmoderated testing
- There is a lack of guidance and support during unmoderated testing. Since unmoderated tests are completely unsupervised, there is no real-time support for test participants, and they will need to solve problems on their own. For that very reason, it’s not recommended that you run unmoderated testing using low-fidelity prototypes because the design might be confusing for them. Even with high-fidelity prototypes, some sessions may end up being unusable or less valuable, depending on any issues that test participants faced along the way.
- There’s no opportunity to ask follow-up questions. When it comes to valuable insights, nothing beats the conversation between moderator and test participant. Unfortunately, in unmoderated testing, it’s impossible to ask follow-up questions to clarify user behavior.
Best practices for unmoderated usability testing
Create simple, crystal clear tasks and mock up a real test situation
Since there is no moderator that can steer participants in the right direction, the outcome of testing sessions will largely depend on the quality of instructions you provide to your test participants. For that very reason, every instruction, task, and question needs to be carefully analyzed to eliminate any potential for misunderstanding.
Test your usability test before inviting real test participants to take part; this will ensure that instructions are clear. If you don’t have test participants, you can try pilot testing your test on a person that represents your target audience but wasn’t involved in the designing of your product.
Recruit more test participants
No-show rates for unmoderated testing can be higher than for moderated usability testing. That’s why it’s recommended that you invite more test participants to take part in order to achieve statistically significant results.
Plan for post-testing activities
In many cases, it’s possible to collect some qualitative or quantitative feedback from test participants by reaching them afterward with a quick survey. Many unmoderated usability testing tools allow predefined follow-up questions to be built into the script. The questions can be shown after a user completes a certain task or at the end of the session.
Should I go for moderated or unmoderated testing?
To choose the right method of testing, you need to focus on the type of data that you need to collect to meet your goals. Moderated and unmoderated testing are not competing, but rather they are complementary; in many cases, you will need both qualitative and quantitative data to create a good product. For that reason, you need to treat usability testing as an iterative process. You can start with moderated testing with a prototype to better understand users’ behavior and ensure that users can actually use your design. As you finalize your design, you can then run an unmoderated test to see how it performs in real-world conditions.