As any UX Designer can confirm, one of the hardest things about in-person usability testing is getting your participants to the physical testing site. No matter how hard we try to accommodate them (taking steps like holding testing hours after work) the fact of the matter is, sometimes you just can’t recruit enough users to test your product onsite.
As UX design best practices have continued to evolve, remote usability testing has become a more popular option. Remote testing is like traditional usability testing except that the participant and facilitator are in two different physical locations; the participant interacts with your design in their own environment, physically separated from the facilitator. This method is becoming more and more popular with the chaotic schedules that we all endure.
It is important to note that on-site usability testing is recommended whenever possible. In-person testing allows facilitators to read participant’s body language and gain a better feel for their overall experience. If we lived in a perfect world, we would always do on-site, in-person usability tests. But the world is not perfect, and so remote usability tests are sometimes a necessity. Here’s everything you need to know about remote usability testing and how to run effective, insightful, and objective remote user tests.
When should you consider remote usability testing?
Before we conduct a remote usability test, it is important to recognize when one is needed. There are certain situations when it makes sense to consider remote testing. Some of these conditions include:
- Conflicting schedules that prevent in-person testing.
- Participants being geographically dispersed, making travel difficult.
- Project has a low budget to put towards user testing.
- Accessibility constraints limit participants to their own environment, such as software or equipment.
These are the most common reasons for remote testing. Most companies face one or more of these obstacles when attempting to recruit users for on-site tests.
What are the different types of remote usability testing?
Now that we are able to recognize when a remote usability test is needed, what kinds of tests are out there? There are two main types of remote usability tests, both with their own advantages and disadvantages: moderated and unmoderated.
During moderated remote testing, the participant and facilitator are both present and are able to communicate back and forth during the test (via phone or video conferencing software). The facilitator is able to ask further questions for deeper clarification from the users as particular issues arise. The facilitator is also able to change, skip, and reorder tasks as needed. The facilitator can make clarifications to the participant so they are less likely to spend time on activities unrelated to the test. This may lead to the situation feeling more natural than the participant talking out loud to themselves.
There are some challenges to moderated remote usability testing. The main challenge is the inability to read a participant’s complete body language, as you are not physically in the same room as them. It is also difficult to decipher the meaning of silence since it may suggest that the participant is confused, thinking, searching, or potentially even distracted. Finding that perfect balance of active listening while not interrupting can be a heightened obstacle in remote moderated usability testing.
The other type of remote user testing is unmoderated. Unmoderated usability tests are, on the contrary, completed alone by the participant without the presence of a facilitator (no phone or video call). The benefits of unmoderated remote testing include being able to test hundreds of people simultaneously while keeping them in their own environment. Unmoderated testing works well in tight time frames or when budget is tight; you do not have the expense of having to pay a facilitator. Unmoderated testing is best used when testing a few specific elements of an experience, like a particular widget or a minor change.
There are some challenges to unmoderated remote usability testing. It is very difficult to dive deeper into issues that come up during the test. Some tools for remote testing allow you to integrate predefined follow-up questions into the study; however, it is never as robust as in-person communication. To supplement this, it is common practice to email a participant follow-up questions after the study to gain further insight into their experience. Besides the inability to ask detailed questions, another challenge is the lack of clarification; users do not have real time support in the event they have a question or need help understanding a question. You are also unable to nudge the participant to think out loud. In the end, they may be too internal with their thoughts and won’t provide very expressive or detailed answers.
Best practices for remote testing
Whether you choose moderated or unmoderated, there are some best practices to keep in mind as you develop your remote usability test to ensure a smooth testing with rich insights:
Step 1: Define goals and target users
First you must define your goals and your target group. Define your target user and decide which specific areas of your site or app you plan on testing. Make sure to define a user goal and create task-based scenarios in order to test and validate areas of your product. Once you have this defined, the next step is to recruit participants.
Step 2: Recruit users
Recruiting testers is the next step in your process. Make sure to recruit users who fit your target group and persona as defined in the previous step. Since tests are remote, you are able to recruit people from just about anywhere! This helps you better pinpoint the target audience you wish to test as geography is no longer a constraint. Social media platforms are great options for contacting people that may be interested in testing out your product. Make sure to offer an incentive for their time which will also help in recruitment. This can be in the form of a gift card or cash payment.
When recruiting users, it is important to know how many participants are sufficient. You want to avoid recruiting too few users yielding in insufficient results; nor do you want to recruit too many users and waste time and resources on additional tests. In short, five users will catch the majority of major usability issues in your prototype. Since attendance is always an issue, it is suggested to schedule at least three extra volunteers to account for people who do not show up.
Step 3: Write your test script
After you define your goals and recruit your target users, it is important to write a test script. When writing a test script, you can divide it into three parts: contextual questions, scenarios and tasks, and questions about their experience.
Contextual questions can act as ice breakers and help participants feel more relaxed and comfortable during the test. For example, if you are designing a transportation app, you may begin the test by asking the users how they get to work, how long it takes them, how they feel about that, etc. Make sure to ask non-leading questions that are not suggestive nor biased. Keep them open ended with room for follow up questions to dive deeper into the user’s response.
Next, write scenarios and tasks for your users. It is up to us to simulate a real use case so the users are able to get into the mindset of using the product. We will give them a hypothetical scenario which will direct a particular task for them to complete with your product. For example, if you are designing a pizza ordering app, this is what the scenario could look like:
- Scenario: You are at home watching a movie on Sunday evening. You are feeling hungry and decide to order a pizza for dinner.
- Task: Using this mobile phone, access our app and order a large pizza with thick crust, pepperoni, olives, and extra cheese and have it delivered to your address.
Lastly, prepare follow up questions about their experience. Now you are able to gather their impressions about your product. You can ask them questions about their experience, what stood out most to them, if they would recommend it to a friend, etc. As you prepare these questions, there are some things to keep in mind:
- Tests should be 15-30 minutes long.
- Focus on 3-5 straightforward, actionable tasks that have well-defined end states.
- Include the system requirements during the screening process to ensure participants have adequate technology to complete the test.
- Include consent forms.
When possible, have a mock test session to practice your script and get yourself accustomed to the remote style test. The more comfortable you are with the script, the more confident and effective you will be on testing day.
Step 4: Conduct the test
Finally, it is time to run the actual test. Using an email template design, make sure to send out reminder emails to all parties either the night before or the morning of testing. Remind them when their sessions begin as well as any pertinent information regarding the software to ensure a smooth testing experience. During a moderated remote usability test, it is recommended to have two people on the call, one to facilitate and one to observe and take notes.
Since you are in a remote environment, it is the moderator’s duty to help create as comfortable and “real” an atmosphere as possible. Make sure to speak slowly and look into the camera, making eye contact as you would in real life. Do not look distracted and remain in the frame throughout to avoid confusing the participant.
Once the participant has successfully connected, it is important to initially orient them with the test. Make it clear that there is no wrong answer and that you are testing the product, not them. Ask participants to think out loud as they execute tasks in order to capture their emotions and thought processes. Lastly, encourage them to be honest and straightforward with their feedback, assuring them that they will not hurt anyone’s feelings. Once you have completed these orientation tasks, you can move onto your test script.
The contextual questions will set the mood as well as expectations for the test. As the user is completing the scenarios and tasks, remind participants to think out loud. If they ask a question, answer their question with a question, as opposed to giving direct answers. For example, if they ask, “Where do I go next?”, you can reply, “Where do you think you should go?”, etc. During the test, the observer will be recording which tasks are successful and the time needed to execute each task. This kind of task analysis can provide meaningful insights about your product.
Step 5: Summarize results
Next, as a team, go through your results. Note any patterns and areas of friction that will help direct future iterations. Consolidate your results into a usability report to share across departments to get everyone up to speed on the test results.
Breaking down the pros and cons of remote usability testing
Now that you know how remote usability testing works and some strategies for going about it successfully, it is important to fully understand the pros and cons of this approach versus on-site testing. This will help you decide when and how you use this remote option, and when it’s more advantageous to invest in a round of on-site testing.
Pros of remote usability testing:
- Cost savings – Plenty of services that are cheap (or even free).
- Availability – Eliminates travel time to and from testing sites and increases participants availability.
- Geography – Ability to test a diverse geography of participants such as multiple users from multiple continents with minimal effort.
- Speed – Third-party testing services help target audiences rather than searching for them on your own. Eliminating these and other logistics helps increase turnover speed of testing.
- Natural environment – Research conducted in a participant’s natural habitat is more accurate and realistic than research in a lab environment.
Cons of remote usability testing:
- Security – Content can often be sensitive. Screen capturing software may not always be 100 percent secure.
- Task limitations – Unmoderated testing works well for small tasks but is not suitable for long and complex tasks.
- Software requirements – Some tests call for special software, making the installation process a burden for users.
- Connection speeds – Connection speed requirements may limit particular demographics from participating in tests.
- Lack of body language – The inability to analyze the participants body language is the biggest burden. This is the richest insight to the counterpart of in-person testing.
Remote usability test tools
As you are developing your remote usability test plan, deciding on which tool to use is a very important part of the process. Most services are quite affordable, while some are even free. For moderated usability tests, the most common tools include:
- Lookback – Recordings are safely stored in the cloud where you can easily test websites and apps. Limitations are that users have to download software.
- Zoom – Very easy to use and free to sign up. A limitation is that participants must download an application.
- Join.Me – Integrates with Google Chrome, making it easily accessible in the browser. Scheduling future meetings is easy as well.
- GoToMeeting – One of the most popular conferencing tools. Has to be downloaded and is more expensive if you want a call recording. Connections can also be unreliable.
- Skype & Hangouts – Most people are familiar with this software however neither support call recordings.
Each of the above tools have their pros and cons but all are heavily used. In terms of unmoderated usability testing services, the most common options include:
- Loop 11 – Includes A/B testing, testing of live websites as well as heat maps and clickstream analysis. Does not have a participant database.
- UserTesting – Key features include Live Conversations and Highlight Reel features. This gathers video recordings of key moments to help make compelling arguments to stakeholders.
- UserZoom – Has an extensive user participant database to help recruit users. Quick turnaround time with testing.
- Userfeel – Gives automatic transcription, timing reports, and highlight videos. Test on desktop, mobile, and tablet.
- Userlytics – Extensive filtering system for the participants allowing you to recruit both on set trait criteria as well as additional screener questions. Criticism is that the data visualization tool is a bit complex and results may be confusing to those not savvy with statistical analysis.
These are just a handful of options as most have similar features. When searching for your ideal tool, pay special attention to the installation requirements per option. Make sure to communicate these installation requirements explicitly to your participants during the screening process.
Conclusion
Remote usability testing has become more and more popular as companies work to cut costs without sacrificing the ability to gain insight into future product success. While they are not as insightful as in-person tests, remote usability testing still offers a ton of value. It is up to us UX designers to know the in’s and out’s of this methodology in order to utilize it most effectively.