Illustration by Nayane de Souza Hablitzel
In this article, we will review the concept of UX benchmarking, the process you need to follow to conduct it, and the top five essential tips for effective benchmarking.
What is UX benchmarking?
UX benchmarking is a procedure of using specific metrics to assess the user experience of a product or compare it with other products available on the market. UX benchmarking helps you assess how your product design is progressing over time and where it falls compared to competitors.
It’s possible to define two types of benchmarking: retrospective and task-based.
- Retrospective benchmarking. In a retrospective study, test participants are asked to recall their most recent experience with a product and answer relevant questions. A user interview is an effective method of retrospective benchmarking. It’s important to remember that the human memory is fallible, and the retrospective approach doesn’t guarantee that test participants will recall all the exact points.
- Task-based benchmarking. In task-based benchmarking, test participants are asked to complete some tasks using a product. By simulating actual interaction with a product, it becomes easier to collect some interaction-based metrics (such as task completion time, error rate, and so on). For this type of benchmarking, it’s essential to invest in creating proper tasks and ensuring that you have well-defined baseline performance (i.e., success criteria for every task). Moderated testing is an effective method of task-based benchmarking. A moderator can observe the user interaction with a product and ask additional questions if something is unclear.
In most cases, it’s recommended to mix retrospective and task-based testing to get the most realistic results.
The UX benchmarking process
UX benchmarking is a part of the user testing process and, similar to other user testing processes, benchmarking requires a structured approach. Here is the process researchers should follow to make the most of their study.
1: Define a baseline
When you start UX benchmarking, you need to create a baseline—a starting point, something that you will measure against. It’s recommended to pick a top feature (or a set of features) you want to measure and improve in your product. For example, when you conduct UX benchmarking for an ecommerce website, you might want to focus on the checkout process. Define basic metrics, such as average time on task or average number of visited pages, and run testing to get baseline usability metrics.
How Engaging Was It?
An example of how to measure user engagement while they interact with a product. Image credit loganjat.Here are two things you need to keep in mind as you define your baseline:
- Benchmarking is a team effort. Reach an agreement on the goals and objectives of your benchmarking study with your team. It’s crucial to have an agreement on what you want to achieve and why before you start. Set clear goals, such as reducing the percentage of users who abandon the checkout process by X%.
- Benchmarking should be tied to KPIs. By tying UX benchmarking metrics to actual business KPIs (e.g., improving revenue or improving customer satisfaction), you’ll be able to communicate UX benchmarking data effectively and efficiently to stakeholders. It will be easier to convince stakeholders to introduce changes if you can use data to say that by reducing the abandonment rate by X% during the checkout process, you will get Y% more revenue.
2: Carefully select test participants
Your research is only as good as your test participants. Invest in hiring people who will represent your target audience (existing or potential customers). Use the information from user personas to evaluate the user demographics and psychological and behavioral attributes.
It’s important to understand that you will need a statistically significant number of test participants. Generally, it’s recommended to invite 20 to 30 test participants because this number will help you to draw more valid conclusions from your data.
3: Define a procedure of testing and keep it consistent
At this step, you need to write test assignments for your UX benchmarking. Ensure that your assignments reflect the goals and objectives you want to achieve.
Once you’ve defined your test assignments and other details of UX benchmarking, be sure to keep all the details of the study the same for each round of testing. The number of test participants, the tasks you ask them to complete, and the devices they use to complete the tasks should be the same for all iterations. Otherwise, it might be really difficult to compare data if the test preconditions change.
4: Make changes based on UX benchmarking
There is no point in running UX benchmarking if you don’t plan to make any changes based on the findings. That’s why it’s vital to make your findings actionable and introduce changes based on the findings. As you make changes, run another round of testing to see how the UX is improving (based on the baseline metrics you’ve defined before).
5: Do benchmarking on a regular basis
UX benchmarking is especially useful when done at regular intervals, such as every month, quarter, or year. By measuring the user experience and comparing it with the previous results, it becomes evident whether your design efforts are actually improving the UX. A special term exists in UX benchmarking for that sort of work: longitudinal benchmarking, which is basically measuring over time.
The interval of benchmarking depends on your product design cycle and how often you’re rolling out product changes. Here are a few questions that can help you select the interval:
- How often do we release versions of our product?
- Do we have the budget to run the UX benchmarking on (interval) basis?
Top five tips for UX benchmarking
Here are some essential tips to remember when conducting the benchmarking process, both when preparing for the study and during testing.
Tip 1: Use benchmarking to identify product strengths and weaknesses
When you conduct UX benchmarking of your key user journeys, it quickly becomes evident where your product strengths and weaknesses lie. For example, by measuring user engagement (e.g., the most visited pages on your website), you will see what areas of your product provide the most value to your users. You can use this knowledge to reinforce your product’s competitive advantages.
Tip 2: Validate your designs against business objectives
User research should always be focused on the most critical business KPIs, such as conversion rate. When you have clear evidence that your design solutions improve the business bottom line, it becomes much easier to engage key stakeholders and secure buy-in on your product design decisions.
It’s essential not only to track how your design solutions improve the business bottom line but also to know why this is happening. It’s recommended to pair quantitative and qualitative testing to discover answers to both questions.
Tip 3: Use benchmarking to evaluate the major design change
Even when you have well-defined intervals for UX benchmarking, run UX benchmarking if you plan to roll-out product redesign that’s outside your product design cycle. As soon as you have a prototype of your solution, you can conduct benchmarking to see how it performs in comparison with existing solutions. The earlier you identify that something is not working as planned, the more time you save on product design.
Tip 4: Stay focused
It’s tempting to include a lot of different metrics in your UX benchmarking but it’s important to resist that temptation. The more you have to measure, the more expensive and more time consuming the procedure of testing becomes. Thus, focus on your top objectives. All metrics you select should help you answer your key questions.
Tip 5: Practice competitive UX benchmarking
UX benchmarking is typically used to understand the current user experience of your digital product. However, a huge part of improving the user experience of your product is knowing how it performs in comparison to your competitors.
Competitive benchmarking compares metrics of your product with those of your direct or indirect competitors.
Use competitive benchmarking to compare how your product performs in comparison to its closest competitor. Image credit MeasuringU.It’s also recommended to conduct competitive UX benchmarking every time your competitors introduce a new feature in their product and you want to create a similar feature for your own product. Prototype a feature and compare the prototype with any number of competitors or non-competitor products. The knowledge you gain after this benchmarking will help you understand whether or not you actually want to introduce a similar feature in your product.
Conclusion
A huge part of improving the user experience of your product is knowing how it performs for the users. You might assume that your product provides the best possible experience, but you’ll never truly know that until you actually measure their experience with your product. Practice UX benchmarking right from the start of the product design process. The sooner you begin benchmarking, the sooner you’ll be able to define the baseline and improve the user experience.