Illustration by Rouba Shabou
As a UX/UI designer, you know that user experience is an essential part of product design. The more you invest in creating a good user experience, the better the chances that your product will be successful. Studies even show that every dollar invested in UX results in a return of $100.
But with so many UX/UI trends to keep up with these days, how do you know if you’ve built a good user experience to begin with? It all comes down to your testing.
We recently surveyed more than 150 design professionals to learn more about how you work, the methods you use, and what skills make you successful. A large part of the survey focused on usability testing, and we found some interesting results. In this article, we’ll review some of the insights we found and how you can implement them into your design process.
What can we learn from user testing?
Good usability means a better chance of product success. But how do we measure usability?
With user testing, we can measure the usability of a product by asking people to complete some targeted operations within the product. Ideally, these people should be a good representation of your target audience. Once you have users perform the targeted task, be sure to incorporate some specific user testing questions to ascertain the most-valuable insights. While there may be many reasons why you want to conduct user testing, here are the two common reasons:
- To identify the areas in your product that need refinement
- To compare your product against your competition
User testing is an essential part of the product design process. Image credit Adobe Stock.The first step to setting up user testing is determining a few quantifiable metrics to track. Without these, you might end up with a jumble of information that can be difficult to understand—or worse, makes you think your product is terrible.
To hone in on this, we asked survey respondents about the metrics they find the most helpful when performing user testing. Below are the top five findings that we discovered.
(Note: While you don’t need to use every metric below, it’s important to at least be aware of it.)
1: Where are the usability issues?
Usability refers to how well a product allows its users to perform a given task. There are usually a few factors involved, but one has a tremendous impact: friction, or something that prevents users from achieving their goals or completing a task.
This friction might be due to poor design (e.g., unclear messages in system dialogs or long waiting time) or a lack of required skills (e.g., users do not have enough skills to use the product). Whatever the cause may be, the more problems users face, the less likely they’ll enjoy the experience.
50% of our survey respondents said that uncovering usability problems helped them the most during their testing, so this is an important metric to think about. User testing scenarios usually represent the most common operations that your users will complete within your product. For example, for an online shop, the first thing we would want to test is whether the user can find and purchase a product. When we conduct usability testing, chances are we’ll uncover the problems that almost every user will encounter.
2: What do users think of the product or website?
We know that user satisfaction is personal and subjective. Is there a way to measure it? Yes. UX practitioners have a particular metric for this called user satisfaction (or task level satisfaction), and 41% of our respondents found this metric very helpful during user testing.
User satisfaction tells us how test participants felt about the tasks you asked them to complete. If the tasks resemble real world scenarios, this information will help you predict the general impression of real users interacting with the system. When people have a bad experience using a product, they are less likely to use it again. That’s why measuring task level satisfaction is critical.
You can uncover this information by asking test participants to fill out a questionnaire at the end of your session. For example, ask them to rate each task using a scale of Very Difficult to Very Easy. This technique is also known as Single Ease Question (or SEQ) and you can read practical tips on how to use it in Jeff Sauro’s article 10 Things to Know About the Single Ease Question (SEQ).
Use a task level satisfaction questionnaire to measure user satisfaction.This metric will show you the general impression of users interacting with the product (and their level of tech-savviness) as well as how the tasks were crafted overall. If a lot of your test participants rate the tasks as Very Difficult, then something might be wrong with tasks themselves, and you probably need to rework them.
3: How many users actually accomplish what we want?
42% of survey participants also listed the success rate as a helpful usability metric. Success rate, also known as completion rate, is a binary metric. That is, 1 means task success while 0 means task failure. It’s easy to understand why it’s so valuable—when users cannot accomplish the task, there may be a big problem.
Conversion level is a related metric that can give you a better understanding of your test results. You can calculate your conversion level with the following formula:
Conversion level = (number of tasks completed successfully) / (total number of tasks) * 100%
For example, if five out of 10 test participants complete a task successfully, the completion rate is 50%. The goal of usability testing is to show a conversion level close to or equal to 100%, which would mean that when real users attempt a task, almost all of them will be able to do it.
For eCommerce products, UX designers use a special metric called conversion rate. This metric measures whether a user can convert (i.e., purchase a product on your online shop), and it will tell you whether your purchase funnel is effective.
4: How long do tasks take?
Time is a precious resource. The reason why so many people love using digital products is that the products improve their productivity and save them time. So the time the participant takes to complete a task successfully can help you measure the efficiency of your product.
The great thing about this metric is that it’s easy to calculate: Simply subtract the start time (the time when users read and understand instructions) from the end time (the time when users successfully completed the task). Typically, the more time it takes to complete a task, the less satisfying the user experience is, especially if it’s a routine task or operation.
40% of survey participants mentioned this metric is helpful during user testing. But keep in mind the usefulness of this metric depends on the nature of your product. For example, the idea that “more time on task means bad UX” doesn’t apply to game design, where you want users to spend more time on it because they enjoy it.
5: How many errors happen along the way?
Only a third of our respondents said the total number of errors is a helpful metric during user testing, and it’s easy to understand why this is. The total number of errors is a measurement that counts the number of user actions that cause error states. The actions can vary from slips (such as mistyped passwords during login) to wrong turns (such as failing to understand the meaning of some part of the app and making incorrect decisions).
But the truth is, the fact that a user encounters a problem at some step of their journey doesn’t mean much. If the user can fix the problem without much effort, they may not think of it as a real problem. That’s why you should classify and categorize every user action that causes an error state. By adding severity ratings to the errors, you’ll get a more holistic picture of your UX.
Conclusion
So, is there one universally accepted set of metrics that a product team can rely on to understand whether their product is usable (or not)? The answer is no. Every product is different, and each requires a different set of metrics.
At the same time, it’s easier to streamline your design process when you are familiar with the approaches that other UX professionals follow. The insights we’ve reviewed above can help you understand the bigger picture of your product and the people who will use it. Be sure to check out more user testing tips and tricks from Adobe XD Ideas.
Methodology
We conducted a survey for design professionals in different disciplines to learn more about:
- Tactics utilized in the design process
- Methods for the different design tactics
- Top skills per field:
We obtained more than 150 responses from designers and utilized their answers for this blog. To see the results, check out the full survey.