The main rule of product design is simple—it’s not enough to design a nice-looking product; it should be usable too. This rule is even more important when a team works on innovative products such as artificial intelligence products.

In this article, we will explore one tool that supports product teams in user research and minimizes the risk of post-release product failure. This tool is called heuristic analysis, and it should be an essential part of your design process.

What are heuristics and what is a heuristic analysis?

Heuristic analysis is an analysis, done by experts, that determines the susceptibility of a system toward a particular risk. A heuristic analysis in UX design is a procedure used to identify a product’s common usability issues. When UX practitioners conduct a heuristic analysis, they compare a digital product’s design to a list of predefined design principles and guidelines, also known as heuristics.

Heuristics are made of design conventions that have been tested or observed over long periods and have proven to be effective. Jakob Nielsen’s “10 Usability Heuristics for User Interface Design” and Ben Shneiderman’s “Eight Golden Rules of Interface Design” are probably the most commonly used set of usability heuristics. Below are just a few examples of heuristics for design from both sets of rules:

  • Visibility of system status–Where am I? The system should always keep users informed about what is going on through appropriate feedback within a reasonable time.
  • Recognition rather than recall–What can I do here? Humans can recognize things much more easily than recalling them from scratch. Product designers should minimize the user’s memory load by making objects, actions, and options visible and clearly communicated. 
  • Match between system and the real world–What will happen if I do this? The system should speak the users’ language, with concepts familiar to the user. It means that the terminology, as well as concepts and metaphors used in digital products, should be natural to users.

At the same time, UX practitioners should not rely solely on these heuristics. Instead, the heuristics coined by Jakob Nielsen and Ben Shneiderman should be used to inform and inspire a designer to develop their own specific heuristics.

How to conduct heuristic analysis

Heuristic analysis is different from any other usability testing because the people who do the analysis (usability experts) inspect the interface instead of using it.

When it comes to establishing design heuristics, there are no fixed recommendations, as each design presents its own set of different needs. However, most experts use between five and ten heuristics when they do an analysis. These items are chosen based on their applicability to the overall usability of the product that is being tested.

Why should you use heuristic analysis?

Jakob Nielsen calls heuristics for design “discount usability engineering methods” because they are good in terms of saving money. Even when you need to hire an expert who will conduct a heuristic analysis, it will usually cost less than conducting usability testing with real users. Additionally, at the end of the evaluation, you will get a set of heuristics that you can use in future analysis.

How to conduct heuristic analysis for AI-driven user experience

Identify the goal of analysis

Know your target audience

No matter what product you design, you should always start a heuristic analysis by finding the answer to the question, “Who are my users?” You need to define user personas and use them during the analysis. Understand user demographics and their tech skills. Are your end users novices or experts? If they are experts, you can use more creative interaction patterns. Are they young or old? If a large part of your users is an older audience, you might want to focus more on the accessibility of your product.

Define the scope

Doing a heuristic analysis of an entire product isn’t a good idea, especially when you are working on a large-scale product. Heuristic analysis of the entire product could take a very long time and therefore become too expensive. Thus, it’s recommended to conduct a highly focused heuristic analysis—focus only on the most crucial areas of the product and the problems that users face there.

In the context of AI products, it’s vital to focus on the following areas:

  • Discoverability – What actions can I do? What is the system capable of? If your product has a screen, you need to ensure what available options are visible to the user. Does the system offer visual or audio guiding that helps users interact with a product? 
  • User control – Can I undo operations? This is especially important when a user interacts with a voice-based AI assistant. When a system doesn’t offer cancel or go-back options, it may not be evident to the user how the system responds to user interactions if they say, for example, “Go back to the previous step.” As a result, the user might skip some types of interactions (such as making online purchases using an AI assistant) because it would be harder for them to predict the outcome from such interactions.
  • Freedom – Can I interact with a system in the same way I interact with humans? The more natural the system feels in terms of interactions, the more users trust the system and are willing to interact with it. 
  • Consistency. Users should not have to wonder whether the same words, visual appearances, or actions mean something different in different parts of the experience.
  • Intuitiveness. The level of intuitiveness of the user interface has a direct impact on the effectiveness of user interactions with AI. As a designer, you need to ensure that the experience is delivered in a way that users would expect to have it. 
  • Contextuality. During the interaction, users should be provided with contextually relevant information (i.e., information relevant to the user’s current task and environment). The system should also maintain short-term memory and offer user-efficient references.
  • Error prevention. Error-prone conditions can be prevented by eliminating states where user actions can cause system errors (for example, do not allow users to input incorrect data in form fields by creating field constraints) and using autocorrect for errors. When error-prone conditions happen, help users recognize, diagnose, and recover from errors.

Define the heuristics

Consider AI design best practices when working on a set of heuristics. Here are a set of fundamental rules that are extremely important to follow when you design an AI-based product:

  • Make it clear what the system can do. Design features with appropriate disclosures built in. When the user asks about a certain feature, clearly explain what and how the user can use it.
  • Support efficient invocation. Make it easy to request AI services when needed. For example, when you design a voice-based system, the invocation process for this system should be as simple as saying, “Hey, [spotter word], do [X].”  
  • Learn from user behavior. The system should learn from user behavior. Remember recent interactions and offer valuable content and features according to the needs of the end user.
  • Create a feedback loop. Explain the system behavior to your users. When a user is confused by the behavior, they should be able to ask the system, “Why do I see what I see?” and the system should then make it clear why it performed as it did. 
  • Notify users about changes over time. When you introduce new features to the system, tell users about them. For example, when a new AI skill for ordering food is supported, the system might reach a user by saying, “John, you can order food for your dinner.”
  • Show contextually relevant information. Provide information or actions based on a specific situation or context of use. For example, when an AI system has access to the user calendar, and it knows that a user has a meeting in an hour, it can offer contextual features such as ordering a taxi or notifying the other person about the meeting.
  • Design for safety. Avoid unintended system behavior that creates harmful risks. The users should not be able to use AI-based systems to cause harm to themselves or other people.

Best practices and guidelines for AI design can help you to define more relevant mechanisms for evaluation of user experience.

Conduct the analysis

Perform heuristic analysis when you have a concept

The perfect moment to conduct a heuristic analysis is right after a team creates the first functional prototype of a product but before the team starts to code it. Conducting a heuristic analysis during this step will help to identify usability issues, and the cost of fixing them will be less than for issues found after development is finished.

Don’t do a heuristic analysis alone

A heuristic analysis relies on the experience and expertise of the evaluator. But no matter how good an expert is, they can suffer from cognitive bias, which is an error in thinking that affects the decisions and judgments that evaluators make. When a few specialists perform a heuristic evaluation, the individual evaluates the product separately. Due to this, the results of evaluations will be less biased because specialists spend time discussing every finding until they come to an agreement.

So how many evaluators (people who run heuristic analysis) should you hire? While every project is different and the number of evaluators may vary, generally it’s recommended to hire five individuals. Five experts will be able to discover approximately 80% of all usability issues.

Line graph for usability research.
The best results come from testing no more than five individuals. Image credit NNGroup.

Remember that your evaluators should not be your end users. They should typically be usability experts that preferably have domain expertise in the product’s industry.

Also, remember that even when you perform a heuristic evaluation with a group of experts, each expert should evaluate the product individually. The evaluations should be independent and unbiased.

Brief your evaluators

Experts should know exactly what they are supposed to do during the evaluation. Hold a briefing session, during which you should ensure the evaluators receive the same instructions. Specify the tasks that you want them to complete and ensure that all experts use the same set of heuristics. A set of heuristics will be used as a reference for the experts when they make their evaluation.

Create a report and assign a proper severity rating to each of the usability issues

An expected deliverable of a heuristic analysis is a consolidated report that contains identified usability issues along with their severity. A clear, prioritized list of usability issues will help UX designers focus on the most critical issues. Typically, those issues create roadblocks for users and prevent them from interacting with the product. For example, users cannot complete a certain task using an AI assistant (i.e., order taxi) because this violates the heuristic “visibility of system status.”

Heuristics analysis report shows the score in comparison to how many rules were followed for each heuristic.
Heuristics analysis report shows the score in comparison to how many rules were followed for each heuristic. Image credit Marli Ritter.

Select a proper reporting tool

Since you plan to share your findings with team members and stakeholders, you need to select a tool that allows you to do so. Depending on the needs of your project, the tool might be Google Docs or specific task management tools such as Jira or Trello. Just ensure that all team members agree to use it.

Be specific when describing the issue

All findings must describe the issues precisely. Vague notes such as “the signup error message doesn’t look good” will require an extra round of clarification, which leads to a decrease in productivity. A better way of phrasing this issue would be: “During the procedure of signup, many users see confusing error messages that prevent them from finding the root cause of the problem. The design of this message violates the #5 heuristic for design (Error prevention).” This approach will help you aggregate the expert’s notes.

Analyze results

At the end of a heuristic analysis, UX practitioners should aggregate the heuristic evaluation reports and build a list of usability issues that need to be fixed. It’s recommended to assign a proper code for each type of problem. For example, in #4 Discoverability, the team will be able to measure how many problems they have in an area, and this will help you create a design strategy that will be focused on eliminating the issues. It’s also recommended to visualize your findings to better communicate valuable information.

Use a heuristic evaluation sheet to simplify the process of feedback categorization.
Use a heuristic evaluation sheet to simplify the process of feedback categorization. Image credit Hsin-Jou Lin.
You can use a spider chart to visualize your findings to make it easier to see the compliance with the heuristics.
You can use a spider chart to visualize your findings to make it easier to see the compliance with the heuristics. Image credit uxmag.

Don’t rely on heuristic analysis as the only way to identify usability problems

It’s impossible to have a deep understanding of a product when you only have one source of data. That’s why it’s recommended to use multiple sources of data—both qualitative and quantitative. By conducting user interviews, running surveys, and analyzing data provided by analytics tools, in addition to performing a heuristic analysis, you will have a better picture of how your product performs in real-world scenarios.

Conclusion

When it comes to product design, it’s important to have a mechanism in place that allows you to evaluate your design decisions. A properly conducted heuristic analysis will help you to identify a substantial number of usability issues even before your users start to use the product. But to achieve optimal results, it’s vital to combine heuristics for design with other usability testing methods such as a cognitive walkthrough, contextual inquiries, and user interviews. By doing this regularly, you will have a big picture of the state of your product design.