Illustration by Erica Fasoli

As much as we would like to think that our users are like us, they usually are not. To design a good user experience, we need to understand who our users are, what they need, and why.

User testing is a type of user research that helps you understand how people interact with your product. You can then use this information to improve your product design. But to get the most from this research, it’s essential to ask the right questions. A wrong set of questions can nullify the benefits of the usability session and lead product development down the wrong path. The right user experience testing questions help you gather unbiased user opinions, which ultimately leads to better user experiences.

In this article, we will explore three major topics related to usability testing:

  • Usability questions to ask before, during, and after the testing;
  • Do’s and don’ts of user testing questions;
  • Three useful techniques for inquiry without leading.

Define the objective of the user testing

Before you start preparing a list of questions, you need to define the objective of the user testing. What do you want to achieve, and why? A clear objective makes it much easier to create a proper list of usability questions.

Generally, user testing can be helpful when you want to:

  • Validate a hypothesis. Product design hypotheses typically sound like “If we [build something], then our users can [do something that brings them value].” You can validate a hypothesis by building a prototype and testing it with your users.
  • Find issues in existing products. You want to identify areas of friction that users face when they interact with your product. In this case, you use existing products and test them with your users.
  • Understand how people interact with similar products. You want to identify the benefits and drawbacks of products created by your rivals. As a part of competitor analysis, you ask test participants to interact with products created by your competitors.

While each case requires a different set of usability test questions, it is still possible to identify core questions that can be useful in every testing scenario. Here are three large groups of user testing questions:

  1. Screening questions. A set of questions intended to evaluate a test participant’s qualifications.
  2. In-test questions. A set of questions directly related to your testing objective. In-test questions include both general questions and specific questions about your product.
  3. Post-test questions. A set of questions you ask at the end of the session. These might include clarifying questions.

1. Screening questions

Finding test participants is the first step in any user testing. The people you invite to testing should represent your target audience—they should be your real or potential users. You need to specify a criterion that defines your ideal participant and evaluate all your candidates according to this standard. The user testing questions that you ask during the screening process should help you understand both the demographics and experience of your candidates.

Screening is the process of selecting the right test participants.
Screening is the process of selecting the right test participants. Image credit Adobe Stock.

Here are a few questions you should consider asking during the screening process:

How old are you?

Confirming someone’s age is important because participants in the wrong age range could make test results unreliable. For example, if you design a product for middle-aged people, it’s better to avoid inviting young people. “How old are you?” will help you verify that test participants are within the desired range.

What’s your highest level of education?

The background of your test participants often informs their actions and opinions as they interact with your product. Education level provides insight into the type of work a test participant may be doing and how they do it. For example, people with higher technical education likely use critical thinking in their work, and this skill makes it easier to interact with the product’s interface. In many cases, they will be able to understand how a feature of a product works, even when this feature isn’t designed very well or doesn’t have a proper introduction.

What’s your income level?

If you offer paid services on your website, you need to ensure that test participants can afford the service. It doesn’t make much sense to invite test participants to interact with a product that offers paid services when they won’t try this product in the future (just because they won’t be able to afford it). In these cases, the test results will be highly biased.

What habits do you have?

Questions like, “Can you tell me about your hobbies?” or “What does your typical day look like?” let test participants talk freely about their daily routine. These questions are icebreakers, and at the same time, they help you gain more insight into your participants’ behaviors (what they like and dislike) and reveal if they have any activities relevant to your research. Ask any lifestyle questions that are related to your topic or product.

How much time do you spend online?

Depending on your objectives, you might need users with varying levels of experience with digital products. Generally, the more time people spend using digital products, the more familiar they are with common interaction techniques and user interface patterns. As a result, experienced users can provide more detailed feedback.

What apps and websites do you use on a regular basis?

This user testing question will help you understand which services and related features the users interact with the most on existing apps or sites. Suppose you want to conduct user testing of your eCommerce app. What you might want to know is how confident the test participant is with online shopping tasks. If this person doesn’t have any eCommerce website on their list, they are probably not interested in online shopping, and you may want to find another candidate.

How often do you [do something]?

This question will help you evaluate the user’s experience level with digital products. For example, if you test an eCommerce app, you might want to ask, “How often do you shop online?” The answer will help you identify the person’s familiarity with online shopping as well as their buying habits.

Which device do you typically use to [do something]?

The answer will determine which devices (desktop, tablet, mobile) users are most comfortable using when they complete a particular task. This information will be helpful when you test a particular type of interaction; for example, how well your eCommerce checkout works for mobile users. If a test participant prefers to use desktop for online purchases and, in very rare cases, switches to mobile, test results with a mobile version of your design won’t be representative for this user.

Have you used this product before?

This question will help you find out if the participant is familiar with your product or not. Existing knowledge may impact participants’ opinions or their ability to use your product. Depending on the goal of your testing, previous experience can either be a benefit or a drawback. For example, when you complete a major redesign of your website, you might want to validate it by testing with existing users to see that everything works fine for them. In this case, previous experience working with your website will be a major plus.

2. In-test questions

These user testing questions will help you discover important insights about user behavior when they interact with your product. In this phase, you allow participants to interact with your product, observe their interactions, and ask relevant questions. Aim to collect data that explains user behavior (i.e., why users make certain actions, why they prefer one option versus another, etc).

Can you name some competitors of this product?

This question will help you identify well-known competitors. It can be insightful to hear who test participants consider your competitors, which may be different from the ones you’ve already identified as such or didn’t think of as competitors.

What features do you find most valuable and why?

Not all of the features that you offer are equally valuable to your target audience. Sometimes it can be surprising to know that a relatively insignificant feature offers a lot of value to users. This question will give you tremendous insights into what the majority of users find the most valuable about your product and why they think this way.

Can you give me a few examples of real situations when you would do [something]?

When test participants give you specific scenarios of interaction, the feedback becomes less abstract and more specific. This question makes test participants think about real use cases rather than generic ones. Oftentimes, participants describe cases that you may have never considered.

What do you think about this design?

Ask this question when you want to learn more about how users feel about your website or app. Open-ended questions like this one are great for collecting personal opinions about specific aspects of your design (such as aesthetics). It’s possible to ask this question when you want to get a general impression of your page/screen or individual elements such as icons or microcopy. It’s easy to turn this qualitative feedback into recommendations for your design.

What prevents you from completing a task?

This question should be asked in the context of an assignment. When you see that your test participants cannot complete certain actions or operations, ask them about the problems that they face along the way. This question will help you identify the roadblocks that users face while interacting with your product. The barriers might be something as simple as a lack of important information or more complex stoppers such as dead-end pages (pages that prevent users from completing a task because the next step is missing).

I noticed you [did something]. Why?

This contextually relevant question comes from observation during the test. It’s an ideal question for the situation when a session moderator notices some interesting user behavior and wants to know the rationale behind it. The answer will help you reveal the motivations behind user actions.

3. Post-test questions

The moment right after the user testing session has been completed is a perfect time to ask test participants if they have questions or anything to add so you can collect more of their personal opinions.

How difficult are these test assignments? (1=very easy, 5=very difficult)

You need to ensure the instructions you share with participants are crystal clear. No matter how much time you invest in polishing test assignments, there is always a chance that something might be hard to understand for users. This question will help you to identify what assignments require additional attention.

How would you describe your overall experience with this product?

By asking this question, you make participants think about their end-to-end user experience. They will likely evaluate their journeys in your product and name a few good areas and a few areas that were bad in terms of UX. This information will help you to prioritize changes in user flows.

Alternatively, it’s possible to ask the question, “What was your overall impression of this product? Rate it on a scale from 1 (Terrible) to 5 (Excellent).” Once participants answer and provide their responses, you can follow up with “why” to allow them to elaborate on their thoughts.

If you could change one thing in this product, what would it be and why?

This compelling question allows you to identify the top-priority change for your test participants. It might be a feature they miss or more intuitive navigation. The insights you get from this question will help you make your product better for your target audience.

What do you expect to see in our product in the future?

Product design backlog should always be aligned with users’ needs. When multiple test participants name the same feature that they want to see in the future, this is a clear signal that you will need to prioritize your backlog.

Do’s and don’ts of user testing

Now that you know what usability test questions to ask your test participants, it’s time to discuss how to do it properly. In this section, I want to share some practical recommendations that will help you to gather more valuable user insights:

Do: Use think-aloud protocol

Think-aloud protocols involve participants thinking out loud as they are interacting with a product and performing specific tasks. It helps researchers understand the user’s state of mind at every important step of the testing process and gives researchers an opportunity to ask contextual questions.

Do: Ask more open-ended questions

If your goal is to collect qualitative data, you should ask more open-ended questions. An open-ended question is a question that cannot be answered with a “Yes” or “No” response. Open-ended usability test questions can help researchers gather more detailed feedback from participants. Thus, instead of asking “Do you like or dislike this [feature]?” ask questions like, “What do you think about this [feature]?”

Do: Watch participant’s facial expressions and body language

Emotions play a significant role in product design. The way a product makes users feel has a direct impact on user engagement. But emotions are a tricky concept to master, and it can be hard to understand the participant’s emotional state because not all people are willing to vocalize their feelings. That’s why it’s vital to watch facial expressions and body language during user testing. Observe user interactions, focus on non-verbal cues, and you will identify areas in your product that people like/dislike. If you’re interested in learning more about decoding human emotions, I highly recommend reading the book Emotions Revealed by Paul Ekman Ph.D.

Do: Ask follow-up questions

The goal of user testing questions is to have discussions with the participants and follow-up with questions such as “Why do you think so?” will help you with that. Even when users have specific thoughts or opinions, not all of them are conscious of the “why” behind their thoughts/opinions. A friendly conversation where the researcher is genuinely interested in hearing a personal opinion will put participants at ease and result in more valuable insights.

Use the “Five Whys” technique, which involves asking “why” five times to understand the participants intention or point of view. Image credit Nick Babich.

Do: Record answers on video

Video recording can be an excellent way to collect information because it allows you to capture every detail of the session. The video can be an excellent source of further in-depth analysis. Of course, any time you want to record your participants, you need to ensure that they are okay with that. Always get written permission before starting the recording and be ready to abandon it if usability questions feel uncomfortable.

Recording the usability session will allow you to refer back to it later.
Recording the usability session will allow you to refer back to it later. Image credit Adobe Stock.

Don’t: Ask leading questions

Questions should guide the conversation, not lead it. People interpret user testing questions differently based on how they are worded. Leading questions (questions that lead participants to specific answers) can be a huge problem for user testing sessions because they can cause incorrect insights.

Here are a few examples of leading questions that you should avoid:

  • Do you like our new design more than the old one? This leading question implies the new design is better than the older one.
  • Does the app help you work more efficiently? Test participants will focus on finding the relevant example of work efficiency.
  • When do you think that feature X would be helpful? This question focuses users on thinking about specific cases for the feature.

Don’t: Provide too many details

The more information you share, the more likely participants will be overwhelmed by the details. Since participants think that all information that you share with them is equally important, they will try to memorize the details. But a human’s short-term memory cannot store a great deal of information, and it’s easy to forget some crucial details when you hear a large amount of information. Thus, try to share only essential details—information that participants need to know to participate in the session successfully.

Don’t: Provide too much guidance

Too much guidance or assistance from the usability session moderator can prevent participants from exploring your product in a natural way. Some participants might even consider too much guidance as a restriction.

Here are a few practical tips to remember to avoid negative impression:

  • Avoid distracting participants when they’re already in the process of completing tasks. By doing that, you increase the risk of user mistakes.
  • If you see that a participant is going to make a mistake, do not interrupt them. Instead, allow them to fail and watch what happens next. By doing that, you will collect more detailed information on user behavior (i.e., see unexpected interaction patterns).
  • When participants ask for your help, you need to encourage them to find the solution by themselves. The only exception to this rule is a situation where there is no solution to their problem yet. In this case, it’s better to tell participants about it to save time.

Don’t: Judge the participants

Resist the urge to judge or educate your participants. You invite people to hear their genuine thoughts and opinions about your product, not the other way around. That’s why it’s always recommended to start the conversation with the sentence: “Remember, when I ask you some questions about this product, I want to hear your honest opinion. There are no right or wrong answers.” and stick to this rule during the testing.

Don’t: Take user words for granted

Not everything that participants say or do during the session represents their real behavior. When users interact with your product in a lab environment, their actions and opinions can be biased. Thus, when you have a clear outcome from a testing session and form a certain product hypothesis, you need to validate this hypothesis using other testing methods.

3 Testing techniques to inquire without leading

What should a researcher do when a participant asks them a question? Well, if you don’t know what to say or are afraid to hint the participant, you can stay quiet. However, it’s possible to do better in this situation. Echo, Boomerang, and Columbo are three useful approaches for usability testing suggested by Kara Pernice of NNGroup. All techniques will help you to respond to questions without stifling.

Effective user testing facilitation techniques. Video credit Nielsen Norman Group.

Let’s quickly overview each approach:

Echo

With the echo technique, the researcher repeats the last phrase or word the participant said and makes it sound like a question to encourage the participant to provide more details. For example, when a participant says, “This feedback form works strange,” the researcher can respond to this phrase with a question: “Works strange?” This question will prompt the participant to provide more detail and clarify their idea.

But don’t overuse this technique. Participants can be easily annoyed to hear an echo question every time they share their opinion. Try to use echo only when the participant shares something that is incoherent.

Boomerang

As you probably guess from the title, this technique is all about returning the participants’ own questions back to them. This technique is good for situations when participants ask for your opinion or help. For example, a participant says: “Should I click this button to complete this operation?” Since the session moderator doesn’t want to guide the participant, they can reply: “What do you think?” or “What would you do if you were alone?” But remember that you need to ask the questions in a polite way. Participants shouldn’t sense any notes of pressure or judgment in your tone.

Columbo

Named after a famous character, detective Columbo, portrayed by Peter Falk, this approach helps create the perception that the researchers are less aware and knowledgeable than they actually are. The goal of this technique is to make users feel like they should help a researcher.

Here is an example of how this technique works:

  • User: “When I click the [x] button in this window, will I leave this app without saving data”
  • Researcher: “So… you want to… (say only the part that you think is safe to say, pause and wait for the participant to fill the silence)
  • User: I’m not sure how [x] works. I believe that when I press [x] the app will save the data automatically. But there is no information about it anywhere in this window. (The user continues the conversation and shares their thoughts about the problem, the insights help you identify the area for improvement.)

The Colombo technique is more advanced than the Echo and Boomerang techniques, and it requires some preparation. You shouldn’t overreact because it will make you look disingenuous. You should always use non-leading questions and do it on the fly (which can occasionally be difficult).

Good user testing leads to better user experience

User testing questions play a key role in learning about your users. When you understand how real users feel and think when they interact with your product, you can use this information to make a user’s interaction with your product efficient and pleasant. Think of user testing as an investment. The more time and effort you invest in it, the more time and effort you save down the road because every product design decision will be based on actual user behavior.