Illustration by Erica Fasoli
When it comes to modern digital product design, we don’t have a shortage of data. Modern analytic tools and UX research methods allow us to collect significant data about our users and how they interact with our products or services. Collecting, analyzing, and properly using data is key to creating good user experiences. When designers don’t take data into account and rely solely on their instincts, they risk wasting time and money on creating ineffective solutions. However, designers who put too much emphasis on data can limit their ability to create innovative solutions. So when should you practice data-driven vs. data-informed decision making? You will find answers in this article.
Why use data in the design process?
Some product designers believe that they know what their users need and want from the beginning of the design process. Needless to say, this way of thinking usually leads to bad design decisions. The False-Consensus effect is a type of cognitive bias that can have a major negative impact on the outcome. Jakob Nielsen summarized the risk of this bias, saying: “One of usability’s most hard-earned lessons is that ‘you are not the user.’”
In most cases, designers are not their target users. Without data that supports a hypothesis about certain experiences, designers often create products based on their own assumptions. Data gives insight to designers, and those insights can help direct product design in the right way.
What’s the difference between data-driven and data-informed design?
Data-driven and data-informed design represents different two approaches of working with data. For data-driven design, data is paramount—the team puts data at the center of their design decisions, and data becomes a primary input. When the team discusses a specific design decision, every solution to a problem is evaluated in accordance with the data the team has.
Data-informed decision making implies that the product team uses data to inform their decisions, but they use it as a reference. In other words, the product team uses data as a source of information along with other inputs such as design intuition and qualitative feedback.
The difference between data-driven vs data-informed design. Image by techsauce.Now let’s review the data-driven vs data-informed decision-making processes.
When to be data-driven
The data-driven approach works well when product teams want to find answers to the “What” and “How many” questions (i.e., “What’s happening?” and “How many users [do something]?”) This approach is beneficial when a team wants to optimize a specific part of the product. For example, if a product team has the goal of reducing the bounce rate on a landing page, quantitative metrics such as average time on a page or time-to-load can help them understand when their users face roadblocks.
When it comes to data-driven design, it’s important to remember that:
- Data-driven design isn’t just about gathering as much data as possible. Data-driven design is about collecting data that can help you find insights about user behavior and then using this knowledge to help make a better product. Thus, it’s essential to understand user needs and business goals first. Invest your time and effort in user research and select a set of metrics that match your business goals.
- A data-driven approach typically requires collecting a significant amount of data, to have a statistically significant result. This technique might not be suitable for small businesses and startups simply because they won’t have enough data.
- It’s important not to go overboard with a data-driven approach. Some companies try to over-optimize simple parts of their products which often leads to design frustration rather than good user experience. A most notable example is Google’s testing 41 shades of blue. It’s always important to remember what you want to test and why.
Now that you know what data-driven design is, here are two approaches that a design team can follow:
A/B testing & multivariate testing
A/B testing is the practice of showing two versions of a product to different users and comparing which variant has better performance. A/B testing requires changing one specific element in a design (typically, it’s small changes such as the color of a call-to-action button). When it comes to measuring performance, teams who perform A/B testing typically measure a level of conversion (e.g., for a landing page, this might be the number of new sign-ups). If you’d like to see some examples of how A/B testing works for real products, I highly recommend checking 37Signals’ A/B testing case study on homepage design.
A/B testing of landing pages conducted by 37Signals. Image by signalvnoiseMultivariate tests, on the other hand, involve changing more than one element at the same time (i.e., the color of the call-to-action button and the hero image). All elements that a team changes have a direct impact on conversion. When teams run multivariate testing, they create a set of elements and test those sets to determine which one performs the best.
Running A/B or multivariate tests on a regular basis will help you to fine-tune your design for the needs of your target audience.
Analytics
Analytics help measure user activity in your product—what parts of your product are more valuable for your users and how exactly they use your product. Analytics typically provide quantitative metrics such as unique page/screen views, how many pages viewed per single session, average time on a page and bounce rate (for websites), etc. Metrics help you measure user interaction (i.e. understand what content your users are interested in).
Analytics can be a first step on the way to creating a big picture view of a user journey for your product. It will help you to understand what user behavior is driving the metric.
When to be data-informed
A data-informed decision making approach works well when product teams want to find answers to the “Why?” questions (“Why is something happening or not happening?” or “Why do users do what they do?”). Figuring out why users behave in certain ways is a significant part of human-computer interaction. That’s why data-informed design is more relevant when it comes to strategic, fundamental design decisions. For example, a product team wants to increase long-term user engagement, so the team focuses on specific product features in an attempt to make the product more valuable. The VMware design team proved the effectiveness of data-informed design in their case study Wavefront by VMware. They used data to increase the free trial activation rate by 30%.
Innovative design is another reason to use data-informed decision making. Companies like Facebook follow this approach to reveal user behavior patterns and create innovative products based on that information.
Qualitative research plays a significant role in data-informed design. User research methodology recommends using contextual inquiries and user flow analysis for data-informed design.
Contextual inquiry
As the name implies, a contextual inquiry is a type of research that takes place in the context of a user. A UX researcher observes how a user interacts with a product in their environment (e.g., their workspace). The researchers might want to clarify some interactions and ask questions like, “Why did you tap on that button?” or “What do you think about this error pop-up message?” This method helps researchers to understand human behavior, habits, and expectations.
Helpful tips on gathering information about users with contextual inquiry. Video Credit UX Mastery.User flows analysis
User flows are the ways users travel through a product, from the entrance point (for a website, this can be the page they first land) to the exit point (for a website, this can be the last page they view before leaving the site). Typically, UX designers have a certain flow that they want the user to follow (the so-called, happy path). However, the actual flows might differ greatly from what’s expected because users can face troubles along the way. The easiest way to understand what flow the user follows when using a product is to ask users to complete a specific task, watch how they do it, and ask specific questions at the end.
Conclusion
Good product design comes from the balance between data and design intuition. That’s why it’s important to use both data-driven and data-informed approaches in your product design. It’s always helpful to think of data not as numbers but as something that supports your design decisions. Collect, analyze, and make design decisions in accordance with data you have, but don’t forget to validate those decisions by testing them with users.