Conversion rate optimization is a typical
product design task. It’s easy to optimize design when you know exactly what
element you need to adjust to achieve great results. But what do you do when
you have a few different elements that affect conversion and have a few
different hypotheses on how to change them? The answer is simple – you need to
validate your design ideas by conducting multivariate testing.
What is multivariate testing
Multivariate testing is a user testing technique for validating a design hypothesis. The product team modifies multiple variables and uses multivariate testing to determine which combination of variations performs the best out of all of the possible combinations.
The word ‘variable’ might sound a bit
jargony, but in reality, it’s an element you will modify. In a multivariate
test, a page/screen is treated as a combination of elements (headlines, images,
buttons, text blocks, etc.). All elements have some impact on the conversion
rate. So when we conduct multivariate testing, we decompose a page/screen into
distinct elements and create variations of those elements. Our goal is to find
a combination that will provide the best conversion.
For example, in the context of landing
pages that advertise a specific service, the conversion goal can be sign-ups.
You can define two variables – an image that demonstrates the service, and a
heading section that provides basic information about it. You might want to
test two versions each of a heading and an image on a webpage. To test all the
versions, you create a combination of all the variations (2 X 2 = 4), as shown
- Image A + Header A
- Image A + Header B
- Image B + Header A
- Image B + Header B
How many variations should I create?
There is no single correct answer to this
question. It all depends on the nature of your product. But you need to select
the elements you want to modify carefully. The more elements you change, the
more variations you get, and the more complex the testing becomes.
The total number of variations in a
multivariate test can be easily calculated using the following formula:
of variations of element A] x [Number of variations on element B] … = [Total number of variations]
Multivariate vs. A/B testing
Even in the UX design and user research
field, many specialists consider multivariate testing and A/B testing
synonymous. Conceptually, the two techniques are similar, but there are crucial
The key difference between multivariate testing and A/B testing is the number of variables tested. This means that A/B testing only tests one variable, but in a multivariate test, multiple variables are tested together.
Another difference is the purpose of
testing. A/B testing is useful in two cases:
- When a team wants to optimize a
particular element on a page. For example, you want to know what color used for
the call-to-action button on your landing page converts better.
- When a team wants to introduce
a radical change in design. For example, a team has two different versions of
landing pages and want to know which one converts better.
Multivariate testing, on the other
hand, uses the same core mechanism as A/B testing but compares a higher number
of variables. As a result, it reveals more information about how these
variables interact with one another. Multivariate testing helps measure the
effectiveness of each design combination and select the one that works best for
But don’t let the differences between A/B
testing and multivariate testing make you think of them as opposites.
Multivariate testing and A/B testing can work excellent together. For example,
you can conduct a series of A/B tests to explore radically different ideas.
After you choose the one design that demonstrates the best conversion, you can
use multivariate testing to understand the elements you need to polish to
generate even better conversion.
Benefits of multivariate testing
Save time on running test
If conducted properly, a multivariate test
can eliminate the need for running a series of A/B tests on the same page with
the same goal. It will be much easier to analyze test results.
Finding design direction
Multivariate testing is a powerful way to
help you target redesign efforts. Designers will know what element (or
elements) on the page have the most impact, and this knowledge will make their
work more focused.
Downsides of multivariate testing
Requires a lot of traffic
The amount of traffic needed to complete
multivariate testing is much larger in comparison with other types of testing.
In a multivariate test, traffic can be split into quarters, sixths, or eighths
segments. It’s hard to conduct this type of testing when your site has a small
number of daily visitors. That’s why this testing doesn’t work very well for
new companies (simply because they don’t have much traffic).
If you don’t have large traffic, consider
using an A/B test instead of a multivariate test. In A/B testing traffic for an
experiment is split in half, with 50 percent of traffic visiting each
Need to select variables carefully
It’s a common situation when one or more of
the variables being tested do not have a measurable effect on the conversion
goal. That’s why it’s so important to review variables before testing. Always
evaluate variables from the user’s perspective. For example, when a user scans
a landing page, they probably will pay more attention to the service
description, rather than the color of the signup button.
How to run multivariate testing
We discussed the multivariate testing rules of thumb before. Here we will extend the list with a few more practical tips.
Define the conversion goal
You need to define a specific conversion
goal for every page/screen you test (in conversion optimization, such goals are
called local). Local goals should be connected to the ultimate goal you want to
achieve. Ultimate goals are usually defined by the sales and marketing team.
For example, on a particular page, you want
users to watch a video about using the service. This video provides all the
essential information about your service and helps them decide to sign up. In
this case, the local goal is to make users watch the video, but the global goal
is to have new signups.
The goals should be clearly stated before
starting testing because you will determine which version of the page is most
effective based on that.
Select pages for testing
It’s recommended to use multivariate
testing on the most important pages on your website – pages that generate
Minimize the number of variables
As was mentioned before, in multivariate
testing, not all variables will have an equal impact on the conversion rate.
That’s why it’s recommended to minimize the number of variables in the test.
Every variable you add makes the test more complex. For example, if you’re
testing a headline and image for your landing page, then there are a total of
four combinations (2 × 2). But if you add a call-to-action button to the test,
there are suddenly eight combinations to test (2 × 2 × 2). The more
combinations, the more traffic you’ll need to get significant results. Thus,
decide which variables are most important to the conversion goal.
Preview combinations before testing
One of the common pitfalls of multivariate
testing is changing the values of variables without reviewing the design after
that. It often leads to odd-looking, or even worse, illogical designs. For
example, by blindly change the description and call-to-action label on a
landing page, you can end up in combinations “Fixed price $20 per month” text
block and call to action button “Get started for only $15 per month.” That’s
why you should always preview your design before sending it to test
Estimate the traffic required for significant results
As mentioned in the section multivariate vs a/b testing, multivariate testing is recommended only for sites that have a substantial amount of daily traffic. That’s why before testing, get a clear idea of how much traffic you’ll need to get statistically significant results. To make things easier, you can use a multivariate testing duration calculator. It will help you to estimate how much traffic your test will require.
Carefully select an automatic tool for testing
Keep in mind that not all A/B testing tools
support multivariate testing, so be sure to check that your tool allows it.
Here are a few tools for multivariate
- Google Optimize. This tool allows you to run tests on your website’s content to learn what works best for your visitors, including A/B and multivariate tests.
- VWO (Visual Website Optimizer). This tool allows you to create a multivariate test visually
Use multivariate testing together with other methods
Don’t think of multivariate testing in
isolation. Think of it as a powerful optimization method that can complement
other testing methods that you can apply. For that, you need to consider how
multivariate testing will fit into the design process and particularly the
testing phase of this process.
Multivariate testing is an excellent
technique that can demonstrate the quantifiable impact of a design change. The
fact that multivariate testing reveals more information about how variables
interact with one another often leads to valuable insights. It’s also an
excellent tool for convincing your team moving into a certain direction – the
data uncovered in multivariate testing removes doubt and uncertainty from the
process of optimization.