Conversion rate optimization is a typical product design task. It’s easy to optimize design when you know exactly what element you need to adjust to achieve great results. But what do you do when you have a few different elements that affect conversion and have a few different hypotheses on how to change them? The answer is simple – you need to validate your design ideas by conducting multivariate testing.

What is multivariate testing

Multivariate testing is a user testing technique for validating a design hypothesis. The product team modifies multiple variables and uses multivariate testing to determine which combination of variations performs the best out of all of the possible combinations.

The word ‘variable’ might sound a bit jargony, but in reality, it’s an element you will modify. In a multivariate test, a page/screen is treated as a combination of elements (headlines, images, buttons, text blocks, etc.). All elements have some impact on the conversion rate. So when we conduct multivariate testing, we decompose a page/screen into distinct elements and create variations of those elements. Our goal is to find a combination that will provide the best conversion.

For example, in the context of landing pages that advertise a specific service, the conversion goal can be sign-ups. You can define two variables – an image that demonstrates the service, and a heading section that provides basic information about it. You might want to test two versions each of a heading and an image on a webpage. To test all the versions, you create a combination of all the variations (2 X 2 = 4), as shown below:

  • Image A + Header A
  • Image A + Header B
  • Image B + Header A
  • Image B + Header B
Image for web
Multivariate testing. Image by vwo.

How many variations should I create?

There is no single correct answer to this question. It all depends on the nature of your product. But you need to select the elements you want to modify carefully. The more elements you change, the more variations you get, and the more complex the testing becomes.

The total number of variations in a multivariate test can be easily calculated using the following formula:

[Number of variations of element A] x [Number of variations on element B] … = [Total number of variations]

Multivariate vs. A/B testing

Even in the UX design and user research field, many specialists consider multivariate testing and A/B testing synonymous. Conceptually, the two techniques are similar, but there are crucial differences.

The key difference between multivariate testing and A/B testing is the number of variables tested. This means that A/B testing only tests one variable, but in a multivariate test, multiple variables are tested together.

Another difference is the purpose of testing. A/B testing is useful in two cases:

  • When a team wants to optimize a particular element on a page. For example, you want to know what color used for the call-to-action button on your landing page converts better.
  • When a team wants to introduce a radical change in design. For example, a team has two different versions of landing pages and want to know which one converts better.

Multivariate testing, on the other hand, uses the same core mechanism as A/B testing but compares a higher number of variables. As a result, it reveals more information about how these variables interact with one another. Multivariate testing helps measure the effectiveness of each design combination and select the one that works best for you.

a/b testing vs multivariate testing model
A/B testing vs. Multivariate testing. Image by Dynamicyield.

But don’t let the differences between A/B testing and multivariate testing make you think of them as opposites. Multivariate testing and A/B testing can work excellent together. For example, you can conduct a series of A/B tests to explore radically different ideas. After you choose the one design that demonstrates the best conversion, you can use multivariate testing to understand the elements you need to polish to generate even better conversion.

Benefits of multivariate testing

Save time on running test

If conducted properly, a multivariate test can eliminate the need for running a series of A/B tests on the same page with the same goal. It will be much easier to analyze test results.

Finding design direction

Multivariate testing is a powerful way to help you target redesign efforts. Designers will know what element (or elements) on the page have the most impact, and this knowledge will make their work more focused.

Downsides of multivariate testing

Requires a lot of traffic

The amount of traffic needed to complete multivariate testing is much larger in comparison with other types of testing. In a multivariate test, traffic can be split into quarters, sixths, or eighths segments. It’s hard to conduct this type of testing when your site has a small number of daily visitors. That’s why this testing doesn’t work very well for new companies (simply because they don’t have much traffic).

If you don’t have large traffic, consider using an A/B test instead of a multivariate test. In A/B testing traffic for an experiment is split in half, with 50 percent of traffic visiting each variation.

Need to select variables carefully

It’s a common situation when one or more of the variables being tested do not have a measurable effect on the conversion goal. That’s why it’s so important to review variables before testing. Always evaluate variables from the user’s perspective. For example, when a user scans a landing page, they probably will pay more attention to the service description, rather than the color of the signup button.

How to run multivariate testing

We discussed the multivariate testing rules of thumb before. Here we will extend the list with a few more practical tips.

Define the conversion goal

You need to define a specific conversion goal for every page/screen you test (in conversion optimization, such goals are called local). Local goals should be connected to the ultimate goal you want to achieve. Ultimate goals are usually defined by the sales and marketing team.

For example, on a particular page, you want users to watch a video about using the service. This video provides all the essential information about your service and helps them decide to sign up. In this case, the local goal is to make users watch the video, but the global goal is to have new signups.

The goals should be clearly stated before starting testing because you will determine which version of the page is most effective based on that.

Select pages for testing

It’s recommended to use multivariate testing on the most important pages on your website – pages that generate conversion.

Minimize the number of variables

As was mentioned before, in multivariate testing, not all variables will have an equal impact on the conversion rate. That’s why it’s recommended to minimize the number of variables in the test. Every variable you add makes the test more complex. For example, if you’re testing a headline and image for your landing page, then there are a total of four combinations (2 × 2). But if you add a call-to-action button to the test, there are suddenly eight combinations to test (2 × 2 × 2). The more combinations, the more traffic you’ll need to get significant results. Thus, decide which variables are most important to the conversion goal.

Preview combinations before testing

One of the common pitfalls of multivariate testing is changing the values of variables without reviewing the design after that. It often leads to odd-looking, or even worse, illogical designs. For example, by blindly change the description and call-to-action label on a landing page, you can end up in combinations “Fixed price $20 per month” text block and call to action button “Get started for only $15 per month.” That’s why you should always preview your design before sending it to test participants.

Estimate the traffic required for significant results

As mentioned in the section multivariate vs a/b testing, multivariate testing is recommended only for sites that have a substantial amount of daily traffic. That’s why before testing, get a clear idea of how much traffic you’ll need to get statistically significant results. To make things easier, you can use a multivariate testing duration calculator. It will help you to estimate how much traffic your test will require.

Carefully select an automatic tool for testing

Keep in mind that not all A/B testing tools support multivariate testing, so be sure to check that your tool allows it.

Here are a few tools for multivariate testing:

  • Google Optimize. This tool allows you to run tests on your website’s content to learn what works best for your visitors, including A/B and multivariate tests.
  • VWO (Visual Website Optimizer). This tool allows you to create a multivariate test visually
screenshot of VWO testing optimizer
Testing combinations using VWO.

Use multivariate testing together with other methods

Don’t think of multivariate testing in isolation. Think of it as a powerful optimization method that can complement other testing methods that you can apply. For that, you need to consider how multivariate testing will fit into the design process and particularly the testing phase of this process.

Conclusion

Multivariate testing is an excellent technique that can demonstrate the quantifiable impact of a design change. The fact that multivariate testing reveals more information about how variables interact with one another often leads to valuable insights. It’s also an excellent tool for convincing your team moving into a certain direction – the data uncovered in multivariate testing removes doubt and uncertainty from the process of optimization.