Every designer, at least once in his or her life, has faced a situation where they had several solutions for the problem and wasn’t sure which solution would work the best for users. Imagine a situation where you’re working on a landing page for a product and have a few different layouts to choose from. All of them look great and it’s hard to choose one that you will use in your final product. Thankfully, there is a simple solution for this problem – A/B testing.

In this article, you will read about the technique of A/B testing and how it is applicable to the product design process.

What is A/B testing

A/B testing (also known as split testing or bucket testing) is the act of running a simultaneous experiment between two or more pages or screens to see which performs the best. And by ‘performs’ we usually mean converts.

A/B testing can improve your bottom line

Proper A/B testing can gather empirical data that will help your team figure out exactly which design decisions or marketing strategies work best for your product. A/B testing is extremely valuable for product design teams because it helps them learn why certain elements of their experiences impact user behavior. This knowledge will help teams make data-informed design decisions as well as be more specific in conversations with your stakeholders (you will use “we know” instead of “we think”).

Where we can use A/B testing

A/B testing is applicable to almost any design decision. Headlines, calls to action, images, search ads – you can test everything that you can change. Of course, the fact that you can test everything doesn’t mean that you should test everything. It’s vital to focus on the design decisions that provide the maximum value for you and your users.

How to run A/B testing

A/B testing is a relatively simple procedure. All you need to do is prepare two (or more) versions of a test page/screen and send users to the pages or screens. Usually, user traffic is randomly assigned to each page variant based upon a predetermined weighting.

Below is a simple 6-step framework for A/B testing:

1. Use analytics data to identify areas for optimization

Your analytics can provide valuable insight into where you can start optimizing. If you want to optimize a conversion rate on your app or website, it’s recommended to start with highly trafficked areas because they will help you gather valuable data faster.

2. Define conversion goals

The goal is an action that you count as a conversion. For example, in the context of a landing page, the goal can be signing up for the product updates.

3. Generate hypotheses on how to improve the conversion

Prepare a list of ideas on how to improve the current conversion rate.

As soon as you prepare the list, you need to review each idea and evaluate it with your team (consider both expected impact and the difficulty of implementation). In the end, you will have a prioritized list of ideas that you will use in your design.

4. Create design variations

Start with the top priority ideas and make the desired changes to an element of your app or website.

5. Run experiment

Allow real-world users to interact with your design variations and track their progress. A few tools where you can experiment with A/B testing are Adobe Target, Optimizely, and Crazy Egg.

6. Analyze results

Once your experiment is complete, analyze the results. The tool that you will use for A/B testing should help you determine whether changing the experience had a positive, negative, or no effect on visitor behavior.

A/B testing checklist

Despite the procedure of A/B testing being relatively simple, it’s vital to remember a few important rules:

Decide what exactly you want to test

The first thing to do when planning an A/B test is to figure out what you want to test. A/B testing works best for one-variable design decisions when you need to test one thing at a time. You need to create two different versions of one piece of content, with changes to a single variable. For example, in the context of a button, a single variable can be a button’s color, shape, label, etc.

Two computer screens with one element changed for A/B testing.
A/B testing checklist. Image by ribkhan.

Testing more than one thing at a time (i.e. headline and call to action buttons) is a multi-variate test that is more complicated to run.

Define metrics you want to collect

Before you start testing you should have a clear idea of the results you’re looking for. It’s vital to know your baseline result (i.e. the current conversion rate).

Run test simultaneously

With A/B testing you can’t test one variation today and another one tomorrow. Why? Because some variables can change drastically due to a time shift. For instance, the number of visitors and their interests can vary drastically depending on the day of testing. If you’re testing your landing page on day one of a promo campaign, the results of the testing can vary significantly with day two. That’s why it’s important to run simultaneously to account for any variations in timing.

Give a test sufficient time

Not giving each test sufficient time to run is a typical problem that many product teams face. This problem happens when the team incorrectly decides the time for the testing. As a result, it has a limited number of test participants and non-representative test results at the end of the testing period. Considering the importance of A/B testing and the audience you have, it’s worth dedicating a few days or even weeks to properly conduct tests.

Generally, the time for testing can be calculated based on two variables:

  • Average Daily Visitors: Average daily unique visitors your tested page received (i.e. 10 000) 
  • Number of variations: Total number of screen or page variations including the control version. (i.e. 3 variants)

It’s good idea to use those parameters as input for A/B test duration calculator. Here is an excellent tool from Abtasty. This calculator will give you the number of days required for running the test.

 Use cookies to maintain the integrity of the test

Visitors who participate in A/B testing should always see the same version of the page.

Consider the state of product

The procedure of A/B testing varies depending on the state of a product. There are two typical scenarios:

  • When you don’t have an established design and have several ideas about which direction to take.
  • When you have an established design but want to try some new ideas out.

In the first case, you want to treat all ideas equally, so you most likely assign equal weight (traffic) to each solution. In the second case, you might want to give your new page variants a smaller percentage of traffic than the existing solution (i.e. 60% traffic will go to the original design, while only 30% to the various) because you want to mitigate the risk inherent with introducing new ideas.

Conduct A/B testing on a regular basis

The effectiveness of anything can change over time and the results of A/B testing are not an exception. Depending on the nature of your product, you might want to run tests anywhere from a few days to a couple of weeks.