A/B Testing

What is it?

A/B testing is a way to get usable data on user preferences. By splitting traffic between two similar sites and comparing which path leads to a more favorable result it is possible to know with certainty which design methods drive the most traffic toward the desired user behaviour.

To explain how A/B testing is done it’s probably a good idea to first explain what the ‘A’ and ‘B’ mean:

‘A’ aka “Control” or “Exploit”

The ‘A’ page is a “Control” site. The control page is the baseline you’re comparing against. It receives the bulk of all site traffic (usually 90%) and is sometimes known as the “Exploit” page.

‘B’ aka “Experiment” or “Explore”

Any ‘B’ pages associated with a page are competing to become the next ‘A’. With every ‘B’ page there is a hypothesis that by making a specific small change the traffic sent will be more likely to perform a defined action. Technically when A/B tests are done with multiple competing ‘B’ pages the appropriate term is Bucket testing or Split testing. In this case – we’ll use the terms interchangeably.

Why Bother?

Often design decisions are made based on intuition – the belief that a certain look-and-feel of a website will have better results. Often though – the behaviour of a crowd of people doesn’t fit nicely into the intuition of a designer. By leaving some flexibility in the design and designing for change a page can be optimized to increase the likelihood of various user behaviours.

Statistics Driven Decision Making

The goal of A/B testing is to make decisions based on user statistics. If a red button makes (for example) 20% more people click it than a gray button, and you want people to click that button – then it’s a good idea to make the button red.

Improved Client Engagement

A/B Testing provides a means of ongoing communication with the client. It allows project managers to be more proactive and reach out rather than just fighting fires. Rather than waiting until something breaks to get in touch with us; we can go to the clients with regular reports on the successful, ongoing changes being made to their site. If customers feel comfortable bringing up change requests in terms of testable alterations then they can be simply added to a testing pool and (over time) – be compared against the baseline.


The goals are fairly clear – pay attention to user statistics to improve results. However – many challenges exist in setting up a system for doing A/B testing en-masse.

Step 1.             Defining Business Goals

Any type of split test has to start with a desired action by the end user. This desired action is known as a business goal and has to be discussed with the business owner and key stakeholders.

Step 2.             Define a User Funnel

Once a known business goal has been defined the path a user must take between entering the site and when they perform a certain action. This pathway is known as the User Funnel. It’s one of the pages along this user funnel that you will be performing an A/B test upon.

This involves creating some form of property object and sending events to a tracking system at important decision points.

Step 3.             Develop a Variant Page

When you have developed a funnel and built up a baseline of user data you will find that some of the pages have higher user dropoff than others. These pages are good candidates for running A/B tests on to reduce that dropoff.

Step 4.             Set Properties and Send Events

In order to evaluate the pages that a user went to during their session you have to know who they are, which page variants they saw and whether they achieved your business goal. When a goal occurs you need to increase or decrease the score of the visited pages in a ranking system (or selecting a winner based on analytics). Amongst other things – you should track who the user is (and any data about them which is known), which pages were viewed, which page variant was viewed, how long the user lingered on a page and any other information which may have modified their decision making process.

Step 5.             Tallying Results

The results should always be compared against the baseline. The baseline page should receive the bulk of the traffic in order to “Exploit” the benefit it provides of higher conversion rates (or whatever the business goal may be). Once all the data is collected – it has to be filtered and sorted in a way that informs the person in charge of analytics which page was more successful. If the results are to be tallied automatically the data collected needs to provide some value to the actions performed by the user.