What it means
The value of an A-B split comes from randomisation. Because people are assigned to the two groups by chance, the groups look alike on average, so the only systematic difference between them is the version they were shown.
The two versions should differ in one meaningful respect. If version B changes the headline, the price and the button colour all at once, a lift in orders tells you that something worked but not what, which is why disciplined teams test one variable at a time.
Results are compared using conversion rates rather than raw counts, because the two groups are rarely exactly the same size. The headline number is usually lift, the percentage improvement of B over A, reported alongside a confidence level that says how likely the difference is to be real rather than noise.
Sample size is where most tests fail. A difference between 3.0% and 3.6% is meaningless on 200 visitors and solid on 24,000, so teams should decide in advance how many observations they need and resist calling a winner on day two.
The term is also used in direct mail for a simple two-way division of a list, and in older advertising language an A-B split meant printing two versions of an advertisement in alternate copies of the same magazine. The underlying idea is identical: divide at random, vary one thing, measure the difference.
In practice
Real-world examples.
Example
A subscription business splits 40,000 trial users and offers group B an annual plan at a 20% discount alongside the usual monthly plan. Group B converts to paid at 9.5% against group A's 8.0%, and the finance team checks that the extra volume more than covers the discount before rolling it out.
Example
A bank tests two versions of a mortgage application form, one asking for income details on the first screen and one on the third. The version that delays the income question raises completed applications from 22% to 27%, worth several hundred extra applications a month at no additional marketing cost.
Example
A charity splits its 60,000-name email list in half and sends one group a $25 suggested donation and the other $40. The higher ask produces fewer donors but a larger average gift, and total income per thousand emails rises from $310 to $368.
Formula
Calculation
Conversion rate = Conversions / Visitors, and Lift = (Conversion rate B - Conversion rate A) / Conversion rate A
An online retailer tests a new checkout page. Traffic is split evenly, giving 12,000 visitors to version A and 12,000 to version B over two weeks.
Version A produces 360 orders, so its conversion rate is 360 / 12,000 = 0.03, or 3.0%.
Version B produces 432 orders, so its conversion rate is 432 / 12,000 = 0.036, or 3.6%.
Lift = (3.6% - 3.0%) / 3.0% = 0.6 / 3.0 = 0.20, a 20% improvement.
In money terms, version B produced 432 - 360 = 72 extra orders during the test. At an average order value of $85 that is 72 x $85 = $6,120 of extra revenue over two weeks, so if traffic continues at the same rate across 26 two-week periods, rolling out version B is worth 26 x $6,120 = $159,120 a year before costs.Case study
Seen in the real world.
Wolder and Pike is a fictional online homeware retailer used here as an illustrative example of an A-B split done properly after being done badly. Its first attempt changed the product photography, the delivery promise and the price display at the same time, saw a 14% lift and rolled all three changes out, only to watch the gain vanish the following month.
The team started again with a single change: showing an exact delivery date rather than a delivery window. They ran the test on 30,000 visitors per version for three weeks, agreed the sample size in advance and refused to look at daily results.
Version B converted at 4.4% against 4.0%, a lift of 10%, which on 30,000 visitors each meant 1,320 orders against 1,200. In this illustrative case the extra 120 orders at an average value of $110 were worth $13,200 over the test window, and because only one thing had changed, Wolder and Pike knew exactly what to apply to the rest of the catalogue.
Watch out
Common mistakes.
- Stopping a test the moment version B is ahead, which reliably produces false winners because early results swing wildly.
- Changing several elements at once, which shows that something worked but leaves the team unable to say which element did it.
- Splitting the audience by something other than chance, such as putting new customers in one group and returning customers in the other, which confuses the groups' behaviour with the test.
Questions
People also ask.
How long should an A-B split run?
Long enough to reach the sample size agreed in advance and to cover at least one full weekly cycle, since weekday and weekend traffic behave differently.
What if neither version wins?
That is a genuine result: it tells you the change does not matter to customers, which saves the cost of rolling it out.
Is an A-B split the same as a multivariate test?
No, an A-B split compares two complete versions, while a multivariate test compares combinations of several elements and needs far more traffic to reach a conclusion.
From the founder's library
Take it further with the book.
Build your financial confidence beyond this definition. Shihan's full-length guide, Accounting Fundamentals, takes the same plain-English approach and turns it into a complete, practical playbook for non-finance managers, business owners and students - with chapter-end quiz answers and presentation slides included.
25% off with code MMHQ25, applied at checkout. Priced in USD - checkout may show the equivalent in your local currency.
View the book and save 25%Related