What it means
While often associated with software and marketing, A/B testing is fundamentally a financial optimisation tool. By systematically testing small changes, businesses can improve conversion rates, boost sales and make sure their spending generates the highest possible return.
Instead of guessing what customers prefer, companies use data to guide their decisions. In practice, this approach replaces opinion-based debates with empirical evidence.
If you want to know whether a 10% discount works better than a free shipping offer, you test both on a subset of your audience. The version that drives more profitable behaviour wins, and you roll it out to your entire customer base.
This method directly affects the bottom line by reducing wasted marketing spend and increasing revenue per user. It removes risk because you test ideas on a small scale before committing significant budget to a full launch.
Even minor improvements found through testing can compound into substantial financial gains over time. For non-finance managers, understanding A/B testing helps in evaluating requests for new projects or campaigns.
When someone suggests a redesign or a pricing change, asking how they plan to test it ensures accountability and fiscal prudence. It shifts the focus from output, such as launching a new page, to outcomes, such as increasing actual profit.
Imagine a chef trying out two different soup recipes on two small groups of diners before deciding which one to put on the main menu permanently.
In practice
Real-world examples
Example 1
An online clothing store tested two checkout page designs. Version A kept the payment button blue, while Version B changed it to bright orange. The orange button increased completed purchases by 12% over two weeks, and the store rolled it out to every shopper.
Example 2
A boutique hotel tested two different subject lines for its monthly email newsletter. Version A offered a free breakfast, while Version B offered a room upgrade. Version B resulted in a 20% higher booking rate, and the hotel also checked that the upgrade offer did not cost more than the extra bookings earned.
Example 3
A software startup tested two pricing page layouts. Version A listed three tiers vertically, while Version B highlighted the middle tier as the most popular. Version B led to a 15% lift in higher-tier subscriptions, which raised average revenue per customer.
Think of it
Imagine a chef trying out two different soup recipes on two small groups of diners before deciding which one to put on the main menu permanently.
Formula
Calculation
Conversion Rate = (Number of Desired Actions / Total Visitors) x 100%
Relative Lift = (Rate of B - Rate of A) / Rate of A x 100%
Worked example. Version A gets 50 sales from 1,000 visitors, and Version B gets 70 sales from 1,000 visitors.
- Version A conversion rate = (50 / 1,000) x 100% = 5%
- Version B conversion rate = (70 / 1,000) x 100% = 7%
- Relative lift = (7% - 5%) / 5% x 100% = 40%
Now put dollars on it. With an average order of $80, Version A earns 50 x $80 = $4,000 per 1,000 visitors, and Version B earns 70 x $80 = $5,600. The gap is $1,600 per 1,000 visitors, so a site with 200,000 visitors a year would gain 200 x $1,600 = $320,000. Before rolling out, check that the sample is large enough for the difference to be statistically significant (unlikely to be down to chance).Case study
Seen in the real world
BrightBooks, a fictional accounting software provider for small businesses, wanted to increase sign-ups for its free trial. The marketing team proposed changing the main landing page headline. Version A, the control, used the headline 'Simple Accounting Software for Small Business'. Version B used 'Save Five Hours a Week on Invoicing'.
They directed equal traffic to both pages over a four-week period, exposing 10,000 potential customers to each version. Version A generated 400 trial sign-ups, a 4% conversion rate. Version B generated 600 trial sign-ups, a 6% conversion rate. Tracking the users over the next three months showed that those from Version B stayed at the same rate as those from Version A, which meant the extra sign-ups were high quality.
By simply changing a headline based on an A/B test, BrightBooks increased its top-of-funnel customer acquisition by 50% without any extra advertising spend. As an illustration of the value, suppose 20% of trial sign-ups become paying customers at $300 a year. The extra 200 sign-ups per 10,000 visitors would then bring 40 customers and $12,000 of annual revenue.
Watch out
Common mistakes
- Ending the test too early before gathering enough data to reach statistical significance.
- Testing too many variables at once, making it impossible to know which specific change caused the result.
- Ignoring the financial cost of the test itself, such as deep discounts that harm profit margins.
Questions
People also ask
How long should an A/B test run?
It should run long enough to account for weekly cycles, usually at least two weeks, and until you have enough sample size to trust the results.
Can A/B testing be used for things other than websites?
Yes, it can be used for emails, pricing strategies, product packaging, phone scripts, and physical store layouts.
What happens if both versions perform exactly the same?
If there is no statistical difference, you usually keep the original version to save implementation costs, or test a bolder change.