What it means
A retailer advertises a weekend offer to 10,000 customers and sees 500 purchases, but some would have bought without the message. To assess lift, it can randomly withhold the campaign from a comparable group and compare purchase rates, adjusting for group sizes.
If the exposed group buys at 5% and the holdout at 4%, the estimated incremental effect is one percentage point, or 100 extra purchases per 10,000 comparable customers, with precision depending on variation and sample size. Google's Conversion Lift guidance describes comparing people who see ads with a control group that does not, then measuring the difference in downstream conversions.
That is a causal design, not a guarantee that every business has access to the same platform tool. A company can also use a suitable geographic experiment or other rigorous design, but must consider spillover, seasonality and differences in customer mix, because a before-and-after chart alone can confuse a campaign with holidays or competitor activity.
Incrementality changes how marketers read attribution, as a channel can receive credit for purchases it merely intercepted, such as a branded search ad shown to a customer already looking for the store. Another activity may build awareness that leads to sales weeks later without receiving last-click credit.
Both attribution and incrementality are useful for different questions: the former describes observed touchpoints, while the latter asks what changed because of the spend. Measure profit, not only incremental revenue, because discounts reduce price on purchases that might have happened anyway.
A campaign that brings 100 extra orders at $20 contribution each creates $2,000 of extra contribution before campaign cost, but giving $10 discounts to 400 existing buyers could erase that benefit. Include creative, media, agency and operational cost, plus returns, since the right calculation depends on what the control group reveals.
Experiment design needs ethics and operations: a holdout may be inappropriate for a required service notice or a promised customer benefit, and randomisation should not quietly violate a contract or discriminate unlawfully. Keep exposures and outcomes consistent and define the test end date before seeing results.
Small tests can produce noisy estimates, so report uncertainty and avoid declaring a tiny difference a win. For owners, choose one decision the test can inform, such as whether to continue a channel, discount or audience, and define the primary outcome, control, budget and stop rule in advance.
Review incremental contribution and customer retention, then repeat when conditions change. Marketing works when it changes behaviour profitably, not merely when a dashboard can connect a sale to an ad.
In practice
Real-world examples.
Example
A merchant holds out a random 10% of its customer list from a promotional email. After two weeks it compares purchase rates between the groups and finds the email lifted orders only slightly.
Example
A brand compares exposed and unexposed regions over a suitable period. It checks that the regions had similar sales before the test and that no competitor promotion distorted the comparison.
Example
Finance subtracts campaign and discount costs from incremental contribution. The campaign is judged on that remaining profit and not on the revenue attributed to it by the dashboard.
Formula
Calculation
Estimated incremental conversion rate = Treatment conversion rate - Comparable control conversion rate
Worked example. In a fictional randomised test, 10,000 exposed customers buy at 5% (500 purchases) and 10,000 controls buy at 4% (400 purchases).
- Estimated lift is one percentage point, or 100 incremental purchases in the exposed group.
- At $30 contribution each, estimated extra contribution is $3,000 before campaign and discount costs.
- Suppose media and creative cost $1,500 and a $5 coupon was redeemed by all 500 buyers, costing $2,500. Net incremental profit is $3,000 - $1,500 - $2,500 = -$1,000, a loss of $1,000.
An attribution report crediting all 500 orders would show $15,000 of contribution (500 x $30) and call the campaign a success. The holdout comparison shows that only 100 of those orders were caused by the campaign, and that the coupon cost more than the extra sales were worth.
Check statistical uncertainty and any spillover before acting on the estimate.Case study
Seen in the real world.
This illustrative and entirely fictional example follows Sable Market, an invented online shop. Its ad dashboard credited 1,000 orders to a coupon campaign, so the owner called it profitable. Finance noticed that loyal customers were using the coupon on purchases they usually made at full price. The team ran a bounded holdout with comparable customers and tracked total contribution. It found a smaller incremental order lift than the attributed total and reduced indiscriminate discounts.
The invented business kept a targeted offer for customers whose behaviour changed. The outcome depended on a measured comparison rather than a claim about every clicked sale. The case shows why attributed revenue and incremental value are different. Sable Market now repeats a small holdout whenever it changes a major discount, so each promotion is judged against what customers would have done anyway.
Watch out
Common mistakes.
- Counting every post-ad purchase as caused by the ad.
- Ignoring discounts given to customers who would buy anyway.
- Declaring a small, noisy test result certain.
Questions
People also ask.
What is incremental lift?
The estimated difference in outcomes caused by the activity versus a credible no-activity comparison.
Is attribution the same thing?
No. Attribution assigns observed credit; incrementality estimates causation.
Why use a holdout?
It gives a comparison for what might have happened without the campaign.
From the founder's library
Take it further with the book.
Build your financial confidence beyond this definition. Shihan's full-length guide, Accounting Fundamentals, takes the same plain-English approach and turns it into a complete, practical playbook for non-finance managers, business owners and students - with chapter-end quiz answers and presentation slides included.
25% off with code MMHQ25, applied at checkout. Priced in USD - checkout may show the equivalent in your local currency.
View the book and save 25%