Holdout Group

A campaign can reach people who would have bought anyway, so counting every sale after an ad as a result overstates its effect, and a holdout provides a comparison for what may have happened without exposure. Meta describes lift studies using test and holdout groups, and Optimizely describes holdouts for feature experiments.

Their implementations differ, but both rely on meaningful separation and measurement. Define the eligible population before assignment, because random assignment can balance known and unknown differences on average while cherry-picking low-value customers for holdout would distort the result.

A fictional retailer randomly assigns 10% of eligible customers to receive no promotion while the other 90% receive an offer, and comparing purchase rates helps estimate incremental response. A fictional team that excludes its least active users from a coupon cannot credit the coupon alone if the rest buy more, because the groups were unequal from the start.

Measure outcomes over the same period with the same definitions, and decide the primary measure before seeing results, since revenue, orders and retention answer different questions. Calculate group rates rather than raw totals when sizes differ: incremental lift is the test rate minus the holdout rate, and relative lift divides that difference by the holdout rate when it is nonzero.

If a fictional campaign has a 12% purchase rate in test and 10% in holdout, absolute lift is 2 percentage points and relative lift is 20%, and the two descriptions should not be confused. A positive observed gap is not automatically statistically reliable, because small groups can produce noisy results.

A fictional team that sees 13 purchases in 100 tested people versus 10 in 100 held out should avoid a strong success claim without a confidence assessment, and more data may be needed. Estimate uncertainty and follow the experiment design before claiming a winner.

Contamination weakens the comparison, because holdout members might see the campaign on another channel or share an account with someone exposed, so these pathways should be recorded where feasible. Holdouts can be at user, account, household or geography level, so pick the unit that matches spillover risk and privacy rules, noting that geographic tests need extra care because areas may differ.

The report should also state exactly what was withheld, since a holdout from email alone measures email on top of other marketing, not the effect of all marketing. Holdout size balances precision against opportunity cost, since withholding an offer from too many customers can sacrifice revenue while too few yields weak evidence, and ethical and contractual constraints also matter.

Set a duration and ending rule, because an evergreen holdout can measure long-term effects but may accumulate differences or face customer fairness concerns, and Optimizely distinguishes local and global holdouts, where a global holdout excludes a unit from several experiments and can answer broader questions but complicate analysis. The holdout estimates an effect for the studied population, channel and period, so record assignment, exposure, dropouts and data gaps, never silently remove low-performing users after the test starts, and use the findings with cost and business context.