What it means
Any test that draws conclusions about a large population from a small sample can go wrong in two directions. Alpha risk is the false positive, where you flag a problem, reject a batch or declare a marketing test a winner when nothing was really there.
Its partner, beta risk, is the false negative, where a genuine effect goes unnoticed. The level you pick for alpha is a business judgement rather than a mathematical truth.
Setting it at 1% means false alarms become rare, but you now demand much stronger evidence before acting, so real problems are missed more often. Auditors know this idea as the risk of incorrect rejection.
An auditor who wrongly concludes that a balance is materially misstated orders extra procedures that cost fees and find nothing, which is expensive and irritating rather than genuinely dangerous. Investors run into alpha risk when testing whether a fund manager or a trading rule has real skill.
Run enough backtests on the same data and some will look impressive purely by chance, which is why a strategy that finally clears a 5% test on its twentieth attempt should be treated with suspicion. The multiple testing trap is the practical version of this.
Alpha describes the risk on a single test, so running many tests at the same threshold makes at least one false positive close to inevitable, and careful analysts tighten the threshold to compensate.
In practice
Real-world examples.
Example
A pharmaceutical quality team samples 200 vials from a production run and rejects the batch when the failure rate looks too high. At a 5% alpha, roughly one perfectly good batch in twenty will be scrapped, which the company accepts because the cost of releasing a bad batch is far higher.
Example
An internal audit team tests 60 expense claims and concludes that the approval control is failing. Additional work shows the control was fine and the sample simply caught an unusual cluster, costing three weeks of audit time and a bruised relationship with the finance team.
Example
A quantitative fund screens 500 technical trading rules and finds twelve that beat the market at a 5% significance level. Since roughly 25 rules would clear that bar by pure chance, the research head insists every survivor is retested on data the screen never touched.
Formula
Calculation
Alpha risk = probability of rejecting the null hypothesis when it is true = the chosen significance level
Probability of at least one false positive across n independent tests = 1 - (1 - alpha) to the power n
A pricing team runs 20 independent website tests in a quarter at a 5% significance level, and suppose that in truth none of the changes affects conversion. The expected number of false positives is 20 x 0.05 = 1, and the probability of getting at least one is 1 - 0.95 to the power 20 = 1 - 0.3585 = 64.15%. In other words, a quarter of pointless tests will almost certainly hand the team a "winner" to celebrate.
Now attach a cost. Rolling out a change costs $40,000 in engineering, training and support, so the expected waste from false positives at a 5% threshold is 1 x $40,000 = $40,000 per quarter. Tightening alpha to 1% cuts the expected count to 20 x 0.01 = 0.2 false positives, or $8,000 of expected waste, and reduces the chance of at least one false winner to 1 - 0.99 to the power 20 = 18.21%. The trade-off is that genuinely good changes now need more traffic and more time to prove themselves.Case study
Seen in the real world.
The following is an illustrative and entirely fictional example. Pellworth Direct, an invented online homeware retailer, built a testing culture and celebrated a "test win" in its weekly meeting whenever a change beat the control at 5% significance. In one quarter the team ran 24 tests and declared five winners, projecting an extra $1,900,000 of annual revenue from the combination.
Actual revenue did not move. A new analytics lead re-examined the tests and found that four of the five winners had been stopped early, the moment the result crossed the threshold, which quietly inflated the true alpha risk well above the stated 5%.
The fictional team changed three things: fixed sample sizes agreed before each test began, a tighter 1% threshold for any change requiring more than $25,000 to build, and a rule that any winner had to be confirmed in a follow-up test. Declared wins fell from five a quarter to about two, but the revenue those wins predicted began showing up in the actual numbers.
Watch out
Common mistakes.
- Treating a 5% significance level as a 5% chance the conclusion is wrong, when it is the chance of a false alarm assuming there was no real effect to begin with.
- Running dozens of tests at the same threshold and reporting the winners without adjusting for how many tests were run.
- Setting alpha low to look rigorous without noticing that beta risk rises, so genuine problems and genuine improvements both get missed.
Questions
People also ask.
Is alpha risk the same as the alpha that measures investment outperformance?
No, these are unrelated uses of the same Greek letter, one being a false positive rate and the other a return above a benchmark.
How do I reduce alpha risk without increasing beta risk?
Increase the sample size, since a larger sample sharpens the test in both directions at the same time, which is why it costs money.
What is a sensible alpha for business decisions?
5% is the common default, but a decision that is cheap to reverse can tolerate 10%, while an expensive or irreversible one deserves 1% or tighter.
From the founder's library
Take it further with the book.
Build your financial confidence beyond this definition. Shihan's full-length guide, Accounting Fundamentals, takes the same plain-English approach and turns it into a complete, practical playbook for non-finance managers, business owners and students - with chapter-end quiz answers and presentation slides included.
25% off with code MMHQ25, applied at checkout. Priced in USD - checkout may show the equivalent in your local currency.
View the book and save 25%