What it means
Three good quarters do not make a skilled manager, yet investors anoint stars on less. Sample-size neglect is the habit of drawing big conclusions from little data.
Tversky and Kahneman documented the bias in their 1971 paper on the belief in the law of small numbers: people expect small samples to mirror the population as faithfully as large ones do. The statistics are unforgiving: the smaller the sample, the wider the range of outcomes pure chance produces, so extreme results are commonest where the data is thinnest.
Investing is saturated with the error: a fund's three-year streak, a stock tip from two successes, a backtest over one market regime, all are small samples dressed as evidence. The regression consequence follows automatically: extreme small-sample results are mostly luck, and luck does not repeat, which is why last year's top performer is a poor guide to next year's.
The bias pairs with its mirror: people who neglect sample size in judging others demand impossible proof for themselves, and both errors come from feeling the story rather than counting the cases. The antidote is arithmetic, not willpower: ask how many observations, how variable the process, and how likely the result is under pure chance before believing any streak.
For a non-finance reader, sample-size neglect is the reason the first hundred trades, hires, or quarters tell you less than you feel they do: small numbers tell big stories, and most of them are fiction. Sports and business recycle the same illusion: the hot hand, the genius quarter, the breakthrough hire, each a small sample granted a narrative that a longer record rarely honours.
Base-rate discipline is the statistical counterweight: start from how often such streaks occur across all managers, and let the specific story adjust that anchor rather than replace it.
In practice
Real-world examples.
Example
A family office backs a fund on eighteen months of returns. A later review shows the streak was within one lucky draw of chance, and the fund gives the gains back in the following year.
Example
An investor hears a tipster's two winning calls and computes how often random tips produce the same record. The answer is often enough that two calls prove nothing, so the investor ignores them until a much longer record exists.
Example
A strategy team sees a backtest covering one market regime and rejects it as evidence for a strategy meant to survive several. It asks for results across rising, falling and flat markets before allocating capital.
Formula
Calculation
Standard error of an average = standard deviation of the individual observations / square root of the number of observations. The standard error shrinks only with the square root of the sample size, so quadrupling observations merely halves the noise around any estimate.
Worked example. A fund's monthly returns have a standard deviation of 4%, and over 16 months it averages 1.5% a month above the market.
- Standard error = 4% / square root of 16 = 4% / 4 = 1%.
- The average of 1.5% is only 1.5 standard errors from zero, which pure chance produces fairly often.
Now suppose the same 1.5% average held over 64 months. The standard error becomes 4% / square root of 64 = 4% / 8 = 0.5%, and the average sits 1.5% / 0.5% = 3 standard errors from zero. Four times the data halved the noise and turned an unconvincing streak into real evidence, which is why a full cycle of data matters more than a good story.Case study
Seen in the real world.
This case study is fictional and illustrative. A made-up family office allocates 5% of its portfolio to a young hedge fund after a spectacular first eighteen months: up 40% in a flat market, with a charismatic founder who explains every win. Eighteen months later the fund has given it all back and the allocation committee holds a post-mortem. The review reconstructs what the committee actually knew: eleven monthly observations above zero, in a single market regime, from a strategy with no track record through a downturn.
The chief risk officer annotates this with the standard-error arithmetic showing the streak was within one lucky draw of pure chance. The founder's skill turns out to be concentrated in one factor bet that reversed the following year. The office's new allocation policy writes the lesson into rules: no strategy earns capital on fewer than a full cycle of data, streaks are discounted by the square root of their length, and the best story in the room gets the smallest cheque until the numbers outlive the narrative. The committee chair's summary is quoted in the next hiring round: we did not lose money to a bad manager, we lost it to our own belief that eleven months could tell us what only a decade can.
Watch out
Common mistakes.
- Trusting short track records; extreme small-sample performance is mostly variance, and variance does not persist.
- Believing more data fixes it cheaply; standard error shrinks with the square root of observations, so precision is expensive in time.
- Scolding others while exempting oneself; the same mind that dismisses a rival's small sample believes its own handful of cases completely.
Questions
People also ask.
What is sample-size neglect?
The cognitive bias of treating small samples as representative as large ones, documented by Tversky and Kahneman as the belief in the law of small numbers.
Why do small samples mislead?
Chance produces extreme results far more often in few observations, so small-sample extremes are mostly luck and do not repeat.
How do investors guard against it?
By demanding full-cycle track records, discounting streaks by the square root of their length, and computing whether chance alone explains the result.
From the founder's library
Take it further with the book.
Build your financial confidence beyond this definition. Shihan's full-length guide, Accounting Fundamentals, takes the same plain-English approach and turns it into a complete, practical playbook for non-finance managers, business owners and students - with chapter-end quiz answers and presentation slides included.
25% off with code MMHQ25, applied at checkout. Priced in USD - checkout may show the equivalent in your local currency.
View the book and save 25%