What it means
When you study a group by looking at a sample, you expect some random difference between what the sample shows and what the whole group would show. That is sampling error, and statistics can estimate its size.
Non-sampling error is everything else that can go wrong, and it is often far more damaging. Common causes include a sampling list that misses part of the population, questions that lead respondents toward an answer, and people who give inaccurate answers, whether by mistake or to look good.
Others are interviewers who influence responses, data entry mistakes and coding or processing errors. Even a perfectly chosen sample can give a poor answer if these problems are present.
The key feature is that more data does not fix it. Doubling a survey from 1,000 to 2,000 people will reduce random sampling error, but if the question is biased, you will simply have twice as many biased answers.
A census, where everyone is counted, is also exposed to these problems. In business, this matters for customer research, employee surveys, market sizing and audit sampling.
A finance team that relies on a survey to forecast demand, or on a sample of transactions to test controls, needs to ask whether the process behind the numbers is sound. A large sample size is not a sign of quality by itself.
The best defence is good design: test questions on a small group first, follow up non-respondents, train staff, validate data entry and compare results with other sources. You usually cannot measure non-sampling error exactly, but you can reduce it and be honest about it.
Auditors and finance teams meet the same issue when they test samples of transactions. If the population of invoices they draw from is incomplete, or if the tester misreads a document, the conclusion can be wrong however many items are tested.
Documenting how the population was built is therefore as important as choosing the sample size.
In practice
Real-world examples.
Example
A retailer emails a satisfaction survey to 20,000 customers and only the happiest ones reply. The average score looks excellent, and the leadership team is pleased. A later phone follow-up with non-respondents shows many were unhappy, and the team revises its view.
Example
A bank's internal audit team samples 150 invoices to test controls, but the list it samples from leaves out invoices raised by one branch. The results look clean, but the missing branch has weak controls. The team widens its list and the results change.
Example
A start-up estimates market size from an online poll of 5,000 people. The poll reaches only frequent internet users, so it overstates interest in a digital product. The founders adjust their revenue forecast downwards before approaching investors.
Formula
Calculation
Non-response bias = non-response rate x (average of respondents - average of non-respondents)
Suppose a company surveys its customers about satisfaction on a 10-point scale. Of those contacted, 60% reply with an average score of 8.0, while the 40% who do not reply would, if asked, have averaged 6.0. The true average = (0.60 x 8.0) + (0.40 x 6.0) = 4.8 + 2.4 = 7.2. The bias in the survey result = 0.40 x (8.0 - 6.0) = 0.8, so the survey overstates satisfaction by 0.8 points, which is 8.0 - 7.2.Case study
Seen in the real world.
Calder and Finch Research is a fictional market research firm created for this illustration. It was commissioned by a drinks company to estimate demand for a new low-sugar product in a large city.
The first survey of 3,000 households reported strong interest, but the interviews were carried out on weekday mornings, when mainly retired people and homeworkers were at home. Later checks showed that the group was older and more health-conscious than the city as a whole.
The illustrative lesson is that a large sample cannot rescue a flawed process. The firm repeated the research at different times of day, reweighted the answers and found demand about 30% lower than first reported, which saved the client from an oversized launch.
Watch out
Common mistakes.
- Believing a bigger sample always means a better answer, when non-sampling error does not fall as the sample grows.
- Assuming only surveys are affected, when audits, accounting estimates and data pipelines have the same kinds of problems.
- Ignoring non-response, when the people who do not answer often differ from those who do.
Questions
People also ask.
How is it different from sampling error?
Sampling error comes from the random chance of who ends up in the sample, while non-sampling error comes from everything else, including design, measurement and processing mistakes.
Can you measure non-sampling error?
Rarely with precision, but you can estimate parts of it, for example through follow-up checks, pilot tests and comparison with trusted data.
Does a census avoid it?
No, because a census removes sampling error but is still exposed to missing records, wrong answers and processing mistakes.
From the founder's library
Take it further with the book.
Build your financial confidence beyond this definition. Shihan's full-length guide, Accounting Fundamentals, takes the same plain-English approach and turns it into a complete, practical playbook for non-finance managers, business owners and students - with chapter-end quiz answers and presentation slides included.
25% off with code MMHQ25, applied at checkout. Priced in USD - checkout may show the equivalent in your local currency.
View the book and save 25%Related