You ran a survey of 500 customers and 32% said they would recommend you. Your finance partner asks whether the real figure could be 28% or 36%. The honest answer is yes, and the reason is sampling error. Every survey result is an estimate, and learning to say how far off it could be is what separates a number you can act on from one you can only quote.
This guide covers examples, the margin of error formula, non-sampling error and sampling frame error. It also corrects a common mix-up: many things called “sampling errors” in older articles are really biases, and they need different fixes. Sampling error is one part of survey data quality; the biases are covered in our guide to survey bias.
Key takeaways
- A sampling error is the chance gap between a survey result and the true population value, and it exists whenever you measure a sample instead of everyone.
- At 95% confidence, a 32% result from 500 respondents has a margin of error of about 4.1 points, so the true value could be 28% to 36%.
- Halving the margin of error takes four times the sample: 100 respondents give about 9.8 points, while 400 respondents give about 4.9 points.
- Sampling error is random and shrinks with sample size, but sampling frame error and other non-sampling errors are biases that a larger sample does not fix.
- Report the sample size, the margin of error and its confidence level together, and say that the margin covers sampling error only.
What is a sampling error in statistics?
Start with two terms. The population is everyone you want to learn about, such as all US adults who bought a laptop last year. The sample is the smaller group you actually measure. A survey result from the sample is an estimate of the true value in the population.
Pick a different random sample and you get a slightly different estimate, because different people landed in it. The gap between an estimate and the true population value that comes from this luck of the draw is the sampling error.
Sampling error is random, so it pushes estimates above and below the truth with no pattern. It is unavoidable whenever you measure a sample instead of everyone. It shrinks as the sample grows, with diminishing returns. And because it is random, you can put a number on it: the margin of error.
What is an example of a sampling error?
Imagine a population of 100,000 customers in which exactly 30% would recommend your brand. You do not know this. You draw five random samples of 500 customers each and get the results below. The numbers are illustrative, not real survey data.
| Sample | Said they would recommend | Gap from the true 30% |
|---|---|---|
| A | 27.4% | -2.6 points |
| B | 31.0% | +1.0 points |
| C | 29.2% | -0.8 points |
| D | 33.0% | +3.0 points |
| E | 28.6% | -1.4 points |
Nothing went wrong with any of these surveys. They differ because 500 people is a small slice of 100,000, and which 500 you got is partly luck. At 95% confidence, a sample of 500 at a true 30% has a margin of error of about ±4.0 points, so these gaps are all within the normal range.
The same logic applies to a brand tracker. If a wave of 500 car intenders finds 32% consider your brand, the next wave might find 29% or 35% with nothing changed in the market.
What are more examples of sampling errors?
Lists of examples often mix true sampling error with problems that are not sampling error at all. Sorting them matters, because the fix differs. Here are common scenarios, classified.
| Scenario | What goes wrong | Sampling error or something else? |
|---|---|---|
| A poll of 200 adults is used to estimate national opinion | The sample is small, so estimates swing widely from sample to sample | Sampling error. A larger sample reduces it |
| A customer survey is sent only to people who bought in the last 30 days | Lapsed customers are never asked and may think differently | Sampling frame error (a coverage problem), not sampling error |
| A political poll surveys only registered members of one party | Other voters have no chance of being picked | Sampling frame error and selection bias, not sampling error |
| Heights are measured only at a shopping mall to estimate a city’s average adult height | People who visit the mall are not a cross-section of the city | Selection bias from convenience sampling |
| 1,000 people are invited and mostly the most enthusiastic customers reply | Responders differ from non-responders | Non-response bias |
| A question is worded in a way that leads people to agree | Answers are pushed in one direction | Measurement error |
Only the first row is sampling error in the strict statistical sense. The rest are real and often much larger, but interviewing more people does not reduce them.
How do you measure sampling error?
The margin of error is the amount you add to and subtract from an estimate to get a range that, at a stated confidence level, is expected to contain the true value. For a percentage, the formula at 95% confidence is:
margin of error = 1.96 × √( p × (1 − p) / n )Here p is the proportion as a decimal (32% is 0.32), n is the number of respondents, and 1.96 is the multiplier for 95% confidence. The formula assumes a simple random sample and a reasonably large n.
Worked example: a 32% result on 500 respondents
- Write down the values: p = 0.32, n = 500.
- Calculate p × (1 − p) = 0.32 × 0.68 = 0.2176.
- Divide by n: 0.2176 / 500 = 0.0004352.
- Take the square root: √0.0004352 ≈ 0.02086. This is the standard error.
- Multiply by 1.96: 0.02086 × 1.96 ≈ 0.0409.
The margin of error is about ±4.1 points. Report the result as 32% ± 4.1 points, a range of roughly 28% to 36% at 95% confidence. That answers the opening question: 28% and 36% are both plausible values for the true figure.
How do sample size, confidence and weighting change it?
Because of the square root, halving the margin of error takes four times the sample: at p = 0.5 and 95% confidence, 100 respondents give about ±9.8 points and 400 give about ±4.9. A higher confidence level widens the range, weighting usually widens it further by lowering the effective sample size, and a gap between two groups needs a larger margin than either group alone. Our guide to the margin of error works through each of these with a sample-size table.
How big a sample do you need?
Rearrange the formula to plan a survey. For a target margin of error E (as a decimal) at 95% confidence, using the worst case p = 0.5, the sample size is:
n = 1.96² × 0.25 / E² = 0.9604 / E²For a margin of ±5 points (E = 0.05), n = 0.9604 / 0.0025 ≈ 385. For ±3 points, n = 0.9604 / 0.0009 ≈ 1,068. For ±2 points, n = 0.9604 / 0.0004 = 2,401.
If you need sub-group results, size for the smallest group you will report on. A survey of 1,000 with a ±3.1 point total margin gives a 100-person sub-group a margin near ±9.8 points.
What is the difference between sampling error and non-sampling error?
Sampling error comes from measuring a sample. Non-sampling error is everything else that can go wrong, even in a census. A bigger sample only fixes the first.
| Sampling error | Non-sampling error | |
|---|---|---|
| Cause | Chance in who is selected | Design, coverage, response, measurement or processing problems |
| Direction | Random, above or below the truth | Often systematic, pushing results one way (bias) |
| Effect of a larger sample | Shrinks | Does not shrink, and can grow |
| Can it be quantified? | Yes, as a margin of error | Rarely. It is managed through design and checks |
| Example | A tracker wave drifts by two points by chance | A frame misses a group, wording leads respondents, many people do not respond |
The main kinds of non-sampling error are:
- Coverage error (including sampling frame error, below): part of the population has no chance of being selected.
- Non-response error: the people who answer differ from those who do not.
- Measurement error: questions, survey mode or respondents introduce inaccuracies, such as confusing wording or answers that are not truthful.
- Processing error: mistakes in coding, data entry or weighting.
Some older explanations treat measurement error as a cause of sampling error. It is not, which is why a tidy margin of error can still sit on top of poor data. For more on checking data quality, see our guide to how to analyze survey data.
What is sampling frame error?
A sampling frame is the list or source you draw your sample from: a customer database, an online panel, a list of phone numbers, a register of voters. Sampling frame error occurs when that list does not match the population you want to describe. People are missing from it, included who should not be, or listed more than once.
Three simple cases:
- You want the views of all customers but draw only from your email subscribers. Customers who never subscribed have no chance of being selected.
- A phone survey draws numbers from a landline directory. Households with only mobile phones are left out.
- A satisfaction survey goes to everyone who contacted support. Customers who gave up before contacting you are absent, so results look better than reality.
Sampling frame error is a form of coverage error, so it is a non-sampling error. It produces bias, and doubling the sample does not help, because you draw more people from the same incomplete list. The fix is a better frame: combine sources, check the list against what you know about the population, and compare respondents with known benchmarks.
How do you reduce sampling error?
You cannot eliminate it, but you can control it.
- Increase the sample size. This is the direct lever, subject to the square-root rule above. Decide how much precision you need before you buy more completes.
- Use random selection where possible, so every member of the population has a known chance of being chosen. This keeps chance from turning into bias, though it does not remove sampling error.
- Stratify the sample. Divide the population into groups, such as regions or age bands, and sample within each. When the groups differ on what you measure, this typically gives more precise estimates than simple random sampling at the same size.
- Think before clustering. Sampling whole stores or households can cut fieldwork cost, but it usually increases sampling error compared with a simple random sample of the same size.
- Keep weighting light. Heavy weighting shrinks the effective sample size, so aim for a balanced sample at fieldwork.
How should you report sampling error?
Put the base and the margin of error where your readers will see them. Specifically:
- State the sample size and, if you weighted, the effective sample size.
- Quote the margin of error with its confidence level, for example “±4.1 points at 95% confidence”.
- On trend charts, show confidence bands, and describe small moves inside the band as “no clear change”.
- Treat small bases with caution. Many teams flag sub-groups under 50 respondents as directional only. That is a convention, not a statistical rule, so set a threshold and apply it consistently.
- Say plainly that the margin covers sampling error only, not frame, response or wording problems. A margin of error describes how precise a result is, not how accurate it is.
How do you do this in practice?
Most of this work happens in tabulation: checking bases, applying weights, and testing whether differences are bigger than chance. Halo Reports from mTab produces crosstabs and banner tables with weighting and significance testing, so you can see which differences in a table are likely to hold, and then present them in charts and interactive report pages. See Halo Reports for what it covers.
What are the most common mistakes with sampling error?
- Treating the margin of error as a guarantee of accuracy. It covers chance only. A sample from the wrong frame can have a small margin and still be badly wrong.
- Calling every survey problem “sampling error”. Bias from who was asked is not fixed by a larger sample. Name the problem so you choose the right remedy.
- Applying the margin to sub-groups. The ±3.1 on a 1,000-person total does not apply to the 100 people in one segment.
- Explaining movements inside the margin. A tracker move from 32% to 34% on 500 respondents is inside a margin of about ±4. Do not write a story for it. A tool such as Pulse monitors trackers and studies and surfaces meaningful movement, so you start from the changes worth explaining.
- Quoting a margin with no confidence level, or using the raw base after weighting. Both make the result look more precise than it is.
Before your next readout, put the base, the margin of error and the confidence level next to every headline number, then check each change you plan to discuss against that margin. If a change is smaller than the margin, say it is not clearly different. If you run a brand tracker, you can track brand health, consideration and competitors continuously and see what changed.