You send a satisfaction survey to every customer, a fifth of them answer, and the score looks healthy. The question nobody asks in the readout is whether the other four-fifths would have said the same. Nonresponse bias is the gap between those two groups, and it is one of the main threats covered in our guide to survey data quality. This article explains how to estimate it, check for it and reduce it with the data you already have.
Key takeaways
- Nonresponse bias happens when people who skip a survey, or a question, differ from those who answer it.
- Bias is roughly the nonresponse rate multiplied by the difference between responders and nonresponders.
- A low response rate does not prove bias, and a high one does not rule it out.
- Detect it by comparing respondents with known population figures, early with late responders, and by following up a sample of nonresponders.
- Weighting to known totals reduces the bias you can see, but not differences within the groups you weight on.
What is nonresponse bias?
Nonresponse bias, also written non-response bias, is a systematic error: it comes from who answers rather than from chance, so it does not average out as you collect more answers. If the customers who reply are happier or more engaged than those who ignore you, every result leans their way.
Unit nonresponse vs item nonresponse
The two kinds of missing answers need different handling.
| Type | What is missing | Typical cause | Main fix |
|---|---|---|---|
| Unit nonresponse | The whole survey | Invitation ignored, survey abandoned early | Fieldwork design, weighting |
| Item nonresponse | One question | Sensitive topic, unclear question, no opinion | Question design, “don’t know” options, clear bases |
Does a low response rate mean your results are biased?
Not on its own. A survey response rate tells you how many people answered, not whether they differ from those who did not. If the 15% who answer think like the 85% who do not, the bias is small. If they think very differently, even a 60% response rate can leave a large error.
The response rate sets how much room there is for bias; the difference between groups decides how much of it is used.
What does nonresponse bias look like in customer feedback?
In customer surveys, the most common pattern is a U-shape. Delighted customers answer to say thanks. Angry customers answer to get something fixed. The indifferent middle ignores the invitation.
Which end is louder depends on the trigger. A survey sent after a support ticket closes may over-represent frustrated customers; one sent to a loyalty-program list may over-represent fans. We see this most often in relationship surveys where the invitation list is “everyone who opened last month’s newsletter”: the engaged are already selected before anyone answers.
How do you calculate nonresponse bias? A nonresponse bias example
In words, the bias in a respondent average is approximately:
Bias = nonresponse rate × (responder value − nonresponder value)
The numbers below are illustrative. A company invites 5,000 customers to a satisfaction survey and 20% respond. Among the 1,000 responders, 75% say they are satisfied. A follow-up of a sample of nonresponders suggests that about 55% of the 4,000 who did not answer are satisfied.
| Group | Customers | Share | Satisfied | Satisfied customers |
|---|---|---|---|---|
| Responders | 1,000 | 20% | 75% | 750 |
| Nonresponders (estimated) | 4,000 | 80% | 55% | 2,200 |
| All customers | 5,000 | 100% | 59% | 2,950 |
The true value combines both groups: 0.20 × 75% + 0.80 × 55% = 15 + 44 = 59%. Counting customers gives the same answer: (750 + 2,200) ÷ 5,000 = 59%.
The survey reported 75%, so the bias is 75 − 59 = 16 points. The formula agrees: 0.80 × (75 − 55) = 0.80 × 20 = 16 points. More responses at the same 20% rate would not change this. The margin of error would shrink, but it would shrink around the wrong number.
How do you detect nonresponse bias?
You rarely know the true value, so look for evidence from several angles.
- Compare respondents with the known population. Your CRM knows every invited customer’s tenure, region, plan and spend; census figures play the same role for general-population studies. If respondents skew toward long-tenure, high-spend customers, you have a measurable gap.
- Compare early and late responders. People who answer only after the second or third reminder are often closer to nonresponders than people who answer on day one. If satisfaction drifts down across reminder waves, the nonresponders are probably lower still.
- Follow up a sample of nonresponders. A short survey, a different channel or a phone call to a small random sample of nonresponders gives you a direct estimate, like the 55% in the example.
- Look at item missingness patterns. Check which questions have the most blanks and who leaves them. If dissatisfied customers skip the renewal question, that figure is biased even when the response rate is fine.
Running these checks by segment is easier with crosstabs and banner tables with weighting and the base shown on every table, because a thin base in one column is often the first sign that a group stopped answering.
How can you reduce nonresponse bias?
At design and fieldwork
Nonresponse is cheaper to prevent than to correct. Send reminders, because they bring in the people who were not going to answer. Make the survey work on a phone, keep it short and say how long it takes. Avoid inviting only self-selected lists.
At analysis: weighting to known totals
Weighting gives each respondent a weight equal to their group’s population share divided by its share of respondents, so under-represented groups count for more. Continuing the illustrative example, the CRM splits the 5,000 customers by tenure:
| Tenure | Population | Pop. share | Respondents | Resp. share | Weight | Satisfied |
|---|---|---|---|---|---|---|
| Under 1 year | 2,000 | 40% | 200 | 20% | 0.40 ÷ 0.20 = 2.0 | 60% |
| 1–3 years | 1,500 | 30% | 300 | 30% | 0.30 ÷ 0.30 = 1.0 | 75% |
| Over 3 years | 1,500 | 30% | 500 | 50% | 0.30 ÷ 0.50 = 0.6 | 81% |
Unweighted, satisfaction is (120 + 225 + 405) ÷ 1,000 = 75%. Weighted, it is 0.40 × 60% + 0.30 × 75% + 0.30 × 81% = 24 + 22.5 + 24.3 = 70.8%.
Weighting removed 4.2 of the 16 points. The remaining 11.8 points come from differences inside each tenure group: new customers who skipped the survey are less satisfied than new customers who answered, and tenure cannot see that. Weighting only corrects for the variables you weight on, and large weights add variance. Report the weighted result with that caveat and keep the scheme fixed across waves. If you run trackers across markets, one governed platform for wide, complex studies helps keep those weighting definitions consistent from wave to wave.
How should you handle item nonresponse?
- Offer an honest “don’t know” or “prefer not to say.” Forcing an answer turns missing data into noise.
- Use imputation sparingly. Predicting blanks from similar respondents can help in modeling, but it invents data, so flag imputed figures.
- Show the base on every table. If 1,000 people took the survey and 640 answered the renewal question, the renewal percentage is based on 640. Say so.
What are the common mistakes with nonresponse bias?
- Treating a high response rate as proof of quality. It limits the possible bias but says nothing about the difference between groups.
- Reporting a margin of error as if it covered bias. It covers sampling error only, not the error from who chose to answer.
- Weighting on what is easy rather than what matters. Weighting by region does little if satisfaction varies by tenure and plan.
- Mixing up who answers and how they answer. Nonresponse bias and response bias need different fixes, and both belong in a wider review of survey bias.
- Changing weights between waves without saying so. A trend that moves only because the weighting changed is not a trend.
What should you do next?
Pull the invitation list for your last customer survey and compare respondents with it on three or four CRM fields, then split the headline score by reminder wave. If either check shows a gap, plan a small nonresponder follow-up before the next wave and decide your weighting variables in advance.