Every survey has a silent group
When a survey is designed, a certain number of people are selected to be asked. Some are never reached because their phone number is out of date, their address has changed, or they simply do not open the message. Some are reached and decline. Some start and stop partway. The people who end up in the final data are only those who got through all of that.
The rest are the non-respondents. In most modern surveys they are a large majority of those contacted, and the data says nothing directly about them. What matters for accuracy is whether they differ from the respondents in ways that relate to the question being asked.
Why it is not just a matter of size
If non-respondents were a random slice of the selected group, their absence would only shrink the sample and widen the margin of error slightly. The problem arises when who answers is related to what the survey is about.
Consider a survey on trust in institutions. People who feel disconnected from public life may be less willing to take part in a survey from an institution-linked organisation. If so, the respondents would typically look more trusting than the population really is, and interviewing ten times as many of them would not fix this. A bigger pile of the same kind of respondent produces a more precise version of the same distortion.
This is the core of non-response bias: the gap between what respondents report and what the whole selected group would have reported, driven by the pattern of who responds.
Who tends to be missing
Research on survey participation has repeatedly found that willingness to respond is not evenly spread, though the details vary by country, method and period. Some patterns that are often reported:
- People with less free time, such as those working long or irregular hours or caring for others, may be harder to reach.
- People who are suspicious of institutions, pollsters or the media may be more likely to decline.
- Younger adults and people who move frequently can be harder to locate with address or phone lists.
- People who are interested in the topic often respond more readily than those who are not.
These are tendencies, not rules, and they interact with how the survey is delivered. A web survey may under-represent people with limited internet access. A phone survey conducted during working hours may under-represent people who are out.
Topic interest and the enthusiasm effect
One subtle form of the problem is that people who care strongly about a subject are often more willing to talk about it. A survey about a political issue may therefore attract an unusually engaged group, and a survey about a hobby may over-represent enthusiasts. The published percentages then describe engaged people more accurately than the general population. Readers meet this whenever a result is presented as “what the public thinks” but came from a survey that was advertised by its topic.
Differential non-response
A related idea is that non-response can change over time and across groups in ways that move the result. If supporters of one side of a debate happen to be more willing to answer on a given day, perhaps after a burst of news that energises them, a poll can show a swing that is partly a change in who responds rather than a change in minds. Researchers sometimes call this differential non-response, and it is one reason why apparent sudden shifts in a single poll are treated cautiously. Wider coverage of media amplification of small shifts shows how such movements can be inflated in the telling.
What researchers do about it
There is no complete fix, but several approaches are common.
Reducing it in the field
Pollsters often make repeated contact attempts, offer more than one way to respond, vary the time of day, and keep surveys short and clear. Some offer small incentives. These steps raise participation, though a higher response rate does not guarantee lower bias, because what counts is whether the remaining gaps relate to the topic.
Comparing with known benchmarks
If a survey’s respondents include noticeably fewer young people, or fewer people in a certain region, than official statistics suggest, that points to a gap. Comparing respondents to reliable external figures is a standard check.
Adjusting afterwards
The most widespread remedy is survey weighting, where respondents from under-represented groups count for more and those from over-represented groups count for less. This can help if the adjustment variables relate to both willingness to respond and the opinion in question. It cannot fully help when the relevant difference is invisible, for example if two people of the same age, sex and education differ in trust in a way that also affects whether they answer.
Following up with non-respondents
Some high-quality surveys take a sample of the people who did not respond and try harder to reach them, sometimes by a different method. Comparing the late responders to the early ones can show how much the picture changes. It is costly, so it tends to be used mainly in large, well-funded studies.
Reading survey results with this in mind
A few questions are useful when you see a survey result:
- Was the survey sent to a defined group, and what share of that group actually answered?
- Is the reported response rate high, low, or not reported at all?
- Does the topic plausibly relate to willingness to respond?
- Did the pollster compare respondents with outside benchmarks or adjust for known gaps?
The key point is that silence is not neutral. The people who never answer are still part of the population, and any serious reading of a poll includes some thought about what their views might be and in which direction their absence could push the numbers. A transparent survey report will say what was done about it, even when the honest answer is that some uncertainty remains.