Two ways of finding respondents
Most polls you see fall into one of two broad families. In the first, the pollster decides who should be asked and uses a randomised procedure to pick them. In the second, people choose for themselves to take part, usually by clicking on a link or joining a panel. These are very different starting points, and they support different kinds of claims.
What a probability sample is
In a probability sample, every person in the target population has a known, non-zero chance of being chosen. The selection might come from a list of addresses, a set of telephone numbers, or a register of some kind, with the final picks made by chance. Because the pollster controls who is invited, the sample is not shaped by who happens to be keen.
This design has a long theoretical pedigree. It lets researchers use standard formulas to estimate sampling uncertainty, which is why a published margin of error is traditionally linked to it. It does not make a poll perfect. The list used for selection may miss some people, and many of those selected will not respond, so real-world probability samples still need careful adjustment. But the underlying logic is clean: the sample is built by a process whose properties are known.
What an opt-in poll is
An opt-in poll draws on people who volunteered. The simplest versions are the quick votes on news sites and social media, where anyone who sees the question can click. More structured versions use large online panels, where people sign up in advance, often in exchange for small rewards, and are then invited to surveys that match certain profiles.
The key feature of all such opt-in polls is self-selection. The pollster does not know the chance that any given person in the wider population would end up in the sample, because that depends on who saw the invitation and who chose to act on it. How survey panels work and who they leave out is a topic in its own right, but the central point is that panel members can differ from non-members in ways that matter, such as how much time they spend online or how interested they are in surveys.
The simplest case: a website vote
A vote embedded in a news article or a social post is the clearest example of what opt-in samples cannot do. The respondents are people who read that particular site, noticed the question, and cared enough to respond. Groups with strong views, or with organised campaigns pushing their supporters to vote, can dominate. The result describes the people who voted and says very little about the general public.
Such votes can still be informative as an illustration of how engaged readers feel, and they are harmless as entertainment. Problems start when the percentages are quoted as if they described everyone. A large number of votes does not repair the problem, since a very large self-selected group is simply a very large version of the same bias.
How good online panels try to compensate
Serious commercial and academic pollsters using online panels do not just publish raw volunteer responses. They typically take several steps:
- Quotas and matching. They recruit or select respondents to resemble the population on features like age, sex, region and education.
- Survey weighting. They adjust the results so that the sample’s profile matches known benchmarks, a process explained further under survey weighting.
- Modelling. Some use statistical models that link panel responses to population characteristics, drawing on outside data.
- Quality checks. They try to detect inattentive respondents, duplicate entries and automated responses.
When these methods are done well, opt-in panels can produce results that track probability-based surveys reasonably well on many topics. But performance is typically uneven. On some measures, especially ones that relate to online behaviour or civic engagement, volunteers can look quite different from the general population, and matching on demographics alone may not capture that.
Probability panels as a middle path
A hybrid exists. Some organisations recruit panel members using probability methods, such as sampling from address lists, and then invite them to complete surveys online, sometimes providing devices or internet access to those who lack it. This keeps the random recruitment step while using the cheaper online mode afterwards. It is more expensive than a standard opt-in panel, and it still faces the usual non-response and attrition issues.
What each can support
The practical difference comes down to the strength of the claim you can reasonably make.
| Type | Typical strength | Typical limitation |
|---|---|---|
| Open website or social vote | Shows how engaged participants feel | Cannot be generalised to a wider public |
| Opt-in online panel with quotas and weighting | Useful, timely estimates, especially for comparing groups or tracking change | Relies on modelling assumptions; may miss people unlike panel members |
| Probability sample | Clearer basis for estimating uncertainty | Costly; affected by missing lists and non-response |
None of these is automatically right or wrong. The question is whether the design fits the claim being made.
Why the margin of error needs care here
A formal margin of error assumes random selection. For an opt-in sample, a printed margin is either an approximation based on a model or a figure borrowed by convention. It can be a helpful guide, but it carries less certainty than the same number attached to a probability sample. Be a little wary when a precise-looking plus-or-minus appears next to a poll that never described how respondents were found.
Questions to ask of any poll
- How were respondents found: invited at random, or did they choose to take part?
- If it was a panel, who runs it and how is it recruited?
- What adjustments were made, and are they described?
- Does the headline claim go beyond what the design can support?
Transparent pollsters answer these questions in a methods note. When a result comes with no explanation of how people were reached, it is reasonable to treat it as a snapshot of whoever happened to answer, rather than a measurement of public opinion at large.