Why raw responses need adjusting
Even a well-designed survey rarely produces a set of respondents that mirrors the population. Some groups answer more readily than others, and some are harder to reach. A typical raw sample might contain too many older people, too many graduates, or too few people from certain regions compared with official population figures.
If those groups differ in their views, the raw percentages will lean towards the over-represented groups. Survey weighting is the standard way to reduce that imbalance. The idea is straightforward: respondents from under-represented groups count for more in the final tally, and respondents from over-represented groups count for less.
A simple illustration
Suppose, purely for illustration, that a population is half group A and half group B, but a sample ends up with three-quarters from group A and one-quarter from group B. To restore the right balance, each group B respondent might be counted as twice a typical respondent and each group A respondent as two-thirds of one. After that adjustment, the two groups carry equal weight in the overall result, matching the population split of one half each.
Check the arithmetic: if the sample has 100 people, 75 from A and 25 from B, then weighting A at two-thirds gives 50, and weighting B at 2 gives 50. The two groups are now equal, as intended. The numbers are invented only to show the mechanism and do not describe any real survey.
Real surveys do this across many characteristics at once.
Common methods
Post-stratification
Respondents are sorted into cells, for example age band by sex by region, and each cell is adjusted to match its known share of the population. It works well when there are not too many cells, but it breaks down when some cells have very few or no respondents.
Raking
Also called iterative proportional fitting, raking adjusts weights so that the sample matches the population on several characteristics separately, such as the overall age mix, the overall education mix and the overall regional mix, without needing every combination to be filled. It repeatedly cycles through the characteristics until the weighted sample matches all of them reasonably closely.
Model-based adjustment
More recent approaches use statistical models that estimate how opinion varies across demographic groups and then project those estimates onto the known makeup of the population. These can be helpful when samples are small or uneven, but they depend on the model assumptions being sound.
Weighting for selection and response
Some adjustments account for the chance of being selected in the first place, such as when households of different sizes have different chances of contributing a respondent, or for patterns of who responds. The reasoning is related to the discussion of non-response bias.
Where the benchmarks come from
Weights need a target. Pollsters typically take population figures from official statistics such as censuses or large government surveys. These benchmarks have their own limits: they may be out of date, based on definitions that differ slightly from the survey’s, or uncertain for smaller groups. If the benchmark is wrong, the weighting carries that error into the result.
What weighting can fix
Weighting works best when the variables used for adjustment are strongly related both to who ends up in the sample and to the opinion being measured. If, say, age strongly affects both willingness to answer and the view in question, correcting the age balance can reduce bias a great deal. It is also a clean tool for fixing known, mechanical imbalances, such as deliberately oversampling a small group to get enough responses from it.
What weighting cannot fix
It cannot create information that is absent. Several limitations are worth knowing.
- Hidden differences. If people of the same age, sex and education differ in a way that matters, for instance in trust or political engagement, and that difference is related to who responds, weighting on demographics will not remove the bias. This is why modern pollsters sometimes also weight on variables like past voting or political interest, though these require reliable benchmarks of their own.
- Empty groups. If nobody from a particular group is in the sample, no weight can stand in for them.
- Measurement problems. Weighting does not correct poor question wording or dishonest answers, including those that stem from social desirability.
- Self-selected samples. Adjusting a volunteer panel can improve it, but it does not turn it into a random sample, a distinction at the heart of the comparison between opt-in polls and random samples.
How adjustment can go wrong
Large weights and added noise
If a few respondents represent a large group that was barely reached, they receive big weights, and the result then hinges on what those few said. This inflates uncertainty. Statisticians describe it as reducing the effective sample size: a survey of a thousand people with very uneven weights can carry less information than a thousand evenly weighted ones. Pollsters often trim extreme weights to limit this, at the cost of leaving some imbalance in place.
Choosing the wrong variables
Weighting on the wrong characteristics, or too many of them, can introduce instability or false confidence. Different reasonable choices lead to different published numbers from the same raw data, which is one reason pollsters can disagree without either being careless.
Overconfidence
A published margin of error that ignores the effect of weighting can understate real uncertainty. Reputable organisations often report a larger figure that accounts for the weights, as discussed in the article on the margin of error.
Adjusting toward a view of the world
If weighting targets, such as the expected makeup of the electorate, reflect a pollster’s assumptions about what will happen, the result partly embeds those assumptions. This is not wrong, but it means the final figure mixes data with judgement.
Reading weighted results
- Look for a methods note describing which characteristics were weighted and where the benchmarks came from.
- Be cautious when the raw sample was very different from the population, since heavy adjustment relies more on assumptions.
- Notice whether the reported uncertainty reflects the weighting.
- Remember that weighting corrects what it can see. It is a repair of known gaps, not a guarantee against all bias.
Used carefully and transparently, weighting makes survey results more representative than raw data would be. Used carelessly, it can give an air of precision to a result that remains uncertain.