The list comes before the sample
To study a large group, researchers usually examine a smaller part of it and extend the result to the whole. The part examined is the sample. But a sample has to be drawn from somewhere, and that somewhere is a list or a method of reaching people: a register of addresses, a set of telephone numbers, a membership database, a platform’s users, a store’s customers.
That list is known as the sampling frame. Anyone who is not on it has no chance of being chosen, however carefully the draw is done afterwards. This is why the frame matters so much: it sets the outer boundary of what a study can possibly find. Many apparent trends are, at least in part, descriptions of the frame.
A simple illustration
Imagine a question about how many people in a town cycle to work. Suppose the researcher draws names at random from the membership list of a local cycling club. The sample might be perfectly random within that list, and large. Yet the finding, that most respondents cycle to work, says almost nothing about the town, because the list consists of people already inclined to cycle.
The error here did not come from small numbers or careless drawing. It came from the list. This is the key point: increasing the number of respondents does not repair a frame that leaves out the wrong people. A million responses from an unrepresentative list can be less informative about a population than a few hundred from a well-built one.
Common frames and what they miss
Different frames have characteristic gaps.
- Address or population registers. Often broad, but may omit people without stable housing, people in institutions, and recent movers whose records are out of date.
- Telephone lists. Coverage depends on who has the kind of phone being called and who answers unknown numbers. As habits change, a list that once covered most households may cover fewer.
- Online panels and opt-in groups. Limited to those who chose to join, as discussed under survey panels and opt-in polls.
- Platform users. Limited to those who use the service, and to those who are active on it, a point covered in platform data.
- Customer or membership databases. Limited to people who already do business with an organisation.
- Street or event interviews. Limited to people who happen to be in a particular place at a particular time.
Each of these can be reasonable for a narrow question about its own members. Problems arise when the result is described as if it applied to everyone.
Under-coverage, over-coverage and duplication
Frames fail in several ways. Under-coverage leaves some people out. Over-coverage includes people who are not part of the target group, such as businesses in a list of households. Duplication gives some people more than one chance of selection, for example someone with two phone numbers or two accounts. Researchers often try to correct for known problems, but the corrections depend on knowing how the frame differs from the target population, which is not always known.
Chance, volunteers and convenience
How people are drawn from the frame matters as well. Selecting by chance, where each person on the list has a known probability of inclusion, allows statistical reasoning about error. Convenience samples, such as whoever answers a website poll or stops in a street, offer no such basis. Those who volunteer often differ from those who do not, which is a form of selection effects. Even a perfect list produces a skewed sample if only the keenest respond, an issue explored under non-response bias.
How the frame turns into a trend
Trend claims depend on comparing groups over time, and here the frame can mislead in two ways.
First, the frame can itself change. If a platform grows in popularity among a new age group, then data from its users will seem to show a cultural shift, when part of what changed is who is present. An apparent rise in a behaviour among “users” might simply reflect that the user base now includes more of the kind of person who does it.
Second, a trend that emerges from a narrow early group may not generalise. Early adopters differ from later ones in income, interests and attitude, so the first people to take up something are rarely a preview of the rest. A study of early adopters can overstate the eventual spread, or describe features that fade as the practice broadens.
Survivorship can be built into frames too. A list of currently active businesses or current members omits those who closed or left, which can make a practice seem more successful or popular than it was, a problem covered under survivorship bias.
Weighting cannot always rescue a bad frame
Statistical adjustments, covered under survey weighting, can rebalance a sample so that its mix of ages, regions and similar characteristics matches known totals. This helps when the frame covers all kinds of people but in the wrong proportions. It helps much less when whole kinds of people are absent, because there is nothing to scale up. And if the people who are present differ from the absent people in ways unrelated to the characteristics used for adjustment, the correction can only be partial.
Questions to ask about any study
- What list or method was used to find people?
- Who could not have been selected under that method?
- Were people picked by chance, or did they choose to take part?
- Does the headline describe the group studied, or a larger population?
- If this claims a change, did the frame stay the same across time?
When the answers are given plainly, a reader can judge how far a finding extends. When they are missing, the safe reading is narrow: this is what was found among the people who could be reached by this method. Sampling does not make findings false, but it does decide how far they can be carried.