A survey measures the question, not just the opinion
When a report says “most people support” or “many people say they use”, the figure is an answer to a specific set of words. Change the words and the answer can change, sometimes noticeably. This does not mean surveys are useless. It means the question is part of the measurement, and a careful reader looks at it before accepting the result.
Survey researchers have long studied these effects, and most professional questionnaires are tested and revised to reduce them. Even so, question wording remains one of the most common reasons two reasonable surveys on the same subject seem to disagree.
Loaded and leading phrasing
Some words carry a built-in judgment. Consider asking whether people support “helping” a group versus supporting “handouts” to the same group. The underlying policy might be identical, yet the words signal different attitudes, and responses often shift accordingly.
Leading questions go further by suggesting the expected answer: “Don’t you agree that…” invites agreement. Prefacing a question with a claim, such as “Given that many experts are concerned about X, do you think…”, can also steer people by supplying a frame before they have thought about it.
A neutral version typically describes the topic plainly, avoids emotive terms and does not signal which answer is preferred.
Double-barrelled questions
A question that asks about two things at once cannot be answered cleanly. “Do you enjoy cooking at home and eating healthily?” leaves a person who loves cooking but not salads with no honest answer. The responses that come back then blend two ideas, and anyone reading the result cannot tell which one drove it. Splitting such items into separate questions is the usual fix.
Answer options shape answers
Responses are not only a reaction to the question; they are also a reaction to the menu of choices.
- Scale ranges. If a question about how many hours someone spends on an activity offers ranges that start low, respondents may infer that the low ranges describe a typical person and place themselves accordingly. Different ranges can lead to different spreads of answers for the same people.
- Missing options. Offering only “agree” or “disagree” forces a choice on people who feel neutral or have not thought about it. Adding “don’t know” or “neither” often changes the share who pick a side.
- Open versus closed. An open question lets people name things in their own words. A closed list prompts them with options they might not have thought of, which often raises the share mentioning those options.
- Balanced versus unbalanced scales. A scale with more positive than negative choices can nudge answers upward.
Order and context effects
Earlier questions can colour later ones. If a survey first asks about a series of worries and then asks about general mood, the earlier items may bring negative topics to mind. If two related questions appear in sequence, people may try to appear consistent, or may avoid repeating themselves. Researchers often randomise the order of items or rotate options to spread these effects, but a published headline rarely says whether they did.
Order also operates within a list of answer choices. In some formats, people tend to pick options shown earlier, and in spoken interviews the last-heard option can have an edge. The size of such effects varies, which is why careful designs rotate them.
Time frames and definitions
Vague words invite different interpretations. “Regularly”, “often” and “recently” mean different things to different people. A question about how often someone eats out might be answered very differently depending on whether it asks about “last week” or “in a typical month”, because memory and estimation work differently over different spans. Terms such as “side hustle” or “plant-based” may also have no single definition, so respondents quietly apply their own. Definitions matter in the same way for the gig economy, where the category itself changes the answer.
Why this matters for trend claims
Tracking change over time depends on asking the identical question each time. If the wording, scale or order changes between waves, a movement in the numbers may reflect the survey rather than the public. That is why tracking surveys usually keep wording fixed and flag changes. Comparing results from two different organisations, each with its own phrasing, is often less reliable than the comparison looks.
Wording also interacts with the honesty problem. A question that makes one answer sound disreputable may encourage people to give the more acceptable reply, a pattern discussed under social desirability. And a question about a topic people barely think about can produce answers that are partly invented on the spot, so the result may reflect the moment of asking more than a settled view.
Questions worth asking of a published survey
- What was the exact wording, and is it available to read?
- Was the question neutral, or did it hint at an answer?
- Were the options balanced, with a way to say “don’t know”?
- Was the same wording used in earlier waves if change is being claimed?
- Does the headline describe the question asked, or a looser paraphrase of it?
A headline that paraphrases “agreed with a statement that…” as “believe that…” can quietly enlarge the claim. Reading the original question is often the quickest way to judge how much weight a figure can bear.