Sampling, Survey Design and Why Most Surveys Are Junk
A number from a survey is only as good as who got asked and how
"73% of people prefer…", "a new poll shows…", "9 out of 10 users…" — survey numbers are everywhere and they carry an air of hard fact. Most of them are junk, and not because someone lied. They're junk because of two invisible things: who got asked (sampling) and how they were asked (question design). Get either wrong and the number is meaningless even if every respondent answered honestly.
This is the survey-side companion to DT-02 (how charts lie) and DT-03 (correlation): the numbers can be "real" and the conclusion still worthless. The good news is that a handful of checks catch the vast majority of bad surveys, and once you can run them, "a study found…" stops being a magic phrase and starts being a claim you can inspect.
What you'll have at the end
- Why who you ask matters more than how many
- The question-design tricks that manufacture the answer they wanted
- Three real surveys taken apart, with the fatal flaw named
Sampling: who got asked
You can't ask everyone, so you ask a sample and generalise to the population. That only works if the sample represents the population — and this is where most surveys die.
- Representative means random. In a proper sample, everyone in the population has a known, equal-ish chance of being picked, so the sample looks like the whole in miniature. Anything that skews who gets in breaks the generalisation.
- The killers are selection biases:
- Self-selection — people opt in. Online polls, "vote in our poll!", reviews, call-in shows: only people with strong feelings or spare time respond, so the result reflects them, not everyone. This is the single most common flaw in surveys you'll see online.
- Convenience — you ask whoever's easy (your followers, people outside one shop, your class). Easy ≠ representative.
- Non-response — the people who don't answer differ systematically from those who do, so even a "random" sample skews toward the type who bothers to reply.
And the counter-intuitive one: a bigger sample does not fix a biased one. The famous case: in 1936 a magazine polled 2.4 million people and confidently predicted the wrong US election winner, because its sample (car and phone owners in the Depression) was skewed rich. A tiny random sample beat a giant biased one. So when a survey brags about its huge number, that tells you nothing about whether it's any good — ask who, not how many.
Question design: how they were asked
Even with a perfect sample, the wording can manufacture the result:
- Leading questions. "Don't you agree that…?" or "How much did you enjoy…?" pushes toward a yes. The neutral version asks whether, not how much, and doesn't signal the wanted answer.
- Loaded words. "Should the government waste money on…?" vs "spend money on…?" — same policy, opposite framing. Emotive words steer the answer.
- Framing. "90% survive" vs "10% die" get different responses to the identical fact (loss aversion, DT-07). How the choice is worded moves it.
- Double-barrelled questions. "Do you support cheaper and faster transport?" — what if you want one, not both? It bundles two questions and you can't answer either cleanly.
- Acquiescence bias. People tend to agree, especially with "do you agree that…?" statements — balanced questions offer both sides evenly.
- Order effects. Earlier questions prime later answers; the order of answer options nudges choices.
- A false scale / missing option. Response choices that leave out where you actually sit, or a scale loaded to one side, force a distorted answer.
A survey designed to get a headline can hit it with all-honest respondents, purely through wording.
The other tells
- Margin of error and confidence. A real poll reports something like "±3%". If two candidates are "48% vs 45%" with ±3%, that's a tie — the gap is inside the noise. A number with no margin of error is a number pretending to more precision than it has.
- Who ran it and who paid. A survey funded by a party with a stake in the answer isn't automatically wrong, but it's a reason to look very hard at the sample and the wording. (Follow the money — DT-09.)
- Was the question even published? If you can't see the exact wording and who was sampled, you can't evaluate it, and "a study found" with no method is a rumour with a percentage.
What this means for you
- A survey number is only as good as who was asked and how — both invisible, both usually where it breaks.
- Representative = random. Self-selection, convenience and non-response are the killers; online opt-in polls are the worst offenders.
- Bigger doesn't fix biased — a huge skewed sample loses to a small random one. Ask who, not how many.
- Wording manufactures answers: leading, loaded, framed, double-barrelled, agree-biased, order effects, rigged scales.
- Check the margin of error, who funded it, and whether the exact question is even shown.
Exercise (35 min, verifiable output)
- Find three real surveys or polls — a news poll, a brand's "9 out of 10", an online/social poll, a "study shows" headline.
- For each, ask: who was sampled (random? self-selected? convenience?), how big and does that matter, can you see the exact question, who paid, is there a margin of error.
- Name the fatal flaw in each (most will have an obvious one — usually self-selection or a leading question).
- Find or imagine one leading/loaded question and rewrite it neutrally — asking whether, not how much, with balanced options.
- Note which of the three, if any, you'd actually trust — and why.
✅ Finish check: three real surveys with the fatal flaw named in each, and one leading question rewritten into a neutral one.
Summary card
- A survey number = who got asked (sampling) × how they were asked (wording); either breaks it, even with honest answers.
- Representative means random. Self-selection / convenience / non-response are the killers; opt-in online polls are worst.
- Bigger ≠ better — a huge biased sample loses to a small random one (1936). Ask who, not how many.
- Wording tricks: leading, loaded words, framing, double-barrelled, acquiescence, order effects, rigged scales.
- Check: margin of error (a gap inside it is a tie), who funded it, and whether the exact question is shown at all.
Sources
- Groves, R. et al. — Survey Methodology, 2009
- Squire, P. — Why the 1936 Literary Digest Poll Failed, 1988
- Schuman, H. & Presser, S. — Questions and Answers in Attitude Surveys, 1981
Next lesson: DT-06 — p-values, Statistical Significance and Effect Size (L3) Related: DT-02 How Charts Lie · DT-01 Reading Data · DT-09 Verifying a Source · DT-03 Correlation Is Not Causation Path: Hard to Fool — extended