p-values, Statistical Significance and Effect Size
"Statistically significant" does not mean what the headline wants you to think
A study is "statistically significant," so the headline treats it as proven, important, and large. It is, at most, evidence of one narrow thing — and none of those three. The phrase is probably the most misused term in all of public reasoning: a scientific-sounding stamp that most people, including many who use it, cannot actually define.
By the end of this lesson you'll be able to define it precisely, say exactly what it can and can't tell you, and never again read "significant" as "true" or "big."
Why nobody explained it this way
Because the phrase is useful left vague. To a journalist it means "real, print it." To a marketer it means "our claim is backed by science." To a researcher under pressure to publish, crossing the significance line is the difference between a career-making paper and a rejected one. Everyone touching the number benefits from it sounding more powerful than it is. Almost no one benefits from explaining that it's a probability about data, not about truth.
1. The setup: what the test actually asks
Statistical tests are built around a deliberately boring starting assumption called the null hypothesis: nothing is going on here. The new drug does nothing; the two groups are the same; the coin is fair.
The test then asks a very specific question: if the null were true, how surprising is the data we actually got? That's it. Everything the p-value means lives inside that sentence.
2. The p-value, defined precisely
The p-value is the probability of seeing data at least as extreme as yours, assuming the null hypothesis is true.
Read that twice, because the meaning is almost the reverse of how it's used:
- A small p-value (say 0.01) means: if nothing were going on, data this extreme would be rare (1% of the time). So the data is surprising under "nothing's going on," which is weak evidence against the null.
- It is NOT the probability that the null is true. It is NOT the probability your result happened by chance. It is NOT the probability you're right. Those are the three things everyone thinks it is, and it is none of them. It's a statement about the data given an assumption, not about the truth given the data.
"Statistically significant" simply means the p-value fell below a chosen threshold — almost always 0.05. And that threshold is a convention, not a law of nature: 0.05 is an arbitrary line someone drew, roughly "less than 1 in 20." Nothing magic happens at 0.049 that doesn't at 0.051.
3. What a p-value cannot tell you
Three things it is completely silent on — and all three are what you actually care about:
- Whether the effect is large. A p-value says a signal is probably there, not how big it is.
- Whether it matters. Statistical significance is not practical importance.
- Whether it will replicate. One significant study is not a settled fact; many fail to reproduce.
And here's the sting that ties them together: with a big enough sample, almost any tiny difference becomes "significant." Test a new font on ten million users and you can prove it raises clicks by 0.001% with p < 0.001 — a rock-solid signal of an utterly meaningless effect. Significance answers "is there a signal?"; it is blind to "is the signal worth anything?"
4. Effect size: the number that was actually the point
The missing partner to the p-value is the effect size — how big the difference actually is, in units you can reason about.
- Sometimes it's plain: "the treatment group scored 8 points higher," "conversions rose from 2% to 3%."
- Sometimes it's standardised (like Cohen's d) so effects can be compared across studies.
The division of labour is clean and worth memorising:
| Question | Answered by |
|---|---|
| Is there a real signal, or could this be noise? | p-value / significance |
| Is the signal big enough to care about? | effect size |
A responsible result reports both. A headline that gives you significance and hides the effect size is hoping you'll assume "significant" means "big." Your first move on any study is to go find the effect size — and if it isn't reported, treat that as a red flag, not a rounding error.
5. The three traps that manufacture significance
Even a real p-value can be gamed, and three mechanisms do it constantly:
- p-hacking. Test twenty things and, by chance, about one will cross 0.05 even if nothing is real. Report only that one, hide the other nineteen, and you have a "significant finding" that's pure noise. (Fishing for a significant subgroup is the same move.)
- Publication bias. Journals and media publish exciting significant results and quietly bin the null ones. So the studies you see are a biased sample skewed toward flukes — a big reason so many findings fail to replicate (Ioannidis's argument).
- Significance–importance swap. Presenting a statistically significant but trivially small effect as though it were meaningful. Technically true, practically empty.
What this means for you
"A study showed, statistically significant, that X" is no longer a full stop — it's the start of three questions you now know to ask: How big is the effect? (find the effect size). Could this be one test among many? (p-hacking, publication bias). Has it replicated? (one study is not a fact). That turns you from someone a significance claim can push around into someone who can read what it actually supports — which is usually far less, and occasionally far more, than the headline says.
Try it: read one real result (25 min)
- Find one study or a news article about one, ideally reporting a p-value ("p < 0.05," "p = 0.03," "statistically significant").
- State what the p-value claims in a single correct sentence — "the probability of data this extreme if there were no real effect" — and one sentence on what it does not claim (that the finding is true / large / certain).
- Find the effect size. How big is the actual difference, in real units? If the article doesn't report it, note that — and go one layer deeper to the study if you can.
- Check the sample size. Very large → ask whether a "significant" effect might be tiny. Very small → ask whether it's likely to replicate.
- Write a two-line verdict: is there probably a real signal, and is it big enough to change any decision? Those are two separate answers.
✅ Finish check: you have, for one real result, a correct one-sentence definition of its p-value, its effect size (or a note that it's missing), and a two-line verdict separating "real" from "big enough to matter."
Summary card
- The null hypothesis is "nothing's going on." A test asks: how surprising is this data if the null were true?
- p-value = probability of data this extreme, assuming the null is true. It is not the probability the claim is true, or that the result was "chance."
- "Significant" = p below a threshold, almost always the arbitrary 0.05.
- A p-value can't tell you if an effect is large, important, or replicable — and with a big enough sample, any tiny effect becomes significant.
- Effect size answers "how big?" A trustworthy result reports both; a missing effect size is a red flag.
- p-hacking, publication bias, and the significance–importance swap manufacture impressive p-values from noise.
- Read any claim with three questions: how big, one-of-many, replicated?
Sources
- American Statistical Association — Statement on p-Values and Statistical Significance (2016)
- Ioannidis, J. — Why Most Published Research Findings Are False (2005)
- Cohen, J. — foundational work on effect size and statistical power
- Wasserstein & Lazar — The ASA's Statement on p-Values: Context, Process, and Purpose
Next lesson: DT-05 — "A Study Showed That…": How to Actually Read a Paper (L3) Related: DT-01 Reading Data (L1) · DT-04 Sampling and Survey Design (L2) · DT-03 Correlation Is Not Causation (L1) Path: Hard to Fool — related