Sampling, Simulation & Inference
A well-chosen thousand can speak for millions, provided you admit how far off you might be.
A poll asks a thousand people and reports 52%.
There are millions of voters. Nobody asked them. Yet the number gets printed as though it were a fact.
Ask a different thousand and you would get a different answer — maybe 50%, maybe 54%. The number moves.
So the honest report is not a number. It is a range, together with how confident you are that the truth is inside it.
That is inference: using a sample to say something about a population, and being explicit about the uncertainty. The margin of error is not an admission of failure — it is the point.
Three kinds of study, three kinds of conclusion
A survey describes a population. It can tell you what people think, and nothing about why.
An observational study watches without interfering, so it can find associations but never causes.
Only a randomised experiment, where the treatment is assigned by chance, can establish cause.
The claim must match the design
An observational study showing coffee drinkers live longer cannot say coffee helps. Something else may cause both, and only random assignment rules that out.
Why random matters more than large
A well-conducted simple random sample avoids selection bias by design, and sampling error usually shrinks as the sample grows. Nonresponse and measurement errors can still introduce bias.
A biased sample is wrong in a fixed direction, and collecting more of it makes the wrong answer more precise, not more true.
Asking only people who answer a landline at noon tells you about that group, however many you ask.
Size cannot rescue bias
Two million biased responses beat nobody. A thousand random ones beat two million biased ones.
The margin of error
For a proportion from a simple random sample, a rough conservative margin-of-error shortcut is 1√n at 95% confidence.
A thousand people gives about 1√1000 ≈ 3%, which is why so many polls report exactly that.
To halve the margin you need four times the people, because of the square root.
What 95% confidence means
Not that there is a 95% chance the truth is in this interval. It means that if you repeated the whole process many times, 95% of the intervals built this way would contain it.
Reading a result honestly
A poll at 52% with a 3% margin means the truth is plausibly anywhere from 49% to 55%.
Since that range includes 50%, the poll does not establish a lead. "Too close to call" is the accurate phrase.
Overlapping individual intervals alone do not decide whether two candidates differ. To compare them, analyse the difference directly, accounting for how the estimates are related.
Check whether the interval crosses the line
Whatever value would change the conclusion — 50% here — ask whether it is inside the interval. If it is, the data has not decided.
Simulation instead of formulas
If a formula is out of reach, simulate: model the situation with coins, dice or a random number generator and run it many times.
Choose the alternative first: unusually many heads is one-sided; unusually many heads or tails is two-sided. Count how often the assumed model produces a result at least that extreme. If that is rare, chance is a poor explanation.
This is how a statistician decides whether a coin is loaded, or a treatment really worked.
What "rare" usually means
Under 5% is the common threshold. It is a convention, not a law of nature, and it is worth remembering that one result in twenty clears it by luck.
Worked examples
Reading a poll
A poll of 1000 reports 52% support with a margin of error of 3%. Can the candidate claim a lead?
- The interval is 52 ± 3, so 49% to 55%. Estimate plus and minus the margin.
- That range includes 50%. 50% is the value that decides a lead.
- So the data does not establish a lead. The true support could plausibly be below half.
No — the interval includes 50%
How many to ask
A margin of error of about 3% needs roughly what sample size? What about 1.5%?
- Margin ≈ 1√n, so 0.03 = 1√n gives √n ≈ 33. Rearrange for √n.
- So n ≈ 1100. Square it.
- Halving the margin needs four times the sample: about 4400. The square root is why it is four times, not two.
about 1100, then about 4400
Judging a claim
A coin lands heads 62 times in 100 flips. A simulation of fair coins gives 62 or more heads in 1.0% of trials. What can you say?
- If the coin were fair, this result would show up about once in a hundred runs. That is what the simulation measured.
- 1% is below the usual 5% threshold. Rare enough to doubt the assumption.
- So there is good evidence the coin is not fair. Evidence, not proof — one run in a hundred does happen.
Good evidence the coin is biased
Words to know
- Population
- Everyone or everything you want to know about.
- Random sample
- A sample where every member had an equal chance of being chosen.
- Bias
- A flaw in how the sample was gathered that tilts the result in one direction.
- Margin of error
- How far the sample result might sit from the truth.
- Confidence interval
- The range formed by the estimate plus and minus the margin of error.
Mistakes to watch for
Thinking a big sample fixes bias. The 1936 Literary Digest poll asked two million people and still called the election wrong, because it only reached car and phone owners. Randomness matters more than size.
Claiming cause from an observational study. Finding that coffee drinkers live longer does not mean coffee helps — something else may cause both. Only random assignment of the treatment supports a causal claim.
Reporting the estimate without the margin. "52% support" sounds decisive; "49% to 55%" shows the race is open. The margin is part of the result, not a footnote.
Reading 95% confidence as a probability about this interval. It describes the method: repeat the process many times and 95% of the intervals would capture the truth. This particular one either does or does not.
Using interval overlap as a universal significance test. If one candidate is 49–55% and the other 45–51%, overlap alone does not settle whether they differ. Test the difference itself; estimates from the same poll are related.
The 1936 Literary Digest poll asked two million people and still got the election wrong, because it surveyed car and telephone owners. A huge biased sample is worse than a small fair one.
Quadrupling the sample size only halves the margin of error, because the margin shrinks with the square root. That is why polls stop at about a thousand — beyond it, accuracy gets expensive fast.
Simulation lets you answer questions with no formula at all: shuffle the data ten thousand times and count how often chance alone produces what you saw.