The sixteen capstones in the previous part all started the same way: someone handed you a spreadsheet. That is the normal situation, and it is also the situation in which the most important decisions have already been made by someone else, usually without a record of why.
No analysis can repair a bad sample. A confidence interval describes uncertainty from drawing a sample; it says nothing about the people your frame never contained or the ones who declined to answer. Those errors do not shrink as n grows, and the arithmetic never announces them.
The Chain From Population to Estimate
Between the group you care about and the number you finally report there are four links, and each one is a place error gets in. Only one of the four is the kind a p-value knows about.
| Error | Where it enters | A concrete example | Does more data help? |
|---|---|---|---|
| Coverage | The frame does not match the population | Surveying customers by email when a fifth of them never gave an address | No |
| Sampling | You drew some members and not others | The 200 people you happened to select differ from the 200 you might have | Yes |
| Non-response | Selected members do not answer | The dissatisfied stop replying, so satisfaction looks higher than it is | No |
| Measurement | The instrument distorts the answer | A leading question, or a scale with no neutral option | No |
Choosing How to Draw
The single most consequential line in the plan is whether every member of the population has a known, non-zero chance of selection. If they do, the sample is a probability sample and the machinery of the previous twenty-seven parts applies. If they do not, that machinery does not apply, and pretending otherwise is the most common error in applied research.
How Big, and How Precise
Sample size is where intuition fails most reliably. Two facts do most of the work.
Precision improves with the square root of n. To halve a margin of error you need four times the sample, not twice. That single relationship sets the budget for most studies.
The fraction of the population barely matters. A sample of 1,000 estimates a national population of 60 million about as precisely as it estimates a town of 60,000. What matters is the absolute size of the sample, not the share it represents, which is why national polls of 1,000 people are not the absurdity they look.
| Sample size | Margin of error on a 50% proportion | Cost relative to n = 400 |
|---|---|---|
| 100 | ± 9.8 points | 0.25× |
| 400 | ± 4.9 points | 1× |
| 1,000 | ± 3.1 points | 2.5× |
| 1,600 | ± 2.5 points | 4× |
| 6,400 | ± 1.2 points | 16× |
Read the last two rows together. Quadrupling the sample from 1,600 to 6,400 buys 1.3 percentage points of precision for four times the money. Whether that is worth it depends entirely on what decision the estimate feeds, which is why the plan states the required precision first and derives n from it, rather than picking a round number and reporting whatever precision falls out.
Clustering costs you. If you sample schools and then students within them, students in a school resemble each other, so each additional student adds less information than a fresh random one would. The design effect is the multiplier: a design effect of 2 means your 1,000 clustered responses carry the information of 500 independent ones. Plans that ignore it are systematically under-powered.
Non-response costs you twice. Once in sample size, which is easy to fix by inviting more people, and once in bias, which is not fixable that way at all. Plan the invitation count from the expected response rate, then plan the analysis for the bias that remains.
The Instrument Is Part of the Design
A questionnaire is a measuring device, and like any measuring device it can be miscalibrated. The data type of every answer is fixed the moment the question is written, and with it the entire analysis that will be possible later.
| Question type | Produces | What you can compute |
|---|---|---|
| Single choice from a list | Nominal | Proportions, chi-square, Cramer's V |
| Choose all that apply | Several binary variables | Proportions per option, but the options are not mutually exclusive and must not be treated as one variable |
| Likert agreement scale | Ordinal | Medians, top-box shares, rank correlations. Not means |
| Numeric entry | Continuous | Means, standard deviations, the full parametric toolkit |
| Ranking task | Ordinal, dependent within respondent | Rank correlation, Kendall's W |
| Open text | Unstructured | Coding into categories, then whatever the coding produces |
Four failures account for most bad instruments, and all four are cheap to avoid at the writing stage and impossible to repair afterwards.
- The leading question. "How much did you enjoy the new booking process?" has already assumed the answer. "How would you rate the new booking process?" has not.
- The double-barrelled question. "Was the service fast and friendly?" cannot be answered by someone who found it fast and rude, and you will never know which half the answer refers to.
- The missing option. A scale with no neutral point forces an opinion from people who have none, and no "not applicable" option turns irrelevance into a fabricated answer.
- Order effects. Ask about a specific irritation and then about overall satisfaction, and the overall figure drops. The instrument changed the thing it was measuring.
After Fielding: Who Did Not Answer
Every survey in this part ends with a section on the people who are not in it. That is not a formality, and it is where the most commonly repeated mistake in survey reporting lives.
A 20 percent response rate is harmless if the 80 percent who declined are like the 20 percent who answered. A 90 percent response rate is dangerous if the missing 10 percent are all the people who had a terrible experience. The response rate tells you nothing on its own. What matters is whether responding is related to what you are trying to measure, and the response rate cannot tell you that.
Three moves are available, in increasing order of usefulness. Compare respondents to the frame on any variable you hold for everyone, since a frame usually carries age, region, tenure or purchase history whether or not the person replied. Weight so the respondent profile matches known population margins, which is what post-stratification and raking do. Follow up a subsample of non-respondents aggressively and compare them to the easy responders, which is the only one of the three that measures the bias rather than assuming it away.
Weighting deserves one warning. It corrects the variables you weighted on and nothing else, and it makes the estimate less precise, because unequal weights waste information. Weights that vary wildly are a sign the sample was badly out of shape to begin with, and no amount of weighting turns a broken sample into a good one.
How the Framework Changes
The twelve-step framework from the previous part still runs, unchanged, from the moment the data exists. What these capstones add is five steps in front of it.
| Step | Decision | What it determines |
|---|---|---|
| D1 · Define | Who exactly is the target population? | What the estimate is an estimate of |
| D2 · Frame | What list will you draw from? | Coverage error, and who is invisible from the start |
| D3 · Method | How will you draw? | Whether standard errors are valid at all |
| D4 · Size | How precise must the answer be? | Budget, and what differences you can detect |
| D5 · Instrument | How will you ask? | The data type of every variable, and so every test |
| 1 to 12 | The analysis framework, unchanged | Everything from here is the previous part |
Notice that D5 determines the data type, and the data type was step 2 of the analysis framework. These capstones simply push the chain back one link: instead of discovering that a variable is ordinal, you chose to make it ordinal, and you could have chosen otherwise.
Sampling in Data Science & AI
Nothing in this part is confined to questionnaires. Every training set is a sample, and almost none of them come with a sampling plan.
| Where it appears | The sampling question nobody asked |
|---|---|
| Training data for a model | What population does this sample represent, and does the deployment population match it? |
| Scraped corpora | The frame is "whatever was reachable and not blocked". Who is systematically absent from that? |
| Benchmark suites | A benchmark is a sample of tasks. Scoring well on it generalizes only as far as the sample does |
| Human labels and preferences | Which annotators, recruited how, and are their judgments the population's judgments? |
| Logged behavior | You observe only the users the current system already serves well. That is non-response by another name |
Dataset shift, the standard explanation for a model that degrades after deployment, is usually a sampling failure that was invisible at training time: the training frame did not cover the population the model eventually met. The discipline that prevents it is the one in this chapter, written down before collection rather than reconstructed afterwards from a post-mortem. A dataset shipped without a statement of what population it represents is a dataset whose limits nobody can check.
The Four Projects Ahead
Four projects, each starting before the data exists. They become clickable as each one is published; the Contents always shows what is live.
🎓 Key Takeaways
- ✓A confidence interval covers sampling error only. Coverage, non-response and measurement error are design problems, and they do not shrink as n grows.
- ✓Probability or not is the decisive line. Known non-zero selection probabilities are what make a standard error mean anything.
- ✓Precision goes as the square root of n, and barely depends on the fraction of the population sampled. Derive n from the precision you need.
- ✓The instrument fixes the data type, and therefore fixes which tests are available. That decision is made when the question is written, not when the analysis starts.
- ✓The response rate is not the problem; whether responding relates to the answer is. Weighting fixes the variables you weighted on and nothing else.
Quiz: Test Yourself
Eight questions on the plan that comes before the data. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved.