Probability sampling is the gold standard, but it needs a frame, a budget, and cooperation. When those are missing, researchers reach for non-probability sampling, where selection is driven by convenience or judgment, not chance. The price is a loss of the one property that makes inference honest.
The deep problem is that bias is systematic: it does not shrink as the sample grows. The notebook shows a convenience sample over-estimating app usage by about +3.5 hours whether n is 100 or 20,000. More data sharpens the wrong answer.
Convenience Sampling
Convenience sampling takes whoever is easiest to reach, mall shoppers, online volunteers, your own social feed. It is the cheapest method and the most biased, because "easy to reach" is rarely the same as "representative".
In the notebook the reachable pool has a mean age of 31 against the population's 47, and since younger users use the app more, every convenience sample over-estimates usage by about 3 to 4 hours. The bias is identical at every sample size, the hallmark of a systematic error.
Voluntary Response Sampling
A voluntary response sample is self-selected: a website pop-up, a call-in poll, a product review. The people who bother to respond are precisely those with the strongest feelings, so the extremes are wildly over-represented.
The notebook makes it concrete: the true mean satisfaction is 6.48, but voluntary responders average 7.82, and the share of extreme ratings (0–2 or 9–10) jumps from 9% in the population to 31% among volunteers. This is why online star ratings look so polarized: the lukewarm middle never clicks "submit".
Quota & Judgment Sampling
Quota sampling is the most sophisticated of the bunch: fix the proportions of visible traits (age, gender, region) to match the population, then fill each quota however is convenient. Judgment (or purposive) sampling goes further and hand-picks "typical" cases. Both fix the margins but leave the interior exposed.
Quota sampling helps: the bias drops from about +3.5 hours (pure convenience) to +2.5 hours in the notebook. But it does not vanish, because quotas only control the variables you quota on. Any lurking variable, here, willingness to respond, still skews the result inside every cell. Better than convenience, still not representative.
Snowball Sampling
For hidden or hard-to-find populations, rare patients, undocumented workers, niche hobbyists, no frame exists. Snowball sampling asks each respondent to refer others, growing the sample through social links. It reaches the unreachable, but it over-samples the well-connected.
In the notebook, snowball respondents average 33 connections against the population's 24, and because connectedness correlates with usage, the estimate runs +1.3 hours high. Snowball sampling is a valuable last resort for invisible populations, but it tilts toward the social core and cannot claim to be representative.
Selection Bias in Machine Learning & AI
Almost every large AI dataset is a giant non-probability sample. Web text, scraped images, app logs, and volunteer labels are all convenience or voluntary-response data, and they carry exactly the biases of this chapter, at enormous scale.
| Method (this chapter) | In ML / AI it appears as | The risk |
|---|---|---|
| Convenience sampling | Web-scraped training data | over-represents whoever posts online; under-represents everyone else |
| Voluntary response | User-submitted reviews / labels | extreme, motivated users dominate the signal |
| Quota sampling | Balanced-by-attribute datasets | balanced margins, hidden within-group bias remains |
| Snowball sampling | Referral / network-crawled data | over-samples hubs, misses the periphery |
| Post-stratification | Reweighting / importance weighting | corrects known imbalances, not unknown ones |
A model trained on a biased sample learns the bias as if it were truth, and at web scale the bias is enormous and invisible. The crucial point from the notebook: more data does not fix selection bias, it entrenches it, which is why a bigger scrape can make a model more confidently skewed. The partial remedy is the same one statisticians use: post-stratification weighting. In the notebook, reweighting a convenience sample by its true age composition pulls the estimate from a biased 17.0 back to 14.0, against a truth of 13.6. Weighting and its cousin importance sampling can correct known imbalances, but they are powerless against the imbalances you never measured. The only real cure is representative collection.
Watch the bias appear in Python
The companion notebook builds a 200,000-person population and runs convenience, voluntary-response, quota, and snowball sampling, measuring the bias each one introduces, showing it survives any sample size, and contrasting it with an unbiased simple random sample.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy and matplotlib and launch
jupyter notebook.
🎓 Key Takeaways
- ✓Non-probability sampling uses unknown, unequal selection chances, so it carries selection bias and untrustworthy error bars.
- ✓Convenience over-samples the easy-to-reach (a constant +3.5 h bias here, at every n).
- ✓Voluntary response over-samples the extremes (extreme share 9% → 31%), inflating polarization.
- ✓Quota fixes the visible margins but leaves residual bias (+2.5 h); snowball reaches hidden groups but favors the well-connected.
- ✓In ML/AI: web-scraped and user-submitted data are giant non-probability samples; more data entrenches bias, and weighting fixes only the imbalances you can measure.
Practice Challenges
Five short challenges, beginner to intermediate. Try them with NumPy before checking the solutions.
Convenience bias is constant
Sample only the young, reachable half of a population at n = 100, 1,000, 10,000 and show the bias does not shrink.
Voluntary response over-samples extremes
On a 0–10 satisfaction scale, let only the extremes respond and compare the extreme share in the population to the responders.
Quota reduces but does not remove bias
Match age-band proportions but pick high-value people within each band. Show the bias is smaller than pure convenience, but not zero.
Snowball over-samples the well-connected
With value rising in social connectedness, sample with probability proportional to connections and show the upward bias.
p = connections / connections.sum().Fixing it with weights
Reweight a convenience sample so each age band counts according to its true population share, and recover the mean (post-stratification).
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on non-probability sampling. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
You now know how to sample well and how sampling goes wrong. The Determining Sample Size chapter answers the practical question every study faces: how many observations do you actually need? We derive the sample-size formulas for a mean and a proportion.