Earlier chapters in this Part treated "the data" as something you sample and clean. But data does not arrive by one route. How it is captured, asked, sensed, observed, logged, or inherited, shapes what it can tell you and which biases it carries. This chapter maps the spectrum before the next chapter dives deep into the most design-intensive mode of all: surveys.
A taxonomy of collection modes, a single design framework that applies to all of them, and five real-world examples, each with a downloadable Excel sheet showing exactly what gets captured and how a statistician would receive it.
The Data-Collection Spectrum
Collection modes range from active (you deliberately elicit or measure each value) to passive (data accumulates as a byproduct, or already exists). They also differ in whether a human is in the loop. No mode is best; each fits different questions.
The crucial divide is designed versus found. Asking, sensing, and structured observing are designed: you decide in advance what to capture. Logs, scrapes, and administrative records are found: they exist for some other reason, so they are cheap and enormous but were never built to answer your question. Most of the biases from the Bias in Data Collection chapter hide in found data.
Designing Any Collection Method
Whatever the mode, a sound collection design answers the same seven questions. A questionnaire's wording, a sensor's calibration, and a scraper's source list are all answers to the very same checklist.
Notice that this is the same arc as the survey chapter's "operationalization", just generalized. A survey operationalizes a construct into items; a sensor system operationalizes a physical quantity into calibrated readings; a scraper operationalizes "media lean" into coded labels. The discipline is identical: define the target, fix the frame, build the instrument, sample deliberately, control quality, check for what is missing, and stay legal.
Five Real-World Examples
Each example below names a mode, explains what is captured, walks through the collection-design decisions, and links a downloadable Excel sheet laid out the analysis-ready way a statistician would receive it (a data sheet, a codebook, and analyst notes).
Production-floor sensor telemetry
This is the data behind predictive maintenance and quality control. Design decisions: which sensors and
where to place them, the sampling frequency (here one reading per minute; too slow and you miss
transients, too fast and you drown in storage), calibration and drift control, and clock
synchronization across machines. Signature risk: an uncalibrated sensor drifts, a systematic
measurement bias, and an outage usually strikes during a fault, making the rows you most need
missing-not-at-random. Keeping machine_state lets you tell a true zero (idle) from
missing data.
Scoring whether news outlets lean left or right
Real media-bias projects (such as AllSides and Ad Fontes Media) do exactly this. Design decisions: the
frame is the list of outlets and the time window, choosing it is a sampling decision. "Lean" is an abstract construct, so it must be operationalized into a coding scheme
just like a survey item (see Survey & Questionnaire Design), and you should have two or more coders label a subset to measure
inter-coder reliability before trusting the labels. Signature risk: paywalled, regional,
or non-indexed outlets are missing, so scraped data is usually a convenience sample with coverage
bias. Record the URL and scrape date for provenance, and respect each site's Terms of Service and
robots.txt.
Store foot-traffic and conversion
Retailers measure this with door counters, cameras, or human observers. Design decisions: a written protocol defining each event ("what counts as entering? as a purchase?"), the time-sampling schedule (you cannot watch every hour, so the slots you pick are a sample over time), and, when humans count, inter-rater reliability from overlapping observers. Signature risk: observer fatigue and undercounting, and reactivity, the Hawthorne effect, where people behave differently because they are being watched. This is the observational end of the Study Design & Data Quality chapter made concrete.
App & web clickstream events
This is the raw material of web analytics and recommendation systems. Design decisions: the
event taxonomy and instrumentation, you only capture events you tagged, so an un-instrumented
action is invisible, plus sessionization (grouping events into visits) and identity stitching
across devices. Signature risk: instrumentation gaps create systematic missing data,
bots inflate counts (filter is_bot), and, most importantly, this is found data, never a
designed sample, so it reflects whoever happened to use the product, with all the selection that implies.
Repurposing records collected by someone else
Administrative data, electronic health records, claims, tax and transaction records, is vast and cheap because
someone already paid to collect it. Design decisions: the central one is fitness-for-use,
a field optimized for billing may not mean what a researcher assumes, so you must check
provenance (source_system, captured_for, received_date)
and handle record linkage carefully. Signature risk: you inherit every one of the
source's collection biases, who shows up, what gets coded, without ever seeing them directly.
Each Mode's Signature Bias & Controls
Modes fail in characteristic ways. Knowing a mode's signature bias tells you which quality control to invest in, and which of the earlier chapters to revisit.
| Mode | What is captured | Signature bias / failure | Primary control |
|---|---|---|---|
| Ask (survey) | self-reported attitudes, facts | wording, social desirability, nonresponse | question design, pretesting (Survey & Questionnaire Design) |
| Sense (telemetry) | physical measurements | sensor drift; outage = missing-not-at-random | calibration, state flags, redundancy |
| Observe | counted behavior | reactivity (Hawthorne), observer error | protocol, inter-rater reliability |
| Log (trace) | behavioral events | instrumentation gaps; not a designed sample | event taxonomy, bot filtering |
| Harvest (scrape) | public web content | coverage bias; label validity | frame definition, coder reliability |
| Reuse (admin) | existing records | fitness-for-use; inherited bias | provenance, linkage checks |
Because every mode is biased differently, combining modes is powerful: a sensor count can validate a human observer; a short survey can calibrate what clickstream logs cannot see (the why behind a click). When two independent modes agree, you can trust the finding far more than either alone.
Modes of Data Collection in Machine Learning & AI
Modern machine learning runs almost entirely on found data, logs, scrapes, and reused records, which is exactly the cheap-but-undesigned end of the spectrum. Knowing the collection mode is the first step in judging whether a model can be trusted.
| Mode (this chapter) | In ML / AI it becomes | What to watch |
|---|---|---|
| Sense (telemetry) | Sensor / IoT features, anomaly detection | drift and calibration shift over time (dataset shift) |
| Harvest (scrape) | Web-scale training corpora (LLMs, vision) | coverage bias and license/ToS provenance |
| Observe | Human annotation / labeling | inter-annotator agreement; labeler reactivity |
| Log (trace) | Clickstream & interaction logs | feedback loops; logged data reflects the old model |
| Reuse (admin) | Repurposed records as labels/features | fitness-for-use; label leakage |
The single most important question about any training set is how it was collected. A web-scraped corpus is a harvest with enormous coverage bias and murky provenance; interaction logs are a trace that reflects the behavior of the previous system, creating feedback loops; human labels are structured observation whose quality lives or dies on inter-annotator agreement. Treating "the data" as a neutral given is the most common and most dangerous mistake in applied ML, because the collection mode, not the model, often decides what the system learns. Documenting provenance (in the spirit of datasheets for datasets) is now considered basic hygiene precisely for this reason.
🎓 Key Takeaways
- ✓Data arrives by many modes: ask, sense, observe, log, harvest, reuse, spanning active/designed to passive/found.
- ✓One design framework fits all: target, frame, instrument, sampling, quality, bias check, ethics/legality.
- ✓Each mode has a signature bias: sensor drift, observer reactivity, instrumentation gaps, coverage bias, inherited bias.
- ✓Found data is cheap but undesigned, it was never built to be a representative sample, so it hides selection.
- ✓In ML/AI, the collection mode often decides what a model learns; documenting provenance is basic hygiene.
Quiz: Test Yourself
Eight quick questions on modes of data collection. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
Of all the modes, asking people is the most design-intensive, the wording, scales, and order of questions can bias every answer. Survey & Questionnaire Design closes the Part: the deep dive into the "ask" mode, complete with real instruments and downloadable examples.