The previous two capstones drew samples, and in both of them the analysis turned on sampling error. This one does not sample at all. It takes everything the job board would show, which removes the only error most analyses know how to measure and leaves every other one intact.
- Setting
- Every open data-analyst vacancy on one public job board for one region and one quarter, collected by scraping within robots.txt and a set rate limit.
- The question
- What does a data analyst in this region earn?
- Why it matters
- The figure would be used to set pay bands. It is exactly the kind of number that gets scraped, published and repeated without anyone asking what population it describes.
- What we do
- Treat the scrape as a census of the frame rather than a sample, resolve listings back to vacancies, show that salary disclosure is missing not at random, benchmark against an outside source, and measure how much of the gap reweighting can repair.
Counting rows overstates the market by 71 percent. The scraped mean salary is $64,153 against an official benchmark of $71,800, low by 10.6 percent. And the standard error is $704, so the naive interval is confidently, precisely wrong.
A Census of the Frame Has No Sampling Error
The target population is every open data-analyst vacancy in the region this quarter. There is no register of vacancies, so the frame became whatever one public job board would show: vacancies visible on that board, not excluded by robots.txt, and reachable inside the rate limit we set.
Every listing matching the search was captured. That makes this a census of the frame, not a sample of it, and that changes which errors are in play.
Sampling error here is zero. Any confidence interval computed from these data is answering a question that was never at issue. The one error a statistician is trained to look for is the one error that is definitely absent, and the three that remain, coverage, duplication and non-disclosure, have no standard formula and produce no warning in any output.
Scraping ten times as many pages would narrow the interval around exactly the same wrong number. More data does not fix a frame problem.
Permissions Come Before Code
The collection plan records what was and was not done, and it belongs in the method section of any scraped study exactly as an ethics approval number would.
| Check | What was done | Why it matters |
|---|---|---|
| robots.txt | Checked and honored | Governs whether you may collect at all |
| Terms of service | Read; collection within permitted use | Legal basis, independent of robots.txt |
| Rate limit | One request per two seconds | Protects the host from your convenience |
| User agent | Identifying string with a contact address | Lets the host reach you instead of blocking you |
| Login walls | Not bypassed | Circumvention changes the legal picture entirely |
| Personal data | Named recruiters and direct contacts discarded | Vacancy text is corporate; the contact block is not |
The distinction that matters is that "technically possible" is not "permitted". None of the six rows above is enforced by your code failing. A scraper that ignores all of them runs perfectly well, which is precisely why the decisions have to be made deliberately and written down.
A Row Is a Listing, Not a Job
This is the first substantive finding, and it comes before any statistics. The crawler ran twice, which is trivial to fix. The real problem is that the same vacancy is advertised repeatedly, sometimes by the employer and sometimes by an agency acting for them under a different name.
| Listings for one vacancy | Vacancies |
|---|---|
| 1 | 458 |
| 2 | 182 |
| 3 | 112 |
| 4 | 56 |
A headline of "1,382 data-analyst vacancies advertised this quarter" would be wrong by 71 percent. Entity resolution is not a tidying step here; it is the difference between 1,382 jobs and 808.
The Salary Is Missing, and Not at Random
Thirty-seven percent of vacancies publish no number, printing "Competitive", "DOE" or "Negotiable" instead. If that were a coin flip it would cost precision and nothing else. It is not a coin flip.
| Seniority | Vacancies | Published a salary | Mean of those disclosed |
|---|---|---|---|
| Junior | 211 | 86.3% | $50,171 |
| Mid | 368 | 64.4% | $66,020 |
| Senior | 165 | 46.1% | $84,388 |
| Lead | 64 | 23.4% | $101,787 |
The probability that a value is missing depends on the value itself. Junior roles publish a number 86 percent of the time; lead roles do so 23 percent of the time, and lead roles pay roughly twice as much.
Dropping the blanks, which is what every default does, therefore does not merely lose data. It removes the top of the distribution, and the mean of what remains is pulled down.
Compare Against Something Outside the Frame
The damage is invisible from inside the dataset. The only way to see it is a measurement that did not come from the job board, and the regional occupational earnings survey covers all employers, advertised or not.
The naive 95 percent interval runs from $62,773 to $65,534. It does not contain the benchmark, and it never would however many pages were scraped, because the error is not sampling error. That is the sentence to carry out of this chapter.
How Much Can Be Repaired
Some of the damage is structure we can see. Disclosure varies by seniority, and seniority is known for every vacancy whether or not it published a salary. That makes it usable as a weighting variable, exactly as age was in Capstone 17.
| Estimate | Value | Gap | Fixes |
|---|---|---|---|
| Unadjusted, disclosed only | $64,153 | −10.6% | nothing |
| Reweighted to the board's seniority mix | $68,465 | −4.6% | the disclosure filter |
| Reweighted to the region's seniority mix | $69,754 | −2.9% | disclosure and coverage skew |
| Official benchmark | $71,800 | — | — |
What remains is unreachable. Within every seniority band, the employers who published a number still pay less than those who did not, so each band mean is itself biased downward. No weighting scheme built from the board can see inside a band it never observed.
Who Never Appeared
Thirty percent of the region's vacancies never reached any public board. Absent vacancies leave no trace in the file, so this is the error the analysis cannot see from the inside at all; the employer census supplies the number.
| Group | Reaches a public board | Pays |
|---|---|---|
| Small employers | 79.2% | around average |
| Medium employers | 74.3% | around average |
| Large employers | 50.5% | above average |
| Junior roles | 75.4% | below average |
| Lead roles | 58.2% | well above average |
Under-coverage removes senior roles at large employers. Non-disclosure removes high salaries within every band. Neither cancels the other and both drag the estimate down.
Had they pointed in opposite directions the scraped mean might have looked correct while resting on two large offsetting errors, which is a more dangerous situation than a visibly wrong number, because nothing would have prompted anyone to look.
The Variable That Survived
Whether a role is remote is printed on essentially every listing, so there is no disclosure filter. Coverage still applies, but remote working is not strongly related to whether a vacancy reaches a public board, so the two-stage filter that wrecked the salary estimate leaves this one nearly intact.
Salary is filtered twice, by coverage and by disclosure. Remote status is filtered once, and roughly neutrally. A dataset is not trustworthy or untrustworthy as a whole. Each variable has to be argued for separately, and the argument is always about the mechanism that decided whether the value would exist at all.
What to Watch
- Permission is not a technicality. robots.txt and the terms of service decide whether collection is allowed; a rate limit decides whether it is considerate. None is enforced by your code working, and a study that cannot state what was checked should not be published.
- Personal data has a different bar entirely. Vacancy text is corporate, but scraped pages routinely carry named recruiters, direct numbers and email addresses. Discarding them is a design decision made before collection, not a cleaning step afterwards.
- The frame is a business artifact. Which vacancies appear publicly is decided by hiring practice, agency contracts and cost. That is a selection mechanism nobody designed for research and nobody documents.
- State the population the data actually describes. Not "data-analyst salaries in the region" but "advertised salary bands, among publicly advertised vacancies that chose to publish one, on one board, during one quarter". The longer sentence is the honest one.
- Scraped data ages badly. The board is a live system: postings expire, get edited and reappear. Two scrapes a week apart are two different frames, so comparisons over time confound real change with changes in what the board was showing.
Scraped Frames in Data Science & AI
Most large modern datasets are scraped, and almost none ship with a statement of what population they represent.
| Where it appears | The frame, stated honestly |
|---|---|
| Web-scale training corpora | Text that was reachable, not paywalled and not blocked, in the languages that dominate the open web |
| Product review datasets | Reviews from customers motivated enough to write one, on platforms that permit collection |
| Public code repositories | Code its authors chose to publish, under licenses permitting reuse |
| Social media panels | Accounts that are public, active, and not rate-limited away during collection |
| Price or listing monitors | Whatever the site rendered to an unauthenticated visitor on the day |
The duplication problem in section 3 has a direct analog at scale. Web corpora contain the same document many times over, mirrored, syndicated and quoted, and near-duplicate removal is standard practice precisely because raw counts overstate coverage and repeated text is over-weighted in training. The salary problem has an analog too: a field is present only when someone chose to publish it, and that choice is rarely independent of the value. Any statistic computed over rows where a field happens to be populated inherits whatever decided the populating.
The full project, step by step
The companion notebook prints the collection plan and its permissions, resolves 1,382 listings to 808 vacancies and shows one vacancy advertised four times under three titles, voids the scraper's parse failures, tabulates salary disclosure against seniority, compares the result to an external benchmark, runs two levels of reweighting and reports what each recovers, quantifies the coverage shortfall it cannot reach, and closes on the one variable that came through intact.
The dataset
(capstone-web-scraping-job-postings.xlsx) holds the raw scrape with every flaw intact: crawler
duplicates, one vacancy under several employer names, parse failures and undisclosed salaries. It also carries
the written collection plan with its permissions record, and an external benchmark sheet. Two written reports
accompany it: a plain-language brief for a research lead, and a technical
report covering entity resolution, the missingness mechanism and the reweighting.
🎓 Key Takeaways
- ✓A census of the frame has zero sampling error and can still be badly wrong. It removes the warning light, not the fault.
- ✓A row is a listing, not a job. 1,382 listings were 808 vacancies; counting rows overstates the market by 71%.
- ✓Salary is missing not at random. Disclosure falls from 86% at junior level to 23% at lead level, so dropping blanks removes the top of the distribution.
- ✓Reweighting recovered 73% of the gap and no more. What is left is inside bands the board never observed.
- ✓Judge each variable separately. The same frame was 10.6% wrong on salary and 1.9% wrong on remote share.
Quiz: Test Yourself
Eight questions on scraped frames, entity resolution and missingness. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved.