Regression Discontinuity and Instrumental Variables

Capstone 23 ended with an estimate nobody should trust, because adjustment can only remove the confounding you measured. This chapter takes the opposite route: stop trying to measure the confounding, and find a piece of variation that never had any.

After deduplication and removing the records where the need index was never assessed, 79,666 students remain. The obvious thing to do with them is the wrong thing.

This is worse than an understatement. The scholarship is means tested, so it goes to students from poorer families, and family circumstances predict completion through routes nobody records. A report quoting that difference would conclude that giving money to students in need makes them less likely to graduate.

Capstone 23 showed what happens if you respond by adjusting for the routes you can see and hoping about the rest. Both designs here refuse that trade. Neither of them assumes that the measured covariates capture selection, because neither of them uses the measured covariates for that purpose at all.

Eligibility is a deterministic function of the need index: eligible at 60.0 or above, not eligible below. A student at 59.9 and a student at 60.1 are, in every respect anyone cares about, the same student. One of them is offered a scholarship, and the reason is a number somebody chose.

That arbitrariness is the whole design. Whatever unmeasured differences exist between high-need and low-need students, those differences change smoothly as the need index rises. Eligibility changes discontinuously. Anything that jumps at 60 is therefore attributable to the rule.

First, no manipulation. If families could nudge the index across the line, the students on either side stop being comparable. The signature is a jump in the density of the running variable: a pile-up just above the cutoff and a hole just below.

Second, covariate smoothness. Anything measured before the scholarship should be continuous at the cutoff. A pre-treatment characteristic that jumps at 60 means something other than the rule changes there.

All three pass, and it is worth being precise about what that means. These checks cannot prove the design works. They can only fail to find evidence that it does not. That is the most any assumption check has ever offered, in this chapter or any other, and a study that reports them as proof is overstating them.

The density result is also substantively reassuring here: the index is computed by an agency from submitted documents rather than self-reported, which is exactly the circumstance in which manipulation is hard. Where a running variable is self-reported, or where the threshold is public and the stakes are high, this check frequently fails, and a failed density test is usually the end of the design rather than something to correct.

Being eligible is not the same as receiving. Take-up among the eligible is 62 percent, so the jump in completion at the cutoff measures the effect of being offered the scholarship. That is an intention to treat, and it is a perfectly good quantity, just not the one the question asked for.

Dividing the outcome jump by the take-up jump rescales the estimate from "offered" to "received". This is the Wald ratio, and it is worth noticing what it is: a fuzzy RDD is an instrumental variables estimator in which the instrument is crossing the threshold. The two designs in this chapter are more closely related than their names suggest, which makes it all the more important that their assumptions are different.

A bandwidth is a researcher's choice, so the honest report is a range rather than the one that looked best. Here the estimate runs from +10.4 to +11.6 points across a fourfold change in window width, and every interval contains the truth. That stability is part of the evidence.

The second design uses different variation entirely. Scholarship places are allocated to high schools by a formula that runs on enrollment counts three years out of date. Two otherwise identical schools can therefore have quite different allocations, for reasons that have nothing to do with this year's students, and a student at a well-allocated school is more likely to end up with a scholarship.

The analysis is restricted to eligible students, since only they can receive the award.

Take-up doubles from the least to the most generously allocated schools. That is the variation the design is going to use, and the reason it is usable is that nothing about it was chosen by, or for, the students.

The first-stage F is 269 times the conventional threshold of 10, which settles relevance and nothing else. A strong first stage is frequently presented as though it validated the instrument. It does not. It establishes that the instrument moves the treatment, which is the easy part.

The exclusion restriction is where an IV study is attacked, and it should be. The argument here is that the formula uses enrollment counts three years stale, so the allocation carries no information about this year's cohort. The obvious objection is that a school's enrollment three years ago might proxy for something about the neighborhood that also affects completion. The honest response has three parts: state the objection, note that the covariate check found the allocation smooth at the cutoff and unrelated to family income, and concede that the assumption cannot be proved. An IV paper that does not do all three is hiding something.

The last chip deserves attention rather than a footnote. Ordinary least squares restricted to the eligible sample gives +11.5 points, which is also close to the truth. That is not luck: within the eligible group, who actually received a scholarship was largely determined by their school's allocation, which is as good as random. Restricting to the eligible removed most of the confounding by itself.

It would be easy to present the IV as having rescued an analysis that a simple subgroup comparison would have handled. The IV's contribution is that it does not depend on that being true. Had take-up among the eligible been driven by how motivated a student was, rather than by an administrative formula, OLS on the eligible would have been badly biased in exactly the way Capstone 23 was, and the IV would still have worked.

Both estimates are also local, and to different localities. The RDD estimates the effect for students near a need index of 60. The IV estimates a local average treatment effect: the effect for students whose receipt was actually shifted by their school's allocation, sometimes called the compliers. Neither is the effect for the average recipient, and neither speaks directly to what would happen if eligibility were widened. Reporting either as "the effect of the scholarship" overstates what was learned.

Capstone 23 got +1.67 from matching and +1.65 from weighting, and that agreement was worth nothing. Both rested on the same untestable assumption, so they were confounded identically and failed together. This is a different situation, and the difference is worth stating precisely.

Read the last column. The RDD assumes that nothing except eligibility changes discontinuously at 60. The IV assumes that a stale funding formula affects completion only by changing who gets a scholarship. Neither assumption implies the other, and neither implies unconfoundedness. They could both be wrong, but they would have to be wrong in unrelated ways and by coincidentally similar amounts. That is what turns two numbers landing near each other into evidence rather than into a coincidence you were hoping for.

• ✓State each design's assumption in a sentence a policy reader can object to. "Nothing else changes at 60" and "a stale formula affects completion only through receipt" can be argued with. An equation cannot.
• ✓A strong first stage validates relevance and nothing else. An F of 2,685 says the instrument moves the treatment. It says nothing whatever about exclusion.
• ✓Report a bandwidth range, not the best one. The range is the honest summary; a single number is a choice the reader cannot see.
• ✓Assumption checks can only fail to find a problem. A smooth density and smooth covariates are consistent with a valid design and do not establish one.
• ✓Do not report a LATE as a program evaluation. If the question is whether to widen eligibility, the relevant population is students who would newly qualify, and neither design here speaks to them.
• ✓A cutoff is a real boundary for real people. Students at 59.9 were refused what students at 60.1 received, and the analysis depends on that being arbitrary. A rule that is analytically convenient is, for the people just below it, hard to justify.
• ✓The instrument is itself a finding. That a three-year-stale formula determined who received a scholarship is a policy result, and arguably a more actionable one than the effect estimate.