A Fairness Audit

Group A and group B are placeholders for a patient characteristic the health system is required to monitor. The mechanism is real and the people are not. What matters is that the two groups do not have equal access to care, which is a fact about the system rather than about the patients.

The substitution at the heart of this is neither lazy nor unusual. Cost is the only number a claims system produces for every patient every month without anyone having to define anything. It correlates strongly with illness, it is auditable, and every incentive points at using it.

Group B carries more chronic conditions and generates less spending. Those two facts together are not a paradox. They are the finding, and everything downstream follows from them.

The bottom-right panel is worth sitting with. The evidence of the bias is present in the data, in columns the model uses as predictors rather than as warnings.

The deployed model uses the five features a claims system has: age, primary-care visits, emergency visits, active medications and prior cost. It identifies the most expensive three percent of the panel with an AUC of 0.920. By any ordinary standard it works.

A ratio of 0.36 against the four-fifths threshold of 0.80. This is where an audit starts, and on its own it settles nothing, because there is an obvious defense: if group B genuinely costs less, a cost model is right to rank them lower and the disparity is a property of the data.

The rest of the chapter is about whether that defense holds.

The standard next step is to check whether the model's predictions are calibrated within each group. If it says twelve thousand dollars, do both groups go on to cost twelve thousand dollars?

The model is close to calibrated on cost, and where it is not, it errs in group B's favor by predicting they will spend more than they do. Read that as an auditor would: every standard check against the label says the model is doing its job, and the one imperfection points away from harm.

A fairness review that audits a model against its own training label cannot find a problem that lives in the label.

The audit that works asks a different question. Take the patients the model considers equally risky, and count how ill they actually are.

That is the defense answered. The model ranks group B lower because the same illness generates fewer dollars for them and the model was told to find dollars. Every patient it treats as equivalent to a group A patient is in fact sicker than them, and the program is allocating on the wrong ordering.

None of this is visible in accuracy, calibration, or any error rate computed against cost.

The two remedies do different jobs and neither does both.

Changing the label helps most on who gets in. Training on next year's condition count raises the share of enrollees who are genuinely high-need to 86.6 percent, the best of the four, and lifts group B's ratio from 0.36 to 0.59. It does not reach parity, because the features the model uses, visits and prior cost, carry the same access gap the label did. You cannot fully repair a label problem through a feature set that has the same problem.

Group-specific thresholds deliver parity by construction and improve targeting barely at all, because they reorder nobody. They take the same ranking and cut it in two places.

Doing both gives parity and gives back the targeting gain. That is the honest result and it is not the tidy one: on this panel the combination is no better at finding high-need patients than the cost model was, it is simply far more evenly distributed.

The recommendation is the label change, because it is the only one that improves the measurement rather than redistributing an unimproved one, with thresholds as a policy decision layered on top and reported as one.

Suppose we tried to fix this purely by choosing thresholds. There is a result that says we cannot: when two groups have different base rates, calibration and equal error rates are mutually incompatible. Here is what that looks like in numbers rather than in algebra.

Equalize the false-positive rate and the precision gap widens to 0.18. Equalize precision instead and the miss rate diverges to 0.35, with group B failing to be identified 76 percent of the time against group A's 41. No threshold pair closes all three, and that is arithmetic rather than a failed search.

Now read the right-hand panel. On the corrected label, once either criterion is equalized every remaining gap is under 0.033. The reason is in the base rates: on cost they are 22.9 percent and 13.1 percent, a gap of nearly ten points that exists because the label is dollars and one group spends fewer of them. On need they are 31.4 and 35.9, and what remains is a real difference in illness.

The severity of the impossibility was itself a symptom. Much of what looked like an unavoidable ethical trade-off had been manufactured by the choice of what to predict, and it went away when that choice was corrected.

Condition count is also a proxy. It comes from diagnoses on a problem list, and a diagnosis requires someone to have been seen. It is less sensitive to access than dollars are, which is why it is an improvement, but its residual bias runs the same way, so the true disparity is probably larger than measured here.

Two groups, one attribute. Real audits handle several attributes and their intersections, where cell sizes fall quickly and the multiple-comparison problem becomes severe.

Nothing here measures the program's effect. Everything is about who gets a place. Whether care management helps, and whether it helps both groups equally, is a causal question this observational panel cannot answer and a randomized rollout could.

Parity was imposed on a fixed capacity. Every place given to one group is taken from another, which is why parity thresholds cost targeting. If the real answer is that the program is too small, no allocation rule fixes that.

The four-fifths rule is a legal screening device, not a definition of fairness. A model can pass it and still do what this one does, and a model can fail it for entirely justified reasons. It starts audits. It does not finish them.

• ✓Write down what the label is a proxy for. Cost for need, clicks for interest, arrests for crime, tenure for performance. If the target is a stand-in, the audit has to be against the thing it stands in for.
• ✓Ask whether the proxy is equally good for everyone. That is the question the whole chapter turns on, and it is answered by comparing groups at a fixed level of the underlying quantity rather than in aggregate.
• ✓Never conclude from calibration on the label alone. A perfectly calibrated model on a biased target is a perfectly calibrated biased model, and it will pass every check you run against its own predictions.
• ✓Treat the four-fifths ratio as a smoke alarm. It tells you to investigate. It does not tell you the model is unfair, and passing it does not tell you the model is fair.
• ✓Expect to trade one criterion against another. When base rates differ you cannot have calibration and equal error rates together. Decide in advance which one the decision cares about and say so.
• ✓Fix the target before tuning the thresholds. Threshold adjustment redistributes an ordering. Only a better label produces a better ordering, and it also shrinks the trade-offs you then have to make.

Almost every consequential model is trained on a proxy, because the thing anyone actually cares about is rarely recorded. The proxy is chosen because it is available, and availability is not neutral.

In every row the model can be accurate, calibrated and audited clean while doing something nobody intended. The failure is upstream of the algorithm, which is why it survives model changes, more data and better validation, and why so much of the fairness literature that focuses on the classifier cannot reach it.