Multiple Regression Done Properly: A Pay Equity Review

Part XXIX was about designing your way to a causal answer. This part is about the craft of the model itself, and it starts with the decision that regression makes easiest to get wrong: which variables belong on the right-hand side.

The most common failure in a pay equity review is producing a single number. There are two, they mean different things, and either one on its own is an answer to a question somebody else asked.

Both get reported. Neither is "the" pay gap, and the analysis plan for this review said so in writing before the salary file was opened, alongside one further commitment: we will not control for a variable the company itself assigns unless we have checked whether it is assigned equally.

That last row is the decision worth defending. Dropping the sixty-nine people with no review on file would have silently changed who the study is about, and people without a recent review are not a random sixty-nine. Filling the rating with the median and adding a perf_missing indicator keeps them in the analysis and keeps the fill visible to the model, so the coefficient on the indicator can tell you whether those employees differ.

The outcome is the log of salary, so coefficients read as approximate percentage differences. Controls go in one at a time, because anyone can produce a small number by adding enough variables and the honest presentation is to show the number moving.

Read the last row against the one above it. Experience, education, department and performance between them take the gap from 14.4 percent to 8.1. Job level alone takes it from 8.1 to 2.3, and the R-squared leaps from 0.47 to 0.88. One variable is doing more work than all the others combined, and it happens to be the only variable on the list that the company assigns to people.

Notice also that performance changed nothing at all. Men and women in this workforce are rated the same, which is worth reporting: it removes one common explanation before anybody offers it.

Age, experience and job level all measure roughly one underlying thing: how far into a career somebody is. Age and experience correlate at 0.829 here, which is unremarkable for a real workforce and enough to matter.

A VIF of 3.4 violates nothing. The thresholds people quote, 5 or 10, are conventions rather than rules, and the practical consequence is easier to see by looking directly at what happens when both variables go in.

Two things happen, and only one of them is what people expect. Putting both in inflates the standard error on experience by about two thirds, and leaves age with nothing left to explain. What does not happen is any movement in the coefficient we care about: the gender gap sits at −0.0234 either way.

We keep experience and drop age, because experience is what the company's pay policy claims to reward, and a coefficient the reader can map onto a policy is worth more than one they cannot.

Job level took the gap from 8.1 percent to 2.3. Before accepting that as the adjusted answer, ask what job level is. It is not a fact about an employee, like their education. It is a decision the company made about them.

If women are promoted more slowly than their experience and performance warrant, then job level is partly a result of unequal treatment. Controlling for it asks whether people are paid fairly once they have arrived at a grade, and treats how they arrived as fixed. That is a legitimate question. It is not the same question, and the regression output looks identical whichever it is answering.

So the plan required checking. At equal experience, education, performance rating and department, what are the odds of an employee sitting at level 4 or above?

This is the finding the adjusted model deletes. Women are 43 percent less likely to sit at senior grades than men who look identical on everything the company says it promotes on. A report giving only the 2.3 percent figure would be arithmetically correct and would leave the mechanism producing most of the gap entirely unmentioned.

The 2.3 percent is itself an average across the whole company, and the plan asked in advance whether the within-level gap is the same at every level. It is not.

The test that the gap is constant across levels returns p = 0.0012, so that pattern is not noise. Quoting 2.3 percent to a woman at level 5 would understate her position by more than half, and quoting it to a woman at level 1 would overstate hers. An average is a summary of a distribution, and it is only a good summary when the distribution has one mode.

All three pass, and the influence line needs reading carefully rather than quoting. One hundred and seventy observations exceed the familiar 4/n cutoff, which sounds alarming and is not. 4/n is a screening rule, not a threshold for concern: in a sample of 3,569 it flags roughly the top few percent by construction, whatever the data look like. The number that matters is the magnitude, and the largest Cook's distance here is 0.007 against conventional cause for concern at 0.5. No single employee is moving this model.

Chapter 187 is about what to do when these checks fail, which they do more often than published papers suggest. Here they pass, and verifying that is worth the four lines it costs.

Three numbers, each with a sentence saying what it means. Not one number, and not three numbers offered as alternatives for the reader to choose between.

Presented that way, the company has a clear picture and a clear place to look. Presented as "our adjusted pay gap is 2.3 percent", it has a defensible-sounding number and no idea where its problem is.

• ✓Never control for a variable the organization assigns without checking how it is assigned. Job level took the gap from 8.1 percent to 2.3. That is either the right adjustment or the erasure of the finding, and only the promotion model can tell you which.
• ✓Show the ladder. Anyone can reach a small number by adding variables, and the sequence is the first thing an opposing expert will reconstruct. Publishing it yourself costs nothing and removes the accusation.
• ✓Collinearity is a problem for the collinear variables. Check whether the standard error that inflated belongs to the coefficient you are reporting. Here it did not, so the VIFs were a formatting decision rather than a threat.
• ✓Test the interaction before averaging. A gap of 2.3 percent that runs from 1.4 to 5.7 across the company describes nobody in it.
• ✓A coefficient is not a finding about any individual. It is evidence about a pattern, and it tells you where to look rather than what any one person is owed. Individual cases need individual review.
• ✓Watch what a small adjusted figure gets used for. It is easy to quote as a clean bill of health, and what it actually says is narrower: within a grade, pay is nearly equal. Whether people reach that grade equally is a separate and less comfortable question.
• ✓Gender here is a recorded binary field. That is what the HR system holds. It excludes non-binary employees and anyone whose record is wrong, and the analysis inherits both limitations rather than solving them.