Survival Analysis: Time to Event

An employee hired thirty months ago who is still here has not given us a tenure. They have given us a lower bound, which is a real observation with a real value, and the entire chapter is about tools that can use one.

The previous chapter was about rows with a gap in them. This looks like the same problem and is not, and the difference decides what you are allowed to do.

Of 3,191 employees, 1,375 resigned, 83 were let go and 1,733 were still employed at the cut-off. For the question being asked, resignation, that leaves 56.9 percent censored. Those people are not missing from the study. They are the study.

The reported figure is not a bad estimate of how long people stay. It is a good estimate of something else: the average tenure of the subset who have already gone, which is guaranteed to be short because anyone who stays a long time has not entered the average yet.

Kaplan-Meier asks a different question at each month: of the people still here at the start of it, what fraction left during it? Someone censored at month 30 contributes to every one of those questions up to month 30 and to none afterward. Nothing is dropped and nothing is invented. Survival at twelve months is 84.2 percent, at twenty-four 58.5 percent and at thirty-six 41.4 percent.

Platform is bottom of the first column and top of the last. On the number HR reports its people last 6.9 months against roughly nineteen everywhere else, which reads as a team in crisis. On a survival curve, 93.4 percent of its hires are still there at twelve months, the best retention in the company.

Both figures are arithmetically correct. Platform was created eighteen months before the cut-off, so its mean follow-up is 8.9 months and only 5.3 percent of its people have resigned at all. The few who did leave left early, because leaving late was not yet possible. The naive statistic measures how long a department has existed at least as much as it measures whether people stay.

The policy question is whether remote work affects retention. The two standard tools both say no.

And then the survival curves, which are what those two numbers were summarizing.

The curves cross between eighteen and twenty-four months. The log-rank test accumulates the difference between them across the whole of follow-up, so when a group is ahead early and behind later the two halves cancel. A p-value of 0.165 here does not mean nothing is happening. It means the two things that are happening point in opposite directions.

Everything Cox reports as a single hazard ratio rests on proportional hazards: the assumption that a covariate's multiplier is the same at month 3 as at month 40. It is not a technicality and it is testable. Correlate each covariate's residuals with time, and a non-zero correlation means the effect is moving.

That is not a marginal warning. It is the model saying that the number it just printed for remote work is not a constant and should not be read as one. The check takes one line and it is the difference between the wrong answer and the right one.

Splitting each employee's follow-up at eighteen months turns 3,191 people into 4,761 rows. Someone who lasted thirty months contributes one row covering months 0 to 18, in which they were remote-and-early, and a second covering months 18 to 30, in which they were remote-and-late. The resignation is attached to whichever row it happened in. Nobody is counted twice.

A factor of four separates the two regimes, and it was reported as 1.08. Nothing about the earlier answers was a computational error. A hazard ratio and a log-rank statistic both compress the whole of follow-up into one number, and when the truth reverses partway through, the one number that fits it is roughly no effect.

There is a third way to get this wrong, and it is the most confident of the lot. Analyze only the 1,375 people who resigned, which discards 1,816 employees, and remote workers appear to have stayed 3.8 months longer, significant at any threshold.

The first is wrong for the same reason the naive tenure figure was. The leavers are not a sample of employees, they are a sample of endings, and which endings have happened yet depends on when people were hired and how long they were going to last.

Only the last analysis used every employee. What it recommends is not a policy reversal but something specific at around the eighteen-month mark for remote employees, and a measurement of whether that moves the second hazard ratio.

Involuntary exits were treated as censoring. Eighty-three people were let go, and the analysis assumes that being let go says nothing about whether that person was about to resign. If the two are related, and they plausibly are, every hazard ratio here is somewhat off. The proper treatment is a competing-risks model that estimates both events.

Eighteen months was chosen by looking at the curves. The split point was picked after seeing where they crossed, which makes the two hazard ratios a little sharper than an honest out-of-sample estimate would be. A pre-registered split, or a smooth time-varying coefficient, would give some of that back.

Sales also fails the proportional-hazards test, at p = 0.006 with a statistic nine times smaller than remote work's. It is small enough that a single hazard ratio is a fair summary and large enough to mention.

Nobody hired before 2020 is in this file. Everything here describes people hired into the company as it has been for six years and says nothing about the people who were already there, who are exactly the long-tenure employees a retention study would most want.

This is an observational comparison. Nobody was assigned to remote work at random. Whatever leads a person to choose it travels with them into the hazard ratio.

• ✓Never average the durations of the ones that ended. Mean time-to-event among those who experienced the event is short by construction, and the more censoring there is the shorter it gets. It is the single most common error in this area.
• ✓Check how long each group has been observed before comparing them. A team, cohort or product line that is younger than the others will look worse on any duration statistic and better on any rate statistic, regardless of what is actually happening.
• ✓Plot the curves before running the test. The log-rank test has almost no power against curves that cross, and will report no difference where the difference is large and reverses.
• ✓Test proportional hazards on every Cox model. One line of code. If a covariate fails, its hazard ratio is an average of a changing effect and should not be quoted as a constant.
• ✓Split follow-up when an effect moves. Episode splitting is simple, uses every subject, and turns an uninterpretable average into two numbers a decision can be made on.
• ✓Say what you treated as censoring. Competing events are not censoring, and calling them that quietly assumes they are unrelated to the event you care about.

Anywhere the outcome is when rather than whether, and the observation window ends before everyone has had their turn, this is the right toolkit and a classifier is the wrong one.

The common failure is to turn the question into a classifier. Predicting "will this customer churn in the next 90 days" throws away when, discards everyone whose 90 days is not yet complete, and relabels the same customer differently depending on when the snapshot was taken. A survival model answers the question the business asked, which is usually how long, and gives a curve rather than a probability.