Churn Prediction and Who Is Worth Contacting

A retention campaign reduced churn by a genuine, well-measured 5.7 percentage points and lost 159,000 dollars doing it. Nothing was wrong with the model. The list was sorted by the wrong thing.

The retention team supplied both before any modeling started. A subscriber who stays is worth $290 of gross margin over the following year. The offer, 20 percent off the next three boxes, costs $30 per account it is sent to.

That sentence quietly rules out the standard approach. A churn model estimates the first quantity and says nothing at all about the second.

The campaign was randomized: half of all active subscribers, chosen by a coin flip on the account number, were sent the offer. Balance across the two arms is clean, with a largest standardized difference of 0.018 against a conventional threshold of 0.10. So the difference in churn between the arms is the causal effect of the offer, and no model is needed to read it.

The campaign worked and lost money. Both halves of that sentence are true and neither is a criticism of the offer. It really did reduce churn, by an amount whose confidence interval is nowhere near zero. It is simply that 5.7 points of avoided churn buys $16.66 of margin, and the offer costs $30.

So the campaign does not need a better model or a better discount. It needs to be sent to fewer people, and the entire question is which ones.

The risk model answers "what is the chance this subscriber cancels if we do nothing", so it is fitted on the control arm only. Three families, same features, same split.

A spread of 0.0009 in ROC-AUC across three quite different model families. There is nothing to choose between them, and a reasonable person would take the logistic regression for its transparency and stop.

Hold that thought. Model selection is where a great deal of project time goes, and by the end of this chapter a different decision will be worth a six-figure sum. It is not this one.

SHAP on the winning model gives a clean global ranking: tenure first, last rating second, skipped weeks third, delivery issues fourth. Support contacts sit near the bottom, tenth of nineteen.

Now remove one column, delivery_issues_90d, and refit.

Nothing changed about how much support contacts matter to a subscriber. What changed is what they add, and that is what a SHAP importance measures. Support calls are largely caused by delivery failures; with the delivery column present there is little left for them to explain, and with it gone they stand in for it.

Both methods explain one prediction by assigning a contribution to each feature, and they are built on different ideas. SHAP allocates the prediction with a rule borrowed from cooperative game theory. LIME fits a simple local model to points sampled around the one you asked about. Nothing guarantees they agree, and on sixty mid-risk subscribers they name a different single most important driver 52 percent of the time.

They agree this subscriber is at risk and they disagree about what to put at the top of the retention agent's screen. Neither is wrong. They answer slightly different questions, and both are descriptions of the model rather than of the subscriber.

That is the honest framing for a business audience: this is what the model responds to, not this is why the customer is leaving. The distance between those two sentences is the rest of this chapter.

Every number so far has been about who cancels. The offer only earns its $30 from people whose behavior it changes. Because the campaign was randomized we can estimate that directly: fit one model on the control arm and one on the offered arm, and the gap between their predictions is the estimated effect of the offer on that subscriber. This is the two-model, or T-learner, approach.

Read the last two chips together. delivery_issues_90d is the clearest signal in the dataset for who cancels, and it points the other way for who to contact. That is not a quirk of the model. A subscriber whose boxes keep arriving late has a service problem, and 20 percent off the next three does not fix a service problem. Meanwhile signup_discount, sixth in the risk ranking, is the strongest single predictor of who responds.

A targeting rule picks a set of accounts using features only. Because assignment was random, the offered and control accounts inside any such set are still comparable, so the value of a rule can be measured rather than modeled.

Read the bottom row first: contacting everybody, which is what the campaign did, loses money on the test set, exactly as section 2 said it would. Then read across. At every depth the uplift ranking is worth more, and at the top of the list it is worth seven times more.

Same data, same features, same model family, same three-model bake-off that made no difference. The only thing that changed is what the list is sorted by.

Quartiles of a model score are hard to hand to an operations team. These four groups can be written down in a spreadsheet, and each one is still a randomized comparison, so each number below is measured against its own control.

The dormant group is the one to look at twice. Long tenure, on autopay, barely opens the app. They cancel less than anybody in the table, and sending them the offer made them more likely to leave, by 8.4 points, with an interval that does not include zero. The most likely explanation is the least sophisticated one: an email about a subscription is a reminder that the subscription exists. Every dormant account that was sent the offer cost about $54.

The delivery-issues group is the trap. They cancel at nearly 49 percent, which puts them at the very top of any risk-ranked list, and targeting them still loses about $6.50 an account. The offer does move them, by 8.1 points, and 8.1 is below 10.3.

The discount signups are the business case. They came in on a promotion, they skip weeks, they respond to a price. Nearly all of the recoverable value in this campaign is in this one group.

• ✓Ask what the decision is before choosing the target variable. "Who will cancel" and "who should we contact" produce different models, and only one of them answers the question that has a budget attached.
• ✓Price the intervention first. The break-even of 10.3 points was computable before any data was loaded, and it turns the whole project from a modeling exercise into an arithmetic one.
• ✓Feature importance is a property of the model. Drop a correlated column and a feature can move five places while the model gets no worse.
• ✓Explanations describe the model, not the world. SHAP and LIME disagreed on the top driver for half the subscribers, and neither was making a causal claim.
• ✓Uplift needs an experiment. None of this was available without a randomized holdout. A retention team that offers to everybody has no way to learn who was worth offering to.
• ✓Keep the holdout permanently. Not as a one-off study. It is the only instrument that measures whether the program still works.
• ✓Some contact does harm. The dormant finding is a subgroup result with a wide interval, and it is exactly the kind of result to confirm with a small test rather than to build into a permanent exclusion rule.