The Offer Works. Send It to a Third of the People.
Last quarter's campaign genuinely reduced cancellations and still lost about 159,000 dollars. The fix is not a better offer or a better model. It is a shorter list.
Recommendation
Keep the offer. Cut the list to about 30 percent of active subscribers. Sending it to everybody cost roughly 159,000 dollars last quarter. Sending it to the right third is worth about 376,000 dollars a year against that, and it is not the third you would pick from a churn model.
What happened last quarter
Half of all subscribers were sent the offer at random and half were not, which is why we can say what follows with confidence rather than with a model. The people who got it canceled less. That part worked.
| Sent the offer | Not sent it | |
|---|---|---|
| Canceled within 90 days | 16.8% | 22.6% |
| Number of subscribers | 11,917 | 11,724 |
So the offer removed about 5.7 percentage points of cancellation. The problem is what that is worth. A subscriber who stays is worth 290 dollars of margin, so 5.7 points of avoided cancellation buys about 16.66 dollars. The offer costs 30 dollars for every account it is sent to, including the very large number of accounts that were never going to leave.
The offer pays for itself only on somebody whose chance of canceling it lowers by more than 10.3 percentage points (30 divided by 290). Averaged over everybody, it managed 5.7. That gap is the whole loss.
Why the obvious list is the wrong list
The instinct is to rank subscribers by how likely they are to cancel and work down. We tried that, and then we tried ranking by how much the offer actually changes each subscriber's behavior, which we can measure because of the random holdout.
| Approach | Who gets contacted | Value over a quarter, per 8,275 subscribers |
|---|---|---|
| Send to everybody (what we did) | all of them | -86,636 dollars |
| Rank by who is likely to cancel | top 40% | +21,519 dollars |
| Rank by who responds to the offer | top 30% | +44,990 dollars |
Same data, same models. The only difference is what the list is sorted by, and it is worth roughly twice as much. Scaled to the full subscriber base, the swing against last quarter's approach is about 376,000 dollars a year.
The four groups worth naming
| Group | Cancel if we leave them alone | What the offer changes | Worth contacting? |
|---|---|---|---|
| Signed up on a promotion | 29.8% | removes 13.2 points | Yes, clearly |
| Two or more delivery problems | 48.6% | removes 8.1 points | No, just short |
| Happy and engaged | 9.7% | removes 7.0 points | No |
| Dormant: long tenure, autopay, rarely logs in | 13.2% | adds 8.4 points | No, it does harm |
The subscribers with delivery problems cancel more than any other group and are still not worth contacting, because a discount is not an answer to a box that arrives late. They need the delivery fixed, which is a different and much larger project. And the dormant group got worse. Long-tenure subscribers on autopay who rarely open the app canceled more when we emailed them. The likeliest explanation is the simplest one: the email reminded them they had a subscription.
What we would like to do next
Two things. First, keep a random holdout every quarter, not as a one-off study but permanently. It is the only instrument that tells us whether the program still works, and it costs a few points of coverage.
Second, confirm the dormant result with a small test before we write it into a rule. It is based on 537 subscribers and the range around it is wide. It is enough to stop emailing that group now, and not enough to build a permanent exclusion on.
One caution on all of the above: everything here measures this offer. A phone call, a free box, or an apology for a late delivery are different things and could easily help the group this offer hurt.