Clustering: Choosing k Honestly

The brief asks for four or five segments with names. It does not ask whether there are any, and no clustering algorithm will ever raise the question, because every one of them returns exactly as many groups as it is told to.

Before any question about method, a question about arithmetic. annual_spend has a standard deviation of 2,135 and discount_share has one of 0.248, a ratio of about 8,600 to 1, entirely because one is denominated in dollars and the other is a proportion. Euclidean distance adds up squared differences, so run k-means on the raw columns and see what happens.

The clusters account for 95 percent of the variation in spend and 7 percent of the variation in how much a customer shops online. Clusters 1 and 2 are indistinguishable on every behavioral column and differ only in size of wallet.

This is not a segmentation. It is a ranking of customers by spend with four names on it, and a sort would have produced it faster.

On the standardized data, silhouette peaks at k = 2 with 0.367, against 0.315 at three and 0.324 at four. Run the identical procedure on uniform random data of the same shape and it peaks at two as well, at 0.095.

There is nothing in that second dataset and the procedure still names a best k. It has to: k-means partitions whatever it is given and silhouette compares partitions. What the comparison does give is a scale. Real data 0.37, structureless data 0.09, a factor of four. That gap is the evidence there is structure, and it is the number nobody quotes.

Stability asks a better question: if these segments are real, they should survive re-estimation on a different sample of customers.

Adjusted Rand index between the full solution and a refit on an 80 percent subsample, averaged over 25 replications.

The contrast between the columns is enormous. On real data a refit on different customers reproduces the same partition almost exactly; on structureless data it agrees with itself barely at all, because the boundaries land somewhere different every time.

Stability is a good test of whether structure exists and a weak test of how much of it there is. It is above 0.98 for every k from two to five, so it cannot separate them, and its maximum is again at two.

So two well-regarded criteria, in agreement, both pointing at two segments. Marketing asked for four or five.

campaign_response records who responded to a price-off campaign run the quarter after this window. It was held out of every clustering. If a partition is picking up something real about how customers behave, its clusters should differ in how they responded. If it is slicing a continuum, they will differ a little and smoothly.

There is the elbow, and it is not subtle. Going from three clusters to four more than doubles the spread in response, from 0.18 to 0.42. Going from four to five adds 0.011, and every further split adds a fraction of a point.

Four segments, not because a fit index peaked there, but because the fourth split separates customers who respond to a price-off campaign at 48 percent from customers who respond at 5 percent, and the fifth split separates nothing.

Every point gets a cluster. There is no way for k-means to report that a customer sits in the middle of the cloud and belongs to nothing in particular, which is a problem if that describes most of the customer base. DBSCAN can: it grows clusters out of dense regions and labels everything else noise.

More than half the customer base is not in any segment. DBSCAN finds four dense regions covering about 45 percent of accounts and declines to classify the rest. That is an unwelcome answer and a useful one: a plan that assigns every customer to a segment and a treatment is planning for a base that does not exist.

Cluster 2 is the interesting failure. Its profile is unremarkable and its response rate of 0.126 is indistinguishable from the 0.185 base rate. It is a dense patch of the continuum rather than a group, and it is exactly the kind of thing that gets a name and a budget if nobody checks. The clue was available without the answer key: it did not separate anything.

Note also how sensitive DBSCAN is to its radius. At 0.7 it finds two clusters and calls 92 percent noise; at 1.5 it merges everything into one. That parameter does the same job k does, and it is chosen the same way: by what the resulting partition separates.

This dataset was generated, so the truth is known and ships with it. On real data this section does not exist, which is the reason for building the argument in the order above before opening it.

Adjusted Rand index against the true segment labels. DBSCAN, on the 2,538 accounts it was willing to classify, scores 0.963.

Everything the external check said turns out to be right and everything the internal indices said turns out to be wrong. Three further things are worth noting.

DBSCAN's noise fraction was 55.4 percent against a true 54.5. It identified the size of the unstructured middle to within a percentage point without being told there was one.

Ward beat k-means at every k from three to six. The true segments are not equally sized or equally spread, and k-means prefers clusters that are.

Silhouette and stability were not useless. They established that structure existed, which is a real question that was worth answering. They could not count it. Counting it needed evidence from outside.

Segments are a description of one year. Nothing here says a customer stays in theirs. A retention plan built on these groups needs the transition rates between them, which requires a second year and is a different analysis.

The scaling choice remains an assumption. There is no test that settles whether a standard deviation of returns rate should count for as much as one of spend, only a business argument, and the boundaries move with the answer.

The external variable validates the partition for one purpose. These four segments separate response to a price-off campaign very well. They may separate response to a loyalty program or a service change badly. A segmentation is not true or false, it is useful, and useful is always useful for something.

And the unstructured middle is a finding, not a failure. Most customer bases are mostly continuum. The interesting question about the 55 percent is not which segment to force them into, but whether there is anything worth doing for them at all.

• ✓Standardize, and say out loud what standardizing assumes. On raw units the segments accounted for 95 percent of spend and 7 percent of online behavior.
• ✓Run your whole procedure on structureless data of the same shape. It will return clusters, and the score it returns is the yardstick your real score has to beat.
• ✓Read the height of a validity index, not the position of its maximum. 0.37 against 0.09 is the finding; the argmax at two was wrong.
• ✓Stability detects structure; it does not count it. Above 0.98 for every k from two to five here.
• ✓Choose k with something the algorithm never saw. If nothing was held back, k is not identified and the number is a decision rather than a result.
• ✓Let a method say "none of the above". k-means must classify everyone; density methods can report that most of the base is not in a group.
• ✓Be ready to deliver fewer segments than were asked for. The brief said four or five with names, and the honest answer included "and most of your customers are in none of them".