Clustering: finding groups with no labels¶
Supervised learning had an answer key. Clustering does not: we hand the algorithm unlabeled data and ask it to discover the natural groups. This notebook segments e-commerce customers three ways, K-means, hierarchical (with a dendrogram), and DBSCAN, and shows how to choose the number of clusters with the elbow and silhouette methods. Library-first with scikit-learn and scipy.
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans, AgglomerativeClustering, DBSCAN
from sklearn.metrics import silhouette_score
from scipy.cluster.hierarchy import linkage, dendrogram
pd.set_option('display.max_columns', 30)
try: df = pd.read_excel('../../data/clustering--shoppers.xlsx', sheet_name='Data')
except FileNotFoundError: df = pd.read_excel(BASE + 'clustering--shoppers.xlsx', sheet_name='Data')
feat = ['annual_income_k','spending_score','age','web_visits_mo']
X = df[feat]
Xs = StandardScaler().fit_transform(X) # z-score each feature: mean 0, sd 1
print(df.shape, 'customers, no label column'); print(X.describe().round(1).T[['mean','std','min','max']])
(330, 5) customers, no label column
mean std min max
annual_income_k 58.0 32.9 12.0 190.0
spending_score 49.7 30.3 5.0 95.0
age 39.6 9.8 18.0 70.0
web_visits_mo 17.8 10.1 0.0 55.0
Why standardize? Clustering groups points by distance, and distance is dominated by whatever feature has the largest numeric range. Here income (recorded in thousands, standard deviation 32.9) and the spending score (30.3) happen to have similar spreads, but age and monthly web visits vary about a third as much (standard deviations near 10), so without scaling those two would barely count. With income in raw dollars it would swamp everything. StandardScaler puts every feature on a common z-score footing (mean 0, standard deviation 1) so each contributes fairly. This is the single most common clustering mistake, and we avoid it up front.
fig, ax = plt.subplots(figsize=(7.2,5.2))
ax.scatter(X['annual_income_k'], X['spending_score'], s=34, c=GREY, edgecolor='white')
ax.set(xlabel='annual income (thousands)', ylabel='spending score (1-100)', title='Customers before clustering: structure is visible')
plt.tight_layout(); plt.show()
Explore first. Even with no labels, a scatter of income against spending shows four obvious clumps near the corners of the plane, low and high income each splitting by whether the customer spends freely or sparingly. Clustering will make this intuition precise and repeatable across all four dimensions at once (the eye can only see two). Always look before you compute.
km = KMeans(n_clusters=4, n_init=10, random_state=0).fit(Xs)
df['kmeans'] = km.labels_
cent = StandardScaler().fit(X).inverse_transform(km.cluster_centers_) # centroids back in original units
fig, ax = plt.subplots(figsize=(7.4,5.4))
ax.scatter(X['annual_income_k'], X['spending_score'], c=km.labels_, cmap='viridis', s=36, edgecolor='white')
ax.scatter(cent[:,0], cent[:,1], c='red', s=260, marker='X', edgecolor='black', label='centroids')
ax.set(xlabel='annual income (thousands)', ylabel='spending score', title='K-means (k=4): four segments and their centroids'); ax.legend()
plt.tight_layout(); plt.show()
print('cluster sizes:'); print(df['kmeans'].value_counts().sort_index().to_string())
cluster sizes: kmeans 0 85 1 81 2 82 3 82
How K-means works. It picks k cluster centers, assigns every point to its nearest center, moves each center to the mean of its members, and repeats until nothing moves. It is fast and scales well, but it assumes clusters are roughly round and similar in size, and you must choose k in advance. The red X's are the final centroids, the average customer of each segment. Next we justify why k=4 was the right choice.
ks = range(2, 9)
inertia = [KMeans(n_clusters=k, n_init=10, random_state=0).fit(Xs).inertia_ for k in ks]
sil = [silhouette_score(Xs, KMeans(n_clusters=k, n_init=10, random_state=0).fit_predict(Xs)) for k in ks]
fig, ax = plt.subplots(1, 2, figsize=(12,4.4))
ax[0].plot(list(ks), inertia, 'o-', color=EM, lw=2.4); ax[0].set(title='Elbow: within-cluster inertia', xlabel='k', ylabel='inertia')
ax[0].axvline(4, color=RED, ls='--')
ax[1].plot(list(ks), sil, 'o-', color=GREEN, lw=2.4); ax[1].set(title='Silhouette score (higher = better)', xlabel='k', ylabel='silhouette')
ax[1].axvline(4, color=RED, ls='--'); plt.tight_layout(); plt.show()
print('inertia for k = 2..8:', np.round(inertia, 1).tolist())
print('best k by silhouette =', list(ks)[int(np.argmax(sil))], '| score =', round(max(sil),3))
inertia for k = 2..8: [687.4, 460.0, 259.7, 235.6, 213.1, 190.8, 169.5] best k by silhouette = 4 | score = 0.552
Two diagnostics agree. The elbow plot shows inertia (total within-cluster distance) always falling as k grows, but the rate of improvement bends sharply at k=4, adding more clusters barely helps. The silhouette score measures how well-separated the clusters are (how much closer each point is to its own cluster than to the next-nearest); it peaks at k=4. When the elbow and the silhouette point to the same number, you can trust it.
Z = linkage(Xs, method='ward') # ward = merge the pair that least increases variance
fig, ax = plt.subplots(figsize=(11,4.6))
dendrogram(Z, truncate_mode='level', p=5, no_labels=True, color_threshold=Z[-3,2], ax=ax)
ax.set(title='Hierarchical clustering dendrogram (Ward linkage)', ylabel='merge distance')
plt.tight_layout(); plt.show()
agg = AgglomerativeClustering(n_clusters=4).fit(Xs)
print('agglomerative vs k-means agreement (crosstab):'); print(pd.crosstab(df['kmeans'], agg.labels_))
agglomerative vs k-means agreement (crosstab): col_0 0 1 2 3 kmeans 0 0 0 0 85 1 0 81 0 0 2 0 0 82 0 3 82 0 0 0
A tree of nested groups. Hierarchical clustering starts with every point alone and repeatedly merges the two closest groups, recording the whole history as a dendrogram. Cutting the tree at a chosen height gives you the clusters, the tall vertical gaps show where the natural cut is (here, four branches). Unlike K-means it needs no k up front and is deterministic, though it is slower on large data. The crosstab shows it recovers the same four segments K-means found. That agreement is reassuring but not independent evidence, because Ward linkage merges whichever pair least increases the within-cluster sum of squares, the same quantity K-means minimizes, and both run on the same standardized features. Checking that segments are real takes something else, such as stability across resamples of the customers or a separate source of data.
dbs = DBSCAN(eps=0.85, min_samples=6).fit(Xs)
df['dbscan'] = dbs.labels_
n_clusters = len(set(dbs.labels_)) - (1 if -1 in dbs.labels_ else 0)
n_noise = int((dbs.labels_ == -1).sum())
fig, ax = plt.subplots(figsize=(7.4,5.4))
mask = dbs.labels_ == -1
ax.scatter(X['annual_income_k'][~mask], X['spending_score'][~mask], c=dbs.labels_[~mask], cmap='viridis', s=36, edgecolor='white')
ax.scatter(X['annual_income_k'][mask], X['spending_score'][mask], c='red', s=90, marker='x', label='noise / outliers')
ax.set(xlabel='annual income (thousands)', ylabel='spending score', title=f'DBSCAN: {n_clusters} clusters + {n_noise} outliers flagged'); ax.legend()
plt.tight_layout(); plt.show()
print(f'DBSCAN found {n_clusters} dense clusters and flagged {n_noise} points as noise (label -1)')
DBSCAN found 4 dense clusters and flagged 5 points as noise (label -1)
Density, not centroids. DBSCAN grows clusters from dense regions: a point with at least min_samples neighbors within radius eps seeds a cluster, and sparse points left over are labeled noise (-1). Its superpowers: it discovers the number of clusters on its own, handles non-round shapes, and, uniquely here, flags outliers instead of forcing every point into a group. The price is sensitivity to eps, too small and everything is noise, too large and clusters merge. It cleanly isolates the oddball customers K-means had to absorb.
profile = df.groupby('kmeans')[feat].mean().round(1)
profile['n'] = df['kmeans'].value_counts().sort_index()
print(profile.to_string())
fig, ax = plt.subplots(figsize=(7.6,4.2))
sns.heatmap(df.groupby('kmeans')[feat].mean().T, annot=True, fmt='.0f', cmap='viridis', cbar_kws={'label':'mean'}, ax=ax)
ax.set(title='Cluster profiles: mean of each feature by segment', xlabel='k-means cluster'); plt.tight_layout(); plt.show()
annual_income_k spending_score age web_visits_mo n kmeans 0 26.7 21.8 39.1 9.8 85 1 89.0 82.5 42.2 25.5 81 2 28.0 76.1 28.2 28.2 82 3 90.0 19.7 49.2 8.3 82
The payoff. Grouping by cluster and averaging each feature turns anonymous labels into a marketing strategy: a low-income high-spending 'Young Spenders' group that browses heavily, a high-income high-spend 'Premium' group to nurture, a high-income low-spend 'Savers' group to win over, and a low-income low-spend 'Budget-Conscious' group. This profiling step, describing each cluster in plain business terms, is what makes clustering actionable. The algorithm finds the groups; you give them meaning.
Clustering, in one view¶
- Always standardize first, distance-based methods are dominated by the largest-scale feature otherwise.
- K-means: fast, assumes round equal-size clusters, you choose k (use the elbow and silhouette together).
- Hierarchical: a dendrogram of nested groups, no k up front, deterministic, slower on big data.
- DBSCAN: density-based, discovers k itself, handles odd shapes, and flags outliers as noise.
- Profiling each cluster's feature means is what turns unlabeled groups into decisions.