Chapter 118 · Practice Challenges, Solved¶
Five clustering exercises, worked in scikit-learn and scipy on the Chapter 118 e-commerce customers.
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans, DBSCAN
from sklearn.metrics import silhouette_score
from scipy.cluster.hierarchy import linkage, fcluster
try: df = pd.read_excel('../../data/clustering--shoppers.xlsx', sheet_name='Data')
except FileNotFoundError: df = pd.read_excel(BASE + 'clustering--shoppers.xlsx', sheet_name='Data')
feat = ['annual_income_k','spending_score','age','web_visits_mo']
Xs = StandardScaler().fit_transform(df[feat])
print(df.shape)
(330, 5)
km = KMeans(n_clusters=4, n_init=10, random_state=0).fit(Xs)
df['cluster'] = km.labels_
print(df['cluster'].value_counts().sort_index().to_string())
cluster 0 85 1 81 2 82 3 82
Solution. Four roughly balanced segments of about 80 customers each, standardizing first matters so age and web visits, whose spreads are about a third of income's and spending's, still count in the distance.
ks = range(2,9)
sil = [silhouette_score(Xs, KMeans(n_clusters=k, n_init=10, random_state=0).fit_predict(Xs)) for k in ks]
best = list(ks)[int(np.argmax(sil))]
print('silhouette by k:', {k: round(s,3) for k,s in zip(ks, sil)})
print('best k =', best)
silhouette by k: {2: np.float64(0.461), 3: np.float64(0.488), 4: np.float64(0.552), 5: np.float64(0.484), 6: np.float64(0.446), 7: np.float64(0.374), 8: np.float64(0.332)}
best k = 4
Solution. The silhouette score peaks at k = 4 (about 0.55), confirming four well-separated segments.
Z = linkage(Xs, method='ward')
hc = fcluster(Z, t=4, criterion='maxclust')
ct = pd.crosstab(df['cluster'], hc)
print(ct)
print('each k-means cluster maps to one hierarchical cluster:', (ct.gt(0).sum(axis=1)==1).all())
col_0 1 2 3 4 cluster 0 85 0 0 0 1 0 0 0 81 2 0 0 82 0 3 0 82 0 0 each k-means cluster maps to one hierarchical cluster: True
Solution. Each K-means cluster lines up with exactly one hierarchical cluster, so the two algorithms recover the same structure. Ward linkage and K-means both minimize within-cluster sum of squares on the same standardized features, so their agreement is reassuring but not independent proof the segments are real.
dbs = DBSCAN(eps=0.85, min_samples=6).fit(Xs)
n_noise = int((dbs.labels_ == -1).sum())
n_clusters = len(set(dbs.labels_)) - (1 if -1 in dbs.labels_ else 0)
print(f'DBSCAN: {n_clusters} clusters, {n_noise} points flagged as noise')
DBSCAN: 4 clusters, 5 points flagged as noise
Solution. With eps = 0.85 DBSCAN recovers 4 dense clusters and isolates the 5 planted outliers as noise (label -1), something K-means cannot do (it forces every point into a cluster).
prof = df.groupby('cluster')[feat].mean().round(1)
prof['score'] = prof['annual_income_k'] + prof['spending_score']
premium = prof['score'].idxmax()
print(prof.to_string())
print('Premium segment = cluster', premium, '| mean income', prof.loc[premium,'annual_income_k'], 'k, spending', prof.loc[premium,'spending_score'])
annual_income_k spending_score age web_visits_mo score cluster 0 26.7 21.8 39.1 9.8 48.5 1 89.0 82.5 42.2 25.5 171.5 2 28.0 76.1 28.2 28.2 104.1 3 90.0 19.7 49.2 8.3 109.7 Premium segment = cluster 1 | mean income 89.0 k, spending 82.5
Solution. The cluster with both the highest income and the highest spending score is the Premium segment, the one to nurture with loyalty perks. Profiling turns an anonymous cluster id into a named, actionable group.