Chapter 114 · Solutions
Clustering · Solutions
Five challenges, each verified in code.
Chapter 114 · Practice Challenges, Solved¶
Five clustering exercises, worked in scikit-learn and scipy on the Chapter 114 e-commerce customers.
In [1]:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
In [2]:
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans, DBSCAN
from sklearn.metrics import silhouette_score
from scipy.cluster.hierarchy import linkage, fcluster
try: df = pd.read_excel('../../data/clustering--shoppers.xlsx', sheet_name='Data')
except FileNotFoundError: df = pd.read_excel(BASE + 'clustering--shoppers.xlsx', sheet_name='Data')
feat = ['annual_income_k','spending_score','age','web_visits_mo']
Xs = StandardScaler().fit_transform(df[feat])
print(df.shape)
(330, 5)
CHALLENGE 1
K-means with k=4
Standardize, fit K-means with 4 clusters, and report the cluster sizes.
In [3]:
km = KMeans(n_clusters=4, n_init=10, random_state=0).fit(Xs)
df['cluster'] = km.labels_
print(df['cluster'].value_counts().sort_index().to_string())
cluster 0 85 1 81 2 82 3 82
Solution. Four roughly balanced segments of about 80 customers each, standardizing first is essential so income does not dominate the distance.
CHALLENGE 2
Choose k
Use the silhouette score across k = 2..8 to justify the number of clusters.
In [4]:
ks = range(2,9)
sil = [silhouette_score(Xs, KMeans(n_clusters=k, n_init=10, random_state=0).fit_predict(Xs)) for k in ks]
best = list(ks)[int(np.argmax(sil))]
print('silhouette by k:', {k: round(s,3) for k,s in zip(ks, sil)})
print('best k =', best)
silhouette by k: {2: 0.461, 3: 0.488, 4: 0.552, 5: 0.484, 6: 0.446, 7: 0.374, 8: 0.332}
best k = 4
Solution. The silhouette score peaks at k = 4 (about 0.55), confirming four well-separated segments.
CHALLENGE 3
Hierarchical agreement
Cut a Ward dendrogram into 4 clusters and check it agrees with K-means.
In [5]:
Z = linkage(Xs, method='ward')
hc = fcluster(Z, t=4, criterion='maxclust')
ct = pd.crosstab(df['cluster'], hc)
print(ct)
print('each k-means cluster maps to one hierarchical cluster:', (ct.gt(0).sum(axis=1)==1).all())
col_0 1 2 3 4 cluster 0 85 0 0 0 1 0 0 0 81 2 0 0 82 0 3 0 82 0 0 each k-means cluster maps to one hierarchical cluster: True
Solution. Each K-means cluster lines up with exactly one hierarchical cluster, two very different algorithms recover the same structure, which is strong evidence the segments are real.
CHALLENGE 4
DBSCAN outliers
Tune DBSCAN to flag the handful of genuine outliers as noise.
In [6]:
dbs = DBSCAN(eps=0.85, min_samples=6).fit(Xs)
n_noise = int((dbs.labels_ == -1).sum())
n_clusters = len(set(dbs.labels_)) - (1 if -1 in dbs.labels_ else 0)
print(f'DBSCAN: {n_clusters} clusters, {n_noise} points flagged as noise')
DBSCAN: 4 clusters, 5 points flagged as noise
Solution. With eps = 0.85 DBSCAN recovers 4 dense clusters and isolates the 5 planted outliers as noise (label -1), something K-means cannot do (it forces every point into a cluster).
CHALLENGE 5
Name the premium segment
Profile the clusters and identify the high-income, high-spending 'Premium' group.
In [7]:
prof = df.groupby('cluster')[feat].mean().round(1)
prof['score'] = prof['annual_income_k'] + prof['spending_score']
premium = prof['score'].idxmax()
print(prof.to_string())
print('Premium segment = cluster', premium, '| mean income', prof.loc[premium,'annual_income_k'], 'k, spending', prof.loc[premium,'spending_score'])
annual_income_k spending_score age web_visits_mo score cluster 0 26.7 21.8 39.1 9.8 48.5 1 89.0 82.5 42.2 25.5 171.5 2 28.0 76.1 28.2 28.2 104.1 3 90.0 19.7 49.2 8.3 109.7 Premium segment = cluster 1 | mean income 89.0 k, spending 82.5
Solution. The cluster with both the highest income and the highest spending score is the Premium segment, the one to nurture with loyalty perks. Profiling turns an anonymous cluster id into a named, actionable group.
Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher