Dimensionality Reduction: many columns into a few¶
Real datasets are wide: dozens of correlated columns that are hard to plot, slow to model, and noisy. Dimensionality reduction compresses them into a handful of informative axes while keeping most of the information. This notebook uses PCA (principal component analysis) on packaged-food nutrition, reads its scree plot, loadings, and biplot, uses it to compress and denoise, and contrasts it with the nonlinear t-SNE. Library-first with scikit-learn.
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
pd.set_option('display.max_columns', 30)
try: df = pd.read_excel('../../data/dimensionality-reduction--foods.xlsx', sheet_name='Data')
except FileNotFoundError: df = pd.read_excel(BASE + 'dimensionality-reduction--foods.xlsx', sheet_name='Data')
num = ['calories','fat_g','satfat_g','protein_g','carbs_g','sugar_g','fiber_g','sodium_mg']
Xs = StandardScaler().fit_transform(df[num]) # PCA needs standardized inputs
fig, ax = plt.subplots(figsize=(7,5.6))
sns.heatmap(df[num].corr(), annot=True, fmt='.2f', cmap='coolwarm', center=0, ax=ax)
ax.set_title('Correlation among the 8 nutrition columns'); plt.tight_layout(); plt.show()
Why reduce? These eight columns are heavily correlated, calories move with fat and carbs, sugar moves against fiber, and so on. That redundancy means the data does not really live in 8 dimensions; a few underlying factors (energy density, sweet vs savory) drive everything. Dimensionality reduction finds those factors. As always with distance- and variance-based methods, we standardize first so a large-range column like sodium does not dominate purely because of its units.
pca = PCA().fit(Xs)
ev = pca.explained_variance_ratio_
cum = np.cumsum(ev)
fig, ax = plt.subplots(figsize=(8,4.4))
ax.bar(range(1,9), ev, color=EM, alpha=0.85, label='each PC')
ax.plot(range(1,9), cum, 'o-', color=RED, lw=2.2, label='cumulative')
ax.axhline(0.9, color=GREY, ls='--'); ax.set(xlabel='principal component', ylabel='share of variance', title='Scree plot: variance explained per component'); ax.legend()
plt.tight_layout(); plt.show()
print('variance explained: PC1 %.0f%% PC2 %.0f%% PC3 %.0f%%' % (ev[0]*100, ev[1]*100, ev[2]*100))
print('first 2 PCs: %.0f%% of all variance | first 3: %.0f%%' % (cum[1]*100, cum[2]*100))
variance explained: PC1 42% PC2 34% PC3 19% first 2 PCs: 76% of all variance | first 3: 95%
Reading the scree plot. PCA rebuilds the data on new perpendicular axes (principal components) ordered so the first captures the most variance, the second the next most, and so on. Here PC1 explains 42% and PC2 34%, so a 2D picture keeps 76% of everything; adding PC3 reaches 95%. The 8-column dataset was really only about 3 dimensions of information. The 'elbow' in the scree plot, and the point where the cumulative curve crosses 90%, tell you how many components to keep.
P = PCA(n_components=2).fit_transform(Xs)
fig, ax = plt.subplots(figsize=(7.6,5.6))
for t,c in zip(['Energy-dense','Sweet','Savory'], [EM, PUR, GREEN]):
m = df['food_type']==t
ax.scatter(P[m,0], P[m,1], s=40, c=c, edgecolor='white', label=t)
ax.set(xlabel='PC1 (42% of variance)', ylabel='PC2 (34% of variance)', title='Foods projected onto their first two principal components'); ax.legend()
plt.tight_layout(); plt.show()
A readable map. Each food is now a single dot in a 2D plane, its coordinates are its scores on PC1 and PC2. Even though PCA never saw the food_type label (it is unsupervised), coloring the dots by type shows the categories fall in different but overlapping regions, savory foods to the upper left, sweet foods low and energy-dense foods to the upper right, so the compression kept much of the structure. This is the everyday magic of PCA, turning an un-plottable 8-column table into one honest picture you can actually look at.
pca2 = PCA(n_components=2).fit(Xs)
comp = pca2.components_
load = pd.DataFrame(comp.T, index=num, columns=['PC1','PC2']).round(2)
print(load.to_string())
fig, ax = plt.subplots(figsize=(7.6,5.8))
ax.scatter(P[:,0], P[:,1], s=18, c=GREY, alpha=0.4)
for i,name in enumerate(num):
ax.arrow(0,0, comp[0,i]*3.2, comp[1,i]*3.2, color=EM, width=0.006, head_width=0.12, length_includes_head=True)
ax.text(comp[0,i]*3.6, comp[1,i]*3.6, name, color=DEEP, fontsize=9, ha='center', fontweight='bold')
ax.axhline(0,color=GRID); ax.axvline(0,color=GRID); ax.set(xlabel='PC1', ylabel='PC2', title='Biplot: how each nutrient loads onto PC1 and PC2')
plt.tight_layout(); plt.show()
PC1 PC2 calories 0.43 0.35 fat_g 0.35 0.44 satfat_g 0.35 0.43 protein_g -0.27 0.37 carbs_g 0.47 -0.13 sugar_g 0.35 -0.33 fiber_g -0.38 0.39 sodium_mg -0.11 0.29
Interpreting the axes. Loadings are the recipe for each component. PC1 loads highest on carbs and calories, equally on sugar, fat and saturated fat, and negatively on fiber and protein, an overall energy-density axis. PC2 sets fat, saturated fat, fiber and protein against sugar, roughly savory and rich against sweet. In the biplot, each arrow is a nutrient: arrows pointing the same way are correlated, opposite arrows are anti-correlated (sugar vs fiber), and a food sitting far along an arrow is high in that nutrient. This is what makes PCA interpretable, the axes are readable combinations of the originals, not a black box.
errs = []
for k in range(1,9):
pk = PCA(n_components=k).fit(Xs)
recon = pk.inverse_transform(pk.transform(Xs))
errs.append(np.mean((Xs - recon)**2))
fig, ax = plt.subplots(figsize=(7.6,4.2))
ax.plot(range(1,9), errs, 'o-', color=EM, lw=2.4)
ax.set(xlabel='components kept', ylabel='reconstruction error (MSE)', title='Fewer components = lossy compression')
ax.axvline(3, color=RED, ls='--'); plt.tight_layout(); plt.show()
print('keeping 3 of 8 components reconstructs the data with error %.3f and retains 95%% of variance' % errs[2])
keeping 3 of 8 components reconstructs the data with error 0.053 and retains 95% of variance
A dial between size and fidelity. Reconstructing each food from only its top-k component scores is lossy compression: fewer components mean a smaller representation but a larger reconstruction error. The curve drops steeply then flattens, keeping just 3 of the 8 components rebuilds the table almost perfectly (95% of the variance). This table was built from three underlying factors, so the discarded components are mostly noise and this step also denoises the data. That holds only when the signal lives in the high-variance directions, because PCA keeps variance, which is not the same as information about a target, which is exactly why PCA is a standard preprocessing step before clustering or supervised models on wide, correlated data.
emb = TSNE(n_components=2, perplexity=30, random_state=0, init='pca').fit_transform(Xs)
fig, ax = plt.subplots(1, 2, figsize=(12,5))
for t,c in zip(['Energy-dense','Sweet','Savory'], [EM, PUR, GREEN]):
m = df['food_type']==t
ax[0].scatter(P[m,0], P[m,1], s=28, c=c, edgecolor='white', label=t)
ax[1].scatter(emb[m,0], emb[m,1], s=28, c=c, edgecolor='white', label=t)
ax[0].set_title('PCA (linear)'); ax[1].set_title('t-SNE (nonlinear)'); ax[0].legend()
plt.tight_layout(); plt.show()
Linear vs nonlinear. PCA can only rotate and stretch, it finds straight axes, which makes it fast, reversible, and interpretable. t-SNE (and its faster cousin UMAP) instead preserve local neighborhoods, unfolding curved manifolds into tidy islands that often separate groups more crisply. The trade-offs: t-SNE and UMAP are for visualization only (the axes have no units, distances between clusters are not meaningful, and t-SNE cannot place new points at all, while UMAP can only approximately). Rule of thumb: reach for PCA to compress, denoise, and interpret; reach for t-SNE/UMAP to make a striking 2D picture of high-dimensional data.
Dimensionality reduction, in one view¶
- Wide, correlated data does not need all its columns, a few latent factors drive it.
- PCA builds ranked perpendicular axes; the scree plot and the 90% cumulative line tell you how many to keep.
- Loadings / biplot make the components interpretable (which original columns build each axis).
- Keeping the top few components compresses, and denoises when the signal lives in the high-variance directions, a standard preprocessing step.
- PCA is linear and interpretable; t-SNE / UMAP are nonlinear and for visualization only.