You cannot plot eight columns at once, and a model given many correlated features is slower and easier to overfit. But wide data is usually not as high-dimensional as it looks: a few underlying factors drive all those columns. Dimensionality reduction finds those factors and rewrites the data in far fewer numbers, keeping the signal and dropping the redundancy.
PCA replaces many correlated columns with a few principal components, ranked by variance: read the scree plot to choose how many to keep, read the loadings/biplot to interpret them, keep the top few to compress and denoise, and reach for nonlinear t-SNE/UMAP when you only need a striking 2D picture.
Why Reduce Dimensions?
Three problems come with wide data. It is impossible to visualize, the eye handles two or three dimensions, not twenty. It suffers the curse of dimensionality, in high dimensions points spread out until every pair looks equidistant, which weakens distance-based methods like clustering and nearest neighbors. And it is redundant: correlated columns repeat the same information, inflating model variance for no gain.
The key insight is that correlated columns are shadows of a smaller set of hidden factors. In a nutrition table, calories, fat, and carbs all move together because they reflect a food's underlying energy density. Dimensionality reduction recovers those factors as new axes. As with all variance- and distance-based methods, we standardize first so a large-unit column (sodium in milligrams) does not dominate purely because of its scale.
PCA: New Axes Ranked by Variance
PCA finds a new set of perpendicular axes, the principal components, oriented so the first points along the direction of greatest variance in the data, the second along the greatest remaining variance perpendicular to it, and so on. Same data, rotated axes, but now the information is front-loaded: the early components carry almost everything, the later ones carry mostly noise.
The scree plot shows how much variance each component explains, and how fast it adds up. You keep enough components to cross a threshold (commonly 90%) and discard the rest.
On the food data PC1 explains 42% of the variance and PC2 34%, so two numbers per food keep 76% of everything; three components reach 95%. An eight-column table was really only about three dimensions of information.
Loadings, Biplots & Compression
A principal component is a weighted blend of the original columns, and those weights, the loadings, are what make it interpretable. The biplot draws each original feature as an arrow: arrows pointing the same way are correlated, opposite arrows are anti-correlated, and a point far along an arrow scores high on that feature.
Here PC1 is a "carb-and-calorie load" axis (high calories, carbs, and sugar; low fiber and protein) and PC2 is a "richness/fat" axis (high fat, saturated fat, and protein). Keeping only the top few components is lossy compression: rebuilding each food from its top 3 scores retains 95% of the variance with tiny error. And because the discarded components are mostly noise, this step also denoises, which is why PCA is a standard preprocessing move before clustering or supervised models on wide data.
Real-World Example: A 2D Map of Foods
Nutrition labels carry many correlated numbers. PCA turns that wide table into a single, honest picture, and t-SNE offers a nonlinear alternative when you only need the picture.
180 packaged foods with eight nutrition columns, calories,
fat_g, satfat_g, protein_g, carbs_g, sugar_g,
fiber_g, sodium_mg, plus a food_type label used only to color plots (PCA
never sees it).
Projecting all 180 foods onto PC1 and PC2 gives a map where each food is one dot. Even though PCA is unsupervised, the food categories land in distinct regions, the compression preserved the real structure. When straight axes are not enough, t-SNE and UMAP take over:
| PCA | t-SNE / UMAP | |
|---|---|---|
| Type | linear (rotate & stretch) | nonlinear (preserves local neighborhoods) |
| Axes | interpretable, have meaning | no units, not interpretable |
| Reversible? | yes, can compress & reconstruct | no, visualization only |
| Best for | compressing, denoising, preprocessing | a crisp 2D picture of clusters |
Rule of thumb: reach for PCA to compress, denoise, and interpret; reach for t-SNE/UMAP only to visualize. They answer different questions.
Dimensionality Reduction in Machine Learning & AI
Reducing dimensions is one of the most-used preprocessing steps in the entire ML pipeline.
| Method / idea | Where it is used |
|---|---|
| PCA | preprocessing, compression, denoising, feature de-correlation before modeling |
| Scree / explained variance | deciding how many components (or features) to keep |
| t-SNE / UMAP | visualizing embeddings, images, single-cell genomics |
| Autoencoders | nonlinear neural compression; the deep-learning generalization of PCA |
| Embeddings | dense low-dimensional vectors for words, images, and users |
Modern AI runs on embeddings, dense, low-dimensional vectors that represent words, images, or users, and dimensionality reduction is the idea behind them. A neural autoencoder is essentially a nonlinear PCA: it squeezes data through a narrow bottleneck and reconstructs it, and that bottleneck is a learned low-dimensional code. When researchers want to see what a large language model has learned, they run t-SNE or UMAP on its internal activations to reveal clusters of concepts. And beating the curse of dimensionality, projecting to a compact, informative space before modeling, remains one of the most reliable ways to make high-dimensional learning both faster and more accurate.
Reduce dimensions in Python
The companion notebook runs PCA on the foods, reads the scree plot, loadings, and biplot, uses the components to compress and denoise, and contrasts the linear PCA with a nonlinear t-SNE, each cell explained.
View opens the rendered notebook instantly. Open in Colab runs it live. To run
locally, install numpy, pandas, scikit-learn, seaborn, and
openpyxl.
🎓 Key Takeaways
- ✓Wide, correlated data is rarely as high-dimensional as it looks, a few latent factors drive many columns.
- ✓PCA builds perpendicular axes ranked by variance; the scree plot and the 90% cumulative line tell you how many to keep.
- ✓Loadings and biplots make the components interpretable, showing which original columns build each axis.
- ✓Keeping the top few components compresses and denoises, a standard preprocessing step (standardize first!).
- ✓PCA is linear, reversible, and interpretable; t-SNE/UMAP are nonlinear and for visualization only.
Practice Challenges
Five short challenges. Try them with scikit-learn before checking the solutions.
How many components?
Standardize, run PCA, and find how many components retain at least 90% of the variance.
np.cumsum(pca.explained_variance_ratio_).Variance in 2D
Report the share of total variance captured by the first two principal components.
pca.explained_variance_ratio_[:2].sum().Read the loadings
Identify which original nutrient loads most strongly on PC1.
pca.components_[0]; take the largest absolute value.Compression error
Keep 3 components, reconstruct the data, and report the reconstruction error.
pca.inverse_transform(pca.transform(Xs)).Nonlinear embedding
Make a t-SNE embedding and note one way it differs from PCA.
TSNE(n_components=2).fit_transform(Xs).A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on dimensionality reduction. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
Clustering grouped rows; PCA compressed columns. Next we look for relationships between items. Association Rule Mining uncovers the "customers who bought X also bought Y" patterns behind market-basket analysis, using support, confidence, and lift.