A score of 85 is good on one test and poor on another. Raw numbers only mean something next to the mean and spread they came from. The z-score bakes both in and gives you a universal unit: standard deviations from the mean.
The z-score Formula
Subtract the mean, divide by the standard deviation. That is the whole move.
Same structure either way: only the symbols change. With real data you usually have a sample, so you use x̄ and s.
If a test has mean 70 and standard deviation 8, then a score of 86 is z = (86 − 70) / 8 = +2.0, two SDs above average. A score of 62 is z = (62 − 70) / 8 = −1.0, one SD below. The points vanish; what remains is position on a common ruler.
Convert every value to its z-score and the new dataset always has mean 0 and standard deviation 1. That is the point of the standard scale: a fixed center and a fixed unit, no matter what you started with.
The Standard Scale & the Normal Curve
z-scores live on one axis centered at 0. When the data is roughly normal, that axis lines up with the familiar bell, and a z translates straight into a percentile through the 68-95-99.7 rule from Measures of Dispersion.
You can compute a z-score for any distribution, it only needs a mean and an SD. But turning that z into a percentile (and the 68-95-99.7 rule itself) is valid only when the data is roughly normal. For skewed or heavy-tailed data, z = +2 is not the 98th percentile.
Why Standardize
Three jobs the standard scale does well:
Compare across scales
SAT vs ACT, math vs reading, cm vs kg: convert each to z and ask who is further above their own average.
Feature scaling for ML
Distance- and gradient-based models (KNN, K-means, SVM, PCA) need features on one scale or the big-numbered one dominates.
Spot outliers
Flag values past a cutoff: |z| > 2 is unusual, |z| > 3 is a common outlier threshold.
You scored 84 on Exam A (mean 78, sd 6) and 85 on Exam B (mean 55, sd 20). Raw, B looks better. But the z-scores are +1.0 and +1.5: relative to each class, you actually did better on Exam B. This is the everyday power of the standard scale.
Pediatric growth charts report a child's height and weight as z-scores (SD-scores) against a reference population, with cutoffs like −2 SD flagging concern. Same idea, life-or-death context.
Standardization vs. Normalization
Two scaling methods get muddled because "normalization" is an overloaded word. They are different transforms with different uses.
| Method | Formula | Result | scikit-learn | Best when |
|---|---|---|---|---|
| Standardization (z-score scaling) | (x − mean) / sd | mean 0, sd 1, unbounded | StandardScaler | roughly Gaussian data; PCA, linear models, regularization |
| Normalization (min-max scaling) | (x − min) / (max − min) | fixed range, usually [0, 1] | MinMaxScaler | you need bounded inputs; but fragile to outliers |
Min-max is sensitive to extremes: one giant value sets the max and squashes everything else toward 0. Standardization spreads that hit across the mean and SD instead.
In machine learning, learn the mean and SD (or min and max) from the training set, then
apply that same scaler to the test set. Computing them over all the data before splitting leaks test
information into training and inflates your scores. A scikit-learn Pipeline handles this
safely inside cross-validation.
This is the most common misconception. Standardization is a linear transform: it shifts and rescales, so the shape, the skew, and the ordering all stay exactly as they were. A right-skewed dataset is just as right-skewed after standardizing. To change shape you need a nonlinear transform like the log (see Shape of a Distribution).
The Altman Z-score is a bankruptcy-prediction formula built from financial ratios. Despite the name, it is unrelated to the statistical z-score in this chapter. Same word, different thing.
Standardization in Machine Learning & AI
Standardization is not an optional nicety in machine learning, it is a required step for a whole class of models.
The z-score you compute by hand here is exactly what StandardScaler applies to every feature.
| Model family | Why scaling matters | If you skip it |
|---|---|---|
| Distance-based (KNN, k-means, SVM) | Distances are dominated by whichever feature has the biggest raw range | A feature in dollars silently outvotes one in years, purely because of units |
| Gradient descent (neural nets, logistic/linear regression) | Wildly different scales make the loss surface stretched and slow to optimize | Training is slower and less stable, and may not converge well |
| Regularized models (ridge, lasso) | The penalty is applied per coefficient, so it must see features on a common scale | Large-scale features are under-penalized and small-scale ones over-penalized |
One rule matters in practice: compute the mean and SD on the training set, then apply that same transform to the test set. Standardizing before you split lets test-set information leak into training, a subtle but real form of data leakage that inflates your reported accuracy.
Real-World Example: Comparing Scores Across Scales
Standardization's original job is comparison. These 300 students each sat three exams marked on very different scales, math out of 100, reading out of 40, science out of 200, so the raw numbers are not comparable: a 35 is superb on reading but poor on science. Converting each exam to a z-score puts all three on one common scale, and the companion notebook uses it to rank a student across subjects.
One row per student: student_id, math_100,
reading_40, and science_200, three exam scores on different maximums. Standardizing
each column is what makes a cross-subject comparison, or a machine-learning model, treat them fairly.
Bring it to life in Python
The companion notebook computes z-scores by hand and with SciPy, proves standardizing keeps the shape, compares two exams on one ruler, hunts outliers with classic and robust z-scores, and scales features for machine learning with a fit-on-train-only StandardScaler.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas,
scipy, scikit-learn, matplotlib and launch jupyter notebook.
🎓 Key Takeaways
- ✓z = (x − mean) / sd: how many standard deviations a value is from the mean. Sign = direction, size = distance.
- ✓Standardized data has mean 0 and sd 1, the standard scale.
- ✓z works on any data; the z-to-percentile mapping (68-95-99.7) holds only when data is roughly normal.
- ✓Standardizing does not change shape: it is linear, so skew and ordering survive.
- ✓For ML, scale features and fit on train only; standardization (z) and min-max normalization are different tools.
Practice Challenges
Five short challenges, beginner to intermediate. Try them on paper or in Python before checking the solutions.
z by hand
A test has mean 70 and standard deviation 8. Compute the z-score for a 86 and for a 62, and say what each means in words.
One common ruler
You scored 88 in History (mean 80, sd 5) and 88 in Physics (mean 75, sd 13). Same raw score: on which exam did you perform relatively better?
z to percentile
Heights are roughly normal with mean 170 cm, sd 7 cm. For a 184 cm person, find the z-score and the approximate percentile. What assumption does the percentile rely on?
scipy.stats.norm.cdf(z); the mapping needs normality.Outlier hunt
For [48, 50, 51, 49, 52, 50, 47, 53, 51, 95], compute z-scores and flag any value with
|z| > 2.5.
Shape check
Standardize a right-skewed sample rng.lognormal(mean=1, sigma=0.7, size=3000). Confirm
mean ≈ 0 and sd ≈ 1, then check whether the skewness changed.
scipy.stats.skew before and after.A fully-worked solutions notebook walks through all five challenges in the same visual style. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on standardization. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.