"How strongly are these two related?" is one of the most asked questions in all of data analysis, in feature selection, in finance, in science. The answer is a correlation coefficient, and there is more than one, each right for a different shape of data.
Pearson measures linear strength; Spearman measures monotonic strength (robust to curves and outliers); partial correlation measures the link after removing a third variable.
Pearson's r: Standardized Covariance
Divide covariance by the spread of each variable and the units cancel, pinning the result into [−1, 1]. Now ±1 is a perfect straight line, 0 is no linear relationship, and any value in between is directly comparable across any pair of variables.
The everyday companion is r², the share of variance explained: an r of 0.7 means about 49% of one variable's variation is linearly shared with the other. It reframes "how strong?" as "how much does knowing x tell me about y?", and is the same r² that reappears in the Simple Linear Regression chapter.
Interpreting r (and Its Traps)
A correlation is a one-number summary, and like all summaries it can mislead. Three habits keep you honest:
| Rough guide (|r|) | Strength | but always… |
|---|---|---|
| 0.0 – 0.1 | negligible | plot it first: Anscombe's quartet shows four very different clouds with the same r |
| 0.1 – 0.3 | weak | r measures only linear association; a strong curve can give r ≈ 0 |
| 0.3 – 0.5 | moderate | outliers can inflate or flip r dramatically |
| 0.5 – 1.0 | strong | correlation is not causation (see Correlation vs. Causation) |
The thresholds are only rough rules of thumb, what counts as "strong" depends on the field. The non-negotiable rule is the first one: always look at the scatterplot before trusting a single r, because a curve, an outlier, or distinct subgroups can hide behind the same number.
Spearman & Partial Correlation
Two more coefficients handle the cases where Pearson stumbles. Spearman's ρ is Pearson's r computed on the ranks, so it captures any monotonic relationship (even a curved one) and barely flinches at outliers. Partial correlation removes a third variable's influence first.
In the notebook, an outlier drags Pearson down to 0.21 while Spearman holds firm at 0.89; and when x and y are both driven by a hidden z, their raw correlation of 0.8 collapses to a partial correlation near 0 once z is controlled. That collapse, computed by correlating the residuals after regressing out z, is the exact tool the next chapter uses to expose confounding.
Real-World Example: A Fitness Correlation Matrix
A gym has 200 members with seven numeric health measures. A correlation heatmap shows every pairwise relationship at a glance, the standard first read of any numeric dataset.
One row per member with age, height_cm,
weight_kg, body_fat_pct, weekly_exercise_hrs, resting_hr, and
vo2max (aerobic fitness).
| Pair (with VO2 max) | Pearson r | Reading |
|---|---|---|
| body fat % | −0.85 | strong negative |
| resting heart rate | −0.77 | strong negative |
| weekly exercise hours | +0.78 | strong positive |
| age | −0.56 | moderate negative |
The heatmap reads instantly: aerobic fitness falls with body fat and resting heart rate, and rises with exercise.
And because the exercise-to-fitness link has diminishing returns (a curve, not a line), Spearman
(≈ 0.81) edges out Pearson (≈ 0.78), the rank measure rewards the monotonic trend the
straight-line measure slightly misses. The notebook draws the heatmap with seaborn and overlays a
LOWESS smooth to reveal the curve.
How to read it: each cell is the Pearson r between its row and column variable. The grid is symmetric, the diagonal is 1.00, and the color is the key, purple for positive, orange for negative, with deeper shades meaning stronger links. Read down the VO2-max row and the story jumps out: aerobic fitness is deep orange with body fat (−0.85) and resting heart rate (−0.77) and deep purple with exercise (+0.78), while height is near white (≈ 0). Why it matters: one glance ranks every relationship in the dataset by strength and direction, which is how analysts decide which variables carry signal, which are redundant (note body fat, weight, and resting heart rate all cluster together), and where to look next.
Correlation in Machine Learning & AI
The correlation matrix is one of the first things a data scientist computes on a new dataset.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| Correlation with the target | Quick feature screening | which inputs relate to what we predict |
| Correlation between features | Multicollinearity detection | drop one of two near-duplicate features |
| Spearman / rank | Monotonic, robust association | ordinal features, heavy-tailed data |
| Correlation heatmap | The standard EDA visual | sns.heatmap(df.corr()) |
A correlation heatmap is the fastest way to understand a fresh dataset: it surfaces which features carry signal about the target and which features are redundant with each other. That second point is multicollinearity, two highly correlated inputs confuse a linear model's coefficients, so one is often dropped or combined. Just remember the chapter's warnings: r only sees linear structure (use Spearman or a scatter for curves), and a high correlation is evidence of association, never proof of cause, the subject of the next chapter.
Compute every correlation in Python
The companion notebook plots r across strengths with scipy, shows Spearman beating Pearson on a
curve and an outlier, demonstrates partial correlation by regressing out a third variable, and loads
correlation-coefficients--fitness.xlsx for a seaborn correlation heatmap.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, seaborn, statsmodels, and openpyxl and launch
jupyter notebook.
🎓 Key Takeaways
- ✓Pearson's r = standardized covariance in [−1, 1]: sign is direction, magnitude is linear strength.
- ✓r² is the share of variance explained (r = 0.7 → ~49%); always plot the scatter before trusting one r.
- ✓Spearman's ρ uses ranks, so it captures monotonic (curved) links and resists outliers and ordinal data.
- ✓Partial correlation removes a third variable's influence (correlate the residuals), the key to spotting confounding.
- ✓In ML/AI: the correlation heatmap screens features and flags multicollinearity, the standard first look at a dataset.
Practice Challenges
Five short challenges, beginner to intermediate. Try them with SciPy and pandas before checking the solutions.
Pearson r and r²
For a pair with ρ = 0.6, compute Pearson's r and report r² as a percentage.
scipy.stats.pearsonr; r² is the variance explained.Spearman beats Pearson on a curve
For y = x³, show Spearman's ρ is near 1 but Pearson's r is lower.
scipy.stats.spearmanr works on ranks.Outlier robustness
Add one outlier to a clean linear pair; compare Pearson and Spearman.
Partial correlation
Make x and y both driven by z; show the partial correlation controlling for z collapses toward 0.
Real data: correlation matrix
Load correlation-coefficients--fitness.xlsx and print every variable's correlation with vo2max, sorted.
df.corr()["vo2max"].sort_values().A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on correlation coefficients. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
You can now measure how strongly two variables move together. Correlation vs. Causation delivers the most important caveat in all of statistics, and shows how confounding, spuriousness, and reverse causation fool the unwary.