You already have the tools: the IQR rule and box plot from Measures of Dispersion, the z-score and the robust modified z from Standardization & Z-Scores. This chapter pulls them together for detection, then tackles the part no earlier chapter did: what to actually do about an outlier.
Three Causes, Three Fates
Before deleting anything, ask why the point is extreme. There are three broad answers, and they lead to very different actions.
The most valuable point in a dataset is sometimes the strangest one: the fraudulent charge, the failing sensor, the breakthrough result. Scrubbing outliers reflexively can erase the very thing you were looking for. And deleting a real extreme value to tidy a chart is data manipulation (see Handling Missing Data).
Detection Methods
Four ways to spot a candidate, each with a rule and a weakness. Start with the picture, then quantify.
| Method | Rule | Robust? | Best / fails when |
|---|---|---|---|
| IQR fence (Tukey) | below Q1 − 1.5·IQR or above Q3 + 1.5·IQR (3·IQR = "far out") | Yes | Distribution-free, the box-plot rule; can over/under-flag on very skewed or tiny samples |
| Z-score | |z| > 3 (sometimes 2.5) | No | Large, roughly normal data; masks on small/skewed data |
| Modified z (MAD) | |0.6745·(x − median)/MAD| > 3.5 | Yes | The recommended default; robust on small and skewed samples |
| Visual | box plot, histogram, scatter, Q-Q plot | n/a | Always do this first; a scatter reveals bivariate and leverage points |
The z-score divides by the standard deviation, but one huge value inflates that SD, dragging its
own z back under 3 so it slips through. In [2,3,3,4,4,4,5,5,6,1000] the 1000 has a plain z of
exactly 3.00 (not flagged), yet a modified z near 670. The median and MAD aren't fooled,
which is why the modified z-score is the safer detector.
The 0.6745 puts the MAD on the same scale as a standard deviation (it is the 0.75 quantile of the normal curve), so the familiar "about 3" cutoff still means something. For many variables at once, a scatter, the Mahalanobis distance, or methods like IsolationForest take over, since per-column checks miss combinations.
Treatment: What to Actually Do
Detection is step one. The real decision is the treatment, and deleting is only one option among several.
| Strategy | How | When | Caveat |
|---|---|---|---|
| Investigate first | Trace the value to its source | Always, before anything | Skipping this is the root mistake |
| Correct | Fix a verifiable entry error | Confirmed, recoverable error | Need evidence, not a guess |
| Remove / trim | Delete the row | Confirmed error, or outside your population | Never "because it's inconvenient"; document it |
| Cap / winsorize | Clip to a percentile or the IQR fence | Keep the row, limit its influence | Changes the distribution, document it |
| Transform | log / sqrt / Box-Cox to pull in a tail | Skewed, positive data | Changes units (more in Transformations) |
| Impute | Treat as missing, then impute | Likely error, true value unrecoverable | Inherits Handling Missing Data's caveats |
| Use robust methods | median & IQR, robust regression, trimmed mean | Real extremes you must not distort | Often the best answer: change the method, not the data |
| Flag / separate | Add an indicator; model outliers apart | When the outliers are the signal | Deleting them destroys the finding |
You rarely have to alter a value at all. Swap the mean and standard deviation for the median and IQR, or use robust regression, and a genuine extreme simply stops dominating. That keeps the data honest and the analysis sturdy at the same time.
A Decision Framework
Put detection and treatment together into one honest loop. The order matters: never jump from "flagged" to "deleted."
Outliers wreck mean- and distance-based models, and a single leverage point can swing a regression line, yet anomaly detection is the entire goal in fraud, fault, and security work. When you do treat outliers for a model, compute the fences, medians, and MAD on the training data only and apply them to validation and test, the same no-leakage rule as scaling and imputation (Chapters 12 and 20). And when the right move is "use a robust method," you change nothing about the data at all.
Outliers in Machine Learning & AI
Outliers pull models the way they pull a mean, only worse, because a squared-error loss punishes big misses hard. Handling them is part detection, part decision, and part choosing a method that resists them.
| Idea | Where it shows up in ML | Why it matters |
|---|---|---|
| Robust methods | RobustScaler, and MAE or Huber loss instead of MSE | They down-weight extremes so a few errors do not dominate training |
| Anomaly detection | Isolation Forest, Local Outlier Factor, one-class SVM | When finding the outliers is the task, fraud, faults, intrusions |
| Capping / winsorizing | Clipping a feature to a percentile before training | Keeps a genuine extreme in the data without letting it distort the fit |
A subtle trap: a couple of huge errors inflate the standard deviation so much that the z-score rule stops flagging the moderate outliers, on this data it catches only 2 where the robust IQR rule catches 14. Prefer the IQR fence or a robust scaler, because they lean on the median and quartiles, which the extremes cannot move.
Real-World Example: Package Weights
These 400 package weights hide two very different kinds of extreme: a few genuine heavy packages (real, keep them) and a couple of data-entry errors, a decimal slip that turned 2 kg into 240. Those errors alone drag the mean from a true 2.0 kg up to 3.2. The companion notebook detects the extremes with the IQR fence and a z-score, then decides what to keep and what to fix.
One row per package: package_id, line (three production
lines), and weight_kg. The weight column mixes normal packages, a handful of genuine heavy
outliers, and a few impossible data-entry errors, so detection and judgment both matter.
Bring it to life in Python
The companion notebook flags points with the IQR fence, shows a z-score masking its own outlier while the modified z catches it, watches one leverage point swing a regression line, winsorizes instead of deleting, and uncovers a multivariate outlier that no single column can see.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas,
scipy, matplotlib and launch jupyter notebook.
🎓 Key Takeaways
- ✓The cause dictates the treatment: error (fix/remove), genuine extreme (keep), or signal (flag). Investigate first.
- ✓The IQR fence and modified z are robust; the plain z-score can mask its own outlier by inflating the SD.
- ✓Flagged is not the same as wrong, and an outlier may be the most valuable point in the data.
- ✓Treatment options include correct, remove, cap/winsorize, transform, impute, flag, or use robust methods, document whatever you do.
- ✓Multivariate outliers hide from per-column checks; in ML, detect and treat using training data only.
Practice Challenges
Five short challenges, beginner to intermediate. Try them on paper or in Python before checking the solutions.
IQR fences
For a small dataset, compute Q1, Q3, the IQR, the 1.5×IQR fences, and list which values are flagged.
data.quantile([0.25,0.75]); fences at Q1−1.5·IQR and Q3+1.5·IQR.z vs modified z
For [10,11,12,11,13,12,10,11,12,500], compute the plain z and modified z of the 500. Does
|z|>3 catch it? Does |modified z|>3.5?
Classify the cause
Name the cause and treatment for: (a) an age of 200; (b) a real CEO paid 50× the staff median; (c) a $9,000 charge on an account averaging $40.
Cap, don't cut
Winsorize a salary column at the 95th percentile with np.clip and compare the mean before
and after. What changed, and what did not?
Invisible to columns
Make correlated height and weight, add a point that is normal on each axis but impossible together, and show neither column's IQR flags it while a scatter reveals it.
A fully-worked solutions notebook walks through all five challenges in the same visual style. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on outliers. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.