⚙️ Setup¶
import numpy as np
import pandas as pd
rng = np.random.default_rng(212)
print("Ready.")
Ready.
data = pd.Series([22, 24, 25, 25, 26, 27, 28, 29, 30, 31, 33, 88])
q1, q3 = data.quantile([0.25, 0.75]); iqr = q3 - q1
lo, hi = q1 - 1.5*iqr, q3 + 1.5*iqr
flagged = data[(data < lo) | (data > hi)]
print(f"Q1={q1:.1f}, Q3={q3:.1f}, IQR={iqr:.1f}")
print(f"fences = [{lo:.1f}, {hi:.1f}]")
print(f"flagged: {list(flagged)}")
Q1=25.0, Q3=30.2, IQR=5.2 fences = [17.1, 38.1] flagged: [88]
Answer: Q1 ≈ 25, Q3 ≈ 30.5, IQR ≈ 5.5, so the fences are about [16.75, 38.75] and 88 is flagged. The IQR rule is robust (it uses quartiles, not the mean/SD) and assumes no particular distribution. Remember that flagged is not the same as wrong: the rule flags roughly 0.7% of perfectly normal data by design, so investigate before acting.
x = np.array([10, 11, 12, 11, 13, 12, 10, 11, 12, 500.0])
z = (x - x.mean()) / x.std()
med = np.median(x); mad = np.median(np.abs(x - med))
mod = 0.6745 * (x - med) / mad
print(f"plain z of 500 = {z[-1]:.2f} -> flagged by |z|>3 ? {abs(z[-1])>3}")
print(f"modified z of 500 = {mod[-1]:.1f} -> flagged by |M|>3.5 ? {abs(mod[-1])>3.5}")
print(f"(the single outlier inflated the SD to {x.std():.1f}, hiding its own z)")
plain z of 500 = 3.00 -> flagged by |z|>3 ? False modified z of 500 = 659.0 -> flagged by |M|>3.5 ? True (the single outlier inflated the SD to 146.6, hiding its own z)
Answer: The plain z of 500 is only about +2.85, so |z|>3 misses it, the 500 inflated the standard deviation so much that it masked itself. The modified z is enormous (well past 3.5), so it is caught. Median and MAD are not pulled by the outlier, which is exactly why the modified z-score is the robust choice.
rows = [
("(a) age = 200", "data-entry error", "correct if recoverable, else remove/impute"),
("(b) real CEO pay 50x", "genuine extreme", "KEEP; report the median, or use robust methods"),
("(c) $9,000 vs $40 avg", "signal (fraud?)", "FLAG & investigate, do NOT delete"),
]
for case, cause, treat in rows: print(f"{case:24} | {cause:18} -> {treat}")
(a) age = 200 | data-entry error -> correct if recoverable, else remove/impute (b) real CEO pay 50x | genuine extreme -> KEEP; report the median, or use robust methods (c) $9,000 vs $40 avg | signal (fraud?) -> FLAG & investigate, do NOT delete
Answer: (a) error -> correct it (or remove/impute if you cannot recover the true age); (b) genuine extreme -> keep it and report the median instead of the mean, or use robust methods; (c) signal -> the outlier is the whole point (possible fraud), so flag and investigate, never quietly delete. The treatment always follows the cause, which is why you investigate first.
salary = np.concatenate([rng.normal(60, 12, 200), [400, 550, 800]]) # $k, a few outliers
capped = np.clip(salary, None, np.percentile(salary, 95))
print(f"raw : mean {salary.mean():7.1f}, median {np.median(salary):6.1f}")
print(f"capped : mean {capped.mean():7.1f}, median {np.median(capped):6.1f}")
raw : mean 68.1, median 59.2 capped : mean 60.5, median 59.2
Answer: Capping at the 95th percentile pulls the mean down noticeably while the median barely moves (it was robust to begin with). Winsorizing keeps every row but limits how much the extremes can dominate, useful before a mean/variance-based step. The catch: it changes the distribution, so you must document that you did it.
import matplotlib.pyplot as plt
h = rng.normal(170, 8, 150)
w = 0.9*(h-170) + 68 + rng.normal(0, 4, 150)
h = np.append(h, 150); w = np.append(w, 95) # short AND heavy: off the trend
def flag(a):
q1,q3 = np.percentile(a,[25,75]); k=1.5*(q3-q1)
return (a[-1] < q1-k) or (a[-1] > q3+k)
print(f"height alone flags it? {flag(h)}")
print(f"weight alone flags it? {flag(w)}")
fig, ax = plt.subplots(figsize=(7,5))
ax.scatter(h[:-1], w[:-1], color="#0891b2", s=24, edgecolor="white")
ax.scatter([h[-1]],[w[-1]], color="#db2777", s=140, edgecolor="white", zorder=5, label="outlier")
ax.set_xlabel("height (cm)"); ax.set_ylabel("weight (kg)"); ax.set_title("Off the height-weight trend"); ax.legend()
plt.tight_layout(); plt.show()
height alone flags it? True weight alone flags it? True
Answer: Neither column flags the point with the IQR rule, its height and weight are each in a normal range, but the scatter shows it sits far off the height-weight relationship. Per-column checks are blind to multivariate outliers; you need a two-variable view or a multivariate method (Mahalanobis distance, or sklearn IsolationForest / LocalOutlierFactor for many dimensions).