⚙️ Setup¶
import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer
rng = np.random.default_rng(202)
print("Ready.")
Ready.
answers = {
"(a) random firmware glitch": "MCAR: missingness unrelated to any value",
"(b) high earners skip income": "MNAR: depends on the unobserved income itself",
"(c) men skip, sex is recorded": "MAR: depends on an OBSERVED variable (sex)",
}
for k,v in answers.items(): print(f"{k:34} -> {v}")
(a) random firmware glitch -> MCAR: missingness unrelated to any value (b) high earners skip income -> MNAR: depends on the unobserved income itself (c) men skip, sex is recorded -> MAR: depends on an OBSERVED variable (sex)
Answer: (a) MCAR, the glitch is unrelated to the data, so dropping those rows is unbiased; (b) MNAR, missingness depends on the very value that is missing (income), the hardest case, where standard imputation is biased; (c) MAR, missingness depends on an observed variable (sex), so imputation conditioning on sex can be approximately unbiased. You cannot read the mechanism off the data; it is a judgment from domain knowledge.
raw = pd.DataFrame({"age":[25,-1,40,-1,33,-1], "score":[88,72,np.nan,90,np.nan,67]})
df = raw.replace(-1, np.nan)
print("count missing:\n", df.isna().sum().to_string())
print("\npercent missing:\n", (df.isna().mean()*100).round(1).to_string())
count missing: age 3 score 2 percent missing: age 50.0 score 33.3
Answer: After replace(-1, np.nan), age has 3 missing (50%) and score has 2 missing (~33%). The sentinel must become real NaN first, otherwise isna() sees nothing and -1 silently drags the mean down. This is the bridge from Chapter 18 (hidden codes) and Chapter 19 (coercion creates NaN).
n=500; df = pd.DataFrame(rng.normal(0,1,(n,3)), columns=list("abc"))
for col in df.columns:
df.loc[rng.choice(n, int(0.10*n), replace=False), col] = np.nan
complete = len(df.dropna())
print(f"rows: {n}")
print(f"complete rows after dropna(): {complete} (lost {n-complete} = {100*(n-complete)/n:.0f}%)")
print(f"(roughly 0.9^3 = {0.9**3:.2f} of rows expected to survive)")
rows: 500 complete rows after dropna(): 366 (lost 134 = 27%) (roughly 0.9^3 = 0.73 of rows expected to survive)
Answer: With three columns each ~10% missing and independent, only about 0.9³ ≈ 73% of rows are complete, so listwise deletion discards roughly a quarter of the data even though no single column is badly missing. Deletion costs compound across columns, which is why "just drop the NaNs" can quietly gut a dataset (and only stays unbiased under MCAR).
full = rng.normal(100, 20, 300)
obs = full.copy(); obs[rng.choice(300, 120, replace=False)] = np.nan
filled = SimpleImputer(strategy="mean").fit_transform(obs.reshape(-1,1)).ravel()
print(f"true SD : {full.std():.2f}")
print(f"observed SD : {np.nanstd(obs):.2f}")
print(f"after mean impute: {filled.std():.2f} <- shrunk")
true SD : 19.36 observed SD : 18.92 after mean impute: 14.65 <- shrunk
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(7, 3.4))
ax.hist(full, bins=25, alpha=0.6, label="true values", color="#059669")
ax.hist(filled, bins=25, alpha=0.6, label="after mean impute", color="#e11d48")
ax.axvline(np.nanmean(obs), color="#1a2138", ls="--", lw=1, label="imputed mean")
ax.set(title="Mean imputation spikes the center and shrinks the spread"); ax.legend()
plt.tight_layout(); plt.show()
Answer: The imputed standard deviation drops well below the true ~20, because 40% of the values became the same number (the mean), adding a tall spike with zero spread. Mean imputation understates variance and biases correlations toward zero. It is fine as a quick baseline, but for anything inferential prefer a method that conditions on other variables (KNN, MICE), and remember even those understate uncertainty unless you do multiple imputation.
from sklearn.model_selection import train_test_split
X = pd.DataFrame({"v": rng.normal(50,10,200)})
X.loc[rng.choice(200, 60, replace=False), "v"] = np.nan
Xtr, Xte = train_test_split(X, test_size=0.3, random_state=1)
imp = SimpleImputer(strategy="mean").fit(Xtr) # learn from TRAIN only
train_mean = imp.statistics_[0]
Xte_filled = imp.transform(Xte)
filled_vals = Xte_filled[Xte["v"].isna().to_numpy()]
print(f"mean learned from train : {train_mean:.3f}")
print(f"value used to fill test : {np.unique(filled_vals.round(3))}")
print(f"match? {np.allclose(filled_vals, train_mean)}")
mean learned from train : 48.744 value used to fill test : [48.744] match? True
Answer: The test gaps are all filled with the training mean, not a mean recomputed on the test set, exactly what fit(Xtr) then transform(Xte) guarantees. Fitting the imputer on the full dataset before splitting would let test information leak into training and inflate your scores. Same discipline as the scaler in Chapter 12; wrap it in a Pipeline so it happens automatically in cross-validation.