The Data-Cleaning Mindset chapter warned about hidden missing codes like -99, and the Finding & Removing Duplicates and Inconsistencies chapter showed that
coercing bad values to numbers creates NaN. Now the gaps are real and visible. The
question is what to do with them, and the honest answer starts with a question of your own: why are they
missing?
Why Missingness Has a Mechanism
It is tempting to just drop the blanks and move on. But whether that is safe depends entirely on why they are blank. Rubin's classic taxonomy names three mechanisms, and the mechanism decides the method.
A test (Little's MCAR test) can give evidence against MCAR, but MAR and MNAR are impossible to tell apart from the observed data alone, because the deciding information is exactly the values you don't have. The mechanism is an assumption you justify with domain knowledge, and you should state it openly.
Detect First, Then Decide on Deletion
Before any fix: convert hidden codes (-99, "N/A") to real NaN, then
count. df.isna().sum() and df.isna().mean()*100 tell you how much is missing and
where; a missingness heatmap shows whether columns go blank together (a clue toward MAR).
The simplest response is deletion, and it is the most common mistake:
| Method | How | Good | Risk |
|---|---|---|---|
| Listwise (complete-case) | df.dropna() drops any row with a gap | Simple; unbiased under MCAR | Can discard a LOT of rows; biased under MAR/MNAR |
| Pairwise | Use rows present for each calculation (e.g. df.corr()) | Keeps more data | Different stats on different subsamples |
| Drop the column | Remove a feature that is mostly empty (rule of thumb >50–60%) | Sheds a near-useless feature | Threshold is arbitrary; missingness may itself be informative |
With three columns each only 10% missing, only about 0.9³ ≈ 73% of rows survive listwise deletion, you can lose a quarter of the data while no single column looks bad. And deletion is unbiased only under MCAR; under MAR or MNAR it systematically distorts your results.
Imputation: Filling the Gaps
Rather than drop, you can impute, estimate the missing values. The methods run from trivial to sophisticated, and each has a catch worth knowing.
| Method | How it works | The catch | scikit-learn |
|---|---|---|---|
| Mean / median / mode | Fill with the column's average (median for skewed, mode for categorical) | Shrinks variance, biases correlations toward 0; ignores other columns | SimpleImputer(strategy=...) |
| Constant + indicator | Fill a fixed value and add a "was-missing" flag column | Keeps the missingness signal; constant fill alone still distorts | add_indicator=True |
| Regression | Predict the gap from other columns | Deterministic version inflates correlations; add a random residual (stochastic) | IterativeImputer |
| KNN | Fill from the k most similar rows | Must standardize first; curse of dimensionality | KNNImputer |
| MICE (iterative) | Model each column from all others, cycle to convergence | The serious default under MAR; sklearn's is experimental & single | IterativeImputer |
Replace every gap with the column mean and the standard deviation drops, a fake spike piles up at the mean, and relationships with other variables wash out. It is a fine quick baseline, but never a basis for careful inference. The notebook shows the variance shrinking before your eyes.
A filled-in value is a guess, but the analysis treats it as if it were truly observed, so
standard errors come out too small and confidence intervals too narrow. Multiple imputation
fixes this: generate several complete datasets, analyze each, and pool the results (Rubin's rules) so the
uncertainty of guessing is carried through. scikit-learn's IterativeImputer is MICE-style but
does single imputation; full multiple imputation lives in R's mice or
statsmodels.
A Decision Framework
Put it together into a repeatable flow. The mechanism drives the method; there is no universal best.
Same discipline as scaling in the Standardization & Z-Scores chapter: learn the fill values (mean, neighbors, MICE model) from the
training set, then apply them to the test set. Imputing over the whole dataset before the
split leaks test information into training and inflates your scores. A scikit-learn Pipeline
handles this automatically inside cross-validation.
Most estimators refuse to run on NaN, so you must deal with gaps before modeling, and how you
deal with them shapes every result that follows. Mishandled missingness is one of the quietest sources of
biased conclusions in all of analysis. Choosing deliberately, and documenting the assumption, is what
separates a defensible analysis from a lucky one. With gaps handled, the next chapter turns to the values
that are present but extreme: detecting and treating outliers.
Missing Data in Machine Learning & AI
Most models refuse to train on missing values, so handling them is a required step, and doing it carelessly leaks information or bakes in bias. The reason a value is missing decides the right fix.
| Approach | The scikit-learn tool | When to reach for it |
|---|---|---|
| Simple imputation | SimpleImputer (mean, median, most-frequent) | A fast, robust default, median for skewed features |
| Model-based imputation | KNNImputer, IterativeImputer | When other features can predict the missing one |
| Missing-indicator | add_indicator=True, or a manual flag column | When the fact that a value is missing is itself informative (MNAR) |
Compute the imputation value (the mean, say) on the training set, then apply it to the test set, exactly as with scaling. Impute before you split and you leak test information into training. And when data is missing not at random, add a missing-indicator, because the missingness carries signal a plain fill would erase.
Real-World Example: A Survey with Gaps
In this survey, income is missing not at random: higher earners are the ones who leave it blank. The result is a trap, the average of the incomes people did report is biased low (about 57,000 dollars against a true 61,000), and mean-imputing that value would spread the bias to everyone. The companion notebook maps the missingness and compares imputation strategies.
One row per respondent: respondent_id, age,
region, hours_online, and income. income is missing not at
random (high earners skip it), while age and hours_online are missing roughly at
random, so the mechanism differs by column.
Bring it to life in Python
The companion notebook converts sentinels to NaN and charts the gaps, measures how many rows
deletion really costs, shows mean imputation shrinking the variance with a fake spike, imputes with KNN
(scaled) and MICE, and fits an imputer on train only to avoid leakage.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas,
scikit-learn, matplotlib (and optionally missingno) and launch
jupyter notebook.
🎓 Key Takeaways
- ✓The mechanism drives the method: MCAR (random), MAR (depends on observed data), MNAR (depends on the missing value itself).
- ✓You can't prove the mechanism from data; MAR vs MNAR needs domain knowledge, so state your assumption.
- ✓Deletion is unbiased only under MCAR and compounds fast across columns; under MAR/MNAR it biases results.
- ✓Mean imputation shrinks variance; single imputation understates uncertainty (multiple imputation fixes it); KNN/MICE use other columns.
- ✓Fit imputers on train only to avoid leakage, and keep a missingness indicator when the gap is informative.
Practice Challenges
Five short challenges, beginner to intermediate. Try them on paper or in Python before checking the solutions.
Name the mechanism
Classify each: (a) a scale randomly drops every 50th reading from a glitch; (b) high earners skip the income question; (c) men skip an item more often, and sex is recorded.
Count the gaps
A frame uses -1 for missing age. Convert it to NaN and report the count and
percent missing per column.
replace(-1, np.nan) first, then isna().sum() / isna().mean()*100.Deletion cost
For a 500-row frame where each of 3 columns is independently 10% missing, how many complete rows survive listwise deletion? Compute it.
len(df.dropna()); compare with 0.9³.Mean imputation cost
Take 300 values ~N(100, 20), blank out 40%, then mean-impute. Compare the standard deviation before and after, and explain what happened.
SimpleImputer(strategy="mean"); the SD shrinks.No leakage
Split a frame with missing values into train/test, then impute without leaking. Show the test set is filled with the value learned from the training data.
SimpleImputer().fit(X_train) then transform(X_test).A fully-worked solutions notebook walks through all five challenges in the same visual style. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on missing data. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.