The Spotify case study was about scale and structure. This one is about judgment. The Ames housing data is the classic teaching set for missing values, because the gaps are not random, they mean different things, and a good analyst reads that meaning before reaching for a fix.
ames_housing.csv records 610 home sales with sale_price, gr_liv_area
(above-grade living area), lot_area, year_built, neighborhood,
fireplace_qual, pool_qc, and more. It is gloriously messy: duplicate rows, dates in
three formats, the famous "big house sold cheap" outliers, a strongly skewed price, and five columns with
missing values ranging from a 0.5% trickle to a 99% flood.
First Contact: The Missingness Is the Story
The audit finds 10 duplicate rows and a date column stuck in text, but the headline is the missing values. And how much is missing hints at why, and that determines the fix:
It is tempting to set a rule like "drop any column over 50% missing". But fireplace_qual is 48%
missing and we will keep it, because the blanks mean "no fireplace", real information. The percentage
tells you where to look; only the reason for the gap tells you what to do. That is the entire lesson of this
chapter.
Describe & Visualize: Price and Its Outliers
Before cleaning, we get to know the target, sale_price. It is strongly right-skewed
(mean $184k, median $173k, skew ≈ +1.2): a few expensive homes pull the average up. And a scatter of living area
against price reveals the dataset's most famous quirk.
Almost every home follows a clean upward trend, bigger means pricier. But four houses over 4,000 sq ft sold for under $185k, far below where their size says they should. These are almost certainly partial or family sales, not open-market deals. A single scatter plot caught what no summary statistic would, and we will deal with them in the prep step (see Charts for Numerical Data and Detecting & Treating Outliers).
Cleaning: Duplicates, Categories & Dates
Three quick fixes clear the deck before the missing-value work.
Duplicates (see Finding & Removing Duplicates and Inconsistencies). Ten rows are exact copies of other sales; left in, they would double-count
those homes. drop_duplicates() takes us from 610 to 600 rows.
Inconsistent category (see Finding & Removing Duplicates and Inconsistencies). central_air was recorded as both "Y" and
"Yes" (and "N" / "No") for the same thing. Until those merge, a group-by
would split one category into two, so we map them to a single convention.
Dates in three formats (see Combining & Reshaping Data). sale_date arrived as text in three different layouts,
"09/07/1955", "1940-03-19", and "19-Dec-1950". A date trapped in a string
is useless: you cannot sort it, difference it, or pull the year. One call,
pd.to_datetime(sale_date, format="mixed"), parses all three at once (with zero failures) and gives us
a real sale_year.
Two more category columns get the same standardizing treatment here: kitchen_qual
(po, PO, Ta … → uppercased) and exterior_color
(where grey and gray are merged). Cleaning them now means they are ready to encode later.
After parsing dates, count the failures (NaT values). A format that silently fails to parse
turns a date into a hidden missing value. Here all 600 rows parsed cleanly, so sale_year is
trustworthy.
Missing Values: One Column, One Decision
This is the heart of the chapter. Five columns are missing values, and each gets a different treatment, driven entirely by why the value is absent.
| Column | Missing | Why it is missing | Decision |
|---|---|---|---|
| pool_qc | 99% | The home has no pool | Drop the column (almost no signal) |
| fireplace_qual | 48% | The home has no fireplace | Recode to "None" (keep as a real category) |
| lot_frontage | 16% | An ordinary measurement gap | Median-impute |
| garage_yr_built | 5% | The home has no garage | Flag has_garage, drop the year |
| sale_price | 0.5% | Simply not recorded | Drop the rows (it is the target) |
Notice that three of these gaps are informative: a blank pool, fireplace, or garage field is
telling you the feature does not exist, not that the data is lost. You never impute a fake pool quality. You
either drop the column (when it is almost all blank, like pool_qc) or recode the blank into a
meaningful category or flag (like fireplace_qual and garage_yr_built). Only
lot_frontage is a true measurement gap worth imputing, and the target,
sale_price, is never imputed at all (see Handling Missing Data).
Preparing: Outliers & Transform
Remove the anomalies (see Detecting & Treating Outliers). The four big-but-cheap houses break the size-to-price relationship, so we remove them. The payoff is immediate: the correlation between living area and price jumps from 0.47 to 0.58 once those four contrarian points are gone.
Transform the skew (see Transformations). Price still has a long right tail (skew ≈ +1.15). A natural-log
transform compresses that tail into a near-symmetric, bell-shaped distribution (skew ≈ +0.02), which is exactly
what regression and most summaries prefer. This is why housing models almost always predict log(price):
Encode by type (see Feature Engineering). The file also carries two more category columns we cleaned along the way:
kitchen_qual (Po / Fa / TA / Gd / Ex) and exterior_color (white / beige / gray / brick /
blue), both in messy casing (including the classic gray vs grey). They are encoded
differently because one has an order and one does not: kitchen_qual is ordinal, so it
becomes integer codes 1 to 5 that keep the quality order; exterior_color and neighborhood
are nominal, so they become one-hot 1/0 columns.
Here is the whole journey, every row accounted for:
Exploring the Clean Data
With the data clean, the measurements and grouped charts finally tell the truth, and they point to a single, valuable conclusion about what drives a home's price.
Measurements: profiling the price
| Measurement | sale_price | What it tells us |
|---|---|---|
| Mean / median / mode (see Measures of Central Tendency) | $184k / $173k / $161k | Mean > median confirms the right skew |
| Std dev / IQR (see Measures of Dispersion & Measures of Position) | $64k / $79k | Wide spread of home values |
| Coeff. of variation (see Measures of Dispersion) | 35% | Prices vary substantially around the mean |
| Quartiles Q1 / Q3 (see Measures of Position) | $139k / $217k | The middle 50% of homes sold here |
| Skewness (see Shape of a Distribution) | +1.15 | Right-skewed, hence the log transform |
The numeric relationships: a correlation heatmap (see Multivariate & Specialized Visuals)
A heatmap only takes numbers, so both categoricals are encoded to join it: neighborhood
by its median price (mean encoding) and kitchen_qual as an ordinal 1 to 5. Now location, kitchen
quality, and the numeric features all sit in one matrix. Scan the price row first.
Both encodings prove their worth. Mean-encoded neighborhood is the strongest correlate of price (0.63), and the ordinal kitchen quality is third (0.52), ahead of bedrooms, a category that would have been invisible to a numbers-only heatmap now reads as a real driver, with its order intact. The size features still form their own collinear block (0.5 to 0.77), and the lot columns stay near zero. (Caution as before: the price-based neighborhood encoding is mildly optimistic and should be cross-fit for a real model; see Feature Engineering.)
Where price really comes from (see Charts for Numerical Data and Multivariate & Specialized Visuals)
The most useful housing chart groups price by neighborhood. The box plots fan out dramatically:
NorthRidge homes sell for nearly double those in OldTown.
Living area is the strongest single numeric driver of price (r ≈ 0.58 after cleaning), but the box plots show neighborhood matters even more: the median jumps from about $146k in OldTown to about $276k in NorthRidge, nearly double, before you account for a single square foot. "Location, location, location" is not folklore here; it is the data (see Charts for Numerical Data and Multivariate & Specialized Visuals).
The last visual check: all the prep work, at a glance
The Ames data is a teaching favorite precisely because its missing values force you to think instead of reaching for a default. Drop, recode, impute, flag, or delete the row, the right answer changes column by column, and only a reason can choose. Master this dataset and you have mastered the single most common, and most consequential, judgment call in applied data work.
Run the whole case study in Python
The companion notebook is the full walkthrough: the first-contact audit, the price summary and the living-area scatter, the deduplication and date parsing, the five different missing-value treatments, the outlier removal (watch the correlation climb) and log transform, then a final exploration beat, a measurements table and the price-by-neighborhood box plots, ending at the analysis-ready table. Every cell explains its decision.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas,
matplotlib and launch jupyter notebook.
🎓 Key Takeaways
- ✓Missingness is the headline: how much is missing is a clue, but why it is missing decides the fix.
- ✓Five gaps, five answers: drop
pool_qc(99%), recodefireplace_qualto "None", median-imputelot_frontage, flaghas_garage, drop rows missing the price target. - ✓Parse messy dates with
format="mixed"and always check the failure count (see Combining & Reshaping Data). - ✓Outliers can be removed with reason: the 4 big-but-cheap homes were partial sales; dropping them lifted the size-price correlation 0.47 → 0.58.
- ✓Log-transform a skewed price (+1.15 → +0.02); then explore, and find that location beats size.
Practice Challenges
Five challenges on the Ames data, following the case study beat by beat. Try them in Python before checking the solutions.
Map the missingness
List every column with missing values and its percentage. For each, decide drop / recode-to-None / impute / flag / drop-rows, and justify it from why the value is missing.
Parse three date formats
Parse sale_date (which holds three layouts) into real datetimes, extract the sale year, and
confirm none failed to parse.
pd.to_datetime(s, format="mixed"); check .isna().sum() for NaT.Big houses, small prices
Find homes over 4,000 sq ft that sold cheaply. Remove them and show the living-area-to-price correlation before and after. Why is removing them justified?
gr_liv_area > 4000; correlation rises ~0.47 → 0.58.Straighten the price
Log-transform sale_price, report the skew before and after, and plot both distributions. Why do
housing models prefer log(price)?
np.log; skew falls from ~+1.2 toward 0.Size vs location
Compute the median price per neighborhood and draw box plots. Which is the bigger price driver, living area or neighborhood? Back it up with numbers.
A fully-worked solutions notebook answers all five on the real data, in the same visual style. Try them yourself first, then compare.
Quiz: Test Yourself
Eight questions on the Ames case study. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.