⚙️ Setup¶
import pandas as pd, numpy as np
BASE_URL = "https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
def load(name):
try: return pd.read_csv(f"../../data/{name}.csv")
except FileNotFoundError: return pd.read_csv(BASE_URL + f"{name}.csv")
spotify, ames, penguins = load("spotify_tracks"), load("ames_housing"), load("penguins")
print("ready")
ready
def health(df):
miss = (df.isna().mean()*100).round(1).sort_values(ascending=False)
return {"rows": len(df), "duplicates": int(df.duplicated().sum()),
"top_missing": miss[miss>0].head(3).to_dict()}
print(health(spotify))
{'rows': 915, 'duplicates': 8, 'top_missing': {'popularity': 8.0, 'tempo': 2.0, 'key': 1.3}}
Answer: A small reusable audit beats eyeballing every file. On Spotify it flags duplicate rows and the columns popularity, tempo, and key as the most missing. Running the same function on every dataset (the idea behind first_contact in the companion notebook) is how professionals stay consistent: the first thing you learn about any table is its shape, its duplicates, and where it is hollow.
zero_tempo = int((spotify["tempo"]==0).sum())
rng = spotify[["danceability","loudness","tempo","duration_ms"]].agg(["min","max"])
skew_pop = spotify["popularity"].skew()
print("tempo == 0 (missing coded as zero):", zero_tempo)
print("\nscales are nothing alike:\n", rng.round(1).to_string())
print(f"\npopularity skew: {skew_pop:+.2f}")
tempo == 0 (missing coded as zero): 23
scales are nothing alike:
danceability loudness tempo duration_ms
min 0.1 -40.0 0.0 97419
max 0.9 -0.2 299.0 18000000
popularity skew: +0.56
Answer: Three different problems, three different tools. tempo == 0 is missing data in disguise (Chapter 20): replace the zeros with NaN, then impute. The features span 0–1, −40–0, and tens of thousands, so any distance- or gradient-based model needs scaling (Chapter 24). And popularity is right-skewed, a candidate for a transformation (Chapter 22) or robust summaries. Naming the tool for each symptom is the whole skill this part is reviewing.
miss = (ames.isna().mean()*100).round(1)
miss = miss[miss>0].sort_values(ascending=False)
print(miss.to_string())
DROP_ABOVE = 80 # % missing
to_drop = miss[miss >= DROP_ABOVE].index.tolist()
to_impute = miss[miss < DROP_ABOVE].index.tolist()
print("\ndrop (or recode to a category):", to_drop)
print("impute:", to_impute)
pool_qc 99.0 fireplace_qual 47.5 lot_frontage 16.1 garage_yr_built 4.8 sale_price 0.5 drop (or recode to a category): ['pool_qc'] impute: ['fireplace_qual', 'lot_frontage', 'garage_yr_built', 'sale_price']
Answer: pool_qc is missing for ~99% of homes, the missingness is informative (the house simply has no pool), so you either drop the column or recode the blanks to a real "None" category, never impute a fake quality. The lightly-missing columns (lot_frontage, fireplace_qual, garage_yr_built) are worth imputing (median for the number, "None"/mode for the categories). A simple percentage threshold is a useful first pass, but the deciding question is always why the value is missing (Chapter 20).
pg = penguins.copy()
pg["sex"] = (pg["sex"].astype("string").str.strip().str.lower()
.replace({".": pd.NA, "": pd.NA}))
print("sex after cleaning:\n", pg["sex"].value_counts(dropna=False).to_string())
measure = ["bill_length_mm","bill_depth_mm","flipper_length_mm","body_mass_g"]
pg = pg.dropna(subset=measure, how="all") # drop the all-missing rows
complete = pg.dropna(subset=measure + ["sex"]) # rows usable for modeling
print(f"\nrows after dropping empty: {len(pg)} fully complete: {len(complete)}")
sex after cleaning: sex male 178 female 158 <NA> 11 rows after dropping empty: 345 fully complete: 336
Answer: Cleaning a text category is three moves: trim whitespace, normalize case, and map junk tokens (".", "") to a true missing value (Chapter 19). Rows where every measurement is missing carry no information and are dropped (Chapter 20). What is left is a clean, analysis-ready frame, and the count of fully-complete rows tells you honestly how much usable data you actually have.
toolbox = {
"Duplicate rows": "Ch 19 · Duplicates & inconsistencies",
"Messy genre / sex labels": "Ch 19 · Standardizing categories",
"Missing tempo / sale_price": "Ch 20 · Handling missing data",
"pool_qc 99% missing": "Ch 20 · Drop vs impute (informative missingness)",
"Impossible flipper = 19.5 cm": "Ch 21 · Outlier detection & treatment",
"Huge-house price outliers": "Ch 21 · Outliers (remove vs keep)",
"Right-skewed popularity/price": "Ch 22 · Transformations (log, Box-Cox)",
"sale_date in 3 text formats": "Ch 23 · Parsing dates / reshaping",
"Features on different scales": "Ch 24 · Scaling / feature engineering",
"Genre / sex to dummy columns": "Ch 24 · Encoding categoricals",
}
for k, v in toolbox.items():
print(f" {k:<32} -> {v}")
Duplicate rows -> Ch 19 · Duplicates & inconsistencies Messy genre / sex labels -> Ch 19 · Standardizing categories Missing tempo / sale_price -> Ch 20 · Handling missing data pool_qc 99% missing -> Ch 20 · Drop vs impute (informative missingness) Impossible flipper = 19.5 cm -> Ch 21 · Outlier detection & treatment Huge-house price outliers -> Ch 21 · Outliers (remove vs keep) Right-skewed popularity/price -> Ch 22 · Transformations (log, Box-Cox) sale_date in 3 text formats -> Ch 23 · Parsing dates / reshaping Features on different scales -> Ch 24 · Scaling / feature engineering Genre / sex to dummy columns -> Ch 24 · Encoding categoricals
Answer: Every messy symptom maps to a tool you already own. That mapping, symptom → technique, is the muscle the next three case studies build. When you can look at a raw file and immediately see "duplicates here, informative missingness there, a skew that wants a log, a category that needs encoding," you are no longer cleaning data by reflex; you are doing exploratory data analysis with intent.