⚙️ Setup¶
import numpy as np, pandas as pd, matplotlib.pyplot as plt
CYAN="#0891b2"; AMBER="#d97706"; PURPLE="#7c3aed"; INK="#1a2138"; GRID="#e6e9f2"
SPC={"Adelie":CYAN,"Chinstrap":AMBER,"Gentoo":PURPLE}
plt.rcParams.update({"figure.dpi":110,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,
"axes.spines.top":False,"axes.spines.right":False,"axes.titleweight":"bold"})
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
try: df = pd.read_csv("../../data/penguins.csv")
except FileNotFoundError: df = pd.read_csv(BASE+"penguins.csv")
print("loaded:", df.shape)
loaded: (347, 10)
d = df.drop_duplicates().copy()
print("suspicious flippers (<50mm):", d.loc[d["flipper_length_mm"]<50,"flipper_length_mm"].to_list())
d.loc[d["flipper_length_mm"]<50, "flipper_length_mm"] *= 10
print("flipper range after fix:", d["flipper_length_mm"].min(), "to", d["flipper_length_mm"].max())
suspicious flippers (<50mm): [19.5] flipper range after fix: 165.0 to 237.0
Answer: 19.5 is impossible for a penguin flipper (~190 mm), and it is exactly one-tenth of a plausible value, the tell-tale sign of centimeters recorded as millimeters. Because we understand the cause, we correct it (×10) rather than deleting a real bird's data. An outlier with a known, fixable cause is repaired, not discarded (Chapter 21).
meas = ["bill_length_mm","bill_depth_mm","flipper_length_mm","body_mass_g"]
before = len(d)
d = d.dropna(subset=meas, how="all")
print(f"rows: {before} -> {len(d)} (dropped {before-len(d)} all-missing rows)")
print("rows still missing at least one measurement:", int(d[meas].isna().any(axis=1).sum()))
rows: 344 -> 342 (dropped 2 all-missing rows) rows still missing at least one measurement: 0
Answer: dropna(subset=meas, how="all") removes only the rows where every measurement is missing, the ones that carry no information. It deliberately keeps rows missing just one value, which can still be imputed or used. Matching the rule to the situation (how="all" vs the default how="any") is the careful move (Chapter 20).
d["sex"] = d["sex"].astype("string").str.strip().str.lower().replace({".":pd.NA,"":pd.NA})
print("after standardizing:", d["sex"].value_counts(dropna=False).to_dict())
n = int(d["sex"].isna().sum())
d["sex"] = d.groupby("species")["sex"].transform(lambda s: s.fillna(s.mode().iloc[0]))
print(f"imputed {n} gaps within species:", d["sex"].value_counts().to_dict())
after standardizing: {'male': 176, 'female': 157, <NA>: 9}
imputed 9 gaps within species: {'male': 183, 'female': 159}
Answer: First standardize (trim, lowercase, map "." and blanks to NaN) so the real categories are just male/female (Chapter 19). Then impute the few remaining gaps with the most common sex within the same species, because the species differ in size and sex ratio, so a species-aware fill is more defensible than a single global mode (Chapter 20). Clean before you impute, always.
sub = d.dropna(subset=["bill_length_mm","bill_depth_mm"])
print("overall r:", round(sub["bill_length_mm"].corr(sub["bill_depth_mm"]),2))
for sp in SPC:
s = sub[sub["species"]==sp]
print(f" within {sp:<10}: r =", round(s["bill_length_mm"].corr(s["bill_depth_mm"]),2))
overall r: -0.29 within Adelie : r = 0.28 within Chinstrap : r = 0.23 within Gentoo : r = 0.3
Answer: Overall the correlation is negative (≈ −0.29), but within every species it is positive (≈ +0.2 to +0.3). This is Simpson's paradox: a hidden grouping variable (species) reverses the apparent relationship. Gentoos have long, shallow bills and form a separate cluster, so pooling all species creates a misleading downward trend. The lesson is permanent: a correlation across mixed groups can lie, so always check whether a lurking category explains it (Chapters 16 and correlation-vs-causation).
d["body_size"] = d["body_size"].astype("string").str.strip().str.lower().replace({"med":"medium"})
d["band_color"] = d["band_color"].astype("string").str.strip().str.lower()
d["body_size_code"] = d["body_size"].map({"small":1,"medium":2,"large":3}) # ORDINAL
band_oh = pd.get_dummies(d["band_color"], prefix="band") # NOMINAL
print("body_size_code values:", sorted(d["body_size_code"].dropna().unique()))
print("one-hot band columns:", list(band_oh.columns))
print("\nmean body mass by band_color:")
print(d.groupby("band_color")["body_mass_g"].mean().round(0).to_string())
body_size_code values: [np.int64(1), np.int64(2), np.int64(3)] one-hot band columns: ['band_blue', 'band_green', 'band_red', 'band_yellow'] mean body mass by band_color: band_color blue 4224.0 green 4239.0 red 4206.0 yellow 4213.0
Answer: body_size has a real order (small < medium < large), so it becomes integer codes 1/2/3. band_color has no order, so it becomes one-hot 1/0 columns, label-encoding it (red=1, blue=2…) would invent a ranking that does not exist (Chapter 24). The no-signal check seals it: mean body mass is ~4,200 g for every color, so the research tag carries no information. Proving a variable is noise is a real finding, it tells you not to waste a model on it.