We close EDA Case Studies where many data-science courses begin, with the Palmer penguins. It is small, clean, and delightful, the perfect dataset to practice the full routine without drowning in mess, and it hides one of the most important cautionary tales in all of statistics.
penguins.csv records 347 penguins of three species (Adelie, Chinstrap, Gentoo) across three
Antarctic islands, with bill_length_mm, bill_depth_mm, flipper_length_mm,
body_mass_g, sex, plus two extra columns to encode: an ordinal body_size
and a nominal band_color (the color of each bird's research tag). A few duplicates, a messy
sex column, two empty rows, and one impossible measurement are all that stand between us and a
clean table.
First Contact: A Small, Almost-Tidy File
Because most of the file is clean, the audit is about finding the few problems precisely.
| What the audit found | Evidence | The fix (and its chapter) |
|---|---|---|
| Duplicate rows | 3 exact-copy rows | Deduplicate · Finding & Removing Duplicates and Inconsistencies |
| Empty rows | 2 rows missing every measurement | Drop the rows · Handling Missing Data |
| Messy + missing category | sex: male, MALE, Female , ., blanks, 9 gaps | Standardize then impute · Finding & Removing Duplicates and Inconsistencies & Handling Missing Data |
| Unit error | flipper_length_mm = 19.5 (cm, not mm) | Correct the value · Detecting & Treating Outliers |
| Messy categories to encode | body_size (ordinal) and band_color (nominal), mixed casing | Clean & encode by type · Finding & Removing Duplicates and Inconsistencies & Feature Engineering |
Visualize First: Three Species, and a Famous Trap
The single most instructive penguin chart plots bill length against bill depth, colored by species. It shows three clean clusters, and a correlation that reverses the moment you account for species.
Pooled across all penguins, bill length and bill depth are negatively correlated (r ≈ −0.27), the dashed line slopes down. But within every species the correlation is positive (Adelie +0.28, Chinstrap +0.26, Gentoo +0.30). The reversal happens because Gentoos (purple) have long, shallow bills and sit apart, dragging the overall line down. A correlation computed across mixed groups can point the opposite way to the truth. This is why we color by the grouping variable and always ask "compared to what?", a lesson that returns in force with correlation versus causation.
Cleaning: Duplicates, a Unit Error, and Empty Rows
Three structural fixes. Deduplicate (see Finding & Removing Duplicates and Inconsistencies) removes 3 exact copies (347 → 344).
Then the standout: one flipper_length_mm reads 19.5 when every other penguin is near
190. It is not an outlier to delete but a unit error to correct, centimeters logged as
millimeters, so multiplying the sub-50 value by 10 restores the true ~195 mm (see Detecting & Treating Outliers). Finally, the 2 rows missing
every measurement carry no information and are dropped (344 → 342, see Handling Missing Data).
Missing & Messy Categories
The sex column is both messy and incomplete, so it gets a two-step treatment. Standardize
first (see Finding & Removing Duplicates and Inconsistencies): trim whitespace, lowercase, and map junk tokens (".", blanks) to a true
NaN, leaving clean male / female and 9 genuine gaps. Then
impute (see Handling Missing Data): fill those gaps with the most common sex within each species, a smarter fill
than a single global mode, because the species differ. The body_size and band_color
columns get the same tidy-up (including med → medium) so they are ready to encode.
Order matters: if you impute before standardizing, the stray "." and "Female "
values count as real categories and corrupt the fill. Standardize to expose the true gaps, then
fill them, and fill them using structure you trust (here, the species) rather than a blunt global value.
Encode by Type, Then Explore
The cleaned categories are encoded by whether they have an order (see Feature Engineering). body_size is
ordinal, so it becomes 1 / 2 / 3; species, island, sex, and
band_color are nominal, so they become one-hot 1/0 columns.
A correlation heatmap, with the ordinal code included
Encoding body_size as 1/2/3 lets an ordered category join the numeric correlation matrix:
Two readings. The ordinal body_size code correlates 0.92 with body mass, confirming the encoding
faithfully captured size order, an ordered category now usable in numeric analysis. And bill_depth is
negative against mass and flipper (≈ −0.58): the same Gentoo effect behind Simpson's paradox, since the
biggest penguins have the shallowest bills. A quick check on the nominal tag closes the loop: mean body mass is
~4,200 g for every band_color, so the tag is pure noise. Confirming a variable
carries no signal is as useful as finding one that does, it stops you chasing a phantom.
The last visual check: the prep-work scorecard
Three datasets, one routine: audit, describe, visualize, clean, encode, and explore, with a reason logged at every step. Penguins added the final lessons, fix a value when you understand why it is wrong, impute using structure you trust, and never trust a pooled correlation without checking the groups. With the whole first half of the book now applied end to end, you are ready for what comes next: probability, the mathematics of the uncertainty that all of this analysis ultimately quantifies.
Run the whole case study in Python
The companion notebook walks the full routine: the first-contact audit, the bill-length-vs-depth scatter that
exposes Simpson's paradox, the deduplication and unit-error fix, dropping the empty rows, standardizing and
imputing sex, encoding body_size (ordinal) and the nominal columns (one-hot), a
correlation heatmap, and the band_color no-signal check, ending at the analysis-ready table.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas,
matplotlib and launch jupyter notebook.
🎓 Key Takeaways
- ✓Visualize first: the bill scatter exposed Simpson's paradox, negative overall, positive within each species.
- ✓Fix, do not delete: the 19.5 mm flipper was a unit error (cm→mm), corrected by ×10, not removed.
- ✓Clean before you impute: standardize
sex(".", casing → NaN), then fill gaps with the mode within species. - ✓Encode by type:
body_sizeordinal (1/2/3); species/island/sex/band_color one-hot (1/0). - ✓A negative result counts:
band_coloris confirmed noise (~4,200 g for every color).
Practice Challenges
Five challenges on the penguin data, following the case study beat by beat. Try them in Python before checking the solutions.
Fix the unit error
One flipper reads 19.5 while the rest are near 195. Show it is a unit error, correct it, and explain why you fix rather than delete it.
Drop only the empty rows
Two rows are missing every measurement. Drop only those (not rows missing just one value) and report the count before and after.
dropna(subset=meas, how="all").Clean and impute sex
Standardize the messy sex column, then impute the gaps with the mode within each species. Why
within species rather than overall?
groupby("species") then fill.Expose Simpson's paradox
Correlate bill length and bill depth overall, then within each species. Explain the reversal and the lesson.
Encode by type, then test for signal
Ordinal-encode body_size and one-hot band_color. Then test whether band_color
predicts body mass. What do you conclude?
A fully-worked solutions notebook answers all five on the real data, in the same visual style. Try them yourself first, then compare.
Quiz: Test Yourself
Eight questions on the penguins case study. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.