←Back to the bookChapter 26 · Review & Case-Study Roadmap

Chapter 26 · Notebook

Meet the Three Datasets 🧭

Before we rescue them one by one, let us run the same first-contact audit on all three case-study datasets. This is the habit every case study starts with: look before you leap.

Statistics, Data Science and AI: A Visual Handbook · John Fisher · 2026

⚙️ Setup

Each dataset loads from the book's data/ folder. The cell tries a local path first (for the rendered book) and falls back to the GitHub raw URL (for Colab).

import pandas as pd, numpy as np
pd.set_option("display.max_columns", 40); pd.set_option("display.width", 200)

BASE_URL = "https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
def load(name):
    try:
        return pd.read_csv(f"../../data/{name}.csv")     # rendered book
    except FileNotFoundError:
        return pd.read_csv(BASE_URL + f"{name}.csv")      # Colab

spotify  = load("spotify_tracks")
ames     = load("ames_housing")
penguins = load("penguins")
print("loaded:",
      "spotify", spotify.shape,
      "| ames", ames.shape,
      "| penguins", penguins.shape)

loaded: spotify (915, 15) | ames (610, 16) | penguins (347, 10)

1 · A reusable first-contact audit

The same five questions open every exploration: how big is it, what are the dtypes, how much is missing, how many rows are duplicated, and what do a few rows actually look like? We wrap that into one function so we can point it at all three.

def first_contact(df, name):
    print(f"=== {name}  =  {df.shape[0]} rows x {df.shape[1]} columns ===")
    miss = df.isna().sum()
    miss = miss[miss > 0].sort_values(ascending=False)
    print("duplicate rows :", int(df.duplicated().sum()))
    print("missing values :")
    if len(miss):
        for c, n in miss.items():
            print(f"    {c:<16} {n:>4}  ({n/len(df)*100:4.1f}%)")
    else:
        print("    none")
    print("dtypes         :", dict(df.dtypes.astype(str).value_counts()))
    print()

for df, nm in [(spotify,"SPOTIFY"), (ames,"AMES"), (penguins,"PENGUINS")]:
    first_contact(df, nm)

=== SPOTIFY  =  915 rows x 15 columns ===
duplicate rows : 8
missing values :
    popularity         73  ( 8.0%)
    tempo              18  ( 2.0%)
    key                12  ( 1.3%)
dtypes         : {'str': np.int64(7), 'float64': np.int64(7), 'int64': np.int64(1)}

=== AMES  =  610 rows x 16 columns ===
duplicate rows : 10
missing values :
    pool_qc           604  (99.0%)
    fireplace_qual    290  (47.5%)
    lot_frontage       98  (16.1%)
    garage_yr_built    29  ( 4.8%)
    sale_price          3  ( 0.5%)
dtypes         : {'str': np.int64(7), 'int64': np.int64(6), 'float64': np.int64(3)}

=== PENGUINS  =  347 rows x 10 columns ===
duplicate rows : 3
missing values :
    sex                 7  ( 2.0%)
    bill_length_mm      2  ( 0.6%)
    bill_depth_mm       2  ( 0.6%)
    flipper_length_mm    2  ( 0.6%)
    body_mass_g         2  ( 0.6%)
    body_size           2  ( 0.6%)
dtypes         : {'str': np.int64(5), 'float64': np.int64(4), 'int64': np.int64(1)}

Already the audit is talking. Spotify has duplicate rows and missing tempo/popularity. Ames has a column that is almost entirely missing (pool_qc) plus mixed-format dates hiding inside an object column. Penguins has a handful of missing rows. None of this is visible from a row count alone, which is exactly why we audit first.

2 · Spotify: a peek at the mess

Look at the first rows and the spread of the numeric features. Notice how the columns live on completely different scales, that is a scaling problem waiting for Chapter 27.

display(spotify.head(4))
print("\nfeature ranges (note the wildly different scales):")
num = ["danceability","energy","loudness","tempo","valence","duration_ms","popularity"]
print(spotify[num].describe().loc[["min","max"]].round(2).to_string())
print("\ngenre labels as stored (same genre, many spellings):")
print(sorted(spotify["genre"].dropna().unique()))

	track_id	track_name	artist	genre	danceability	energy	loudness	tempo	valence	duration_ms	key	mode	popularity	audio_quality	mood
0	TRK00063	Silent Summer	Foxglove	pop	0.890	0.707	-21.31	143.448	0.436	215091	0.0	Major	10.0	standard	happy
1	TRK00677	Velvet Gravity	The Paper Kites	rock	0.532	0.808	-16.58	117.839	0.274	280893	4.0	Major	20.0	low	sad
2	TRK00140	Broken Waves	Northbound	r&b	0.722	0.368	-38.31	136.771	0.476	210682	7.0	Major	38.0	high	chill
3	TRK00053	Silent Machine	Ember Road	r&b	0.728	0.159	-40.00	99.454	0.291	169086	3.0	Minor	9.0	standard	sad

feature ranges (note the wildly different scales):
     danceability  energy  loudness  tempo  valence  duration_ms  popularity
min          0.08    0.02    -40.00    0.0     0.01      97419.0         0.0
max          0.95    0.98     -0.25  299.0     1.00   18000000.0        81.0

genre labels as stored (same genre, many spellings):
['EDM', 'Electronic', 'Hip-Hop', 'POP', 'Pop', 'R&B', 'classical', 'country', 'electronic', 'hip hop', 'hip-hop', 'hiphop', 'jazz', 'pop', 'pop ', 'r and b', 'r&b', 'rnb', 'rock']

3 · Ames: the 99%-missing column

The classic Ames lesson lives in the missingness pattern: some columns are missing a little (impute), one is missing almost everything (drop or recode).

miss_pct = (ames.isna().mean()*100).round(1)
print("missing % per column (worst first):")
print(miss_pct[miss_pct>0].sort_values(ascending=False).to_string())
print("\nsale_date is text in three different formats:")
print(ames["sale_date"].head(6).to_list())

missing % per column (worst first):
pool_qc            99.0
fireplace_qual     47.5
lot_frontage       16.1
garage_yr_built     4.8
sale_price          0.5

sale_date is text in three different formats:
['09/07/1955', '1940-03-19', '19-Dec-1950', '1945-02-25', '20-Sep-1902', '1967-03-05']

4 · Penguins: small, almost tidy

The friendliest of the three. A few rows are entirely missing, sex is inconsistent, and one flipper value is impossible, but most rows are clean.

print("species counts:\n", penguins["species"].value_counts().to_string())
print("\nsex column as stored (note blanks, \".\", and casing):")
print(penguins["sex"].value_counts(dropna=False).to_string())
print("\nflipper_length_mm range (one value is in cm by mistake):")
print("   min", penguins["flipper_length_mm"].min(), " max", penguins["flipper_length_mm"].max())

species counts:
 species
Adelie       154
Gentoo       124
Chinstrap     69

sex column as stored (note blanks, ".", and casing):
sex
male       174
female     155
NaN          7
.            4
MALE         4
Female       3

flipper_length_mm range (one value is in cm by mistake):
   min 19.5  max 237.0

---

🎯 What comes next

You have now done the single most important step of any analysis on all three datasets: you looked. Each of the next three chapters takes one of these from this raw, messy state through describing, visualizing, and cleaning, until it is ready for analysis, explaining every decision along the way.

Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher

←Back to the bookChapter 26 · Review & Case-Study Roadmap