⚙️ Setup¶
import numpy as np, pandas as pd, matplotlib.pyplot as plt
CYAN="#0891b2"; AMBER="#d97706"; GREEN="#059669"; PURPLE="#7c3aed"; INK="#1a2138"; GRID="#e6e9f2"
plt.rcParams.update({"figure.dpi":110,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,
"axes.spines.top":False,"axes.spines.right":False,"axes.titleweight":"bold"})
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
try: df = pd.read_csv("../../data/spotify_tracks.csv")
except FileNotFoundError: df = pd.read_csv(BASE+"spotify_tracks.csv")
print("loaded:", df.shape)
loaded: (915, 15)
print("duplicates:", int(df.duplicated().sum()))
d = df.drop_duplicates().copy()
gmap = {"Pop":"pop","POP":"pop","pop ":"pop","hip hop":"hip-hop","Hip-Hop":"hip-hop","hiphop":"hip-hop",
"R&B":"r&b","rnb":"r&b","r and b":"r&b","Electronic":"electronic","EDM":"electronic"}
before = d["genre"].nunique()
d["genre"] = d["genre"].str.strip().replace(gmap)
print(f"genre labels: {before} -> {d['genre'].nunique()}")
print(d["genre"].value_counts().to_string())
duplicates: 8 genre labels: 19 -> 8 genre pop 180 rock 164 hip-hop 163 electronic 123 country 83 r&b 80 jazz 67 classical 47
Answer: Eight rows were exact duplicates (they would double-count those songs), so drop_duplicates() removes them. Then trimming whitespace and mapping the spelling variants collapses nineteen genre labels into the eight real genres. Standardizing categories before any group-by is essential, otherwise "Pop", "pop ", and "POP" count as three different genres (Chapter 19).
d["tempo"] = d["tempo"].replace(0, np.nan)
n = int(d["tempo"].isna().sum())
med, mean = d["tempo"].median(), d["tempo"].mean()
d["tempo"] = d["tempo"].fillna(med)
print(f"missing tempo after recoding 0 -> NaN: {n}")
print(f"median {med:.1f} vs mean {mean:.1f} (imputed with the median)")
missing tempo after recoding 0 -> NaN: 41 median 117.6 vs mean 118.3 (imputed with the median)
Answer: Zeros that mean "not recorded" must become NaN first, otherwise they drag every statistic toward zero. After recoding, the gaps are filled with the median, which (unlike the mean) is not pulled around by extreme tempo values, so a few very fast or very slow songs cannot distort the fill value. This is the standard treatment for a numeric measurement gap (Chapter 20).
q1,q3 = d["duration_ms"].quantile([.25,.75]); iqr=q3-q1; fence=q3+1.5*iqr
flagged = int((d["duration_ms"]>fence).sum())
errors = d[d["duration_ms"]>600000] # > 10 minutes = corrupt
print(f"IQR fence: {fence/60000:.1f} min; flagged: {flagged}")
print("impossible (minutes):", (errors["duration_ms"]/60000).round(1).to_list())
d = d[d["duration_ms"]<=600000].copy()
print("rows after removing the 3 errors:", len(d))
IQR fence: 6.3 min; flagged: 22 impossible (minutes): [300.0, 25.0, 32.0] rows after removing the 3 errors: 904
Answer: The IQR rule flags a couple dozen tracks, but most are legitimately long (extended mixes, classical works). Only three are physically impossible, up to five hours, and those are the corrupt values worth removing. The lesson: an outlier rule is a detector, not a verdict. You inspect what it flags and decide; here we delete the three errors and keep the genuine long songs (Chapter 21).
num = ["danceability","energy","loudness","tempo","valence","duration_ms","popularity"]
corr = d[num].corr()
c = corr.where(~np.eye(len(num),dtype=bool)).abs()
i,j = np.unravel_index(np.nanargmax(c.values), c.shape)
print(f"strongest pair: {num[i]} & {num[j]} (r = {corr.iloc[i,j]:.2f})")
print("\npopularity vs audio features (all near zero):")
print(corr["popularity"].drop("popularity").round(2).to_string())
strongest pair: energy & loudness (r = 0.94) popularity vs audio features (all near zero): danceability -0.01 energy -0.01 loudness -0.01 tempo -0.06 valence 0.05 duration_ms -0.03
Answer: energy and loudness correlate at r ≈ 0.94, they are almost the same column. For a linear model that is multicollinearity: it inflates coefficient variance and makes the individual effects uninterpretable, so you would drop one or combine them. The second finding is just as useful: popularity barely correlates with any audio feature here, a strong hint that what makes a song popular in this data is not its sound (Chapters 24 & the EDA chapter).
from sklearn.preprocessing import StandardScaler
before = d["duration_ms"].skew()
d["log_duration"] = np.log(d["duration_ms"])
after = d["log_duration"].skew()
print(f"duration skew: {before:+.2f} -> log skew: {after:+.2f}")
feat = ["danceability","energy","loudness","tempo","valence","log_duration"]
z = pd.DataFrame(StandardScaler().fit_transform(d[feat]), columns=feat)
print("\nstandardized means (~0):", z.mean().round(2).to_dict())
print("standardized stds (~1):", z.std().round(2).to_dict())
duration skew: +0.92 -> log skew: +0.03
standardized means (~0): {'danceability': 0.0, 'energy': 0.0, 'loudness': 0.0, 'tempo': -0.0, 'valence': 0.0, 'log_duration': -0.0}
standardized stds (~1): {'danceability': 1.0, 'energy': 1.0, 'loudness': 1.0, 'tempo': 1.0, 'valence': 1.0, 'log_duration': 1.0}
Answer: The natural log pulls duration's right skew from about +0.9 to roughly 0, near-symmetric, which steadies any model that assumes well-behaved inputs (Chapter 22). Then standardizing rescales every feature to mean 0 and SD 1 so none dominates a distance or gradient simply because its raw numbers are bigger (loudness in the tens, duration in the hundreds of thousands). Transform first, then scale: the order matters (Chapter 24).