⚙️ Setup¶
import numpy as np
import pandas as pd
print("Ready.")
Ready.
df = pd.DataFrame({"item":["A","B","C"], "price":["1,200","350","2,000"], "qty":["10","4","7"]})
print(df.dtypes)
print("\nTrying price.mean():")
try:
print(df["price"].mean())
except Exception as e:
print(" ERROR:", type(e).__name__)
item str price str qty str dtype: object Trying price.mean(): ERROR: TypeError
Answer: Both price and qty are object (text), not numbers. price carries thousands commas ("1,200"), so a CSV reader keeps it as a string; qty may have been read as text for similar formatting reasons. Numbers stored as text silently break arithmetic and sorting. The fix (next chapters) is to strip the commas and convert with pd.to_numeric, but first you have to notice via dtypes.
temp = pd.Series([21.4, 22.0, -999, 20.8, -999, 23.1, 19.9])
print(f"isna() count : {temp.isna().sum()} (sees nothing!)")
print(f"hidden -999 count : {(temp==-999).sum()}")
print(f"naive mean : {temp.mean():.1f} <- corrupted")
print(f"mean ignoring -999 : {temp[temp!=-999].mean():.1f}")
isna() count : 0 (sees nothing!) hidden -999 count : 2 naive mean : -270.1 <- corrupted mean ignoring -999 : 21.4
Answer: isna() finds 0 missing because -999 is a valid float. Really there are 2 missing readings. The naive mean (about -273) is nonsense; excluding the sentinel gives a sensible ~21.4. Lesson: you must know the dataset's missing codes and search for them, blanks are not the only kind of missing.
country = pd.Series(["USA","usa","U.S.A.","United States","UK","uk","U.K.","USA"])
print(country.value_counts())
print(f"\ndistinct labels: {country.nunique()} but real countries: 2 (USA and UK)")
USA 2 usa 1 U.S.A. 1 United States 1 UK 1 uk 1 U.K. 1 Name: count, dtype: int64 distinct labels: 7 but real countries: 2 (USA and UK)
Answer: value_counts() shows several labels (USA, usa, U.S.A., United States, UK, uk, U.K.) that collapse to just 2 real categories. Inconsistent casing and punctuation fragment one country into many, which would split counts and break group-bys. Standardizing these is Chapter 19; the mindset here is to surface them first.
df = pd.DataFrame({
"id":[1,2,2,4,5],
"name":["Ann","Bo","Bo","Cy","Di"],
"age":[31, 27, 27, 250, 44],
})
print(f"exact duplicate rows: {df.duplicated().sum()}")
bad = df[(df["age"]<0)|(df["age"]>120)]
print(f"impossible ages : {list(bad['age'])}")
exact duplicate rows: 1 impossible ages : [250]
Answer: There is 1 exact duplicate row (the second "Bo, 27"), and one impossible age of 250. duplicated() flags repeated rows; a simple range rule flags values that cannot be real. Whether to drop the duplicate or investigate the 250 is a judgment call for later, the audit just makes them visible.
wide = pd.DataFrame({"store":["North","South"], "2021":[120,90], "2022":[135,110], "2023":[150,95]})
print("MESSY (years are column headers):"); print(wide)
tidy = wide.melt(id_vars="store", var_name="year", value_name="sales")
print("\nTIDY (each variable a column, each observation a row):"); print(tidy)
MESSY (years are column headers): store 2021 2022 2023 0 North 120 135 150 1 South 90 110 95 TIDY (each variable a column, each observation a row): store year sales 0 North 2021 120 1 South 2021 90 2 North 2022 135 3 South 2022 110 4 North 2023 150 5 South 2023 95
import matplotlib.pyplot as plt
tidy.pivot(index="year", columns="store", values="sales").plot(kind="bar", color=["#0891b2", "#7c3aed"])
plt.title("Tidy data plots directly: sales by store and year"); plt.ylabel("sales")
plt.xticks(rotation=0); plt.tight_layout(); plt.show()
Answer: The years 2021, 2022, 2023 are values, not variable names, so the wide table breaks tidy rule 1 ("each variable forms a column"). pd.melt pulls them into a single year column with a matching sales column, so every row is now one observation (a store-year). Tidy data is about shape, not typos, and it is what lets grouping, plotting, and modeling just work.