⚙️ Setup¶
import numpy as np
import pandas as pd
from scipy import stats
rng = np.random.default_rng(222)
print("Ready.")
Ready.
x = rng.lognormal(mean=2, sigma=0.9, size=3000)
print(f"raw skew : {stats.skew(x):+.2f}")
print(f"log skew : {stats.skew(np.log(x)):+.2f}")
raw skew : +3.83 log skew : -0.00
Answer: The raw data is strongly right-skewed (skew well above 1); after np.log the skew drops to near 0. Log is a step down the ladder of powers, which is the direction that pulls in a right tail. It requires strictly positive values because the log of zero is undefined and the log of a negative is not real, so for data with zeros use log1p or a shift, and for negatives use Yeo-Johnson.
rows = [
("(a) right-skewed income", "DOWN", "log (or sqrt)"),
("(b) left-skewed exam scores", "UP", "square (x^2)"),
("(c) Poisson counts", "DOWN", "square root"),
]
for case, dirn, t in rows: print(f"{case:30} | go {dirn:4} -> {t}")
(a) right-skewed income | go DOWN -> log (or sqrt) (b) left-skewed exam scores | go UP -> square (x^2) (c) Poisson counts | go DOWN -> square root
Answer: (a) down the ladder (log or sqrt) to pull in the right tail; (b) up the ladder (square) to pull in the left tail; (c) down to the square root, the variance-stabilizing transform for counts (Poisson variance is about equal to the mean). Right skew goes down, left skew goes up, that is the whole rule.
low = rng.poisson(5, 3000)
high = rng.poisson(100, 3000)
print(f"raw low : mean {low.mean():.1f}, var {low.var():.1f}")
print(f"raw high: mean {high.mean():.1f}, var {high.var():.1f}")
print(f"sqrt low : var {np.sqrt(low).var():.3f}")
print(f"sqrt high: var {np.sqrt(high).var():.3f}")
raw low : mean 5.0, var 5.1 raw high: mean 99.9, var 99.5 sqrt low : var 0.297 sqrt high: var 0.251
Answer: Raw, the high-mean group has roughly 20x the variance of the low group, because for Poisson data the variance equals the mean. After the square root, both variances sit near 0.25, comparable regardless of the mean. That is why the square root is the classic variance-stabilizing transform for counts.
x = rng.lognormal(mean=1.5, sigma=0.7, size=3000)
bc, lam = stats.boxcox(x)
print(f"chosen lambda = {lam:.3f}")
print(f"skew: raw {stats.skew(x):+.2f} -> Box-Cox {stats.skew(bc):+.2f}")
chosen lambda = -0.001 skew: raw +3.82 -> Box-Cox -0.00
import matplotlib.pyplot as plt
fig, ax = plt.subplots(1, 2, figsize=(10, 3.4))
ax[0].hist(x, bins=40, color="#e11d48", alpha=0.75); ax[0].set_title(f"Raw (skew {stats.skew(x):+.2f})")
ax[1].hist(bc, bins=40, color="#059669", alpha=0.75); ax[1].set_title(f"Box-Cox (skew {stats.skew(bc):+.2f})")
plt.tight_layout(); plt.show()
Answer: Box-Cox returns the maximum-likelihood lambda that best normalizes the data (here near 0, so essentially a log). The lambda maps onto the ladder: lambda≈1 means no transform needed, lambda≈0.5 is square-root-like, lambda≈0 is a log, and lambda≈−1 is a reciprocal. Box-Cox needs positive data; for zeros or negatives use Yeo-Johnson (sklearn PowerTransformer).
x = rng.lognormal(mean=3, sigma=1.0, size=5000)
arith = x.mean()
geo = np.exp(np.log(x).mean())
print(f"arithmetic mean : {arith:8.1f}")
print(f"exp(mean of logs) : {geo:8.1f} (geometric mean)")
print(f"median : {np.median(x):8.1f}")
arithmetic mean : 32.8 exp(mean of logs) : 19.8 (geometric mean) median : 19.3
Answer: They are not equal: exp(mean(log x)) is the geometric mean, which sits below the arithmetic mean (Jensen's inequality, because log is concave). Back-transforming a mean-of-logs systematically underestimates the arithmetic mean. For a right-skewed quantity, report the median (monotone transforms preserve it), or apply a bias correction such as Duan's smearing estimator. The headline lesson: results live on the transformed scale until you back-transform, and the mean does not survive the round trip cleanly.