You have already met three regressions in this Part: a straight line for numbers, an S-curve for probabilities, and now a log-scale model for counts. The generalized linear model shows they are the same machine with two dials swapped, a choice of distribution and a choice of link.
Step 1 gains one decision, choose the family and link to match the outcome (number, probability, count). Step 4 gains one check for counts, overdispersion. Everything else, fit by maximum likelihood, evaluate, interpret, carries straight over.
The GLM Umbrella
The genius of the GLM is separating what you are predicting from how the predictors combine. The predictors always combine linearly into η = Xβ. The link function then bends η onto the right scale for the outcome, and the random component says how the outcome scatters around its mean.
This is why everything in Regression Analysis is really one idea. Swapping the family and link is all it takes to move between
a house price, a default probability, and a claim count, and the same statsmodels call,
glm(formula, family=…), fits them all. The most common families:
| Outcome | Family | Link | Model |
|---|---|---|---|
| Continuous number | Gaussian (Normal) | identity | linear regression (see Simple Linear Regression) |
| Yes / no | Binomial | logit | logistic regression (see Logistic Regression) |
| Count / rate | Poisson | log | Poisson regression (this chapter) |
| Overdispersed count | Negative Binomial | log | the overdispersion fix (Section 3) |
Poisson Regression for Counts
When the outcome is a count, how many claims, visits, defects, linear regression is wrong: it can predict negative counts and assumes constant variance. Poisson regression models the log of the expected count, so predictions stay positive and multiply rather than add.
Just as logistic coefficients exponentiate to odds ratios, Poisson coefficients exponentiate to rate ratios. Two practical notes. First, when observations are exposed for different lengths of time (or area, or population), add an offset, log(exposure), so you model a rate rather than a raw count. Second, the model is still fit by maximum likelihood and reads out like any other GLM. In the claims data, each extra 10 years of driver age carries a rate ratio of about 0.79 (a 21% lower claim rate), and urban drivers file claims at about 2.4× the rural rate.
Overdispersion: the Count-Model Pitfall
Poisson regression makes one strong, easily-violated assumption: the variance of the count equals its mean. Real counts are usually overdispersed, more variable than that, because of unmeasured heterogeneity (some drivers are simply riskier). When you ignore it, the coefficients stay roughly right but the standard errors come out far too small, making everything look more significant than it is.
The tell is simple: bin the data and the variance sits well above the mean, and the Pearson chi-square divided by its degrees of freedom is much larger than 1 (here about 4.8, when 1 is ideal). The remedy is to swap the family for one that lets variance exceed the mean, the negative binomial (or a quasi-Poisson correction). It fits the excess zeros and the long tail that Poisson misses:
The negative binomial does not change the story about which factors matter, but it tells the truth about how sure we are. On the claims data it slashes the AIC from 2881 to 2068 and roughly doubles the standard errors, turning Poisson's over-confident intervals into honest ones. Checking for overdispersion is to Poisson regression what checking for heteroscedasticity is to linear regression: the diagnostic that keeps your p-values believable.
Real-World Example: Insurance Claim Counts
We model the number of claims per policy from driver age, car age, and region, with years insured as
the exposure offset, then run the workflow to its overdispersion fix.
One row per policy with driver_age, car_age,
region, exposure_years (the offset), and the claims count.
Step 1, check the data. The outcome is a count, mostly small with a long right tail. The claims average about 2.8 per policy but their variance is roughly 20, and about a third of policies have zero claims. That variance-far-above-the-mean is already a hint that a plain Poisson model (which assumes they are equal) may not fit.
Step 2, fit. We fit a Poisson GLM with a log(exposure_years) offset, so we model a claim
rate per year rather than a raw count:
glm("claims ~ driver_age + car_age + C(region)", family=Poisson(), offset=log(exposure)).
Step 3, evaluate. The rate ratios tell a clean story: younger drivers and urban policies file more claims (each +10 years of age carries a rate ratio of 0.79, and urban policies file at about 2.4× the rural rate).
Step 4, check the conditions. The key count-model condition is that the variance equals the mean. Here it is badly violated, the counts are overdispersed, so we refit as negative binomial:
| Check | Poisson | Negative Binomial |
|---|---|---|
| Variance vs mean | assumes equal; data ratio ≈ 7× | allows variance > mean ✓ |
| Pearson χ² / df | 4.8 ✗ overdispersed | ≈ 1 ✓ |
| AIC (lower is better) | 2881 | 2068 |
| Std. error (urban) | 0.089 (too small) | 0.166 (honest) |
Step 5, interpret and predict. The negative binomial keeps the same rate ratios but reports honest, roughly-doubled standard errors. To predict, the model gives an expected count through the log link:
For a 40-year-old urban driver with a 5-year-old car and one full year of exposure: exp(1.61 − 0.96 + 0.14 + 0.86) = exp(1.65) ≈ 5.2 claims per year. The fitted rate falls smoothly with driver age, tracking the observed rates by age band:
Same coefficients, very different certainty. The Poisson model would have reported the urban effect with an interval half as wide as it should be, overstating the evidence. The negative binomial keeps the rate ratios and fixes the error bars, which is exactly what a trustworthy count model must do.
GLMs in Machine Learning & AI
The GLM is the bridge between classical statistics and the loss functions of machine learning.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| Link function | The output activation of a network | identity (regression), sigmoid, softmax, exp |
| Random component | The likelihood, hence the loss | MSE (Normal), cross-entropy (Bernoulli), Poisson loss |
| Poisson / count models | Count & rate prediction | demand, clicks, event forecasting |
| Overdispersion | Modeling uncertainty honestly | negative-binomial heads, distributional forecasting |
| Exponential family | A unifying view of losses | the theory behind why these losses work |
Every regression and classification loss you use is a GLM in disguise. Mean-squared error is the Normal likelihood; binary cross-entropy is the Bernoulli likelihood with a logit link; the softmax output layer is the multinomial GLM; and a network predicting counts uses a Poisson or negative-binomial head. The link function is literally the last activation, and the choice of loss is a choice of random component. Seeing this, that the loss encodes an assumption about the noise, is what lets researchers design principled objectives for new problems instead of guessing. And the overdispersion lesson generalizes to one of the deepest issues in applied ML: a model can predict the right average while being dangerously overconfident about its spread, which is why distributional and uncertainty-aware models matter wherever the stakes are real.
Fit the whole GLM family in Python
The companion notebook fits linear, logistic, and Poisson models with the same statsmodels
glm() call to show they are one framework, adds an exposure offset, reads out
rate ratios, detects overdispersion (Pearson chi-square, variance-vs-mean),
and refits as negative binomial on generalized-linear-models--claims.xlsx.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, seaborn, statsmodels, and openpyxl and launch
jupyter notebook.
🎓 Key Takeaways
- ✓A GLM = random component + linear predictor + link function; choosing the family and link recovers linear, logistic, and Poisson regression from one recipe.
- ✓Poisson regression models log(count), so coefficients exponentiate to rate ratios; use a log(exposure) offset to model a rate.
- ✓Overdispersion (variance > mean, Pearson χ²/df ≫ 1) breaks Poisson's standard errors, making results look too significant.
- ✓Fix it with the negative binomial (or quasi-Poisson): same rate ratios, honest, wider standard errors, and a much lower AIC.
- ✓Real data: the claims counts had variance ≈ 7× the mean; switching to negative binomial cut AIC from 2881 to 2068 and doubled the standard errors.
Practice Challenges
Five short challenges with statsmodels GLMs.
Fit a Poisson GLM with an offset
Model claims on driver age, car age, and region with a log(exposure_years) offset.
smf.glm(..., family=sm.families.Poisson(), offset=np.log(df.exposure_years)).Read a rate ratio
Exponentiate the driver-age coefficient and state the rate ratio per 10 years in a sentence.
np.exp(coef * 10).Detect overdispersion
Compute Pearson chi-square / df and compare the variance and mean of the counts.
model.pearson_chi2 / model.df_resid should be near 1 if Poisson holds.Fix it with negative binomial
Refit as negative binomial; compare AIC and the standard errors with Poisson.
sm.families.NegativeBinomial(); lower AIC, wider SEs.Logistic is a GLM too
Fit a Binomial-family GLM with a logit link and confirm it matches smf.logit from the Logistic Regression chapter.
family=sm.families.Binomial() on a 0/1 outcome.A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on generalized linear models. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
You now see linear, logistic, and count regression as one framework. Econometrics & Panel Data closes the Part with the regression tools built for causal questions and repeated-measures data.