Does joining a union raise your wage? Run a plain regression and you will get a big, confident number, and it will be wrong, because the kind of worker who unionizes differs in ways you never measured. Econometrics is regression that takes that problem seriously.
Every chapter checked linearity, variance, and normality. Econometrics confronts the condition they assumed away, exogeneity (predictors uncorrelated with the error). When it fails, no amount of clean residuals saves you; you need a better design.
Panel Data and Fixed Effects
Panel data follow the same units, people, firms, countries, over multiple periods. That repetition is a gift: it lets you compare each unit to itself over time, canceling out everything about it that does not change. Fixed effects do exactly this, adding a dummy for each unit so the model uses only the within-unit variation.
The picture is the whole idea. Within every worker the effect is positive (each colored line slopes up), but the pooled regression that ignores who is who traces a misleading line through the clusters. By absorbing a per-worker intercept, fixed effects strip out time-invariant confounders, unmeasured ability, motivation, family background, without ever measuring them. The cost is that fixed effects cannot estimate the effect of anything that does not change over time. When you are willing to assume the unit effects are unrelated to the predictors, random effects are more efficient; a Hausman test chooses between them, but when in doubt, fixed effects are the safer, more credible default.
Endogeneity and Instrumental Variables
Fixed effects only remove confounders that stay constant over time. For endogeneity from a time-varying confounder, reverse causation, or measurement error, the classic tool is an instrumental variable: a variable Z that moves the troublesome predictor X but affects the outcome Y only through X.
A valid instrument needs two things: relevance (it really shifts X) and the exclusion restriction (it touches Y through no other path, in particular it is unrelated to the confounder U). Given both, two-stage least squares uses only the part of X that Z explains, the part scrubbed clean of the confounder, to estimate the causal effect. The catch is that good instruments are rare and the exclusion restriction cannot be proven, only argued, which is why an instrument is only as convincing as the story behind it. A related design, difference-in-differences, is next.
Natural Experiments: Difference-in-Differences
When a policy hits one group and not another, a natural experiment appears. Difference-in-differences (DID) compares the change in the treated group with the change in an untreated control group, so any trend common to both cancels out.
The engine is the parallel-trends assumption: absent the intervention, the two groups would have moved in lockstep. The control group's change estimates what would have happened to the treated group (the dashed counterfactual), and the gap that opens above it is the causal effect. DID is a workhorse of modern policy evaluation precisely because it needs no instrument, only a credible comparison group and a parallel past. A final econometric staple ties back to the Transformations chapter: log transformations. Regressing log(y) on log(x) makes the coefficient an elasticity (a percent-per-percent effect), the natural language of economics.
Real-World Example: the Union Wage Premium
We estimate how much union membership raises wages, using a panel of workers observed over four years. The dataset secretly includes each worker's latent ability, so we can prove that fixed effects do their job.
A panel: worker_id, year, tenure,
union (time-varying), log_wage, and the normally-hidden ability_index
confounder, included so you can check the method.
Step 1, check the data and the threat. This is a panel: 160 workers observed over four years. The true union premium built into the data is 0.15 in log wages (about a 16% raise). The catch: union status is correlated with unobserved ability (r ≈ 0.37), so the exogeneity condition fails, and a plain regression that cannot see ability will be biased.
Steps 2–3, fit naively and diagnose the bias. Watch the estimate come home as we control for the confounder, first directly, then with fixed effects that never measure it:
| Model | Union coefficient | vs truth (0.15) |
|---|---|---|
| Pooled OLS (ability omitted) | 0.54 (95% CI 0.46 to 0.63) | ✗ biased up 3.6× |
| OLS controlling for ability | 0.16 | ✓ correct (ability observed) |
| Fixed effects (within worker) | 0.15 (95% CI 0.12 to 0.18) | ✓ correct, without observing ability |
Step 4, fix the design. Pooled OLS is biased because ability is omitted. Fixed effects add a per-worker intercept, so the model uses only within-worker variation and differences out anything about the worker that does not change, ability included. It recovers the true 0.15, and matches what you get by controlling for ability directly, proof the method works.
Step 5, interpret. The union coefficient of 0.15 is on the log scale, so exponentiate it to read it as a wage multiplier:
Fixed effects put the union premium at about 16%; the pooled 0.54 would have claimed a wildly inflated 72%.
This is the chapter in one table. The pooled model is confidently wrong, its tight confidence interval does not include the truth, because its error is bias, not variance, and no sample size fixes bias. Fixed effects recover the true 15% premium using nothing but each worker compared to himself. When the design is right, the coefficient is trustworthy; when it is wrong, precision only makes the wrong answer look more convincing.
Econometrics in Machine Learning & AI
The fastest-growing corner of applied ML is causal, and it is built directly on these econometric foundations.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| Endogeneity / OVB | Confounding in observational ML | why a predictive model must not be read causally |
| Instrumental variables | Causal ML with instruments | deep IV, double/debiased machine learning |
| Fixed effects | Entity embeddings / within estimators | controlling for user or item in recommender models |
| Difference-in-differences | Policy & product experimentation | measuring a launch effect without a clean A/B test |
| Treatment effects | Heterogeneous / uplift modeling | causal forests, who does an intervention help? |
Machine learning is superb at prediction and, on its own, silent about causation, the exact gap the Correlation vs. Causation chapter opened and this chapter closes. The frontier field of causal machine learning grafts these econometric designs onto flexible models: double machine learning uses instruments and controls with ML nuisance estimators to get unbiased treatment effects; causal forests estimate who benefits from an intervention; entity embeddings are fixed effects learned by gradient descent. The lesson is durable: a model that predicts a metric perfectly can still be useless for deciding what to do, because acting changes the world in ways only a causal design can anticipate. Prediction tells you what will happen; econometrics tells you what happens if you intervene, and knowing which question you are answering is the whole game.
Correct the bias in Python
The companion notebook fits pooled OLS and watches the union premium balloon, adds worker
fixed effects to recover the truth, confirms it matches directly controlling for ability,
predicts the sign of the omitted-variable bias, and runs a two-stage least squares
instrumental-variable demo, all on econometrics-and-panel-data--panel.xlsx with statsmodels.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, seaborn, statsmodels, and openpyxl and launch
jupyter notebook.
🎓 Key Takeaways
- ✓Endogeneity (a predictor correlated with the error) biases OLS; it is the condition the earlier chapters assumed and this one confronts.
- ✓Fixed effects use panel data to compare each unit to itself, removing all time-invariant confounders without measuring them.
- ✓Instrumental variables (2SLS) need relevance and the exclusion restriction to recover a causal effect from a variable that shifts X but not Y directly.
- ✓Difference-in-differences reads a causal effect off a natural experiment via the parallel-trends assumption; log-log models give elasticities.
- ✓Real data: pooled OLS put the union premium at 0.54, but fixed effects (and controlling for ability) recovered the true 0.15, bias, not variance, was the enemy.
Practice Challenges
Five short challenges on panel and causal methods with statsmodels.
The biased pooled estimate
Fit pooled OLS of log_wage on union and tenure; note the (over-confident) CI.
ols("log_wage ~ union + tenure", panel).fit().Add fixed effects
Add C(worker_id) and show the union coefficient falls to the true value.
Reveal the confounder
Control for ability_index directly and confirm it matches the fixed-effects estimate.
Predict the bias direction
From the sign of corr(union, ability) and ability's effect on wage, predict whether pooled OLS is biased up or down.
Two-stage least squares
Simulate an endogenous X with a valid instrument Z; show OLS is biased and 2SLS recovers the true slope.
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on econometrics and panel data. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
From a single straight line to causal inference on panel data, you now command the whole regression toolkit, and the five-step workflow that keeps it honest. Next, Regression Case Study puts it all to work in an end-to-end regression case study on the Ames housing data, from cleaning to a diagnosed, interpreted model.