Two blind spots sit inside the standard regression you have used all along. First, it assumes independent observations, but students share schools, patients share clinics, and repeated measurements share a person, so rows come in clusters. Second, it assumes your variables are observed, but ideas like motivation, satisfaction, or ability are latent, visible only through several imperfect indicators. This chapter's two model families fix exactly these gaps: mixed models for nested data and structural equation models for latent variables and the paths between them.
Both families are the linear model stretched to fit reality. Mixed models let the intercept (and slope) vary by group instead of forcing one line through everyone. SEM lets a variable be measured indirectly and lets effects flow through chains and mediators rather than a single outcome. Master the linear model and both are natural next steps, not new worlds.
When Rows Cluster and Constructs Hide
Picture 1,000 students spread across 40 schools. Two students in the same school are more alike than two students picked at random, they share teachers, a building, a neighborhood. Ordinary regression, which treats all 1,000 as independent, gets the structure of the data wrong, which distorts its standard errors and hides the fact that much of the variation lives between schools, not between students.
Now picture a survey trying to measure “study motivation.” There is no motivation column; there are six questionnaire items, each a noisy echo of the same underlying trait. Averaging them blindly throws away information about how well each item measures the construct. A latent variable model treats motivation as a real but unobserved quantity that causes the answers, and estimates it from their shared variance.
- ●Nested / clustered data (students in schools, visits per patient, employees in firms) needs mixed models, also called multilevel or hierarchical models.
- ●Latent constructs and mediated effects (motivation, trust, ability, and how they influence outcomes through chains) need structural equation models.
Structural Equation Modeling: Latent Variables and Paths
SEM has two halves. The measurement model says how a latent variable connects to its observed indicators; the structural model says how latent and observed variables cause one another. Drawn as a diagram, ovals are latent, rectangles are observed, and arrows are directed effects.
Reading the picture: Study Motivation is a latent oval measured by three survey items, its arrows carry loadings (how strongly each item reflects the trait, here about 0.78, 0.70, 0.70). Separately, a structural path runs from motivation to the observed test score, its coefficient is the estimated effect of the construct on the outcome. SEM fits the whole system at once, so it can express path analysis (chains of direct and indirect effects, including mediation) and correct estimates for the fact that the predictor was measured imperfectly. You judge the fit not by a single R squared but by fit indices (CFI, RMSEA, and the chi-square test) that compare the model's implied covariance structure to the observed one.
Mixed Models: Random Intercepts and Slopes
A mixed model splits an effect into a part shared by everyone (fixed) and a part that varies by group (random). The simplest version, a random intercept, lets each school sit at its own baseline while sharing one slope: parallel lines at different heights.
Three things make this worth the trouble. First, the model reports how much variance lives between groups versus within them, summarized by the intraclass correlation (ICC); a large ICC means ignoring the grouping is a real mistake. Second, it does partial pooling: a small school's estimate is shrunk toward the overall average, borrowing strength from the others, which beats both a single pooled line and 40 separate noisy fits. Third, a random slope can let the effect itself (not just the baseline) vary by group, so the hours-to-score relationship can be steeper in some schools than others.
Real-World Example: Students Nested in Schools
The companion notebook fits a multilevel model to student test scores and shows what ordinary regression misses, all with statsmodels.
1,016 students across 40 schools (about 25 per school). Three columns:
school (the grouping / level-2 unit), hours_studied (a student-level predictor), and
test_score (the outcome, 0 to 100). A textbook two-level structure.
- ●Clustering is large: the random-intercept model gives an ICC of 0.30, so 30 percent of the leftover variance is between schools (between-school SD about 5.4, within-student SD about 8.2 points).
- ●The effect: each extra hour of study is worth about +2.31 points (the fixed slope), holding the school baseline aside.
- ●Why it matters: plain OLS treats all 1,016 students as independent and reports a similar slope (about 2.37) but is blind to the 30 percent of variance the schools carry, so its uncertainty and any school-level story are wrong.
The one-line fix captures it: mixedlm("test_score ~ hours_studied", df, groups="school") adds a random
intercept per school and immediately reports the variance split. Add re_formula="~hours_studied" and each
school also gets its own slope, though here the slopes barely vary, a useful negative result: the payoff of studying is
roughly the same everywhere, but the starting point is not.
SEM & Mixed Models in Machine Learning & AI
Latent variables and grouped data are everywhere in machine learning; these classical models are the interpretable ancestors of ideas that reappear, scaled up, across modern AI.
| Where it shows up | What it does | Examples |
|---|---|---|
| Hierarchical Bayesian models | Partial pooling and varying effects, the Bayesian face of mixed models | Multilevel regression with poststratification, varying-slope models (see the Bayesian Inference chapter) |
| Latent-variable representation learning | Learn hidden factors that generate the data, SEM's measurement model at scale | Factor analysis, PCA, autoencoders, variational autoencoders (VAEs) |
| Topic & embedding models | Documents or users as mixtures of latent themes or traits | LDA topic models, matrix factorization, embeddings |
| Mixed models meet trees | Random effects bolted onto gradient boosting and forests for clustered data | GPBoost, MERF, mixed-effects random forests |
| Structural causal models | SEM's directed paths formalized for causal inference | Causal DAGs and do-calculus (see the Causal Inference chapter) |
| Psychometrics & IRT | Latent ability estimated from item responses, a measurement model | Item response theory, adaptive testing, ability scoring |
The through-line from these models to modern AI is the latent variable: the idea that observed data is generated by hidden factors is the beating heart of representation learning, from VAEs to the embedding spaces inside large language models. Active work fuses the traditions, deep latent variable models that keep SEM's interpretability, hierarchical Bayesian deep learning that scales partial pooling to millions of groups, and neural structural causal models that learn the paths themselves. The classical versions in this chapter are the readable blueprint for all of it.
Fit these models in Python
The companion notebook fits a random-intercept and a random-slope model to the school data with statsmodels, reads off the fixed effect and the ICC, visualizes the per-school intercepts and partial-pooling shrinkage, and contrasts it all with a naive OLS fit; it then builds a small measurement model, extracting one latent factor from six survey items with scikit-learn and reading its loadings, every step with a plot.
View opens the rendered notebook instantly.
Open in Colab runs it live. To run locally, install numpy, pandas,
matplotlib, statsmodels, scikit-learn, and openpyxl (for full SEM,
the semopy library adds latent-path fitting and fit indices).
🎓 Key Takeaways
- ✓Two broken assumptions: ordinary regression assumes independent rows and directly observed variables; mixed models and SEM relax them.
- ✓Mixed models add random effects (group intercepts and slopes) to fixed effects, the right tool for nested or clustered data.
- ✓The ICC is the share of variance between groups; a large one (here 0.30) means ignoring the grouping is a real error.
- ✓Partial pooling shrinks small-group estimates toward the average, beating both one pooled line and many separate fits.
- ✓SEM pairs a measurement model (indicators load on a latent variable) with a structural model (paths, including mediation).
- ✓Latent variables are the bridge to modern AI: representation learning, embeddings, and VAEs all learn hidden factors.
Practice Challenges
Five exercises on the school data (and one latent-variable demo). Full solutions are in the companion solutions notebook.
Pooled vs grouped
Fit a plain OLS of score on hours, then a random-intercept mixed model. Compare the slope and what each says about schools.
smf.ols vs smf.mixedlm(..., groups="school").Compute the ICC
From the mixed model, extract the between-school and residual variances and compute the ICC. Interpret it.
See the shrinkage
Plot each school's raw mean score against its mixed-model (shrunk) intercept. Which schools move most?
Add a random slope
Refit with re_formula="~hours_studied". Does the hours effect vary meaningfully across schools?
cov_re.A latent factor
Simulate six correlated survey items, extract one latent factor with FactorAnalysis, and read the loadings. Which items measure the trait best?
FactorAnalysis(n_components=1).fit(...).components_.Solutions notebook
All five challenges worked in code, pooled versus grouped, the ICC, partial-pooling shrinkage, a random slope, and a latent factor from survey items, each with a short explanation.
Quiz: Test Yourself
Eight questions on random effects, the ICC, partial pooling, latent variables, and paths. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved.
Latent variables pointed the way from classical models to learned representations. Next, we step fully into that world. Chapter 146 · Deep Learning Primer builds neural networks from the ground up, layers, activations, backpropagation, and the training loop behind modern AI.