Until now a random variable lived alone. But weather affects your commute, height tracks weight, one word predicts the next. To capture how two variables behave together, we need the joint distribution, and from it, everything else follows.
This Part is the advanced, mathematical-statistics track. It is optional for a first read, but it is where the machinery underneath estimation, regression, and modern generative AI is built. We start with the most fundamental object of multivariate probability: the joint distribution.
The Joint Distribution
A joint distribution assigns a probability to every pair of outcomes. For two discrete variables it is a table; for two continuous variables it is a surface, and probability becomes volume under that surface.
The joint table captures everything about how the two variables behave together: here P(rain and late) = 0.20. The four cells sum to 1, and from this single object we can recover every marginal and conditional in the rest of the chapter.
Marginal Distributions
To describe one variable on its own, ignore the other by summing it out (integrating, for densities). The result, written in the margin of the table, is the marginal distribution.
Summing each row gives the marginal for weather (30% rain, 70% clear); summing each column gives the marginal for commute (35% late, 65% on-time). The marginal is simply each variable considered alone, with the other averaged away.
Conditional Distributions and Independence
To find the distribution of one variable given a fixed value of the other, take that row of the joint and re-normalize it to sum to 1: f(y | x) = f(x, y) / f(x).
Given rain, the chance of being late leaps from the unconditional 35% to 67%. Because conditioning changed the distribution, the variables are dependent. The formal test for independence is whether the joint factors into the product of the marginals, f(x, y) = f(x) × f(y); here it does not.
Two variables are independent exactly when their joint distribution is the product of their marginals. This is the multivariate version of P(A and B) = P(A)P(B) from the Rules of Probability chapter, and it is the assumption that makes naive Bayes "naive" and that lets many models treat features as separable.
Covariance and the Bivariate Normal
For continuous variables, the most important joint distribution is the bivariate normal: two correlated Gaussians whose shape is set by a covariance matrix. The correlation ρ tilts the cloud.
The cloud tilts because X and Y are correlated. The pink line is the conditional mean E[Y | X = x], and for a bivariate normal it is exactly a straight line. That fact, the conditional mean of jointly-normal data is linear, is the theoretical seed of linear regression (the Regression Analysis part). Covariance measures how the two vary together; dividing by the standard deviations gives the unitless correlation ρ.
Joint Densities in Machine Learning & AI
The joint-versus-conditional distinction is one of the deepest divides in machine learning, separating whole families of models, and covariance matrices sit at the heart of many algorithms.
| Idea (this chapter) | In ML / AI it appears as | Concrete example |
|---|---|---|
| Conditional P(y | x) | Discriminative models | logistic regression; an LLM's P(next token | context) |
| Joint P(x, y) | Generative models | naive Bayes, VAEs, diffusion models sample the joint |
| Independence / factorization | Simplifying assumptions | the "naive" in naive Bayes |
| Covariance matrix | Structure of the data | PCA, Gaussian models, Mahalanobis distance |
| Linear conditional mean | Why linear regression works | E[Y | X] is a line for jointly-normal data |
The single most important taxonomy in machine learning falls straight out of this chapter. Discriminative models, logistic regression, most classifiers, and the language models that predict P(next token | context), learn only the conditional P(y | x). Generative models, naive Bayes, variational autoencoders, and diffusion models, learn the full joint P(x, y) and can therefore sample new data. The covariance matrix of a joint distribution is what PCA diagonalizes to find principal directions, what a Gaussian model stores, and what Mahalanobis distance uses to measure "unusual". And the fact that a bivariate normal's conditional mean is linear is the deep reason linear regression is so natural. Understand the joint distribution and you understand the shape of the entire modeling landscape.
Build joint distributions in Python
The companion notebook builds a joint PMF and visualizes it as a heatmap, recovers the marginals by summing, conditions to get one variable given another (and tests independence by factorization), simulates a bivariate normal to recover its covariance and correlation, and shows the joint-versus-conditional split that separates generative from discriminative models.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy and matplotlib and launch
jupyter notebook.
🎓 Key Takeaways
- ✓The joint distribution f(x, y) is the master object: the probability of every combination of values.
- ✓Marginals come from summing (integrating) out the other variable: each variable considered alone.
- ✓Conditionals f(y | x) = f(x, y)/f(x) re-normalize a slice of the joint; independence means the joint factors into the marginals.
- ✓The bivariate normal is set by a covariance matrix; its conditional mean E[Y | X] is a straight line, the seed of regression.
- ✓In ML/AI: discriminative models learn P(y | x), generative models learn the joint P(x, y); covariance powers PCA and Gaussian methods.
Practice Challenges
Five short challenges, beginner to intermediate. Try them with NumPy before checking the solutions. All use the 3×3 joint of X, Y ∈ {0, 1, 2} given in the solutions notebook.
Read the joint
Using a 3×3 joint PMF, find P(X=2, Y=2) and P(X=Y).
Marginals
Find the marginal distributions of X and Y from the joint.
Conditional
Find the conditional distribution P(Y | X=2).
Independent?
Decide whether X and Y are independent by comparing the joint to the product of the marginals.
Recover correlation
Simulate 5,000 draws from a bivariate normal with correlation 0.6 and recover it from the sample.
np.corrcoef on the simulated columns.A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on joint, marginal, and conditional distributions. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.