Statistics begins with an unknown, the true mean, the true rate, the true effect, and a sample of data. Estimation theory is the study of how to turn that data into a best guess of the unknown, and how to measure the quality of the guess. It is the formal foundation of everything that fits a model to data.
This chapter ties the advanced track together. The conditional expectation, moments, and inequalities of the last chapters all feed into one practical question, how to estimate parameters well, and the answer turns out to be the very definition of training a machine-learning model.
Estimators and Their Properties
An estimator is itself a random variable: feed it different samples and it returns different values. Two properties summarize its quality. Bias is how far its average sits from the truth; variance is how much it scatters. The classic picture is a dartboard.
The sample mean is unbiased: in the notebook its estimates center exactly on the true value 10, with a spread (standard error) of 0.47. The ideal estimator is the top-left dartboard, low bias and low variance, but as the next sections show, the two often trade off against each other.
Bias, Variance, and Mean Squared Error
The single number that combines both is the mean squared error, and it splits cleanly: MSE = bias² + variance. A great estimator can sometimes accept a little bias to win a large reduction in variance, lowering the total error.
Not every natural estimator is unbiased. Estimating a variance by dividing the squared deviations by n under-estimates it (by the factor (n−1)/n, since the data hugs the sample mean), which is exactly why the sample variance divides by n − 1, Bessel's correction. In the notebook the /n version averages to 3.2 against a true 4, while /(n−1) lands on 4.
Maximum Likelihood Estimation
How do you build a good estimator in the first place? The dominant recipe is maximum likelihood: choose the parameter that makes the observed data most probable. In practice you maximize the log-likelihood, and the peak is the estimate.
For exponential data the log-likelihood peaks at rate = 1/mean; in the notebook the grid search and the closed form agree at 0.52 (true 0.5). Maximum likelihood is the workhorse of statistics, and almost every estimator you already know, the sample mean, the sample proportion, least-squares coefficients, is an MLE in disguise.
How Good Can an Estimator Be?
Estimation theory also says when to stop improving. Two ideas set the standards. Consistency means the estimate converges to the truth as data accumulates; efficiency means it achieves the lowest possible variance.
| Property | Meaning | For the MLE |
|---|---|---|
| Unbiased | centers on the true value | often (exactly or asymptotically) |
| Consistent | converges to the truth as n → ∞ | yes, under mild conditions |
| Efficient | lowest possible variance | asymptotically efficient (hits the Cramer-Rao bound) |
In the notebook the MLE of a rate marches from 0.56 at n = 10 to 0.500 at n = 100,000, its scatter shrinking like 1/√n, the visible signature of consistency. There is even a hard floor on variance, the Cramer-Rao lower bound, set by the Fisher information in the data; no unbiased estimator can beat it, and for large samples the MLE reaches it. Estimation theory thus tells you both how to estimate and how well it is possible to do.
Estimation Theory in Machine Learning & AI
This is the chapter where mathematical statistics becomes machine learning. Training a model is parameter estimation, and the two pillars, maximum likelihood and the bias-variance trade-off, are the theory behind every fit.
| Idea (this chapter) | In ML / AI it becomes | Concrete example |
|---|---|---|
| Maximum likelihood | The training objective | cross-entropy = Bernoulli NLL; MSE = Gaussian NLL |
| Bias-variance trade-off | Underfitting vs overfitting | model capacity, the U-shaped test-error curve |
| A little bias for less variance | Regularization | ridge/lasso, weight decay, early stopping |
| MLE + a prior | MAP estimation | regularized loss = MLE with a prior penalty |
| Consistency & efficiency | Why more (clean) data helps | estimates tighten like 1/√n |
Training a model is estimation. Minimizing cross-entropy is maximizing a Bernoulli likelihood; minimizing squared error is maximizing a Gaussian likelihood, so "fit the parameters" literally means "find the maximum-likelihood estimate". The bias-variance trade-off is the master diagnostic of model capacity: in the notebook a degree-1 polynomial underfits (test error 0.48, high bias), a degree-5 fits well (0.02), and a degree-12 overfits catastrophically (test error 83, runaway variance). Regularization, ridge, lasso, weight decay, early stopping, is the deliberate injection of a little bias to slash variance, and adding a prior turns MLE into MAP estimation, which is exactly a regularized loss. Every lever you pull when training a model is an idea from this chapter.
Estimate and diagnose in Python
The companion notebook measures an estimator's bias and variance, demonstrates Bessel's n−1 correction, finds a maximum-likelihood estimate by climbing the log-likelihood, watches the MLE converge as n grows (consistency), and traces the bias-variance trade-off across polynomial degrees, the U-shaped test-error curve at the heart of overfitting.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy and matplotlib and launch
jupyter notebook.
🎓 Key Takeaways
- ✓An estimator is a rule for guessing a parameter; judge it by bias (off-center?) and variance (jumpy?).
- ✓MSE = bias² + variance; a little bias can buy a big drop in variance. The sample variance divides by n − 1 to stay unbiased.
- ✓Maximum likelihood picks the parameter that makes the data most probable; most familiar estimators are MLEs.
- ✓Good estimators are consistent (converge as n grows) and efficient (hit the Cramer-Rao variance floor).
- ✓In ML/AI: training is MLE (cross-entropy, MSE), the bias-variance trade-off is overfitting vs underfitting, and regularization is bias-for-variance.
Practice Challenges
Five short challenges, beginner to intermediate. Try them with NumPy before checking the solutions.
Unbiased?
Is the sample mean unbiased for the population mean? Simulate samples of size 20 from Normal(50, 10) and check.
Bessel's correction
For samples of size 4 from Normal(0, 3) (variance 9), compare the variance estimator with /n to the one with /(n−1).
np.var(ddof=0) vs ddof=1.MLE
Compute the maximum-likelihood estimate of an exponential rate from a sample (true rate 0.4).
Consistency
Show the estimate of a normal mean gets more accurate as n grows from 10 to 10,000.
MSE decomposition
For a biased estimator (the sample mean shrunk by 10%), compute bias, variance, and verify MSE = bias² + variance.
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on estimation theory. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
You have built the advanced machinery: joint and conditional densities, transformations, conditional expectation, moments and inequalities, and now estimation theory, the bridge to model fitting. Next, Sampling & Data Collection returns to practical ground: how to gather data worth analyzing in the first place.