Fitting a model sounds like magic, but it is a single mechanical idea repeated a few thousand times: measure how wrong you are, figure out which way is downhill, and take a step. Do that until you reach the bottom. That is gradient descent, and it is how essentially every model in this book, and every neural network, actually learns.
Training = rolling to the bottom of a loss surface: step opposite the gradient, tune the learning rate (too small crawls, too large diverges), use mini-batches to scale, and let momentum / Adam get there faster.
The Loss Surface & Gradient Descent
Start with the loss function: a number that says how wrong the model's predictions are, for regression, the mean squared error. Plotted against the model's parameters, it forms a bowl-shaped surface. Training is nothing more than finding the parameters at the very bottom.
The gradient is the vector of slopes of the loss with respect to each parameter; it points uphill, so we step the opposite way. In the notebook we code this in a few lines and watch the parameters march down the bowl, arriving at exactly the closed-form least-squares answer from the Regression Analysis part (w = 54.9, b = 139.0, to the decimal). The point is not a new answer, it is a method that scales: for a model with billions of parameters we cannot solve for the minimum directly, but we can always follow the gradient downhill.
The Learning Rate
The learning rate sets the size of each step, and it is the single most important hyperparameter in all of machine learning. Get it wrong in either direction and training fails.
The notebook runs all three. A rate that is too small (0.02) inches toward the answer and wastes time; a good rate (0.3) descends fast and settles; a rate that is too large (1.05) overshoots the bottom every step so the loss oscillates and explodes to infinity (a divergence you can watch shoot up on a log scale, reaching 1016 in the challenges). There is no universal value, so it is the first thing to tune, and learning-rate schedules that start larger and shrink over time capture the best of both worlds.
Stochastic, Mini-Batch, and Better Optimizers
Plain gradient descent uses the entire dataset to compute each step, which is impossible when the data has millions of rows. The fix is to estimate the gradient from a sample.
| Variant | Data per step | Character |
|---|---|---|
| Batch GD | all rows | accurate but slow; smooth loss curve |
| Stochastic GD (SGD) | one row | noisy but very cheap; fast early progress |
| Mini-batch GD | a few dozen rows | the practical default; smooth enough and fast |
In the notebook, batch descent gives a clean curve, SGD's is jumpy but drops quickly, and mini-batch splits the difference, exactly how neural networks train on data far too large to fit in memory. On top of this sit smarter optimizers. Momentum accumulates a running average of past gradients (a ball gaining speed downhill) to power through narrow valleys; Adam adds a per-parameter adaptive step size, making it robust to badly-scaled features. Both reach the bottom in noticeably fewer steps than plain GD, which is why Adam is the default optimizer for deep learning.
Real-World Example: Fitting by SGD
You almost never hand-code gradient descent, the libraries run it for you. On a small advertising table,
scikit-learn's stochastic solver reproduces the models from earlier chapters, but with an engine that
scales to datasets that would never fit in memory.
One row per campaign with ad_spend and social_reach, the
regression target sales, and a binary label hit (a top-selling campaign) for the
classifier demo.
| Model | What it minimizes | Result |
|---|---|---|
| Gradient descent (from scratch) | mean squared error | w = 54.9, b = 139.0 (matches least squares exactly) |
SGDRegressor | squared error, by SGD | R² = 0.95 on sales |
SGDClassifier(loss="log_loss") | log loss (cross-entropy) | 97.6% accuracy on hit |
The loss argument is the whole story: squared error gives regression,
log loss gives logistic classification, hinge gives an SVM, all trained by the
identical descent loop with built-in learning-rate schedules. That single, uniform engine, pick a
loss, follow the gradient, is what makes it possible to scale from a 250-row spreadsheet to a model with
billions of parameters.
Optimization in Machine Learning & AI
Gradient descent is not one technique among many, it is the substrate the entire field runs on.
| Idea (this chapter) | In deep learning & AI it becomes |
|---|---|
| Loss function | cross-entropy for classifiers, next-token loss for language models |
| Computing the gradient | backpropagation, the chain rule applied through a deep network |
| Learning rate | the top hyperparameter; warmup and decay schedules |
| Mini-batch SGD | how every neural network is trained on huge data |
| Adam optimizer | the default for training vision models and LLMs |
Training a neural network, up to and including a large language model, is exactly this loop, at scale: feed a mini-batch, compute the loss, use backpropagation (the chain rule) to get the gradient with respect to every weight, and take an Adam step. The differences from this chapter are quantity, not kind: billions of parameters, a loss landscape that is non-convex (full of saddle points and many local minima rather than one clean bowl), and clever schedules and normalization to keep it stable. The counterintuitive gift is that the noise in SGD actually helps, it lets training jump out of poor minima. Understanding this one loop is understanding how modern AI is built.
Watch a model learn in Python
The companion notebook plots the loss bowl, codes gradient descent from scratch
with its descent path across the loss surface, compares three learning rates (including one
that diverges), races batch vs stochastic vs mini-batch, pits plain GD against
momentum and Adam, and finishes with SGDRegressor and SGDClassifier,
annotated line by line.
View opens the rendered notebook instantly. Open in Colab runs it live. To run
locally, install numpy, pandas, scikit-learn, matplotlib,
seaborn, and openpyxl.
🎓 Key Takeaways
- ✓Training = minimizing a loss: the loss surface is a bowl, and gradient descent slides to the bottom.
- ✓Step opposite the gradient, scaled by the learning rate; for a convex loss this reaches the exact optimum.
- ✓The learning rate is the key knob: too small crawls, too large diverges; schedules help.
- ✓SGD and mini-batches estimate the gradient from a sample so training scales to huge data; momentum and Adam converge faster.
- ✓The same loop trains everything: scikit-learn's SGD and every neural network (via backpropagation) are gradient descent, scaled up.
Practice Challenges
Five short challenges on the advertising table. Try them before checking the solutions.
Find the bottom of the bowl
Scan the slope of a standardized 1-feature model and report the loss-minimizing value.
mean((w*x - y)**2); it equals the correlation.Gradient descent from scratch
Implement GD for sales = w*x + b and confirm it matches LinearRegression.
gw = mean(2*(pred-y)*x), then w -= lr*gw.A too-large rate diverges
Show that a large learning rate makes the final loss blow up instead of settling.
Batch vs stochastic
Compare full-batch and single-row (stochastic) updates over a few epochs.
Let the library optimize
Fit an SGDRegressor for sales and an SGDClassifier for hit; report R² and accuracy.
SGDClassifier(loss="log_loss").A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on optimization and gradient descent. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
A model has learned by minimizing a loss, but low loss is not the same as a good model. the Model Evaluation chapter covers how to actually judge it, the confusion matrix, precision and recall, ROC-AUC, and the regression error metrics.