←Back to the bookChapter 110 · Take the quiz & aim for 100%

Chapter 110 · Solutions

Optimization & Gradient Descent · Solutions

Five challenges, each verified in code.

Chapter 110 · Practice Challenges, Solved

Five exercises on optimization, worked on the Chapter 110 advertising table.

import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns   # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
   "axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
   "axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"

import warnings; warnings.filterwarnings('ignore')
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression, SGDRegressor, SGDClassifier
from sklearn.metrics import r2_score, accuracy_score
try: df = pd.read_excel('../../data/optimization-and-gradient-descent--ads.xlsx', sheet_name='Data')
except FileNotFoundError: df = pd.read_excel(BASE + 'optimization-and-gradient-descent--ads.xlsx', sheet_name='Data')
x = StandardScaler().fit_transform(df[['ad_spend']]).ravel(); y = df['sales'].values
print('loaded', df.shape)

loaded (250, 5)

CHALLENGE 1

Find the bottom of the bowl

Scan the slope of a standardized 1-feature model and report the loss-minimizing value.

ys = StandardScaler().fit_transform(df[['sales']]).ravel()
ws = np.linspace(-1,2,300); best = ws[np.argmin([np.mean((w*x-ys)**2) for w in ws])]
print(f'best slope = {best:.3f}  (equals the correlation for standardized data)')

best slope = 0.957  (equals the correlation for standardized data)

Solution. The MSE is a convex bowl; its minimum for standardized data sits at the correlation between the variables.

CHALLENGE 2

Gradient descent from scratch

Implement gradient descent for sales = w*x + b and confirm it matches LinearRegression.

w=b=0.0; lr=0.3
for _ in range(80):
    p=w*x+b; w-=lr*np.mean(2*(p-y)*x); b-=lr*np.mean(2*(p-y))
lin=LinearRegression().fit(x.reshape(-1,1),y)
print(f'GD: w={w:.2f} b={b:.2f}   |   closed form: w={lin.coef_[0]:.2f} b={lin.intercept_:.2f}')

GD: w=54.90 b=138.99   |   closed form: w=54.90 b=138.99

Solution. Stepping opposite the gradient converges to the exact least-squares solution.

CHALLENGE 3

A too-large learning rate diverges

Show that a large step size makes the loss increase instead of decrease.

def final_loss(lr):
    w=b=0.0
    for _ in range(40): p=w*x+b; w-=lr*np.mean(2*(p-y)*x); b-=lr*np.mean(2*(p-y))
    return np.mean((w*x+b-y)**2)
print(f'lr=0.3  final MSE = {final_loss(0.3):.1f}   (converges)')
print(f'lr=1.2  final MSE = {final_loss(1.2):.1e}   (diverges to huge values)')

lr=0.3  final MSE = 260.2   (converges)
lr=1.2  final MSE = 1.1e+16   (diverges to huge values)

Solution. Above a stability threshold each step overshoots, so the loss blows up rather than settling.

CHALLENGE 4

Batch vs stochastic

Compare full-batch and single-row (stochastic) updates over a few epochs.

X=StandardScaler().fit_transform(df[['ad_spend','social_reach']]); yv=df['sales'].values; m=len(yv)
def train(batch, lr=0.05, epochs=25, seed=0):
    rng=np.random.default_rng(seed); w=np.zeros(2); b=0.0
    for _ in range(epochs):
        for s in range(0,m,batch):
            bi=rng.permutation(m)[s:s+batch]; e=X[bi]@w+b-yv[bi]; w-=lr*2*X[bi].T@e/len(bi); b-=lr*2*e.mean()
    return np.mean((X@w+b-yv)**2)
print(f'batch final MSE = {train(m):.1f}   |   stochastic final MSE = {train(1):.1f}')

batch final MSE = 270.3   |   stochastic final MSE = 196.0

Solution. Both reach a low loss; stochastic updates are noisier per step but far cheaper on large data.

CHALLENGE 5

Let the library optimize

Fit an SGDRegressor for sales and an SGDClassifier for hit; report R-squared and accuracy.

X=StandardScaler().fit_transform(df[['ad_spend','social_reach']])
r=SGDRegressor(max_iter=2000, random_state=0).fit(X, df['sales']); print(f'SGDRegressor R2 = {r2_score(df.sales, r.predict(X)):.3f}')
c=SGDClassifier(loss='log_loss', max_iter=2000, random_state=0).fit(X, df['hit']); print(f'SGDClassifier accuracy = {accuracy_score(df.hit, c.predict(X)):.3f}')

SGDRegressor R2 = 0.952
SGDClassifier accuracy = 0.976

Solution. scikit-learn runs the same gradient-descent loop with tuned schedules; the loss argument switches between regression and classification.

---

Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher

←Back to the bookChapter 110 · Take the quiz & aim for 100%