Chapter 114 · Solutions
Optimization & Gradient Descent · Solutions
Five challenges, each verified in code.
Chapter 114 · Practice Challenges, Solved¶
Five exercises on optimization, worked on the Chapter 114 advertising table.
In [1]:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
In [2]:
import warnings; warnings.filterwarnings('ignore')
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression, SGDRegressor, SGDClassifier
from sklearn.metrics import r2_score, accuracy_score
try: df = pd.read_excel('../../data/optimization-and-gradient-descent--ads.xlsx', sheet_name='Data')
except FileNotFoundError: df = pd.read_excel(BASE + 'optimization-and-gradient-descent--ads.xlsx', sheet_name='Data')
x = StandardScaler().fit_transform(df[['ad_spend']]).ravel(); y = df['sales'].values
print('loaded', df.shape)
loaded (250, 5)
CHALLENGE 1
Find the bottom of the bowl
Scan the slope of a standardized 1-feature model and report the loss-minimizing value.
In [3]:
ys = StandardScaler().fit_transform(df[['sales']]).ravel()
ws = np.linspace(-1,2,300); best = ws[np.argmin([np.mean((w*x-ys)**2) for w in ws])]
print(f'best slope on the grid = {best:.3f}, correlation = {np.corrcoef(x, ys)[0,1]:.3f} (they agree to the grid spacing of 0.01)')
best slope on the grid = 0.957, correlation = 0.959 (they agree to the grid spacing of 0.01)
Solution. The MSE is a convex bowl; its minimum for standardized data sits at the correlation between the variables.
CHALLENGE 2
Gradient descent from scratch
Implement gradient descent for sales = w*x + b and confirm it matches LinearRegression.
In [4]:
w=b=0.0; lr=0.3
for _ in range(80):
p=w*x+b; w-=lr*np.mean(2*(p-y)*x); b-=lr*np.mean(2*(p-y))
lin=LinearRegression().fit(x.reshape(-1,1),y)
print(f'GD: w={w:.2f} b={b:.2f} | closed form: w={lin.coef_[0]:.2f} b={lin.intercept_:.2f}')
GD: w=54.90 b=138.99 | closed form: w=54.90 b=138.99
Solution. Stepping opposite the gradient converges to the exact least-squares solution.
CHALLENGE 3
A too-large learning rate diverges
Show that a large step size makes the loss increase instead of decrease.
In [5]:
def final_loss(lr):
w=b=0.0
for _ in range(40): p=w*x+b; w-=lr*np.mean(2*(p-y)*x); b-=lr*np.mean(2*(p-y))
return np.mean((w*x+b-y)**2)
print(f'lr=0.3 final MSE = {final_loss(0.3):.1f} (converges)')
print(f'lr=1.2 final MSE = {final_loss(1.2):.1e} (diverges to huge values)')
lr=0.3 final MSE = 260.2 (converges) lr=1.2 final MSE = 1.1e+16 (diverges to huge values)
Solution. Above a stability threshold each step overshoots, so the loss blows up rather than settling.
CHALLENGE 4
Batch vs stochastic
Compare full-batch and single-row (stochastic) updates over a few epochs.
In [6]:
X=StandardScaler().fit_transform(df[['ad_spend','social_reach']]); yv=df['sales'].values; m=len(yv)
def train(batch, lr=0.05, epochs=25, seed=0):
rng=np.random.default_rng(seed); w=np.zeros(2); b=0.0
for _ in range(epochs):
for s in range(0,m,batch):
bi=rng.permutation(m)[s:s+batch]; e=X[bi]@w+b-yv[bi]; w-=lr*2*X[bi].T@e/len(bi); b-=lr*2*e.mean()
return np.mean((X@w+b-yv)**2)
print(f'batch final MSE = {train(m):.1f} | stochastic final MSE = {train(1):.1f}')
batch final MSE = 270.3 | stochastic final MSE = 196.0
Solution. Both reach a low loss; stochastic updates are noisier per step but far cheaper on large data.
CHALLENGE 5
Let the library optimize
Fit an SGDRegressor for sales and an SGDClassifier for hit; report R-squared and accuracy.
In [7]:
X=StandardScaler().fit_transform(df[['ad_spend','social_reach']])
r=SGDRegressor(max_iter=2000, random_state=0).fit(X, df['sales']); print(f'SGDRegressor R2 = {r2_score(df.sales, r.predict(X)):.3f}')
c=SGDClassifier(loss='log_loss', max_iter=2000, random_state=0).fit(X, df['hit']); print(f'SGDClassifier accuracy = {accuracy_score(df.hit, c.predict(X)):.3f}')
SGDRegressor R2 = 0.952 SGDClassifier accuracy = 0.976
Solution. scikit-learn runs the same gradient-descent loop with tuned schedules; the loss argument switches between regression and classification.
Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher