An End-to-End ML Project: Take It Further¶
Five extensions of the Chapter 124 cancellation model, each worked in code with a picture and a plain explanation: tune the decision threshold to a business cost, engineer an interaction feature, tune a gradient-boosting rival, check probability calibration, and score a new booking end to end. We start by rebuilding the cleaned data and the fitted pipeline.
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.calibration import calibration_curve
from sklearn.metrics import roc_auc_score, precision_recall_curve, accuracy_score
import warnings; warnings.filterwarnings('ignore')
try: df = pd.read_csv('../../data/hotel_bookings.csv')
except FileNotFoundError: df = pd.read_csv(BASE + 'hotel_bookings.csv')
df = df.drop_duplicates('booking_id'); df = df[df.lead_time>=0].copy()
tidy = lambda s: s.astype(str).str.strip().str.lower()
dep = {'no deposit':'No Deposit','non refund':'Non Refund','non-refund':'Non Refund','non-refundable':'Non Refund','refundable':'Refundable'}
df['deposit_type'] = tidy(df.deposit_type).map(lambda v: dep.get(v, v.title()))
seg = {'online ta':'Online TA','online travel agent':'Online TA','offline ta':'Offline TA','offline travel agent':'Offline TA','direct':'Direct','corporate':'Corporate','groups':'Groups','group':'Groups'}
df['market_segment'] = tidy(df.market_segment).map(lambda v: seg.get(v, v.title()))
numeric = ['lead_time','nights','adults','children','prior_cancellations','booking_changes','adr','total_special_requests','is_repeated_guest']
categorical = ['deposit_type','market_segment','customer_type']
def make_prep(num): return ColumnTransformer([('num', Pipeline([('i',SimpleImputer(strategy='median')),('s',StandardScaler())]), num), ('cat', OneHotEncoder(handle_unknown='ignore'), categorical)])
X = df[numeric+categorical]; y = df['is_canceled']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y)
model = Pipeline([('prep', make_prep(numeric)), ('clf', LogisticRegression(max_iter=3000, C=0.3))]).fit(X_train, y_train)
prob = model.predict_proba(X_test)[:,1]
print('recap: test AUC = %.3f (matches Chapter 124)' % roc_auc_score(y_test, prob))
recap: test AUC = 0.846 (matches Chapter 124)
# choose the threshold on out-of-fold predictions for the TRAINING rows, then score it on the test set once
from sklearn.model_selection import cross_val_predict
COST_MISS = 200 # a missed cancellation: a room left empty that could have been re-sold
COST_FALSE = 25 # a false alarm: an unnecessary confirmation call / small incentive
def total_cost(p, y_true, t):
pred = (p >= t).astype(int)
return ((pred==0) & (y_true==1)).sum()*COST_MISS + ((pred==1) & (y_true==0)).sum()*COST_FALSE
oof = cross_val_predict(Pipeline([('prep', make_prep(numeric)), ('clf', LogisticRegression(max_iter=3000, C=0.3))]), X_train, y_train, cv=5, method='predict_proba')[:,1]
ths = np.linspace(0.05, 0.95, 91)
costs = [total_cost(oof, y_train.values, t) for t in ths]
best_t = ths[int(np.argmin(costs))]
fig, ax = plt.subplots(figsize=(7.6,4.2))
ax.plot(ths, costs, color=EM, lw=2.4); ax.axvline(0.5, color=GREY, ls='--', label='default 0.5')
ax.axvline(best_t, color=RED, ls='--', label=f'cost-optimal {best_t:.2f}'); ax.set(xlabel='decision threshold', ylabel='total cost on training folds', title='Choosing the threshold by business cost'); ax.legend()
plt.tight_layout(); plt.show()
print(f'cost-optimal threshold (training folds) = {best_t:.2f}')
print(f'test-set cost at 0.5 = {total_cost(prob, y_test.values, 0.5):.0f} | at {best_t:.2f} = {total_cost(prob, y_test.values, best_t):.0f}')
cost-optimal threshold (training folds) = 0.09 test-set cost at 0.5 = 6675 | at 0.09 = 3075
The threshold is a business decision, not a default. Because a missed cancellation costs far more than a needless confirmation call, the expected-cost curve, computed on out-of-fold predictions so the test set stays out of the choice, bottoms out well below 0.5: the hotel should flag a booking as risky at a lower probability, accepting more false alarms to catch more real cancellations. The model outputs a probability; you choose the cutoff to match what each type of error actually costs.
def add_interaction(frame):
f = frame.copy(); f['lead_x_nodeposit'] = f['lead_time'] * (f['deposit_type']=='No Deposit').astype(int); return f
Xtr2, Xte2 = add_interaction(X_train), add_interaction(X_test)
num2 = numeric + ['lead_x_nodeposit']
base_auc = cross_val_score(Pipeline([('p',make_prep(numeric)),('c',LogisticRegression(max_iter=3000,C=0.3))]), X_train, y_train, cv=5, scoring='roc_auc').mean()
int_auc = cross_val_score(Pipeline([('p',make_prep(num2)),('c',LogisticRegression(max_iter=3000,C=0.3))]), Xtr2, y_train, cv=5, scoring='roc_auc').mean()
print('CV AUC without interaction = %.3f' % base_auc)
print('CV AUC with lead x no-deposit = %.3f' % int_auc)
fig, ax = plt.subplots(figsize=(6.2,4))
ax.bar(['base','+ interaction'], [base_auc, int_auc], color=[LIGHT, EM]); ax.set(ylim=(0.75,0.85), ylabel='5-fold CV AUC', title='Does the engineered interaction help?')
plt.tight_layout(); plt.show()
CV AUC without interaction = 0.812 CV AUC with lead x no-deposit = 0.815
Feature engineering can close the gap. A linear model cannot represent 'lead time matters more for no-deposit bookings' unless you build that product term for it, exactly the interaction a tree discovers automatically. Adding lead_time x no-deposit moves the logistic model's CV AUC only from 0.812 to 0.815, well inside the fold-to-fold spread, so on this data the interaction adds little. Building such terms is still the way to give a linear model what a tree finds on its own.
gb_grid = GridSearchCV(Pipeline([('prep', make_prep(numeric)), ('clf', GradientBoostingClassifier(random_state=0))]),
{'clf__n_estimators':[150,300], 'clf__max_depth':[2,3], 'clf__learning_rate':[0.03,0.1]}, cv=5, scoring='roc_auc', n_jobs=-1).fit(X_train, y_train)
gb_auc = roc_auc_score(y_test, gb_grid.predict_proba(X_test)[:,1])
print('tuned GBM best params:', {k.replace('clf__',''):v for k,v in gb_grid.best_params_.items()})
print('tuned GBM CV AUC = %.3f | logistic CV AUC = %.3f (the comparison)' % (gb_grid.best_score_, base_auc))
print('tuned GBM test AUC = %.3f' % gb_auc)
print('logistic test AUC = %.3f' % roc_auc_score(y_test, prob))
fig, ax = plt.subplots(figsize=(6,4))
ax.bar(['logistic','tuned GBM'], [roc_auc_score(y_test, prob), gb_auc], color=[EM, PUR]); ax.set(ylim=(0.75,0.9), ylabel='test AUC', title='Simple vs flexible, on the held-out test set')
plt.tight_layout(); plt.show()
tuned GBM best params: {'learning_rate': 0.03, 'max_depth': 2, 'n_estimators': 300}
tuned GBM CV AUC = 0.802 | logistic CV AUC = 0.812 (the comparison)
tuned GBM test AUC = 0.850
logistic test AUC = 0.846
Flexible is not automatically better. Even after a full grid search, the gradient-boosting model lands right around the logistic regression on this clean, mostly-linear data, slightly below it by cross-validation (0.802 against 0.812) and slightly above it on the test set (0.850 against 0.846). When a simple model matches a complex one, prefer the simple one, it is faster, easier to explain, and less likely to break in production. Complexity should have to earn its place.
frac_pos, mean_pred = calibration_curve(y_test, prob, n_bins=8)
fig, ax = plt.subplots(figsize=(6.2,5))
ax.plot([0,1],[0,1],'--',color=GREY,label='perfectly calibrated')
ax.plot(mean_pred, frac_pos, 'o-', color=EM, label='our model')
ax.set(xlabel='predicted cancellation probability', ylabel='actual fraction canceled', title='Reliability curve'); ax.legend()
plt.tight_layout(); plt.show()
print('the points track the diagonal -> when the model says 0.7, about 70% really do cancel')
the points track the diagonal -> when the model says 0.7, about 70% really do cancel
Calibrated probabilities can be trusted as probabilities. A ranking metric like AUC only cares about order; but if you are going to act on 'this booking has a 70% chance of canceling', that number had better be truthful. Logistic regression is naturally well-calibrated, the reliability curve hugs the diagonal, so the probabilities are safe to use directly in the cost calculation from Extension 1. (For models that are not calibrated, wrap them in CalibratedClassifierCV.)
risky = pd.DataFrame([{'lead_time':240,'nights':2,'adults':2,'children':0,'prior_cancellations':1,'booking_changes':0,
'adr':160,'total_special_requests':0,'is_repeated_guest':0,'deposit_type':'No Deposit','market_segment':'Groups','customer_type':'Transient'}])
safe = pd.DataFrame([{'lead_time':10,'nights':3,'adults':2,'children':1,'prior_cancellations':0,'booking_changes':1,
'adr':120,'total_special_requests':3,'is_repeated_guest':1,'deposit_type':'Non Refund','market_segment':'Direct','customer_type':'Transient'}])
print('high-risk booking -> cancel probability %.0f%%' % (model.predict_proba(risky)[0,1]*100))
print('low-risk booking -> cancel probability %.0f%%' % (model.predict_proba(safe)[0,1]*100))
high-risk booking -> cancel probability 95% low-risk booking -> cancel probability 0%
This is what deployment delivers. Feed the fitted pipeline a single new booking and it returns a cancellation probability, no manual preprocessing, because the pipeline carries it. A long-lead, no-deposit group booking with a prior cancellation scores very high; a last-minute, non-refundable, repeat-guest booking with special requests scores very low. Combine that score with the cost-based threshold and the hotel has an automated, explainable early-warning system.
Take-it-further summary, in plain terms¶
- The 0.5 cutoff is arbitrary, set the threshold by what a miss vs a false alarm actually costs.
- A hand-built interaction barely moved the simple model's CV AUC on this data.
- A tuned boosting model tied the logistic one, so we keep the simpler, explainable model.
- The probabilities are calibrated, '70%' really means about 70%, so they are safe to act on.
- Scoring a booking is one line, the deployed pipeline turns a raw record into a decision.