←Back to the bookChapter 120 · An End-to-End ML Project

Chapter 120 · Case Study: An End-to-End ML Project

Case Study: An End-to-End ML Project

The complete machine-learning workflow on a messy real-world file: predict hotel booking cancellations the right way, a proper train/validation/test split, a baseline to beat, cross-validation to compare models, hyperparameter tuning, and a leakage-free pipeline, then interpret and deploy. The method is the lesson. Library-first with scikit-learn, and heavily visual.

Statistics, Data Science and AI: A Visual Handbook · John Fisher · 2026

An End-to-End Machine Learning Project

This is the whole workflow in one place: take a messy real-world file and turn it into a trustworthy model, disciplined at every step. We predict hotel booking cancellations and, more importantly, do it the right way, a proper train / validation / test split, a baseline to beat, cross-validation to compare models honestly, hyperparameter tuning, and a leakage-free pipeline. The star of this chapter is not the algorithm; it is the method. Library-first with scikit-learn.

import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns   # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
   "axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
   "axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"

from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV, learning_curve
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.dummy import DummyClassifier
from sklearn.metrics import (accuracy_score, roc_auc_score, precision_score, recall_score,
     f1_score, confusion_matrix, RocCurveDisplay, classification_report)
import warnings; warnings.filterwarnings('ignore'); pd.set_option('display.max_columns', 40)

STEP 1 · DEFINE THE OBJECTIVE

What are we predicting, and why?

Frame the problem before touching data.

The goal. A hotel loses money on a cancellation: an empty room it could have sold, and staffing planned for a guest who never arrives. If the hotel could flag high-risk bookings in advance, it could overbook wisely, send a deposit reminder, or target a confirmation offer. So the objective is a binary classification: predict is_canceled (1 = the booking will be canceled, 0 = it will be honored) for each new booking, and understand what drives the risk.

STEP 2 · COLLECT THE DATA

One line to load a CSV export

The same pandas readers work for SQL, Excel, or an API.

try: df = pd.read_csv('../../data/hotel_bookings.csv')
except FileNotFoundError: df = pd.read_csv(BASE + 'hotel_bookings.csv')
print('raw shape:', df.shape); df.head(3)

raw shape: (1322, 14)

	booking_id	lead_time	nights	adults	children	prior_cancellations	booking_changes	deposit_type	market_segment	customer_type	adr	total_special_requests	is_repeated_guest	is_canceled
0	B21186	124	9	3	1.0	2	2	No Deposit	Direct	Transient-Party	77.36	1	1	0
1	B20480	118	10	2	0.0	0	1	NO DEPOSIT	Direct	Transient	141.40	0	0	1
2	B20522	103	13	1	2.0	1	0	REFUNDABLE	Offline TA	Transient	91.54	2	0	0

Where data comes from. Here it is a CSV export from the property-management system. In practice you might pull it from a SQL warehouse (pd.read_sql), an Excel workbook (pd.read_excel), or a REST API (requests then pd.json_normalize), the loading line changes, the rest of the project does not.

STEP 3 · INSPECT

Look before you leap

shape, dtypes, missing values, and category spellings.

print(df.shape)
print('\nmissing values:'); print(df.isna().sum()[df.isna().sum()>0])
print('\ndeposit_type spellings:', sorted(df.deposit_type.unique()))
print('\nany duplicate booking_ids?', df.booking_id.duplicated().sum())
print('impossible lead_time (<0)?', (df.lead_time<0).sum())

(1322, 14)

missing values:
children    34
adr         16
dtype: int64

deposit_type spellings: ['NO DEPOSIT', 'No Deposit', 'No deposit', 'Non Refund', 'Non-Refund', 'REFUNDABLE', 'Refundable', 'no deposit', 'non refund', 'non-refundable', 'refundable']

any duplicate booking_ids? 22
impossible lead_time (<0)? 9

What inspection reveals. A quick look surfaces the usual real-world mess: duplicate bookings, a handful of impossible negative lead times (data-entry errors), missing children and adr values, and the same category written many ways (No Deposit, no deposit, NO DEPOSIT, ...). None of this is optional to fix, a model will happily treat No Deposit and no deposit as two different things.

STEP 4 · CLEAN

Fix the mess, once, reproducibly

Dedupe, drop impossible rows, standardize categories.

before = len(df)
df = df.drop_duplicates('booking_id')                       # 1) remove duplicate bookings
df = df[df.lead_time >= 0].copy()                           # 2) drop impossible negative lead times
def tidy(s): return s.astype(str).str.strip().str.lower()
dep = {'no deposit':'No Deposit','non refund':'Non Refund','non-refund':'Non Refund','non-refundable':'Non Refund','refundable':'Refundable'}
df['deposit_type'] = tidy(df.deposit_type).map(lambda v: dep.get(v, v.title()))   # 3) standardize categories
seg = {'online ta':'Online TA','online travel agent':'Online TA','offline ta':'Offline TA','offline travel agent':'Offline TA','direct':'Direct','corporate':'Corporate','groups':'Groups','group':'Groups'}
df['market_segment'] = tidy(df.market_segment).map(lambda v: seg.get(v, v.title()))
print(f'{before} rows -> {len(df)} rows after dedupe + dropping bad lead times')
print('deposit_type now:', sorted(df.deposit_type.unique()))
print('market_segment now:', sorted(df.market_segment.unique()))
print('cancellation rate: %.1f%%' % (df.is_canceled.mean()*100))

1322 rows -> 1291 rows after dedupe + dropping bad lead times
deposit_type now: ['No Deposit', 'Non Refund', 'Refundable']
market_segment now: ['Corporate', 'Direct', 'Groups', 'Offline TA', 'Online TA']
cancellation rate: 45.4%

A crucial restraint. We fixed duplicates, impossible values, and inconsistent labels here. But we did not yet fill the missing children and adr values, and that is deliberate. Imputation learns something from the data (the median), and if we learn it from the whole dataset before splitting, information from the test set leaks into training. So imputation and scaling go inside the modeling pipeline (Step 6), where they are fit on training data only. After cleaning, 1,291 bookings remain, about 45% canceled, a healthy, near-balanced target.

STEP 5 · VISUALIZE

See the signal

Which features separate cancellations from honored bookings?

fig, ax = plt.subplots(1, 3, figsize=(14,4))
df.is_canceled.value_counts().rename({0:'honored',1:'canceled'}).plot.bar(ax=ax[0], color=[GREEN, RED])
ax[0].set(title='Target balance', ylabel='bookings'); ax[0].tick_params(axis='x', rotation=0)
(df.groupby('deposit_type').is_canceled.mean()*100).sort_values().plot.barh(ax=ax[1], color=EM)
ax[1].set(title='Cancellation rate by deposit type', xlabel='% canceled')
df.groupby(pd.cut(df.lead_time, [0,30,90,180,500])).is_canceled.mean().mul(100).plot.bar(ax=ax[2], color=DEEP)
ax[2].set(title='Cancellation rate by lead time', ylabel='% canceled', xlabel='days before arrival'); ax[2].tick_params(axis='x', rotation=20)
plt.tight_layout(); plt.show()

The story the data tells. Cancellations are common but not the majority (a real baseline to beat). The risk climbs steeply with lead time, a booking made six months out is far likelier to fall through than a last-minute one, and it depends strongly on deposit type: guests who put money down (Non Refund) rarely cancel, while No Deposit bookings are the riskiest. This is exactly the kind of separable structure a model can learn.

STEP 6 · TRANSFORM (LEAKAGE-FREE)

Build the preprocessing into a pipeline

Impute, scale, and encode, fit on training data only.

numeric = ['lead_time','nights','adults','children','prior_cancellations','booking_changes','adr','total_special_requests','is_repeated_guest']
categorical = ['deposit_type','market_segment','customer_type']
preprocess = ColumnTransformer([
    ('num', Pipeline([('impute', SimpleImputer(strategy='median')), ('scale', StandardScaler())]), numeric),
    ('cat', OneHotEncoder(handle_unknown='ignore'), categorical)])
print('numeric features:', len(numeric), '| categorical features:', len(categorical))
print('Every step (impute, scale, one-hot) lives INSIDE the pipeline, so it is fit on the training fold only -> no leakage.')

numeric features: 9 | categorical features: 3
Every step (impute, scale, one-hot) lives INSIDE the pipeline, so it is fit on the training fold only -> no leakage.

Why a pipeline, not manual steps. A ColumnTransformer median-imputes and standardizes the numeric columns and one-hot-encodes the categoricals; wrapping it in a Pipeline with the model means every cross-validation fold re-fits the preprocessing on just its own training data. Impute or scale before splitting and the median/mean is computed from rows that later sit in the test set, a subtle data leak that makes your model look better than it really is. The pipeline makes leakage almost impossible by construction.

STEP 7 · ANALYZE + SPLIT

Train / validation / test

Quarantine a test set before doing anything else.

X = df[numeric + categorical]; y = df['is_canceled']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=0, stratify=y)
print('training set:', X_train.shape[0], 'bookings   (used to fit AND tune the model)')
print('test set    :', X_test.shape[0], 'bookings   (LOCKED AWAY until the very end)')
print('\nThe three roles:')
print('  TRAIN      -> fit model parameters')
print('  VALIDATION -> compare models & pick hyperparameters (here: k-fold CV inside the training set)')
print('  TEST       -> one honest, final estimate of real-world performance (used ONCE)')

training set: 1032 bookings   (used to fit AND tune the model)
test set    : 259 bookings   (LOCKED AWAY until the very end)

The three roles:
  TRAIN      -> fit model parameters
  VALIDATION -> compare models & pick hyperparameters (here: k-fold CV inside the training set)
  TEST       -> one honest, final estimate of real-world performance (used ONCE)

The golden rule of honest evaluation. We immediately set aside 20% as a test set and do not look at it again until the end. Model comparison and tuning happen on the training set only. Rather than carving off a single fixed validation set, we use k-fold cross-validation on the training data, it plays the validation role but rotates through the data, giving a more stable estimate and using every row. The test set is our incorruptible final judge.

STEP 8 · BUILD: A BASELINE + CANDIDATES

Never trust a model without a baseline

Compare three models by cross-validation.

baseline = DummyClassifier(strategy='most_frequent').fit(X_train, y_train)
print('baseline (always predict the majority class) test accuracy = %.3f' % accuracy_score(y_test, baseline.predict(X_test)))
candidates = {'Logistic Regression': LogisticRegression(max_iter=3000),
              'Random Forest': RandomForestClassifier(random_state=0),
              'Gradient Boosting': GradientBoostingClassifier(random_state=0)}
cv_results = {}
for name, clf in candidates.items():
    pipe = Pipeline([('prep', preprocess), ('clf', clf)])
    scores = cross_val_score(pipe, X_train, y_train, cv=5, scoring='roc_auc')
    cv_results[name] = scores
    print(f'{name:22s} 5-fold CV AUC = {scores.mean():.3f} (+/- {scores.std():.3f})')
fig, ax = plt.subplots(figsize=(7.6,4.2))
ax.bar(cv_results.keys(), [s.mean() for s in cv_results.values()], yerr=[s.std() for s in cv_results.values()], color=[EM, BLUE, PUR], capsize=5)
ax.axhline(0.5, color=GREY, ls='--', label='random (AUC 0.5)'); ax.set(ylabel='cross-validated AUC', title='Model comparison by 5-fold CV', ylim=(0.5,0.9)); ax.legend()
plt.xticks(rotation=12); plt.tight_layout(); plt.show()
# learning curve: would more data (or a more flexible model) help?
sizes, tr_sc, va_sc = learning_curve(Pipeline([('prep',preprocess),('clf',LogisticRegression(max_iter=3000,C=0.3))]),
    X_train, y_train, cv=5, scoring='roc_auc', train_sizes=np.linspace(0.2,1.0,6), random_state=0)
fig, ax = plt.subplots(figsize=(7.4,4.2))
ax.plot(sizes, tr_sc.mean(1), 'o-', color=BLUE, label='training AUC')
ax.plot(sizes, va_sc.mean(1), 'o-', color=EM, label='cross-validation AUC')
ax.set(xlabel='training examples used', ylabel='AUC', title='Learning curve for the logistic model'); ax.legend()
plt.tight_layout(); plt.show()

baseline (always predict the majority class) test accuracy = 0.544

Logistic Regression    5-fold CV AUC = 0.812 (+/- 0.039)

Random Forest          5-fold CV AUC = 0.789 (+/- 0.033)

Gradient Boosting      5-fold CV AUC = 0.789 (+/- 0.028)

The baseline matters. A model that always predicts 'not canceled' is already 54% accurate, so any accuracy near that is worthless. That is why we score by AUC (ranking quality, baseline 0.5) and always compare against the dummy. The surprise: logistic regression (CV AUC 0.81) beats both tree ensembles (0.79). On clean, mostly-linear tabular data a well-regularized linear model is often the strongest and the most interpretable, do not reach for a boosted forest reflexively. The learning curve confirms the choice: the training and cross-validation scores converge and flatten, so the model is not badly overfit and a lot more data would help only a little, the ceiling here is the signal in the features, not the sample size.

STEP 9 · VALIDATE: TUNE, THEN THE FINAL TEST

Tune on CV, judge once on the test set

GridSearchCV picks the regularization; the test set gives the verdict.

grid = GridSearchCV(Pipeline([('prep', preprocess), ('clf', LogisticRegression(max_iter=3000))]),
                    param_grid={'clf__C': [0.01, 0.03, 0.1, 0.3, 1, 3, 10]}, cv=5, scoring='roc_auc')
grid.fit(X_train, y_train)
print('best regularization C = %s | best CV AUC = %.3f' % (grid.best_params_['clf__C'], grid.best_score_))
res = pd.DataFrame(grid.cv_results_)
fig, ax = plt.subplots(figsize=(7.4,4.0))
ax.semilogx(res['param_clf__C'].astype(float), res['mean_test_score'], 'o-', color=EM)
ax.axvline(grid.best_params_['clf__C'], color=RED, ls='--', label=f"best C = {grid.best_params_['clf__C']}")
ax.set(xlabel='regularization strength C (log scale)', ylabel='5-fold CV AUC', title='Hyperparameter tuning: validation AUC across C'); ax.legend()
plt.tight_layout(); plt.show()
# --- the moment of truth: evaluate ONCE on the locked-away test set ---
y_pred = grid.predict(X_test); y_prob = grid.predict_proba(X_test)[:,1]
print('\n=== FINAL held-out TEST performance ===')
print('accuracy  %.3f' % accuracy_score(y_test, y_pred))
print('ROC-AUC   %.3f' % roc_auc_score(y_test, y_prob))
print('precision %.3f | recall %.3f | F1 %.3f' % (precision_score(y_test,y_pred), recall_score(y_test,y_pred), f1_score(y_test,y_pred)))
fig, ax = plt.subplots(1, 2, figsize=(12,4.6))
cm = confusion_matrix(y_test, y_pred)
sns.heatmap(cm, annot=True, fmt='d', cmap='Purples', xticklabels=['honored','canceled'], yticklabels=['honored','canceled'], ax=ax[0])
ax[0].set(title='Confusion matrix (test set)', xlabel='predicted', ylabel='actual')
RocCurveDisplay.from_estimator(grid, X_test, y_test, ax=ax[1])
ax[1].lines[0].set_color(EM); ax[1].plot([0,1],[0,1],'--',color=GREY); ax[1].set_title('ROC curve (test set)')
plt.tight_layout(); plt.show()

best regularization C = 0.3 | best CV AUC = 0.812

=== FINAL held-out TEST performance ===
accuracy  0.753
ROC-AUC   0.846
precision 0.718 | recall 0.754 | F1 0.736

Reading the verdict. Tuning nudged the regularization to C = 0.3. On the untouched test set the model scores AUC 0.85 and 75% accuracy, comfortably above the 54% baseline, with recall 0.75 (it catches three-quarters of real cancellations) at precision 0.72. Because we tuned only on cross-validation and touched the test set exactly once, this number is an honest estimate of how the model will do on next month's bookings, not an optimistic illusion.

STEP 10 · INTERPRET

What drives a cancellation?

Read the coefficients as odds.

best = grid.best_estimator_
ohe = best.named_steps['prep'].named_transformers_['cat']
feat_names = numeric + list(ohe.get_feature_names_out(categorical))
coefs = pd.Series(best.named_steps['clf'].coef_[0], index=feat_names).sort_values()
fig, ax = plt.subplots(figsize=(8,5))
colors = [RED if c>0 else GREEN for c in coefs.values]
coefs.plot.barh(ax=ax, color=colors); ax.axvline(0, color=INK)
ax.set(title='What pushes a booking toward cancellation (right) or not (left)', xlabel='logistic coefficient (log-odds)')
plt.tight_layout(); plt.show()
print('biggest cancel drivers:', ', '.join(coefs.tail(3).index[::-1]))
print('biggest anti-cancel factors:', ', '.join(coefs.head(3).index))

biggest cancel drivers: deposit_type_No Deposit, market_segment_Groups, lead_time
biggest anti-cancel factors: deposit_type_Non Refund, market_segment_Corporate, total_special_requests

The model is legible. Positive bars raise cancellation odds, negative bars lower them. The strongest risk factors are No Deposit bookings, Groups, long lead times, and a history of prior cancellations, while Non Refund deposits, Corporate bookings, and more special requests (a sign of genuine intent) mark reliable guests. Every one of these matches hotel-industry intuition, which is a good sign the model learned real structure, not noise. This interpretability is why the simple logistic model was worth keeping.

STEP 11 · DEPLOY

From notebook to production

Persist the whole pipeline, then serve, monitor, and retrain.

import joblib
joblib.dump(best, 'cancellation_model.joblib')      # the ENTIRE pipeline: preprocessing + model in one object
reloaded = joblib.load('cancellation_model.joblib')
new_booking = X_test.iloc[[0]]
print('reloaded model predicts cancel-probability = %.2f for a sample booking' % reloaded.predict_proba(new_booking)[0,1])
print('\nProduction checklist: serve behind an API (predict_proba) | log inputs+scores |')
print('monitor feature DRIFT (do live bookings still look like training data?) | retrain on a schedule | keep a human in the loop for high-stakes actions')

reloaded model predicts cancel-probability = 0.68 for a sample booking

Production checklist: serve behind an API (predict_proba) | log inputs+scores |
monitor feature DRIFT (do live bookings still look like training data?) | retrain on a schedule | keep a human in the loop for high-stakes actions

Deployment is a beginning, not an end. Saving the fitted Pipeline with joblib bundles preprocessing and model together, so production applies the exact same transformations as training (no skew). The model then lives behind a service that returns a cancellation probability for each new booking. The real work starts after launch: watch for data drift (a new market segment, a price change), track live accuracy, and retrain on fresh data before performance decays. Chapter 126 is a full case study on this operational lifecycle.

The end-to-end method, in one view

1. Define the objective as a concrete prediction.
2. Collect the data (CSV, SQL, API, and so on).
3. Inspect the shape, types, and gaps.
4. Clean: dedupe, fix impossible values, standardize categories.
5. Visualize the signal.
6. Transform inside a pipeline so preprocessing never leaks.
7. Split off a test set and use cross-validation as the validation set.
8. Build a baseline plus candidate models and compare them.
9. Tune with GridSearchCV, then evaluate once on the test set.
10. Interpret the drivers.
11. Deploy and monitor for drift.
12. Communicate the result in plain English.

The discipline, not the algorithm, is what makes a model trustworthy.

---

Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher

←Back to the bookChapter 120 · An End-to-End ML Project