From Model to Decision: evaluating and communicating a model¶
The earlier case studies each built a model. This one is about everything around the model, the judgment that turns a trained classifier into a trustworthy business decision. Using an employee-attrition problem, we will catch a data leak that makes a model look perfect, compare models without being fooled by noise, choose the right metric for the question (accuracy will actively mislead us), turn the score into a decision with a cost matrix, and communicate it. The lessons here apply to every model in the book. Library-first with scikit-learn.
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.dummy import DummyClassifier
from sklearn.metrics import (roc_auc_score, average_precision_score, accuracy_score, recall_score,
precision_score, roc_curve)
import warnings; warnings.filterwarnings('ignore'); pd.set_option('display.max_columns', 40)
The real goal. An HR team wants to reduce attrition by offering retention support to employees at risk of leaving. The model's job is not to be 'accurate', it is to help decide who to talk to, given that losing an employee is expensive and a retention conversation is cheap. Keeping that decision in view is what makes every downstream choice, metric, threshold, model, correct or wrong.
try: df = pd.read_csv('../../data/employee_attrition.csv')
except FileNotFoundError: df = pd.read_csv(BASE + 'employee_attrition.csv')
df = df.drop_duplicates('employee_id')
print('shape:', df.shape, '| attrition rate: %.1f%%' % (df.attrition.mean()*100)); df.head(3)
shape: (4000, 13) | attrition rate: 18.6%
| employee_id | age | tenure_years | monthly_salary | satisfaction | work_life_balance | overtime | num_promotions | years_since_promotion | commute_km | job_role | exit_process_started | attrition | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | E71694 | 35 | 4.2 | 7833 | 2.0 | 3 | No | 1 | 1.7 | 13 | Engineering | 1 | 1 |
| 1 | E71189 | 21 | 1.0 | 4379 | 5.0 | 4 | No | 1 | 3.0 | 2 | Support | 0 | 0 |
| 2 | E70091 | 37 | 5.7 | 9831 | 4.0 | 2 | No | 1 | 4.3 | 25 | Sales | 0 | 0 |
A moderately rare target. About 19% of employees left, so a model that predicts 'stays' for everyone is already 81% accurate, our first warning that accuracy will be a poor guide. The columns are the usual HR signals, satisfaction, work-life balance, salary, overtime, promotions, commute, plus one that deserves suspicion.
print('correlation of each feature with attrition:')
num = df.select_dtypes('number').drop(columns=['attrition'])
print(num.corrwith(df.attrition).abs().sort_values(ascending=False).round(3).to_string())
print('\nattrition rate by exit_process_started:')
print(df.groupby('exit_process_started').attrition.mean().round(3).to_string())
correlation of each feature with attrition: exit_process_started 0.918 satisfaction 0.260 work_life_balance 0.177 commute_km 0.062 years_since_promotion 0.060 monthly_salary 0.048 tenure_years 0.045 num_promotions 0.029 age 0.006 attrition rate by exit_process_started: exit_process_started 0 0.013 1 0.923
A predictor that is almost the answer. exit_process_started correlates far more strongly with attrition than any real signal, employees for whom it is 1 leave over 90% of the time. That is the classic fingerprint of a leak: the offboarding process mostly starts after someone has already resigned, so a current employee who has not resigned would almost always show 0 at scoring time (the 8% of flagged employees who stayed are exits that did not go through). Using it would build a model that is useless in practice but looks spectacular in testing.
numeric = ['age','tenure_years','monthly_salary','satisfaction','work_life_balance','num_promotions','years_since_promotion','commute_km']
categorical = ['overtime','job_role']
# exit_process_started is DELIBERATELY excluded: it is a post-outcome leak
preprocess = ColumnTransformer([
('num', Pipeline([('impute', SimpleImputer(strategy='median')), ('scale', StandardScaler())]), numeric),
('cat', OneHotEncoder(handle_unknown='ignore'), categorical)])
print('features used:', len(numeric)+len(categorical), '(exit_process_started left out on purpose)')
features used: 10 (exit_process_started left out on purpose)
y = df['attrition']
X_leaky = df[numeric+['exit_process_started']+categorical]; X_honest = df[numeric+categorical]
Xl_tr,Xl_te,yl_tr,yl_te = train_test_split(X_leaky, y, test_size=0.3, random_state=0, stratify=y)
Xh_tr,Xh_te,yh_tr,yh_te = train_test_split(X_honest, y, test_size=0.3, random_state=0, stratify=y)
pre_leaky = ColumnTransformer([('num', Pipeline([('i',SimpleImputer(strategy='median')),('s',StandardScaler())]), numeric+['exit_process_started']), ('cat', OneHotEncoder(handle_unknown='ignore'), categorical)])
leaky = Pipeline([('p',pre_leaky),('c',LogisticRegression(max_iter=2000,class_weight='balanced'))]).fit(Xl_tr,yl_tr)
honest = Pipeline([('p',preprocess),('c',LogisticRegression(max_iter=2000,class_weight='balanced'))]).fit(Xh_tr,yh_tr)
auc_leaky = roc_auc_score(yl_te, leaky.predict_proba(Xl_te)[:,1]); auc_honest = roc_auc_score(yh_te, honest.predict_proba(Xh_te)[:,1])
print('WITH the leaky feature: ROC-AUC %.3f <- looks amazing, and is a lie' % auc_leaky)
print('honest model (no leak): ROC-AUC %.3f <- what you would actually get in production' % auc_honest)
fig, ax = plt.subplots(figsize=(6.4,5))
for m,X,yt,lab,c in [(leaky,Xl_te,yl_te,'with leak',RED),(honest,Xh_te,yh_te,'honest',EM)]:
fpr,tpr,_ = roc_curve(yt, m.predict_proba(X)[:,1]); ax.plot(fpr,tpr,color=c,lw=2.6,label=f'{lab} (AUC {roc_auc_score(yt,m.predict_proba(X)[:,1]):.2f})')
ax.plot([0,1],[0,1],'--',color=GREY); ax.set(title='A data leak inflates the score', xlabel='false positive rate', ylabel='true positive rate'); ax.legend(loc='lower right')
plt.tight_layout(); plt.show()
WITH the leaky feature: ROC-AUC 0.987 <- looks amazing, and is a lie honest model (no leak): ROC-AUC 0.756 <- what you would actually get in production
Leakage is the most dangerous evaluation mistake. Including exit_process_started rockets the AUC to 0.99, a model that would collapse to 0.76 the moment it met a real, current employee. The tell was the too-good score, and the fix was understanding when each feature becomes known. Before trusting any high number, ask: could this feature have been recorded only after the outcome? We proceed with the honest feature set.
X_train, X_test, y_train, y_test = train_test_split(X_honest, y, test_size=0.3, random_state=0, stratify=y)
cv = StratifiedKFold(5, shuffle=True, random_state=0); scores = {}
for name, clf in [('Logistic', LogisticRegression(max_iter=2000, class_weight='balanced')),
('Random Forest', RandomForestClassifier(n_estimators=300, class_weight='balanced', random_state=0)),
('Gradient Boosting', GradientBoostingClassifier(random_state=0))]:
s = cross_val_score(Pipeline([('p',preprocess),('c',clf)]), X_train, y_train, cv=cv, scoring='roc_auc')
scores[name] = s; print(f'{name:20s} CV ROC-AUC {s.mean():.3f} +/- {s.std():.3f}')
diff = scores['Logistic'] - scores['Gradient Boosting']
print(f'\nLogistic minus Gradient Boosting, per fold: {np.round(diff,3)}')
print(f'logistic wins in all 5 folds: {bool((diff>0).all())} -> the edge is consistent, not a lucky split')
fig, ax = plt.subplots(figsize=(7.8,4.4))
means=[s.mean() for s in scores.values()]; stds=[s.std() for s in scores.values()]
ax.bar(scores.keys(), means, yerr=stds, capsize=6, color=[EM,BLUE,PUR])
for i,s in enumerate(scores.values()): ax.plot([i]*len(s), s, 'o', color='k', alpha=0.5)
ax.set(ylabel='cross-validated ROC-AUC', title='Model comparison (dots = individual folds)', ylim=(0.65,0.8)); plt.tight_layout(); plt.show()
Logistic CV ROC-AUC 0.747 +/- 0.014
Random Forest CV ROC-AUC 0.705 +/- 0.015
Gradient Boosting CV ROC-AUC 0.723 +/- 0.017 Logistic minus Gradient Boosting, per fold: [0.03 0.023 0.014 0.02 0.03 ] logistic wins in all 5 folds: True -> the edge is consistent, not a lucky split
Do not over-read a single split. The three models land within a few points of each other, close enough that on one lucky train/test split any of them could look best. The disciplined check is cross-validation, examined fold by fold: logistic regression beats gradient boosting in all five folds (mean edge about 0.02), so its advantage is real and consistent, not noise, and it is the simpler, more interpretable model. That combination makes it the clear choice. Had the per-fold differences flipped signs, we would have called it a tie and picked the simpler model anyway.
model = Pipeline([('p',preprocess),('c',LogisticRegression(max_iter=2000, class_weight='balanced'))]).fit(X_train, y_train)
prob = model.predict_proba(X_test)[:,1]; yv = y_test.values
do_nothing = np.zeros_like(yv)
pred_05 = (prob>=0.5).astype(int)
print('predict everyone STAYS: accuracy %.3f, but recall %.3f (catches ZERO leavers)' % (accuracy_score(yv,do_nothing), recall_score(yv,do_nothing)))
print('the model at 0.5: accuracy %.3f, recall %.3f' % (accuracy_score(yv,pred_05), recall_score(yv,pred_05)))
print('\naccuracy actually RANKS do-nothing above the model, yet do-nothing helps no one.')
print('ROC-AUC %.3f | PR-AUC %.3f <- these reward ranking leavers correctly' % (roc_auc_score(yv,prob), average_precision_score(yv,prob)))
predict everyone STAYS: accuracy 0.814, but recall 0.000 (catches ZERO leavers) the model at 0.5: accuracy 0.705, recall 0.664 accuracy actually RANKS do-nothing above the model, yet do-nothing helps no one. ROC-AUC 0.756 | PR-AUC 0.441 <- these reward ranking leavers correctly
Accuracy is the wrong yardstick. Because only 19% leave, predicting 'stays' for everyone scores 81% accuracy, higher than our working model, while identifying not a single at-risk employee. If we optimized accuracy we would ship the do-nothing model and help nobody. The metric has to reflect the goal: we care about catching leavers (recall) and ranking them (ROC-AUC, PR-AUC), and ultimately about the dollar value of the decision, which comes next.
from sklearn.model_selection import cross_val_predict
COST_LOSS = 25000 # cost to replace an employee who leaves
COST_OUTREACH = 1500 # cost of a retention conversation / incentive
SAVE_RATE = 0.5 # a retention effort keeps ~half of the would-be leavers
def campaign_value(p, y_true, t):
flag = p >= t # leavers caught x save rate x replacement cost, minus the cost of every conversation
return (flag & (y_true==1)).sum()*SAVE_RATE*COST_LOSS - flag.sum()*COST_OUTREACH
# choose the threshold on out-of-fold predictions for the TRAINING rows; the test set only scores the choice
oof = cross_val_predict(model, X_train, y_train, cv=cv, method='predict_proba')[:,1]; ytr = y_train.values
ths = np.linspace(0.05, 0.9, 86)
value = [campaign_value(oof, ytr, t) for t in ths]
acc = [accuracy_score(ytr, (oof>=t).astype(int)) for t in ths]
best_t = ths[int(np.argmax(value))]
fig, ax = plt.subplots(figsize=(8,4.4)); ax2 = ax.twinx()
ax.plot(ths, np.array(value)/1000, color=EM, lw=2.6, label='business value ($k)')
ax2.plot(ths, acc, color=AMBER, lw=2.0, ls='--', label='accuracy')
ax.axvline(best_t, color=RED, ls=':', lw=2); ax.axvline(0.5, color=GREY, ls=':')
ax.set(xlabel='decision threshold (flag if risk above)', ylabel='campaign value on training folds ($k)', title='Value peaks where accuracy does not')
ax2.set_ylabel('accuracy'); ax.legend(ax.get_lines()[:1] + ax2.get_lines()[:1], ['business value ($k)', 'accuracy'], loc='lower center')
plt.tight_layout(); plt.show()
print(f'threshold chosen on the training folds = {best_t:.2f} (accuracy on those folds ranges {min(acc):.2f} to {max(acc):.2f})')
for t in [0.5, best_t]:
f=(prob>=t).astype(int)
print('test set at %.2f: accuracy %.2f | recall %.2f | flags %.0f%% of employees | value $%s' % (t, accuracy_score(yv,f), recall_score(yv,f), f.mean()*100, f'{campaign_value(prob, yv, t):,.0f}'))
threshold chosen on the training folds = 0.30 (accuracy on those folds ranges 0.19 to 0.82) test set at 0.50: accuracy 0.70 | recall 0.66 | flags 36% of employees | value $1,209,500 test set at 0.30: accuracy 0.49 | recall 0.89 | flags 66% of employees | value $1,301,000
Accuracy and value point in opposite directions. With a departure costing far more than a conversation, the value curve, computed on out-of-fold predictions for the training rows, peaks at a threshold of 0.30, well below 0.5. On the test set that cutoff catches 89% of leavers and adds about 92,000 dollars over the 0.5 cutoff, but it flags 66% of employees, and accuracy falls to 0.49. The dashed accuracy line rises as the threshold climbs and value falls, and it is highest when almost no one is flagged, a choice that creates almost no value. The lesson of the chapter in one picture: choose the threshold by the decision's economics, and never by accuracy.
coefs = pd.Series(model.named_steps['c'].coef_[0][:len(numeric)], index=numeric).sort_values()
fig, ax = plt.subplots(figsize=(8,4.6))
coefs.plot.barh(ax=ax, color=[RED if c>0 else GREEN for c in coefs.values]); ax.axvline(0, color=INK)
ax.set(title='Numeric features associated with leaving (right) or staying (left)', xlabel='logistic coefficient (log-odds)')
plt.tight_layout(); plt.show()
print('strongest numeric signals (by magnitude):', ', '.join(coefs.abs().sort_values(ascending=False).head(4).index))
strongest numeric signals (by magnitude): satisfaction, work_life_balance, years_since_promotion, commute_km
Insight, not just a score. Low satisfaction and poor work-life balance are the strongest departure signals, followed by a long time since the last promotion and a long commute, while higher salary and tenure go with staying (the chart shows the numeric features only). These are associations in past data, so they point HR at factors worth testing, career progression, workload and flexibility among them, and an experiment would show whether changing them reduces attrition.
import joblib; joblib.dump(model, 'attrition_model.joblib'); svc = joblib.load('attrition_model.joblib')
new = pd.DataFrame([
{'age':29,'tenure_years':1.5,'monthly_salary':4200,'satisfaction':2,'work_life_balance':1,'num_promotions':0,'years_since_promotion':3.0,'commute_km':45,'overtime':'Yes','job_role':'Support'},
{'age':45,'tenure_years':12,'monthly_salary':11000,'satisfaction':5,'work_life_balance':4,'num_promotions':4,'years_since_promotion':1.0,'commute_km':8,'overtime':'No','job_role':'Engineering'}])
for j in range(len(new)):
p = svc.predict_proba(new.iloc[[j]])[0,1]
action = 'FLAG for a retention conversation' if p >= best_t else 'no action'
print(f'employee {j+1}: attrition risk {p:.0%} -> {action}')
print(f'\nSTAKEHOLDER SUMMARY: the model flags the {(prob >= best_t).mean():.0%} of employees most at risk of leaving,')
print('so HR can focus retention effort where it pays. It is right often enough to be worth acting on,')
print('the factors associated with leaving are satisfaction, work-life balance and career progress, and a control group will show the impact.')
employee 1: attrition risk 88% -> FLAG for a retention conversation employee 2: attrition risk 5% -> no action STAKEHOLDER SUMMARY: the model flags the 66% of employees most at risk of leaving, so HR can focus retention effort where it pays. It is right often enough to be worth acting on, the factors associated with leaving are satisfaction, work-life balance and career progress, and a control group will show the impact.
Close the loop in plain language. The saved model scores a new employee, the unhappy, overworked junior with a long commute is flagged; the well-paid, satisfied, recently-promoted senior is not, and the decision comes with a one-paragraph summary a manager can act on: what it does, how good it is, what drives it, and what to do. That translation, from probabilities to a sentence a decision-maker trusts, is the final and most under-rated step of any modeling project.
From model to decision, in one view¶
- Hunt for leakage first: a feature known only after the outcome (here
exit_process_started) inflated AUC from an honest 0.76 to a fake 0.99. - Compare models fold by fold: logistic beat the ensembles in all five CV folds, a consistent, real edge, and it is simpler.
- Pick the metric that matches the decision: accuracy ranked 'do nothing' first while catching zero leavers; recall, PR-AUC, and value are the right lenses.
- Set the threshold by economics: chosen on the training folds, business value peaked well below 0.5, at a threshold where accuracy is far below its best.
- Interpret and communicate: the factors associated with leaving point to levers worth testing, and a one-paragraph summary turns the model into a decision people will act on.
A model is not a decision. The value is created in how you evaluate it, threshold it, and explain it.