Model Interpretability & Explainability: opening the black box¶
A powerful model is often an opaque one: it predicts well but cannot say why. For trust, debugging, fairness, and regulation, we need to open it up. This notebook explains a credit-risk model four ways: permutation importance and partial dependence (global, how the model behaves overall) and SHAP and LIME (local, why it made this prediction). Library-first with scikit-learn, shap, and lime.
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
import warnings; warnings.filterwarnings('ignore')
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.inspection import permutation_importance, PartialDependenceDisplay
import shap, lime, lime.lime_tabular
pd.set_option('display.max_columns', 30)
try: df = pd.read_excel('../../data/model-interpretability-and-explainability--credit.xlsx', sheet_name='Data')
except FileNotFoundError: df = pd.read_excel(BASE + 'model-interpretability-and-explainability--credit.xlsx', sheet_name='Data')
feat = ['credit_score','debt_to_income','income_k','loan_amount_k','num_late_payments','employment_years']
X, y = df[feat], df['default']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
model = GradientBoostingClassifier(random_state=0).fit(X_train, y_train)
from sklearn.metrics import roc_auc_score
print(f'gradient-boosting test accuracy = {model.score(X_test, y_test):.3f}')
print(f'always predicting "repaid" = {(y_test == 0).mean():.3f}, test AUC = {roc_auc_score(y_test, model.predict_proba(X_test)[:,1]):.3f}')
print('...but if the bank denies a loan, it must explain WHY. That is what this chapter delivers.')
gradient-boosting test accuracy = 0.710 always predicting "repaid" = 0.680, test AUC = 0.723 ...but if the bank denies a loan, it must explain WHY. That is what this chapter delivers.
Two kinds of explanation. The gradient-boosting model is useful but modest. Its test accuracy beats the score of always predicting repayment by a few points, and its AUC shows it ranks defaulters above repayers well short of perfectly. It is also a black box, since a hundred trees give no single readable rule. Interpretability tools come in two flavors. Global explanations describe how the model behaves overall (which features matter, and how). Local explanations justify one specific prediction (why was this applicant denied?). We need both, and we will build both.
r = permutation_importance(model, X_test, y_test, n_repeats=20, random_state=0)
imp = pd.Series(r.importances_mean, index=feat).sort_values()
fig, ax = plt.subplots(figsize=(7.4,4.2))
ax.barh(imp.index, imp.values, color=EM)
ax.set(title='Permutation importance (drop in accuracy when a feature is shuffled)', xlabel='importance'); plt.tight_layout(); plt.show()
print(imp.sort_values(ascending=False).round(4).to_string())
credit_score 0.0745 income_k 0.0242 debt_to_income 0.0217 num_late_payments 0.0095 loan_amount_k 0.0092 employment_years -0.0118
What the code does. Permutation importance measures how much the model's accuracy falls when a single feature's values are randomly shuffled, breaking its link to the target. A big drop means the model leans on that feature heavily. It is model-agnostic and more trustworthy than a tree's built-in impurity importances (which can be biased toward high-cardinality features). Here credit score dominates, costing about 0.07 of accuracy when shuffled. Income and debt-to-income come next at about 0.02 each, loan amount and late payments matter little, and shuffling employment years slightly raises accuracy, so the model gains nothing from it on this test set.
fig, ax = plt.subplots(1, 2, figsize=(12,4.2))
PartialDependenceDisplay.from_estimator(model, X_test.astype(float), ['credit_score','debt_to_income'], ax=ax, line_kw={'color': EM, 'linewidth':2.5})
ax[0].set_title('Default risk vs credit score'); ax[1].set_title('Default risk vs debt-to-income')
plt.tight_layout(); plt.show()
What the code does. A partial dependence plot (PDP) shows how the model's average output changes as one feature sweeps across its range, averaging over all the others. For this gradient-boosting classifier scikit-learn plots the output on the log-odds scale the trees work on, which is why the axis runs below zero. The shapes have the expected direction and are clearly non-linear: predicted default falls as credit score rises and climbs as debt-to-income grows, and you can read the thresholds where the effect is steepest. (Its cousin, the ICE plot, draws one line per individual instead of the average, revealing whether the effect differs across applicants.) Importance tells you which features matter; partial dependence tells you how.
explainer = shap.TreeExplainer(model)
shap_values = explainer(X_test)
plt.figure()
shap.plots.beeswarm(shap_values, show=False, max_display=6)
plt.title('SHAP summary: each dot is one applicant, colored by feature value'); plt.tight_layout(); plt.show()
What SHAP does. SHAP (SHapley Additive exPlanations) borrows a fair-payout idea from game theory: it splits each prediction into an additive contribution from every feature, in a way that is mathematically consistent. The beeswarm plot is a global view built from local pieces: each dot is one applicant's SHAP value for a feature, and the color is the feature's value (blue low, red high). You can read both magnitude and direction at once, high debt-to-income (red) pushes strongly toward default (right), while a high credit score (red) pushes away from it (left). It is the richest single summary an interpretability tool offers.
i = int(np.argmax(model.predict_proba(X_test)[:,1])) # the highest-risk applicant
print('applicant profile:'); print(X_test.iloc[i].to_string())
print(f'predicted default probability = {model.predict_proba(X_test.iloc[[i]])[0,1]:.2f}')
sv_i = pd.Series(shap_values[i].values, index=feat)
base = shap_values[i].base_values
print(f'SHAP starting point (average log-odds) = {base:.2f}, probability {1/(1+np.exp(-base)):.2f}')
print('SHAP contributions (log-odds):'); print(sv_i.reindex(sv_i.abs().sort_values(ascending=False).index).round(2).to_string())
print(f'start + contributions = {base + sv_i.sum():.2f} log-odds')
plt.figure()
shap.plots.waterfall(shap_values[i], show=False, max_display=6)
plt.title('Why this applicant was flagged high-risk'); plt.tight_layout(); plt.show()
applicant profile: credit_score 551.000 debt_to_income 0.682 income_k 43.000 loan_amount_k 42.600 num_late_payments 0.000 employment_years 4.300 predicted default probability = 0.91 SHAP starting point (average log-odds) = -1.00, probability 0.27 SHAP contributions (log-odds): loan_amount_k 1.56 credit_score 1.40 debt_to_income 0.39 num_late_payments -0.20 income_k 0.13 employment_years 0.04 start + contributions = 2.32 log-odds
The local story. The waterfall plot explains a single prediction: it starts at the model's average output and adds each feature's push, up (red, toward default) or down (blue, away), until it reaches this applicant's score. For a gradient-boosting classifier the pushes are in log-odds, and they add up exactly there, so the waterfall ends at about 2.32 log-odds, the 0.91 probability printed above. For our highest-risk applicant a large loan and a low credit score did most of the damage, debt-to-income added a little more, and having no late payments was the only push the other way. This is the artifact you hand a customer or an auditor: not a bare denial, but a ranked, quantitative reason.
lt = lime.lime_tabular.LimeTabularExplainer(X_train.values, feature_names=feat,
class_names=['repaid','default'], mode='classification', random_state=0)
exp = lt.explain_instance(X_test.values[i], model.predict_proba, num_features=6)
fig = exp.as_pyplot_figure(); fig.set_size_inches(7.6,3.8)
plt.title('LIME explanation for the same applicant'); plt.tight_layout(); plt.show()
print('LIME feature contributions:'); [print(f' {f}: {w:+.3f}') for f,w in exp.as_list()]
LIME feature contributions: credit_score <= 634.75: +0.262 debt_to_income > 0.40: +0.161 loan_amount_k > 29.32: +0.149 num_late_payments <= 0.00: -0.041 income_k <= 51.00: +0.039 3.50 < employment_years <= 6.30: +0.005
[None, None, None, None, None, None]
How LIME differs. LIME (Local Interpretable Model-agnostic Explanations) takes a different route to the same goal: it probes the black box with many small perturbations around one applicant, then fits a simple, readable linear model to that neighborhood. The bars show each feature pushing the local prediction toward default (positive) or repayment (negative). LIME is fully model-agnostic (it never looks inside the model, only its predictions) and works on tabular data, text, and images alike. Here LIME's top three features are the same as SHAP's (credit score, debt-to-income and loan size), though in a different order. That agreement is a useful consistency check. It does not prove either explanation faithful to the model, and LIME's weights shift with its random seed and the width of the neighborhood it samples.
Interpretability, in one view¶
- Global, which features matter: permutation importance (model-agnostic, more reliable than impurity importances).
- Global, how a feature acts: partial dependence (average effect) and ICE (per-instance).
- Local + global, one consistent measure: SHAP, additive feature contributions per prediction; beeswarm for the whole model, waterfall for one decision.
- Local, model-agnostic: LIME, a simple linear model fit around a single prediction.
- Interpretability is not optional: it is the foundation of trust, debugging, fairness, and regulatory compliance in deployed models.