←Back to the bookChapter 113 · Model Interpretability & Explainability

Chapter 113 · Model Interpretability & Explainability

Model Interpretability & Explainability

Opening the black box of a credit-risk model four ways: permutation importance and partial dependence for the global picture, and SHAP and LIME for why a single applicant was flagged, all library-first with scikit-learn, shap, and lime, and heavily visual.

Statistics, Data Science and AI: A Visual Handbook · John Fisher · 2026

Model Interpretability & Explainability: opening the black box

A powerful model is often an opaque one: it predicts well but cannot say why. For trust, debugging, fairness, and regulation, we need to open it up. This notebook explains a credit-risk model four ways: permutation importance and partial dependence (global, how the model behaves overall) and SHAP and LIME (local, why it made this prediction). Library-first with scikit-learn, shap, and lime.

import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns   # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
   "axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
   "axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"

import warnings; warnings.filterwarnings('ignore')
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.inspection import permutation_importance, PartialDependenceDisplay
import shap, lime, lime.lime_tabular
pd.set_option('display.max_columns', 30)

DEMO 1 · A GOOD MODEL, BUT WHY?

Accuracy is not an explanation

Train the model we will interrogate.

try: df = pd.read_excel('../../data/model-interpretability-and-explainability--credit.xlsx', sheet_name='Data')
except FileNotFoundError: df = pd.read_excel(BASE + 'model-interpretability-and-explainability--credit.xlsx', sheet_name='Data')
feat = ['credit_score','debt_to_income','income_k','loan_amount_k','num_late_payments','employment_years']
X, y = df[feat], df['default']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
model = GradientBoostingClassifier(random_state=0).fit(X_train, y_train)
print(f'gradient-boosting test accuracy = {model.score(X_test, y_test):.3f}')
print('...but if the bank denies a loan, it must explain WHY. That is what this chapter delivers.')

gradient-boosting test accuracy = 0.710
...but if the bank denies a loan, it must explain WHY. That is what this chapter delivers.

Two kinds of explanation. A gradient-boosting model predicts default well, yet it is a black box, hundreds of trees give no single readable rule. Interpretability tools come in two flavors. Global explanations describe how the model behaves overall (which features matter, and how). Local explanations justify one specific prediction (why was this applicant denied?). We need both, and we will build both.

DEMO 2 · GLOBAL: PERMUTATION IMPORTANCE

Which features drive the model?

Shuffle a feature and watch accuracy drop.

r = permutation_importance(model, X_test, y_test, n_repeats=20, random_state=0)
imp = pd.Series(r.importances_mean, index=feat).sort_values()
fig, ax = plt.subplots(figsize=(7.4,4.2))
ax.barh(imp.index, imp.values, color=EM)
ax.set(title='Permutation importance (drop in accuracy when a feature is shuffled)', xlabel='importance'); plt.tight_layout(); plt.show()
print(imp.sort_values(ascending=False).round(4).to_string())

credit_score         0.0745
income_k             0.0242
debt_to_income       0.0220
loan_amount_k        0.0097
num_late_payments    0.0095
employment_years    -0.0118

What the code does. Permutation importance measures how much the model's accuracy falls when a single feature's values are randomly shuffled, breaking its link to the target. A big drop means the model leans on that feature heavily. It is model-agnostic and more trustworthy than a tree's built-in impurity importances (which can be biased toward high-cardinality features). Here debt-to-income, credit score, and late payments dominate, exactly the drivers a credit analyst would name.

DEMO 3 · GLOBAL: PARTIAL DEPENDENCE

How does a feature change the prediction?

Importance says which; partial dependence says how.

fig, ax = plt.subplots(1, 2, figsize=(12,4.2))
PartialDependenceDisplay.from_estimator(model, X_test.astype(float), ['credit_score','debt_to_income'], ax=ax, line_kw={'color': EM, 'linewidth':2.5})
ax[0].set_title('Default risk vs credit score'); ax[1].set_title('Default risk vs debt-to-income')
plt.tight_layout(); plt.show()

What the code does. A partial dependence plot (PDP) shows how the model's predicted probability changes as one feature sweeps across its range, averaging over all the others. The shapes are exactly right and clearly non-linear: predicted default falls as credit score rises and climbs as debt-to-income grows, and you can read the thresholds where the effect is steepest. (Its cousin, the ICE plot, draws one line per individual instead of the average, revealing whether the effect differs across applicants.) Importance tells you which features matter; partial dependence tells you how.

DEMO 4 · LOCAL & GLOBAL: SHAP

Every prediction, decomposed into feature contributions

SHAP unifies global and local with one consistent measure.

explainer = shap.TreeExplainer(model)
shap_values = explainer(X_test)
plt.figure()
shap.plots.beeswarm(shap_values, show=False, max_display=6)
plt.title('SHAP summary: each dot is one applicant, colored by feature value'); plt.tight_layout(); plt.show()

What SHAP does. SHAP (SHapley Additive exPlanations) borrows a fair-payout idea from game theory: it splits each prediction into an additive contribution from every feature, in a way that is mathematically consistent. The beeswarm plot is a global view built from local pieces: each dot is one applicant's SHAP value for a feature, and the color is the feature's value (blue low, red high). You can read both magnitude and direction at once, high debt-to-income (red) pushes strongly toward default (right), while a high credit score (red) pushes away from it (left). It is the richest single summary an interpretability tool offers.

DEMO 5 · SHAP FOR ONE DECISION

Why was THIS applicant flagged?

The local explanation a regulator can read.

i = int(np.argmax(model.predict_proba(X_test)[:,1]))     # the highest-risk applicant
print('applicant profile:'); print(X_test.iloc[i].to_string())
print(f'predicted default probability = {model.predict_proba(X_test.iloc[[i]])[0,1]:.2f}')
plt.figure()
shap.plots.waterfall(shap_values[i], show=False, max_display=6)
plt.title('Why this applicant was flagged high-risk'); plt.tight_layout(); plt.show()

applicant profile:
credit_score         551.000
debt_to_income         0.682
income_k              43.000
loan_amount_k         42.600
num_late_payments      0.000
employment_years       4.300
predicted default probability = 0.91

The local story. The waterfall plot explains a single prediction: it starts at the average prediction and adds each feature's push, up (red, toward default) or down (blue, away), until it reaches this applicant's score. For our highest-risk applicant you can see exactly which factors drove the decision, a large loan, an elevated debt-to-income ratio, and a low credit score piling on risk, only partly offset by the rest. This is the artifact you hand a customer or an auditor: not a bare denial, but a ranked, quantitative reason.

DEMO 6 · LIME: A LOCAL LINEAR SURROGATE

A second opinion on the same decision

LIME fits a simple model near one point.

lt = lime.lime_tabular.LimeTabularExplainer(X_train.values, feature_names=feat,
        class_names=['repaid','default'], mode='classification', random_state=0)
exp = lt.explain_instance(X_test.values[i], model.predict_proba, num_features=6)
fig = exp.as_pyplot_figure(); fig.set_size_inches(7.6,3.8)
plt.title('LIME explanation for the same applicant'); plt.tight_layout(); plt.show()
print('LIME feature contributions:'); [print(f'  {f}: {w:+.3f}') for f,w in exp.as_list()]

LIME feature contributions:
  credit_score <= 634.75: +0.262
  debt_to_income > 0.40: +0.161
  loan_amount_k > 29.32: +0.149
  num_late_payments <= 0.00: -0.041
  income_k <= 51.00: +0.039
  3.50 < employment_years <= 6.30: +0.005

[None, None, None, None, None, None]

How LIME differs. LIME (Local Interpretable Model-agnostic Explanations) takes a different route to the same goal: it probes the black box with many small perturbations around one applicant, then fits a simple, readable linear model to that neighborhood. The bars show each feature pushing the local prediction toward default (positive) or repayment (negative). LIME is fully model-agnostic (it never looks inside the model, only its predictions) and works on tabular data, text, and images alike. When SHAP and LIME surface the same set of drivers, as they do here (credit score, loan size, and debt-to-income), you can trust the explanation.

Interpretability, in one view

• Global, which features matter: permutation importance (model-agnostic, more reliable than impurity importances).
• Global, how a feature acts: partial dependence (average effect) and ICE (per-instance).
• Local + global, one consistent measure: SHAP, additive feature contributions per prediction; beeswarm for the whole model, waterfall for one decision.
• Local, model-agnostic: LIME, a simple linear model fit around a single prediction.
• Interpretability is not optional: it is the foundation of trust, debugging, fairness, and regulatory compliance in deployed models.

---

Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher

←Back to the bookChapter 113 · Model Interpretability & Explainability