The most accurate models, gradient-boosted forests, deep networks, are the least transparent: hundreds of trees or millions of weights offer no readable rule. Yet a bank that denies a loan, a hospital that flags a patient, or a court that scores a defendant must be able to explain the decision. Interpretability is how we get accuracy and accountability.
Explain a model two ways: globally (which features matter, and how, via permutation importance and partial dependence) and locally (why this one prediction, via SHAP and LIME), because trust, debugging, fairness, and the law all demand a reason, not just a number.
Why Interpretability, and Two Kinds
An unexplained model cannot be trusted, debugged, or defended. Interpretability serves four concrete needs: trust (would you accept a diagnosis with no reasoning?), debugging (is the model right for the wrong reason, leaning on a leaked or biased feature?), fairness (does it discriminate?), and regulation (laws increasingly require reasons for automated decisions).
Explanations come at two scales, and you need both.
Global explanations describe how the model behaves across all data, its overall priorities and learned relationships. Local explanations justify one specific prediction. A loan applicant does not care which features matter on average; they want to know why they were denied. We build both.
Global: Importance & Partial Dependence
Two global tools answer two questions. Permutation importance answers which features matter: it shuffles one feature's values, breaking its link to the target, and measures how far the model's accuracy falls. A big drop means the model relies on it heavily. It is model-agnostic and more trustworthy than a tree's built-in impurity importances. On the credit model, credit score leads by a wide margin (shuffling it costs about 0.07 of accuracy), with income and debt-to-income next at about 0.02 each, the ranking a credit analyst would expect.
Partial dependence answers how a feature acts: it sweeps one feature across its range and plots the model's average output, which for this gradient-boosting model is the log-odds of default. In the notebook, predicted default falls as credit score rises and climbs with debt-to-income, non-linear curves whose steepest regions reveal the model's decision thresholds. (Its per-instance cousin, the ICE plot, draws one line per applicant to check whether the effect is the same for everyone.) Together they give the model's overall logic; but they cannot explain a single denial. For that we go local.
Local: SHAP & LIME
SHAP (SHapley Additive exPlanations) borrows a fair-credit-assignment idea from game theory: it splits each prediction into an additive contribution from every feature, in a mathematically consistent way. A single applicant's prediction becomes a running total, start at the average, then add each feature's push, up toward default or down toward repayment, until you reach their score.
The waterfall above is the artifact you hand a customer or an auditor: not a bare denial, but a ranked, quantitative reason. A large loan (+1.56 log-odds) and a low credit score (+1.40) did most of the damage, debt-to-income added a little more (+0.39), and having no late payments was the only push the other way (−0.20). SHAP's pushes add up exactly on the log-odds scale the model works on, so the waterfall ends at 2.32, which is the 0.91 probability. LIME (Local Interpretable Model-agnostic Explanations) reaches the same goal differently: it probes the black box with small perturbations around one applicant and fits a simple linear model to that neighborhood. Here LIME's top three features are the same as SHAP's (credit score, debt-to-income and loan size), though in a different order. That agreement is a useful consistency check. It does not prove either explanation faithful to the model, and LIME's weights shift with its random seed and the width of the neighborhood it samples.
Real-World Example: Explaining a Credit Decision
A lender trains a gradient-boosting model on past applicants and, by law, must give any rejected applicant the reasons. Interpretability turns an opaque score into a defensible decision.
One row per applicant with credit_score, debt_to_income,
income_k, loan_amount_k, num_late_payments, and
employment_years, plus the label default.
| Question | Tool | Answer on the credit model |
|---|---|---|
| Which features matter overall? | permutation importance | credit score first, then income and debt-to-income |
| How does a feature act? | partial dependence | risk falls with credit score, rises with debt-to-income |
| Why was this applicant flagged (0.91)? | SHAP waterfall | a large loan and a low credit score drove the risk |
| A second opinion on that applicant? | LIME | the same top three features, in a different order |
The workflow is the point: global first to sanity-check that the model learned sensible relationships (and is not leaning on a biased or leaked feature), then local to generate the specific, ranked reason codes each decision requires. That is how a black-box model becomes something you can trust, debug, and defend in front of a regulator.
Interpretability in Machine Learning & AI
Explainable AI (XAI) has grown from a nice-to-have into a requirement, technically and legally.
| Tool (this chapter) | Where it is used |
|---|---|
| Permutation importance | model debugging, feature engineering, leak detection |
| Partial dependence / ICE | validating that learned relationships make sense |
| SHAP | the industry-standard explainer; regulatory reason codes |
| LIME | model-agnostic local explanations for tabular, text, and images |
| Interpretability broadly | fairness audits, legal duties to give reasons for automated decisions |
The more powerful the model, the more it needs explaining, and the harder it gets. Rules such as the GDPR, which gives people rights over decisions made solely by automated means, including meaningful information about the logic involved, and US fair-lending rules, which require a lender to state the main reasons it denied credit, push automated decisions toward being explainable. SHAP-style reason codes are now common in banking and insurance. (This describes the general principle and is not legal advice.) At the frontier, an entire field of mechanistic interpretability tries to reverse-engineer what individual neurons and attention heads inside large language models actually compute, work that is central to AI safety, because we cannot align or trust a system we do not understand. The gap between a model's capability and our ability to explain it is one of the defining problems of modern AI, and the tools in this chapter are where learning to close it begins.
Explain a model in Python
The companion notebook trains a credit-risk model, then explains it with permutation importance, partial dependence plots, a SHAP summary (beeswarm) and a per-applicant SHAP waterfall, and a LIME local explanation, nothing left as a black box.
View opens the rendered notebook instantly. Open in Colab runs it live. To run
locally, install numpy, pandas, scikit-learn, shap,
lime, matplotlib, and openpyxl.
🎓 Key Takeaways
- ✓Interpretability is not optional: it underpins trust, debugging, fairness, and legal compliance for deployed models.
- ✓Global explanations describe the whole model: permutation importance (which features matter) and partial dependence (how they act).
- ✓Permutation importance is model-agnostic and more reliable than a tree's built-in impurity importances.
- ✓SHAP decomposes any single prediction into additive, consistent per-feature contributions, a beeswarm globally, a waterfall for one decision.
- ✓LIME explains one prediction by fitting a simple linear model in its local neighborhood; when SHAP and LIME agree, the reason has passed one consistency check.
Practice Challenges
Five short challenges. Try them with scikit-learn, shap, and lime before checking the solutions.
Permutation importance
Rank the features by permutation importance and name the most important.
permutation_importance(model, X, y).Partial dependence direction
Show the model's predicted risk falls as credit score rises.
partial_dependence(model, X, ["credit_score"]); cast to float.Global SHAP importance
Rank the features by mean absolute SHAP value.
np.abs(shap_values.values).mean(axis=0).SHAP for one applicant
For the highest-risk applicant, find the feature contributing most to the default prediction.
LIME agreement
Explain the same applicant with LIME and compare its top driver to SHAP.
LimeTabularExplainer(...).explain_instance(...).A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on interpretability. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
That completes supervised learning, every chapter so far has had a labeled target. Next we drop the labels. Clustering opens Unsupervised Learning, where the goal is to find structure, groups, patterns, and anomalies, with no answer key at all.