Interpretability and Explainable AI

Explanatory targets

• Predictor and behavior - Inspect the scoring rule, explain one output, or measure predictive performance. Removal-based explanations distinguish output contributions from loss-based importance; these targets need different comparisons. Covert et al. 2021.
• Training data - Influence methods estimate how training observations affect predictions or loss. Koh and Liang approximate parameter responses; deletion-and-retraining is a separate validation target. Koh and Liang 2017.
• Representation and computation - Probes test decodable information; causal abstractions test proposed computations through aligned interventions. Accessible information alone does not establish its use. Hewitt and Liang 2019, Geiger et al. 2021.
• Decision changes - Recourse concerns feasible actions and their downstream effects. Intervening on model activations tests the program; intervening on people or physical systems requires assumptions about the world. Karimi et al. 2020.

Interpretable models

• Additive models and interactions - GAMs sum feature functions on a link/score scale; GA²M adds pairwise terms. EBMs learn them by boosting. Additivity on that scale need not imply additive probability changes. Lou et al. 2013; docs.
• RETAIN - Uses reverse-time, two-level attention and computes visit/variable contributions with attention held fixed. Interpretation relies on this architecture's contribution calculation. Choi et al. 2016; code.
• Rule lists - Ordered conditions followed by a default prediction. CORELS certifies optimal regularized training risk over its supplied rule candidates when optimization completes. Angelino et al. 2018.
• Sparse scoring systems - Learn small integer coefficients with sparsity and application-specific constraints. The score is the predictor itself. Ustun and Rudin 2019.
• Neural additive models - Jointly train one neural network per input and sum their outputs on the score/link scale. The original architecture exposes feature functions without explicit cross-feature interaction terms. Agarwal et al. 2021.

Feature effects and global importance

• Individual conditional expectation (ICE) - Plots a separate prediction curve for each observation, exposing heterogeneity hidden by average partial dependence. Goldstein et al. 2015.
• Partial dependence (PDP) - Averages model predictions while varying selected inputs and retaining other observed values. Correlated inputs can produce unrealistic combinations. docs.
• Model class reliance - Ranges over feature importance across a specified class of sufficiently accurate predictors. Different models can rely on different inputs; the class and performance tolerance define the range. Fisher et al. 2019.
• Accumulated local effects (ALE) - Accumulates locally averaged prediction differences, reducing extrapolation caused by dependent inputs. Feature effects describe the fitted model, not causal effects. Apley and Zhu 2020.
• Permutation importance - Measures a fitted model's score change after shuffling one input. Results depend on the model, score, evaluation data, and correlated substitutes. docs.
• SAGE - Assigns Shapley values to features using their contribution to expected predictive loss reduction. This measures predictive power rather than averaging local prediction attributions. Covert et al. 2020.
• Removal-based explanations - Covert et al. connect SHAP, LIME, permutation tests and other methods through three choices: how features are removed, which behavior is measured, and how removal effects are summarized. Covert et al. 2021.
• Dependence and extrapolation - Breaking feature relationships can make importance and effect plots reflect behavior in sparsely observed regions. Hooker and Mentch analyze this failure and alternatives. Hooker et al. 2021.
• Conditional subgroup importance - Forms interpretable subgroups with trees, then permutes within them to approximate conditional feature importance; also adapts PDPs. Molnar et al. 2023.

Attribution and local explanations

Feature attribution

• Layer-wise relevance propagation (LRP) - Redistributes a prediction backward through layers using relevance-propagation rules. The propagation rule determines the decomposition. Bach et al. 2015.
• SHAP - Decomposes an individual prediction into additive Shapley contributions. Aggregation summarizes contributions over chosen observations; it does not turn them into loss-based importance. Lundberg and Lee 2017; code.
  • TreeSHAP - Computes tree-model attributions under specified assumptions about missing or dependent features. Background data and perturbation mode affect the explanation. Lundberg et al. 2018; docs.
  • Shapley-value choices - The value function and treatment of absent features define the attribution problem; different constructions can disagree. Sundararajan and Najmi 2020.
  • Duplicate-input example (constructed) - Let X2=X1 be a fair bit and f(x)=x1. At (1,1), conditional Shapley gives (0.25,0.25); marginal replacement gives (0.5,0). Game definitions change whether an unused but informative input receives credit. Sundararajan and Najmi 2020.
• Integrated Gradients - Integrates input gradients along a straight path from a baseline to the observation. Attributions sum to the output difference; the baseline is part of the explanation. Sundararajan et al. 2017.
• DeepLIFT - Propagates activation differences relative to reference activations to assign input contributions. Shrikumar et al. 2017; code.
• Attribution guarantees - IG completeness constrains a sum; Shapley axioms characterize an allocation for a specified game. Neither property alone identifies a unique internal computation. Sundararajan et al. 2017, Sundararajan and Najmi 2020.
• Grad-CAM - Weights convolutional feature maps with target gradients to produce a coarse spatial attribution map. A heatmap's visual appeal does not establish faithfulness. Selvaraju et al. 2019.

Local surrogates and rules

• LIME - Fits an interpretable surrogate around one prediction using weighted perturbed samples. Sampling, neighborhood weights, and surrogate capacity shape the result. Ribeiro et al. 2016; code.
  • MeLIME - Extends LIME with data-distribution-aware sampling and alternative local interpretable models. Its changes address the perturbation and surrogate-fitting stages. Botari et al. 2020.
• Anchors - Searches for rules that preserve a prediction with high probability under a specified perturbation distribution. Precision depends on that distribution. Ribeiro et al. 2018; code.

Formal sufficient explanations

• Abductive explanations - Use a model/domain encoding and sound solver to find assignments preserving a prediction for all admissible completions. Subset-minimal means none can be removed; minimum-cardinality means the fewest assignments. Ignatiev et al. 2019.

Counterfactual explanations and recourse

• Counterfactual explanations - Search for nearby input changes that produce a desired model output. A changed prediction alone does not establish an actionable intervention. Wachter et al. 2018.
• Actionable recourse - Optimizes changes to actionable inputs for linear classifiers. Karimi et al. distinguish input edits from interventions with downstream causal effects. Ustun et al. 2019, Karimi et al. 2020.
• Diverse counterfactuals - DiCE balances proximity and diversity to offer several ways to change a prediction. Feature constraints specify permitted changes. Mothilal et al. 2020; code.

Training-data attribution

• Influence functions - Approximate how upweighting a training example changes a test prediction or loss using derivatives of the fitted model. Deep-network approximations require validation. Koh and Liang 2017.
  • Influence-estimation failures - Network depth, regularization, and test-point choice affect approximation accuracy in the investigated neural networks. Reported influence is not an exact retraining result. Basu et al. 2021.
• TracIn - Estimates training-example influence using gradient inner products across saved training checkpoints. This differs from a single-checkpoint Hessian approximation. Pruthi et al. 2020.
• Datamodels - Predict a target example's model output from which training examples were included. Ilyas et al. fit these surrogates using many subset-trained models, enabling dataset counterfactuals; the training algorithm and subset distribution define the target. Ilyas et al. 2022.
• TRAK - Approximates training-data effects using randomly projected model gradients and a kernel-based calculation. Evaluated against retraining-based prediction changes across vision and language tasks; these scores remain approximations to dataset interventions. Park et al. 2023.

Concepts and representations

• Visual representations - Feature visualization optimizes inputs for activation; Network Dissection measures unit alignment with labeled concepts (2017). Activation patterns and semantic alignment do not by themselves test causal use. Olah et al. 2017, Bau et al. 2017.
• TCAV - Learns concept directions from examples and measures prediction sensitivity through directional derivatives. Results depend on concept examples and the chosen representation layer. Kim et al. 2018.
• ProtoPNet - Predicts by matching image regions to learned prototypes associated with training-image patches. Prototype similarity forms part of the predictor, rather than a separate local surrogate. Chen et al. 2019.
• Probing with controls - Test what a representation makes decodable while checking what the probe itself can memorize. Decodability alone does not establish that the original model uses that information. Hewitt and Liang 2019.
• Concept bottleneck models - Predict annotated concepts before predicting the target; editing predicted concept values changes downstream predictions. These are interventions inside the model. Koh et al. 2020.
  • Concept leakage - Soft concept outputs can encode task information beyond their labels, even with sequential training. Mahinpei et al. demonstrate this on synthetic and MNIST tasks; concept supervision does not ensure semantic isolation. Mahinpei et al. 2021.

Comparing representations

• SVCCA - Reduces activation matrices by SVD, then compares retained subspaces with canonical correlations. Measures shared representation across layers or models on the same examples. Raghu et al. 2017.
• CKA - Compares centered example-similarity matrices. Linear CKA is invariant to orthogonal transforms and uniform scaling, but not arbitrary invertible transforms; it compares representation geometry, not causal model use. Kornblith et al. 2019.
• Representation-similarity evidence - ReSi finds no consistently best measure across its tests and graph, language and vision models. Its prediction-grounded tests use last-layer features; rankings depend on what counts as similar. Klabunde et al. 2025.

Mechanistic interpretability

• Causal abstraction - Aligns neural representations with variables in a proposed causal model and tests corresponding interchange interventions. This evaluates a specified computational hypothesis, not causal relations in the world. Geiger et al. 2021.

Candidate representations

• Superposition - Controlled toy models store sparse features in overlapping directions, yielding polysemantic neurons. This motivates sparse dictionaries; it does not establish that a learned dictionary recovers a real network's true features. Elhage et al. 2022.
• Sparse autoencoders - Reconstruct activations as sparse combinations of learned directions, seeking features distributed across neurons. Bricken et al. 2023, Gao et al. 2024.
  • End-to-end sparse autoencoders - Train sparse dictionaries through the remaining model. On GPT-2 small and TinyStories-1M, output-preserving training needs fewer active features at matched model loss. Downstream reconstruction limits changes to intermediate computation. Braun et al. 2024.
• Transcoders - Approximate dense feed-forward layers with sparse networks, enabling feature-level circuit tracing. Tests on 120M-1.4B language models find comparable sparsity, fidelity and interpretability to sparse autoencoders. Dunefsky et al. 2024.
• Feature absorption - A child concept can absorb a parent concept in an SAE, causing the parent latent to miss examples. Larger dictionaries or greater sparsity need not resolve this failure. Chanin et al. 2025.
• Native feature baselines - Feed-forward key/value features compete with SAEs and transcoders on the tested interpretability measures. Dictionary widths and available checkpoints constrain the comparison. Ye et al. 2025.
• SAE seed sensitivity - Different initializations on the same model/data yield different feature dictionaries. Overlap depends on architecture and matching; one fitted SAE is not a unique feature inventory. Paulo and Belrose 2025.
• Neurons versus transcoders - On mainly English data, sparsified neurons need a learned output correction for competitive replacement KL; transcoders reconstruct better. Neurons match top-activation concept tests but trail across broader activation quantiles. Shabalin and Piotrowski 2026.

Interventions and circuit discovery

• Transformer circuit framework - Decomposes attention computation into query-key and output-value paths, including interactions across layers. The initial analysis studies small attention-only models. Elhage et al. 2021.
• Circuit discovery case study - Identifies attention-head roles in GPT-2 small's indirect-object identification task and tests faithfulness, completeness, and minimality. Findings concern that model and task. Wang et al. 2022.
• ACDC - Removes computational edges through activation-patching tests while seeking to retain a specified behavior. The recovered circuit depends on the dataset, intervention and metric. Conmy et al. 2023.
• Activation patching - Replaces internal activations between runs to test their effect on model behavior. Zhang and Nanda show that corruption choices and evaluation metrics can change localization results. Zhang and Nanda 2024.
• Sparse Feature Circuits / SHIFT - Humans select task-irrelevant SAE circuit features for ablation, then retrain the classifier head. Marks et al. improve profession classification on gender-balanced biographies in their case study. Marks et al. 2025.
• Circuit tracing - Builds prompt-specific attribution graphs using transcoders that approximate model components. Reconstruction error and frozen attention limit the original method. Ameisen et al. 2025.
  • Attention-path extensions - Traces feature interactions through attention query-key computations omitted by the original circuit-tracing method. Kamath et al. 2025.
• Neuron circuits - MLP activations with RelP attribution yield sparse circuits competitive with SAE circuits on tested tasks. Activation site and attribution rule both matter; sparsity does not establish monosemanticity. Arora et al. 2026.
• Patching interaction effects - Noising and denoising can measure different effects because a patched component interacts with unpatched pathways. A mediation decomposition isolates this interaction; a small individual patching effect need not imply an irrelevant component. Vaidyanathan et al. 2026.
• INNSteer - Learns an invertible transformation from paired behavioral activations. A fixed shift in its latent space becomes an input-dependent nonlinear edit after inversion. Measures behavioral control, not recovery of a causal explanation. Nguyen and Le 2026.
• TopKLoRA circuits - Sparse adapters yield smaller known-trigger backdoor circuits in 30 matched comparisons with dense LoRA on two models. Removing circuits from multi-layer adapters preserves instruction-following in an LLM-judged test. Masiak et al. 2026.

Mechanism evaluation

• MIB - On four tasks and five models, attribution and mask optimization lead circuit localization; supervised distributed alignment search leads causal-variable localization. Tested SAE features do not outperform neurons on the latter track. Mueller et al. 2025.
• SAEBench - Evaluates sparse autoencoders across eight metrics spanning interpretation, disentanglement, and downstream uses; reconstruction gains need not improve other scores. Karvonen et al. 2025; code.
  • SAEBench reliability - Chanin's audit tests reseed noise, synthetic ground-truth agreement and training-trajectory discrimination. Targeted Probe Perturbation and Spurious Correlation Removal fail multiple tests at canonical settings. Chanin 2026.
• Circuit-score misranking - Geng et al. find intervention scores favoring equally sized circuits with worse held-out behavioral agreement. Restoring intact-model signals reverses selected failures; mechanism identity remains unverified. Geng et al. 2026.

• Neural Chameleons - Fine-tuning to evade benign-concept probes transfers to unseen safety probes when triggered. Monitors are trained on the final frozen checkpoint; ensembles mitigate but do not eliminate evasion. This tests trained evasion, not spontaneous concealment. McGuinness et al. 2026.

Evaluation and comparative evidence

• Evaluation objectives - Distinguish application-grounded, human-grounded, and functionally grounded evaluation. A proxy score and usefulness to a person answer different questions. Doshi-Velez and Kim 2017.
• Sources of disagreement - Separate explainer approximation noise, changes from model fitting, and different accurate models. Changing the target is distinct from estimating the same target unreliably. Sundararajan and Najmi 2020, Fisher et al. 2019.
• Human usefulness - Transparent, few-feature apartment-price models improved prediction simulation, but not advice-following; transparency sometimes impaired error correction. 3,800 participants; effects depended on presentation. Poursabzi-Sangdeh et al. 2021.

Faithfulness, stability, and perturbations

• Saliency sanity checks - Randomize model parameters or training labels to test whether explanations depend on learned behavior. Some tested methods remained visually similar after randomization. Adebayo et al. 2018.
• Infidelity and sensitivity - Compare attributed changes with output changes under perturbations, and measure explanation variation. A constant explanation can minimize sensitivity without being informative. Yeh et al. 2019.
• Remove-and-retrain (ROAR) - Masks highly ranked inputs and retrains to assess retained predictive information. This tests a different question from perturbing a fixed predictor. Hooker et al. 2019.
• Faithfulness versus plausibility - An explanation's agreement with model behavior differs from whether a person finds it convincing. Jacovi and Goldberg distinguish these evaluation targets. Jacovi and Goldberg 2020.
• Adversarial explanations - Classifiers can behave differently on real and perturbed inputs, concealing biased predictions from LIME/SHAP in the studied attacks. Slack et al. 2020.
• Out-of-distribution perturbations - Masking can make feature-importance scores and evaluation metrics reflect behavior outside the training distribution. Perturbation design is part of the evaluation. Hase et al. 2021.

Attention and language explanations

• Attention as explanation - Jain and Wallace find alternative attention weights with similar predictions; Wiegreffe and Pinter challenge the inference and propose additional tests (2019). The result depends on architecture and evaluation. Jain and Wallace 2019, Wiegreffe and Pinter 2019.
• Interpretation pitfalls - Feature dependence, interactions, poor generalization, uncertainty, and unjustified causal readings can invalidate conclusions. Molnar et al. 2022; docs.
• Chain-of-thought faithfulness - Models can use prompt hints or biases without reporting them in their explanations. Chen et al. test reasoning models. Findings are conditional on the tested tasks and hints. Turpin et al. 2023, Chen et al. 2025.

Software

• Interpretable models - InterpretML provides Explainable Boosting Machines and post-hoc methods; framework paper. Nori et al. 2019; code.
• Feature effects and attribution - scikit-learn for PDP/ICE and permutation importance; SHAP, LIME, ELI5 for model explanations. docs, code, code, docs.
• Neural-network attribution - Captum provides input, neuron, and layer attribution plus evaluation metrics for PyTorch; DeepLIFT supplies reference-based propagation. Kokhlikyan et al. 2020; docs, code.
• Explanation toolkits - AIX360 and Alibi collect multiple explanation methods; DiCE generates counterfactual examples. code, code, code.
• Mechanistic analysis - TransformerLens exposes activations and interventions; SAELens trains and analyzes sparse autoencoders; circuit-tracer constructs attribution graphs and supports interventions. code, code, code.
• Explanation evaluation - Quantus collects metrics for neural-network explanations; metric assumptions and perturbation choices determine what is evaluated. Hedström et al. 2023; code.
• TSViz - Combines input/filter influence, filter clustering, and inverse optimization to inspect convolutional time-series models. Siddiqui et al. 2019.

Surveys and perspectives

• Interpretable models versus post-hoc explanations - Rudin argues for directly interpretable predictors in high-stakes settings where suitable models can be built. Rudin 2019.
• Explanatory Model Analysis - Model diagnostics and local/global explanations, with R and Python examples. Biecek and Burzykowski 2021.
• Interpretable Machine Learning - Molnar. Interpretable models, feature effects, attribution, examples, and limitations. Molnar 2025.
• Already Clean - Goodfire research essay argues that better visualizations and concept descriptions explain apparent polysemanticity. Its neuron examples motivate direct baselines; broad cleanliness claims remain preliminary. Cammarata 2026.

Applications

• Glucose forecasting - Applies RETAIN to longitudinal glucose prediction and examines its explanations. Evidence concerns the studied type-2 diabetes cohort. De Bois et al. 2020.

Related topics

• Feature selection
• Feature extraction
• Causal inference
• Neural networks
• Ensemble learning
• Models