Captum: PyTorch Interpretability Library

• Captum is a unified PyTorch interpretability library providing gradient and perturbation attribution methods for model explanation.
• It supports multimodal, classification, regression, and graph-structured models with an extensible, user-friendly API.
• Captum offers robust evaluation metrics and interactive visualization tools for scalable, domain-specific model analysis.

Searching arXiv for recent and foundational papers on Captum and closely related extensions/applications. Captum is a unified, open-source model interpretability library for PyTorch that provides generic implementations of gradient-based and perturbation-based attribution algorithms, together with evaluation metrics and visualization tooling. It is designed for classification and non-classification models, including graph-structured neural networks, and its defining characteristics are multimodality, extensibility, and ease of use. Within the supplied literature, Captum appears both as a foundational library for feature, neuron, and layer attribution and as a practical backend for subsequent work on generative LLMs, temporal explanation, benchmarking, and domain-specific post hoc analysis (Kokhlikyan et al., 2020, Miglani et al., 2023).

1. Origins, scope, and design characteristics

Captum was introduced as a unified and generic model interpretability library for PyTorch in order to fill a gap in existing interpretability frameworks, which unified attribution methods in general but did not adequately support PyTorch models. Its motivation is tied to the opacity of modern neural networks and to the need for reliable explanation methods in high-stakes domains such as healthcare, finance, and autonomous driving (Kokhlikyan et al., 2020).

The library is explicitly described as supporting both classification and non-classification models, including regression models and graph-structured neural networks. Its modality coverage includes image, text, audio, video, categorical features, and dense-feature inputs. For multi-input and multi-modal models, Captum can aggregate importance by modality, thereby revealing which source of information is driving a prediction (Kokhlikyan et al., 2020).

Three design characteristics recur in the foundational description. Multimodality denotes applicability across heterogeneous input types and multi-input models. Extensibility denotes the ability to add new algorithms and features without redesigning the framework. Ease of use denotes a unified API across explanation families, so that primary, neuron, and layer explanations can be invoked in a consistent way regardless of whether the method is gradient-based or perturbation-based (Kokhlikyan et al., 2020).

A plausible implication is that Captum’s importance in practice derives less from any single attribution rule than from its role as a stable implementation substrate. Later papers use it in precisely this way: as a centralized PyTorch interface for established interpretability methods, and as a base that can be extended to settings not foregrounded in the original release, including generative language modeling and time-series explanation (Miglani et al., 2023, Enguehard, 2023).

2. Attribution model, method families, and mathematical foundations

At its core, Captum organizes attribution into three levels. Primary attribution methods explain model outputs in terms of model inputs. Neuron attribution methods explain internal hidden neurons in terms of inputs. Layer attribution methods explain outputs in terms of all neurons in a hidden layer. The foundational paper emphasizes that many neuron and layer methods are slight variants of primary methods, but that Captum exposes them through a unified API (Kokhlikyan et al., 2020).

A later formulation describes attribution methods abstractly as mappings from a model and an input to a vector of feature scores:

$$\phi : \mathbb{F} \times \mathbb{R}^d \rightarrow \mathbb{R}^d.$$

For an input vector $X \in \mathbb{R}^d$, each output coordinate corresponds to the contribution of one input feature. Many methods also require a baseline or reference input $B$, and the notation $X_S$ denotes a hybrid input in which features in $S \subseteq D$ come from $X$ and the rest come from $B$ (Miglani et al., 2023).

The supplied literature repeatedly distinguishes two major method families. Perturbation-based methods estimate feature importance by repeatedly evaluating the model on perturbed inputs and do not require access to model weights. Gradient-based methods use backpropagated gradient information and therefore require differentiable access to model internals (Miglani et al., 2023). In the foundational paper, Captum is described as including Integrated Gradients, DeepLift, SHAP variants, Grad-CAM, Guided Grad-CAM, Guided Backprop, Deconvolution, Occlusion, Feature Ablation, and smoothing methods such as SmoothGrad, SmoothGrad Square, and VarGrad through NoiseTunnel (Kokhlikyan et al., 2020). A later paper describes Captum as already supporting a broad set of methods such as Integrated Gradients, LIME, DeepLift, TracIn, and TCAV (Miglani et al., 2023).

Integrated Gradients is the attribution rule most consistently associated with Captum across the supplied papers. Its standard definition is given as

$$\phi_i (f, X) = (X_i - B_i) \times \int_{\alpha = 0}^1 \frac{f'(B + (X - B)\alpha)}{\partial x_i} \, d\alpha.$$

The same literature emphasizes that baseline choice is central to the meaning of the resulting attribution, especially for Integrated Gradients, DeepLift, SHAP-style methods, Feature Ablation, and Occlusion (Miglani et al., 2023, Kokhlikyan et al., 2020).

Captum’s connection to causation-oriented analysis is particularly clear in the NLP tutorial literature. There, Captum is presented not as a theoretical method in itself but as a practical library that supports workflows involving Saliency, Integrated Gradients, Layer Conductance, SHAP, and interaction-aware methods such as Integrated Hessians, Shapley Taylor Index, and Archipelago. The tutorial’s framing is that Captum helps operationalize questions about which input features or neurons influence a prediction and what happens when those components are ablated or manipulated (Sajjad et al., 2021).

3. Evaluation, scalability, and interactive tooling

Captum is not limited to producing attribution scores. The foundational paper emphasizes two generic quantitative evaluation metrics: infidelity and maximum sensitivity. Infidelity measures how well an attribution explains the change in model output under perturbation, while maximum sensitivity measures how much the attribution changes when the input is slightly perturbed. These metrics are presented as complements to subjective visual inspection, which the authors note can be misleading (Kokhlikyan et al., 2020).

The same paper places substantial weight on scalability and memory efficiency. For algorithms that internally expand inputs, such as Integrated Gradients and Layer Conductance, Captum can slice inputs into smaller chunks, run computations sequentially, and aggregate the results. This is explicitly presented as a way to prevent out-of-memory errors and to make very fine-grained integral approximations feasible. For perturbation-based methods, Captum can perturb multiple features together in a batch when memory allows, thereby reducing runtime. It also supports PyTorch DataParallel, enabling simultaneous forward and backward passes on multiple GPUs (Kokhlikyan et al., 2020).

Captum Insights is the library’s interactive visualization tool. It is built on top of Captum and is described as supporting sample-based debugging and visualization using feature-importance metrics. Users can sub-sample inputs, choose different attribution methods, and inspect how various output classes are attributed to input features. In the foundational account, Captum Insights is positioned as a debugging aid rather than as a separate explanation theory (Kokhlikyan et al., 2020).

A plausible implication is that Captum’s evaluation and systems features are as important as its method inventory. Several later frameworks use Captum as a backend precisely because they need method implementations plus stable engineering support. EvalxNLP, for example, treats Captum as the implementation backend for gradient-based methods and adds a benchmarking scaffold on top for transformer-based NLP tasks, including metrics for faithfulness, plausibility, and complexity (Dhaini et al., 2 May 2025).

4. NLP, neuron analysis, and generative LLMs

Within the NLP interpretability literature, Captum is presented as one of the key open-source toolkits for model interpretability in practical workflows for fine-grained interpretation and causation analysis. The tutorial literature positions it as a bridge between theory and analysis: a means to explain a model’s decision, identify important neurons in a layer, compare attribution methods, and support interventions such as ablation or manipulation (Sajjad et al., 2021).

That tutorial places Captum at the intersection of two analytic programs. The first is fine-grained interpretation, which studies individual neurons and groups of neurons with respect to linguistic or task-related properties. The second is causation analysis, which studies how input features and internal components influence predictions. In this setting, Captum is relevant to input-feature attribution, layer and neuron attribution, gradient-based and perturbation-based explanations, and interaction-aware attribution. The same literature links these analyses to applications such as model distillation, domain adaptation, efficient feature selection, removing unimportant neurons, facilitating architecture search, and mitigating bias by identifying neurons associated with sensitive attributes such as gender, race, or politeness (Sajjad et al., 2021).

A major subsequent extension concerns generative LLMs. The generative-language-model paper presents Captum as a comprehensive library for model explainability in PyTorch and introduces Captum v0.7 functionality specifically designed to analyze the behavior of generative LLMs. Two wrappers are introduced: LLMAttribution for perturbation-based methods and LLMGradientAttribution for gradient-based methods (Miglani et al., 2023).

These extensions address several issues that arise specifically in text generation. First, users can define attribution features at the level of tokens, words, phrases, full sentences, or custom semantic segments. Second, baselines for text features can be specified as a single baseline, a distribution of baselines, a list, or a sampling function; the paper recommends baselines that stay close to the natural data distribution, such as replacing a city with another city rather than padding or deleting it. Third, correlated features can be grouped and perturbed together, as in city-and-state or name-and-pronoun groupings. Fourth, attribution targets can be tied either to the model’s most likely decoded sequence or to a user-specified target string, in which case attribution is computed with respect to the log probability of that output sequence. Fifth, because token inputs are non-differentiable, gradients are computed with respect to embedding outputs and then aggregated across embedding dimensions to recover per-token attribution scores (Miglani et al., 2023).

The same paper also introduces visualization utilities for language-model attributions, including a heatmap in which prompt tokens appear across the top, target tokens along the side, and attribution scores in the matrix cells. Its worked examples use Shapley Value Sampling to study learned associations in prompts such as “David lives in Palm Coast, FL and is a lawyer. His personal interests include ...” and to assess whether few-shot demonstrations in a prompt-based sentiment-classification setting strengthen or reduce confidence in a desired output (Miglani et al., 2023).

5. Extensions, benchmarking layers, and adjacent toolkits

One line of development extends Captum to temporal settings. The library $\texttt{time\_interpret}$ is introduced as an extension of Captum with a specific focus on temporal data. It retains Captum’s model-agnostic attribution style but adds time-series-specific explanation methods, datasets, evaluation metrics, and model wrappers tailored to sequential prediction tasks. It is designed to work with any PyTorch model and uses PyTorch Lightning to make training and wrapping models easier (Enguehard, 2023).

The central technical issue in that extension is temporal leakage. Generic attribution methods can inadvertently use future time points when explaining a prediction at time $t$. To address this, $X \in \mathbb{R}^d$0 introduces time-aware wrappers, notably Time Forward Tunnel, which adapts any Captum attribution method to temporal data by cropping the input up to time $X \in \mathbb{R}^d$1. It also introduces Temporal Integrated Gradients, LOF LIME, LOF Kernel SHAP, and NonLinearitiesTunnel. The library provides datasets such as Arma, BioBank, Hawkes, Hidden Markov Model, and Mimic-III; models such as MLP, CNN, RNN, TransformerEncoder, Bert, DistilBert, and RoBerta; and evaluation methods for both known-saliency and unknown-saliency settings (Enguehard, 2023).

A second line of development surrounds Captum with benchmarking infrastructure. EvalxNLP is described as a framework for benchmarking post hoc feature-attribution methods on transformer-based NLP models. In that ecosystem view, Captum supplies several gradient-based explainers, while EvalxNLP adds datasets, metrics, visualization, and an LLM-based textual explanation layer. The benchmark compares eight methods and evaluates them along faithfulness, plausibility, and complexity dimensions, with gradient methods implemented using Captum (Dhaini et al., 2 May 2025).

A third line of comparison appears in ICX360, which focuses on prompt- and context-level explanation for LLM-generated text. That paper explicitly contrasts Captum with an LLM-specific toolkit that supports truly black-box text-only settings, contrastive explanations, and segment-level prompt explanations. In that comparison, Captum remains the closer analog for gradient saliency, integrated gradients, and classical attribution on differentiable models, but it is said not to handle the truly black-box text-only case and often to produce token-level attributions for each output token, which can lead to fragmented explanations (Wei et al., 14 Nov 2025).

These developments suggest an increasingly modular interpretability ecosystem. Captum remains the core PyTorch attribution library in the supplied literature, while later systems either specialize it for a data type, benchmark it against alternatives, or extend its abstractions to new interaction patterns.

6. Domain-specific uses, interpretive value, and methodological limitations

Application papers in the supplied literature consistently use Captum as a post hoc explanation layer rather than as part of the predictive pipeline itself. The table summarizes representative uses.

Domain	Model or input	Role of Captum
Home wound referral (Fard et al., 22 Jan 2025)	DeBERTa-base clinical notes in DM-WAT	Integrated Gradients identifies predictive words in notes
Production scheduling (Fischer et al., 2024)	DRL scheduling agent state vector	Input x Gradient ranks features supporting or discouraging an action
Bitcoin volatility forecasting (Herremans et al., 2022)	Synthesizer Transformer inputs from CryptoQuant and whale-alert tweets	Feature Ablation identifies the most important forecast features
Log anomaly detection (Balasubramanian et al., 26 Aug 2025)	RoBERTa-base log classifier	Token attributions complement BERTViz attention views

In DM-WAT, Captum is the text-interpretability component for clinical notes, while Score-CAM serves the analogous role for wound images. The system uses Integrated Gradients to highlight tokens in clinical notes that significantly impact the model’s predictions, with green text marking tokens that support the prediction and red text marking tokens that distract from it. These explanations are tied to three referral categories—Continue treatment, Change treatment non-urgently, and Change treatment urgently—and are presented as a way to improve transparency, trustworthiness, clinician confidence, and sensemaking for non-specialist visiting nurses (Fard et al., 22 Jan 2025).

In production scheduling, Captum is the gradient-based, model-specific framework used to inspect why a trained DRL agent selected particular actions. The paper uses Input x Gradient, rebuilds the original network in PyTorch for compatibility, and computes top features for actions associated with product choices. The authors explicitly treat these attributions as quantitative clues to whether the agent’s decisions align with two natural-language hypotheses derived from the reward logic: a criticality hypothesis and a setup-effort hypothesis. They also emphasize that Input x Gradient is correlative, not causal, that it is sensitive to scaling, and that its outputs for rare actions can be too small to interpret meaningfully (Fischer et al., 2024).

In Bitcoin volatility forecasting, Captum is used after training to identify which daily inputs drive next-day volatility forecasts in Synthesizer Transformer models. The specific method is Feature Ablation. The paper reports that among the most frequently important features are HL_sprd, volume, taker_buy_volume, taker_sell_volume, exchange_outflow_mean_ma7, exchange_transactions_count_inflow, close, and the whale-alert feature USDminus (Herremans et al., 2022).

In log anomaly detection, Captum supplies token-level attributions that are complementary to BERTViz attention visualizations. The paper explicitly distinguishes these roles: BERTViz shows where the transformer attended, whereas Captum shows which input tokens were most influential for the anomaly decision. This distinction is presented as especially useful for security analysts who need actionable evidence rather than only attention structure (Balasubramanian et al., 26 Aug 2025).

The methodological limitations are equally prominent in the supplied literature. Baseline choice is repeatedly described as crucial to attribution meaning (Kokhlikyan et al., 2020, Miglani et al., 2023). In addition, the paper on Integrated Gradients saturation argues that gradients in saturated regions of the path can contribute disproportionately to standard Integrated Gradients attributions. It proposes Split IG, separating a path segment in which the output meaningfully changes from a later saturated segment. On ImageNet models, the unsaturated segment often produced more faithful and less noise-sensitive explanations; for Inception-v3, the reported ABPC values are 0.2837 for Left IG, 0.1570 for Right IG, and 0.2486 for Full IG, while the corresponding sensitivity values are 0.5341, 0.6706, and 0.5711. The same paper advises explaining logits rather than post-softmax probabilities, because softmax can further saturate outputs (Miglani et al., 2020).

A broader controversy concerns the status of attribution as explanation. Several supplied papers treat Captum as operationally valuable but insufficient on its own for trustworthy interpretation. The production-scheduling study argues that hypotheses-based verification is needed to make explanations falsifiable (Fischer et al., 2024). The ICX360 comparison suggests that Captum’s abstractions are strongest when direct model access is available, but less suitable for truly black-box text-only settings and for cohesive segment-level explanation of full generations (Wei et al., 14 Nov 2025). Taken together, these arguments suggest that Captum is best understood as a mature attribution infrastructure whose scientific value depends on baseline design, target definition, evaluation protocol, and task-specific validation rather than on attribution scores alone.