Spectral Integrated Gradients for Coarse-to-Fine Feature Attribution

Abstract: Integrated Gradients (IG) is a widely adopted feature attribution method that satisfies desirable axiomatic properties. However, the choice of integration path significantly affects the quality of attributions, and the standard straight-line path introduces all input features simultaneously, often accumulating noisy gradients along the way. To address this limitation, we propose Spectral Integrated Gradients, which constructs integration paths based on singular value decomposition (SVD) of the baseline-to-input difference. By progressively activating singular components from largest to smallest, SIG introduces global structure before fine-grained details, naturally following a coarse-to-fine progression. Through extensive evaluation across diverse image classification datasets, we demonstrate that SIG produces cleaner attribution maps with reduced noise and achieves improved quantitative performance compared to existing path-based attribution methods. Our code is available at https://github.com/leekwoon/sig/.

• The paper introduces a novel attribution method that uses SVD to prioritize dominant features, resulting in cleaner, semantically aligned attributions.
• It demonstrates that the coarse-to-fine spectral path outperforms standard IG by reducing noise and improving localization across diverse image datasets.
• Empirical results validate SIG’s efficiency and theoretical guarantees, showing significant gains in faithfulness, robustness, and computational cost.

Spectral Integrated Gradients for Coarse-to-Fine Feature Attribution

Introduction and Motivation

The Spectral Integrated Gradients (SIG) method, introduced in "Spectral Integrated Gradients for Coarse-to-Fine Feature Attribution" (2605.19607), addresses a critical limitation in the design of path-based attribution algorithms, specifically Integrated Gradients (IG). While IG is axiomatic and widely adopted, its linear interpolation path accumulates gradients from all input differences uniformly, introducing high-frequency noise and resulting in scattered, unreliable attributions. SIG formulates a new integration path via singular value decomposition (SVD) of the baseline-to-input difference and prioritizes the most structurally significant components first, progressing in a coarse-to-fine manner. This spectral ordering aligns more faithfully with the semantic structure present in natural images and produces cleaner, more localized attribution maps.

Figure 1: SIG constructs integration paths based on SVD, prioritizing low-rank (structurally dominant) components before fine-grained (high-rank) detail, yielding cleaner attributions compared to standard IG.

Spectral Path Construction

SIG begins by decomposing the baseline-to-input difference $\Delta = x - x'$ using SVD, such that $$\Delta = U \Sigma V^\top = \sum_{i=1}^R \sigma_i u_i v_i^\top$$, where singular values are arranged in descending order. The rationale is that large singular values correspond to dominant, low-frequency semantic structure; trailing values encode diminishing fine-grained details and noise.

SIG defines an integration path where, at each step $\alpha \in [0,1]$, the path incrementally activates a subset of singular components up to a complexity budget determined by $\alpha$. A continuous relaxation is employed for stability: each singular component $i$ is gated by a smooth schedule $\phi_i(\alpha; \omega)$, such that the transition from inactive to fully active is controlled by an overlap parameter $\omega$. For $\omega \to 1$, SIG recovers the standard IG linear path; for $\omega \to 0$, path activation is strictly sequential from the largest to smallest singular values.

This coarse-to-fine spectral activation ensures that the integration path first traverses regions of the input space implicated in robust semantic content, only later introducing potentially spurious high-frequency or noisy gradients. The theoretical justification follows from the Eckart-Young-Mirsky theorem, guaranteeing that truncated SVD provides the optimal rank-$k$ approximation under the squared error.

Empirical Analysis: Spectral Behavior and Attribution Quality

A frequency-domain analysis reveals the distinct traversal of IG versus SIG. Standard IG interpolates all frequency components simultaneously, causing high-frequency signals (often corresponding to noise) to be introduced too early along the path. In contrast, SIG aggregates low-frequency structure before gradually incorporating higher-frequency details.

Figure 2: Frequency-domain analysis of IG and SIG paths; SIG exhibits a coarse-to-fine mechanism where low-frequency structure emerges first before introducing high-frequency detail.

Qualitative comparisons on various image classification datasets and model architectures (e.g., InceptionV3/ImageNet, ResNet18/Oxford-IIIT Pet, VGG16/Oxford 102 Flower) demonstrate that SIG produces attribution maps with visibly reduced background noise and stronger alignment to salient object regions.

Figure 3: SIG attribution maps are significantly sparser and more focused on semantically relevant regions compared to competing methods across multiple datasets and classifiers.

Path analysis supports that, during early integration steps, SIG suppresses feature contributions until coarse structural components are present, whereas IG and methods such as BIG (blur-based path) accumulate noise or emphasize edges, often missing object interiors.

Figure 4: SIG suppresses early-stage noise and selectively aggregates attribution only after core structure emerges, in contrast to IG and BIG.

Quantitative Evaluation

Extensive evaluation across faithfulness, robustness, and efficiency metrics was conducted:

• Faithfulness (DiffID, Insertion/Deletion, ROAR): Across ImageNet, Oxford-IIIT Pet, and Oxford 102 Flower for ResNet18, VGG16, and InceptionV3, SIG consistently matches or outperforms baseline and state-of-the-art path methods (BIG, SAMP, GIG, MIG, EIG). Specifically, SIG yields uniformly superior DiffID scores and attains the lowest area under the curve (AUC) in ROAR evaluations, indicating accurate identification of truly discriminative features.

  Figure 5: On ROAR (CIFAR-10), SIG achieves the steepest accuracy degradation post-retraining, pinpointing maximal removal of discriminative input features.

• Leakage-aware Perturbation (ROAD) and Localization: SIG achieves the best GAP and LeRF scores on leakage-calibrated ROAD metrics and outperforms alternative methods in localization accuracy against both SAM~2 pseudo-masks and ground-truth bounding boxes.
• Efficiency: The increase in computational overhead over standard IG is marginal ($$\Delta = U \Sigma V^\top = \sum_{i=1}^R \sigma_i u_i v_i^\top$$0 for a 1.2s average inference time per image on ImageNet2012/ResNet18). SIG maintains a Pareto-optimal position, balancing explanation quality (DiffID) with inference cost, significantly outperforming more expensive path algorithms like SAMP.

  Figure 6: SIG achieves the highest DiffID with minimal computational overhead, lying on the Pareto frontier for explanation quality versus inference time.

Theoretical Properties and Guarantees

SIG retains all axiomatic properties of IG, including completeness, sensitivity, and implementation invariance. Importantly, SIG is proven to be equivariant under discrete geometric permutations (rotations, reflections), preserving the spatial consistency of attributions—a property not guaranteed by many adaptive or data-driven path methods.

Implications and Future Directions

SIG provides an efficient, theoretically grounded, and empirically robust refinement of path-based attribution for high-dimensional perception models. The use of SVD guarantees an optimal, data-adaptive ordering of input difference components while maintaining ease of integration with any Aumann-Shapley-style attribution rule. The method is well suited to natural image domains, where spectral ordering typically aligns with semantic structure.

Potential extensions include joint multi-channel SVD to exploit cross-channel correlations, adaptive or data-distribution-matched baselines, and alternative spectral decompositions for input domains where SVD's empirical relevance weakens (e.g., NLP or tabular domains). The method is domain-agnostic in principle, but its coarse-to-fine advantage is most pronounced in scenarios where energy ordering meaningfully tracks semantic granularity—a property thoroughly validated for standard vision tasks but less so for tasks such as text classification, where SIG converges to IG under high overlap.

Conclusion

Spectral Integrated Gradients introduces a structurally principled, computationally efficient refinement to the IG family of attribution methods by traversing the input space in a data-adaptive, coarse-to-fine manner. SIG produces attribution maps that are empirically cleaner, more faithful, and more computationally efficient than previously established path-based alternatives while inheriting desirable theoretical guarantees from the IG framework. Its implications support more reliable and interpretable diagnostic analyses of deep models, particularly in safety-critical or semantically dense application spaces. Further generalization and incorporation of domain-specific decompositions may extend SIG’s advantages beyond vision to other complex modalities.