Explainable AI Relevance Mapping

• Explainable AI Relevance Mapping is a framework that quantifies input feature contributions using structured attributions like heatmaps to explain model outputs.
• It employs techniques such as LRP, integrated gradients, and Shapley values to decompose model decisions and highlight key input regions.
• It extends to multimodal, time-series, and domain-specific applications, providing actionable insights for debugging, fairness, and regulatory compliance.

Explainable AI Relevance Mapping refers to a suite of methods for assigning, quantifying, and interpreting the contributions of specific inputs, components, or concepts to the outputs of complex machine learning models. These approaches generate structured attributions (e.g., heatmaps or region-level decompositions) that surface the components of the input or intermediate representations most responsible for a given decision, with the dual goal of providing insight into model reasoning and enabling practical trust, validation, and debugging. Modern relevance mapping frameworks span a progression from fine-grained feature attributions (such as pixels or time steps), through “middle-level” or concept-level explanations, to relevance quantification in latent spaces or entire modalities in multimodal models.

1. Mathematical Foundations of Relevance Mapping

At the heart of relevance mapping is the assignment of a relevance value $R_i$ to each feature, region, or component $x_i$ of a model’s input (or, in some frameworks, an intermediate or latent representation), reflecting its contribution to the output $f(x)$. For a model $f: \mathbb{R}^d \to \mathbb{R}$, this is canonically realized as an additive decomposition:

$f(x) = \sum_{i=1}^d R_i(x)$

Layer-wise relevance propagation (LRP) is the foundational algorithmic paradigm that ensures conservation of relevance through layers, with each neuron receiving a portion of the total relevance proportional to its role in the forward computation. For a neuron $j$ in layer $l+1$ and its input $i$ in layer $l$, the generic LRP redistribution rule is:

$$R^{(l)}_i = \sum_j \frac{z_{ij}}{z_j + \varepsilon \,\mathrm{sign}(z_j)} R^{(l+1)}_j$$

where $z_{ij} = a_i w_{ij}$ is the product of activation and weight, and $\varepsilon$ is a stabilizing term (Samek et al., 2017, Bharadhwaj, 2018, Agarwal et al., 2020). Extensions such as the $\alpha$–$\beta$ rule and variants for max/avg pooling preserve conservation while modulating sensitivity to positive and negative evidence.

Beyond LRP, alternative quantifications include saliency maps (raw gradients), integrated gradients (averaging gradients along a path from a baseline input), and the family of Shapley value decompositions, where each feature’s contribution is defined as its average marginal effect over all possible context sets (Janzing et al., 2019). Notably, causally sound relevance mapping via Shapley values demands defining the effect of “dropping” a feature as an interventional (do-operator) marginalization, not a conditional expectation (Janzing et al., 2019).

2. Classical and Modern Algorithms

Table: Key Relevance Mapping Algorithms

Method	Input Granularity	Mechanism/Backend
Saliency/Gradient	pixel, input feature	$\partial f/\partial x_i$
LRP	pixel, neuron	Backprop relevance with conservation
Deep Taylor Decomp.	pixel, voxel, frame	Taylor expansion at root, LRP variant
Integrated Gradients	pixel, region	Path-averaged gradients
SHAP	pixel/feature	Marginal Shapley value, interventional
Grad-CAM	feature map/pixel	Channel-weighted gradient, ReLU
PRCA/DRSA	concept/subspace	Max-relevance subspace projection
CRP	pixel, concept, region	LRP w/ concept masking, clustering

Classical relevance mapping in images (e.g., LRP, Grad-CAM, Deep Taylor) produces heatmaps that directly indicate the regions or features critical for a model output (Samek et al., 2017, Bharadhwaj, 2018, Hiley et al., 2019). In temporal and multimodal domains, virtual inspection layers and spectral transformations enable attribution in interpretable feature domains (frequency, latent, concept) (Vielhaben et al., 2023, Roshdi et al., 2024). Middle-level mapping introduces higher-order primitives (superpixels, dictionary atoms) as attribution units, shifting the focus from raw features to semantically meaningful input components (Apicella et al., 2020).

Advances in model-architecture–aware explainability include explainable segmentation from classification by transforming pixel-wise relevance maps into semantic segmentations (Ma et al., 6 Aug 2025), and fast-relevance mapping proxies enabling real-time heatmaps for large vision–LLMs (Stan et al., 2024).

3. Compositional and Disentangled Relevance

Modern relevance mapping frameworks move beyond monolithic attributions to yield compositional, multiplexed, or concept-disentangled explanations. Principal Relevant Component Analysis (PRCA) and Disentangled Relevant Subspace Analysis (DRSA) extract orthogonal, maximally-relevant subspaces at intermediate layers, isolating distinct decision-making factors (e.g., object vs. background, texture vs. shape) (Chormai et al., 2022). These methods optimize subspace selection by

$\max_U \; \text{Tr}(U^T \Sigma U), \; U^TU=I$

where $\Sigma = \mathbb{E}_n[a_nc_n^T + c_na_n^T]$ aggregates activations and local attribution context.

Concept Relevance Propagation (CRP) extends LRP by allowing relevance to be propagated not just through the output, but optionally conditioned or masked to a set of “concept neurons,” enabling both “where” and “what” explanations for each input. Concept clusters can be discovered by activation or attribution similarity, producing atlases and composition graphs that visualize how spatial regions in the input correspond to specific semantic factors (Achtibat et al., 2022).

Multipath-attribution mappings in disentangled representation learning (e.g., $\beta$-TCVAE backbones) allow attributions to flow from input to latent to output, enabling causal hypothesis testing about which semantically disentangled factors were responsible for predictions, and surfacing shortcuts and spurious correlations (Klein et al., 2023).

4. Multimodal, Time-Series, and Application-Specific Techniques

Relevance mapping is not confined to vision. Recent frameworks extend to:

• Time-series: Virtual inspection layers (e.g., DFT-LRP) propagate relevance into frequency or time–frequency domains, making sequential model strategies transparent and surfacing dependence on, for example, spectral features in ECG or audio (Vielhaben et al., 2023).
• Multimodal and generative models: FastRM uses a learned proxy, operating on final hidden states, to predict patch-level relevancy maps from vision–language transformers. Unlike gradient-based attributions—which require saving high-dimensional layer-wise activations and backward passes—FastRM achieves 99.8% reduction in computation time and reduces memory use by 44.4%, closely matching the reference attention-gradient maps in both patch-level accuracy (99.3% on VQA) and F1 (0.72 on VQA, ≥0.81 on GQA/POPE) (Stan et al., 2024).
• Medical imaging: Integrated Gradients combined with post-processing (Otsu-morphology, Normalized Cuts, DenseCRF) turn XAI relevance maps into segmentation masks, outperforming unsupervised segmentation on benchmarks such as CBIS-DDSM and NuInsSeg (Ma et al., 6 Aug 2025).
• Physics and domain science: Augmenting DNNs with domain expert variables (“XAUGs”) and applying LRP enables the ranking of both low- and high-level features by their decision relevance, with the potential to robustly quantify uncertainties via model ensembling (Agarwal et al., 2020).

5. Quantitative and Qualitative Assessment of Explanations

Quantitative evaluation of relevance maps employs several strategies:

• Perturbation tests: Iteratively mask or perturb top-k% (or bottom-k%) of relevant features/regions, measuring performance degradation to calibrate faithfulness (Samek et al., 2017, Bharadhwaj, 2018, Stan et al., 2024).
• Fidelity and concentration: Metrics such as area under patch-flipping curves (AUPC), F1-score against reference maps, or localization statistics for synthetically-controlled tasks (Chormai et al., 2022, Vielhaben et al., 2023).
• Human interpretability: Proxy user studies comparing explanation-driven identification of artifacts, or matching with human attention (e.g., normalized scanpath saliency, Pearson correlation vs. human eye movements) (Achtibat et al., 2022, Roshdi et al., 2024).
• Model calibration: Classification accuracy, precision, recall, and F1 at varying thresholds for binarizing predicted relevancy maps (Stan et al., 2024).

Qualitative evaluation includes overlaying heatmaps on input data, concept-atlas visualization, latent traversals for disentangled factors, and composition graphs showing how hierarchical concepts flow through network layers (Chormai et al., 2022, Achtibat et al., 2022, Klein et al., 2023).

6. Limitations, Open Problems, and Prospects

Despite advances, several limitations are recognized:

• Many approaches (e.g., FastRM) only attribute from final hidden states, potentially missing fine-grained cross-layer causal patterns or recovering only binary (not absolute) relevance (Stan et al., 2024).
• Disentangled and compositional explanations often require additional computations (e.g., subspace optimization or clustering) and may depend on the quality of underlying concept extraction (Chormai et al., 2022, Achtibat et al., 2022).
• For time series, extension to multichannel settings and appropriate choice of invariants (e.g., time–frequency resolution) remain open (Vielhaben et al., 2023).
• Faithfulness of explanations under distribution shift, or when adversarially perturbed, is an ongoing area of study, with multipath and concept-level attributions providing tools for shortcut detection but not yet guaranteeing robustness (Klein et al., 2023).
• Scalability to large models or real-time deployment is addressed by proxy-based surrogates such as FastRM, but generalization to diverse model families remains a subject for future work (Stan et al., 2024).

Emerging directions include tighter integration of causal and relevance mapping frameworks (e.g., ensuring that attributions correspond to interventional, not merely conditional, effects (Janzing et al., 2019)), concept-level aggregation and user-tailored explanation pipelines (Hashemi et al., 2023), and domain-specific evaluation protocols that align explainability with real-world trust and safety requirements.

7. Impact and Applications

Explainable AI relevance mapping is foundational to model transparency across a broad spectrum of applications:

• Safety-critical domains: Real-time, on-the-fly relevancy mapping (as in FastRM) is enabling trust and diagnostic monitoring in autonomous driving, medical diagnosis, and scientific research (Stan et al., 2024, Roshdi et al., 2024, Agarwal et al., 2020).
• Regulatory and stakeholder alignment: Taxonomies of XAI methods (e.g., mapping user needs to relevance frameworks) facilitate alignment between model developers, regulators, end-users, and domain experts (Hashemi et al., 2023, Arrieta et al., 2019).
• Model debugging and bias detection: Compositional and concept-level relevance mapping tools surface spurious cues (e.g., background artifacts, dataset shortcuts), improving model robustness and fairness (Chormai et al., 2022, Klein et al., 2023).

Systematic methodological advances in relevance mapping are crucial for trustworthy, interpretable deployment of deep and multimodal models, and remain an active area of research focusing on scalability, causal validity, and actionable interpretability.