R-LRP: Enhanced Neural Explainability

• R-LRP is a collection of advanced techniques that adapt classical LRP for diverse neural architectures by incorporating specialized rules for recurrence, residual paths, and positional encodings.
• The methods employ tailored propagation rules, such as multiplicative gate handling for RNNs and multi-sink conservation in Transformers, to maintain numerical stability and enforce relevance conservation.
• Empirical evaluations show that R-LRP variants achieve superior attribution fidelity, with significant improvements in pixel and word relevance tests compared to traditional LRP approaches.

R-LRP encompasses a family of Layer-wise Relevance Propagation (LRP) extensions designed for improved neural network explainability, addressing both architectural idiosyncrasies (e.g., recurrence, residual connections, or positional encodings) and numerical robustness. The term “R-LRP” has been used in at least three distinct contexts: (1) Recurrent LRP for RNNs, (2) Relative LRP designed for broad robustness in feed-forward and convolutional architectures, and (3) Positional-Aware LRP for Transformer models. All R-LRP variants aim to provide more faithful, numerically stable, or structurally complete explanations for neural decision-making by addressing previously unhandled or problematic model structures and operations.

1. Motivations and Conceptual Foundations

Layer-wise Relevance Propagation distributes a scalar model output backward through a network, assigning input-level scores quantifying the contribution (“relevance”) of each input feature, pixel, or token to the output. Classical LRP enforces a local conservation principle such that the sum of inputs’ relevances approximately matches the original prediction score at each layer. However, standard LRP can exhibit numerical instability (especially division by near-zero activations), and typically lacks rules for specialized architectures such as RNNs with gates, deep residual structures, or position-sensitive transformer blocks.

The core motivations for the various R-LRP proposals include:

• Eliminating the need to divide by small or canceling values, which in classical LRP causes spurious attributions and unstable explanations.
• Ensuring conservation of total relevance in the presence of complex operations such as multiplicative gates (RNNs) or additive skip connections (ResNets).
• Incorporating all critical input and structural features—such as positional encodings in Transformers—so that the relevance distribution is comprehensive and theoretically consistent (Nyiri et al., 24 Jan 2025, Arras et al., 2017, Bakish et al., 2 Jun 2025).

2. Formal Definitions and Propagation Rules

The following table summarizes the main R-LRP variants by model type and their defining propagation rules:

Architecture	R-LRP Formulation	Key Propagation Rule(s)
RNN / LSTM / GRU	Recurrent-LRP	Weighted & multiplicative (gate) rules
CNN, ResNet, Dense NN	Relative-LRP	Global fan-in normalization, no local div.
Transformer	Positional-Aware LRP	Position-token multi-sink, PE-specific

2.1 Recurrent Layer-wise Relevance Propagation (R-LRP for RNNs)

Recurrent-LRP extends standard LRP by explicitly handling multiplicative connections from gates (e.g., $g_t$ in LSTM/GRU). For an affine connection,

$$R_{i \leftarrow j} = \frac{z_i w_{ij} + \frac{\epsilon\,\mathrm{sign}(z_j) + \delta b_j}{N}}{z_j + \epsilon\,\mathrm{sign}(z_j)} R_j;$$

for a multiplicative interaction ($z_j = g_t \odot s_t$), the entire relevance flows to $s_t$. This deterministic, signed, and one-pass propagation can distinguish evidence “for” and “against” a class and empirically outperforms gradient-based sensitivity (SA) in saliency tests (Arras et al., 2017).

2.2 Relative Layer-wise Relevance Propagation (Relative-LRP)

Relative-LRP modifies LRP for CNNs, dense, and residual networks. The pairwise edge contribution is

$$z_{i\to j,k}^{(\ell)} = \frac{1}{M_j^{(\ell+1)}}\,w_{ij}^{(\ell)}\,x_i^{(\ell)}\,R_{j,k}^{(\ell+1)},$$

with input-level relevance post-normalization:

$$R_{i,k}^{(\ell)} = \frac{|\mathrm{Succ}(i)|}{N^{(\ell+1)}} \sum_{j \in \mathrm{Succ}(i)} z_{i\to j,k}^{(\ell)}.$$

No divisors depend on activations, ensuring numerical stability. ResNet skip/residual branches are treated independently and re-normalized to enforce global conservation. No hyperparameters are required (Nyiri et al., 24 Jan 2025).

2.3 Positional-Aware LRP for Transformers (Editor’s term: “R-LRP for Transformers”)

Standard LRP for Transformers omits relevance through positional encodings (PEs), breaking the conservation property. R-LRP for Transformers (or PA-LRP) solves this by reformulating the input space as position-token pairs, introducing specialized addition-matrix-multiplication rules per PE type (learnable, absolute, rotary, ALiBi). For example, for learnable PE,

$$\mathcal{R}(P'_i) = P'_i \frac{\mathcal{R}(z_i)}{P'_i + E_i + \epsilon};$$

in RoPE, relevance flows back into rotation matrices, which are then unflattened into position-buckets. Inductive proofs ensure conservation per layer (Bakish et al., 2 Jun 2025).

3. Implementation and Algorithmic Outlines

The R-LRP workflow consists of a standard forward inference to store activations, followed by a single backward relevance pass. Key implementation details by architecture include:

• Recurrent-LRP: Forward and reverse traversal through all LSTM/GRU gates and cell states, applying weighted and multiplicative rules per step. Aggregate all word-embedding scores for final explanation (Arras et al., 2017).
• Relative-LRP (CNN/Dense/ResNet): At each layer, compute edge contributions, aggregate and normalize by fan-in/out. For residual blocks, treat each path (residual/skip) separately, then globally renormalize to match conservation (Nyiri et al., 24 Jan 2025).
• Transformer R-LRP (PA-LRP): Store all input embeddings, positional encodings, and rotation matrices. Backpropagate relevance per standard or PE-specific rule. Accumulate multi-sink scores, apply ReLU gating, and aggregate for heatmap outputs (Bakish et al., 2 Jun 2025).

All R-LRP methods remain hyperparameter-free, require no retraining, and use only elementary network parameters such as weights, activations, and integer branch fan-ins per layer.

4. Quantitative and Qualitative Evaluation

Evaluation methodologies for R-LRP typically rely on ablation and perturbation metrics:

• Word-level relevance in RNNs: In word deletion experiments, removing top-$k$ R-LRP-identified words causes classification accuracy to drop much sharper than with gradient-based methods, indicating better faithfulness of relevance assignment. Conversely, deleting “against”-class words can effectively restore predictions (Arras et al., 2017).
• Pixel-level relevance in images: For R-LRP on CNNs and ResNets, retaining only the top fraction of relevant pixels as per the relevance map allows models to recover nearly full accuracy, while LRP-0, LRP-ε, and other variants degrade faster. On modified MNIST, R-LRP achieves 98.2% accuracy at 5% most relevant pixels; other variants perform substantially worse at this sparseness. On Cat vs. Dog and ImageNet, R-LRP at top-$p$-percentile pixel masks outperforms counterparts by 10–30 absolute points (Nyiri et al., 24 Jan 2025).
• Pointing game/segmentation: R-LRP correctly aligns heatmaps with ground-truth object regions, as measured by intersection with object masks (pointing score ∼0.8 vs. 0.35 in classical LRP); average mask-border distances confirm sharper localization.
• NLP and vision tasks for Transformers: On NLP tasks, PA-LRP achieves statistically significant reductions in attribution AU-MSE (up to 51% on Tiny-LLaMA), and increases AUAC vs. AttnLRP. In vision, segment-based metrics (mIoU, pixel accuracy) and perturbation-based AUC favor the positional-aware formulation (Bakish et al., 2 Jun 2025).

5. Practical Considerations and Limitations

R-LRP variants are post-hoc, model-agnostic explanation algorithms requiring only access to a model’s forward and backward computational graph. For Transformer applications, minor overhead (~10–20% additional computation) arises from explicit PE relevance tracking. PA-LRP incurs modest additional memory for rotation matrices (RoPE).

While R-LRP variants bring rigor and robustness, some limitations persist:

• The Relative-LRP conservation factor is uniform within layers; more granular normalization might yield sharper attribution.
• Negative relevance propagation for suppressive evidence remains unresolved in Relative-LRP; current formulations focus on positive evidence.
• Comparisons relying on object masks in vision implicitly assume that the classifier’s “trustworthy region” aligns exactly with annotated objects; this may not be justified for models exploiting background/context.

This suggests future research could refine scaling procedures, develop signed-relevance propagation for “against” evidence, and generalize the framework to architectures such as attention-based or graph neural networks by extending the propagation rules to arbitrary directed acyclic graphs (Nyiri et al., 24 Jan 2025).

6. Extensions and Directions for Future Research

R-LRP’s principled approach to numerically robust and structurally aware attribution lends itself naturally to ongoing extension:

• For residual networks beyond ResNet50: Relative-LRP’s global conservation and fan-in normalization remain stable, but additional normalization steps are necessary where skip and residual branches differ in magnitude.
• For Transformer explainability, R-LRP (PA-LRP) closes the conservation gap by including positional sinks and can be extended to multi-head attention and positional mechanisms beyond RoPE and ALiBi.
• Expansion to signed relevance and trans-model occlusion-based validation offers further avenues for refinement.
• A plausible implication is that as more networks integrate complex skip, gate, or composition structures, only family-unified frameworks such as R-LRP—which tailor propagation per operation—will scale in robustness and faithfulness.

7. Summary Table of R-LRP Variants

Variant	Target Architecture	Main Innovation	Empirical Advantage
Recurrent LRP	LSTM/GRU RNNs	Deterministic gate relevance rules, signed evidence	Superior word-level faithfulness (Arras et al., 2017)
Relative LRP	CNN, ResNet, Dense	Division-free, fan-in norm, skip-path normalization	Robust pixel attributions, high accuracy (Nyiri et al., 24 Jan 2025)
Positional-Aware LRP	Transformers	Multi-sink conservation, position-token coupling, PE rules	Restores conservation, better attribution (Bakish et al., 2 Jun 2025)

All R-LRP approaches preserve the efficiency and post-hoc nature of LRP. Their robust handling of network-specific operations, together with superior empirical performance in both explanatory and quantitative tests, establish R-LRP as a leading framework for neural network interpretability in modern deep learning.