Modified RISE-eval in Speaker Recognition

• The paper introduces Modified RISE-eval, a deletion-only evaluation method that mitigates overmasking in sparse attention maps for speaker recognition.
• It applies thresholding with achieved mask ratios and early-drop–weighted scoring to assess GradCAM and LayerCAM explanations from a ResNet34 model on mel spectrograms.
• Empirical findings reveal layer-dependent differences, indicating shallow maps cause steeper performance drops, while deeper maps provide more class-discriminative localization.

Modified RISE-eval is a principled modification of the Randomised Input Sampling for Explanation – Evaluation algorithm, proposed for quantitative evaluation of attention maps in speaker recognition systems. It is designed to make perturbation-based evaluation more discriminative and less confounded than earlier RISE-eval variants by restricting evaluation to deletion, thresholding attention maps before masking, tracking the achieved rather than requested mask ratio, and replacing area-under-curve scoring with an early-drop–weighted score that emphasizes rapid performance degradation at small mask ratios. In the reported study, Modified RISE-eval is applied to GradCAM and LayerCAM explanations of a ResNet34-based speaker recognizer operating on mel spectrograms, and it reveals layer-dependent differences that are not exposed by earlier insertion-based evaluations (Xu et al., 22 Jun 2026).

1. Conceptual setting and motivation

Modified RISE-eval arises in the broader context of explainable AI for neural speaker recognition. In this setting, the model input is a $2$-D time–frequency representation, typically a mel spectrogram, and a CAM-based explanation method produces an attention map $\widetilde{A} \in \mathbb{R}^{h \times w}$ whose values are interpreted as indicating which time–frequency regions are selectively processed when the network identifies a speaker. The central evaluation assumption is perturbation-based: if an attention map correctly identifies regions that truly influence the model’s decision, then perturbing or removing those regions should cause a large degradation in model performance (Xu et al., 22 Jun 2026).

The original RISE work framed explanation quality through map-guided deletion and insertion curves, computed by progressively removing or revealing high-saliency pixels and measuring the effect on model output (Petsiuk et al., 2018). In the speaker-recognition adaptation reviewed by the Modified RISE-eval paper, these ideas were instantiated at the dataset level by tracking classification accuracy as a function of the mask ratio. That adaptation exposed two difficulties. First, insertion-mode curves for GradCAM++, LayerCAM, and ScoreCAM were reported to be almost indistinguishable, which made the evaluation non-discriminative despite substantial computational cost. Second, forcing masking to continue until $100\%$ of pixels had been processed caused late-stage masking to drift into low-valued or near-zero regions, so the evaluation became contaminated by effectively random masking rather than by the structure of the attention map itself (Xu et al., 22 Jun 2026).

These issues are especially acute for sparse CAMs. In such maps, a relatively small set of high-valued regions carries most of the explanatory mass, whereas the remaining regions are weakly weighted. Once the high-valued support has been exhausted, any ranking among the remaining near-zero regions is largely arbitrary. Modified RISE-eval therefore reformulates the evaluation so that it remains tied to the portion of the map that the explainer actually identifies as important.

2. Relation to original RISE-eval

The original RISE paper introduced a black-box saliency method based on randomized masking and also proposed automatic explanation metrics based on deletion and insertion trajectories (Petsiuk et al., 2018). In that framework, pixels are sorted by importance, then progressively deleted from the original image or inserted into a blurred baseline, and the model’s class score is tracked as the fraction of modified pixels increases. The resulting curve is summarized by an area under the curve. For deletion, a steeper drop and smaller AUC indicate a better explanation; for insertion, a steeper rise and larger AUC indicate a better explanation (Petsiuk et al., 2018).

The speaker-recognition literature discussed in the Modified RISE-eval study distinguishes two practical formulations of RISE-eval. In the Petsiuk et al. image setting, each input yields a per-example curve

$$c_{\mathrm{Petsiuk}}\!\left(\frac{(j+1)\,n_{\mathrm{step}}}{wh}; \widetilde{A}_i\right) = f\big(\widetilde{X}_i[j]\big)[y_i],$$

where $f(\widetilde{X}_i[j])[y_i]$ is the probability assigned to the target class. In the Li et al. speaker-recognition adaptation, the curve is defined at the dataset level: $$c_{\mathrm{Li}}\!\left(\frac{(j+1)\,n_{\mathrm{step}}}{wh}; \widetilde{\mathbb{A}}\right) = \frac{1}{n}\sum_{i=1}^n \mathbf{1}\!\left[\arg\max f(\widetilde{X}_i[j]) = y_i\right],$$ so the ordinate is classification accuracy rather than per-instance class probability (Xu et al., 22 Jun 2026).

The paper identifies two limitations in this speaker-recognition setting. The first is the empirical failure of insertion-mode evaluation to separate methods: LayerCAM, GradCAM++, and ScoreCAM produced insertion curves that “practically overlap.” The second is overmasking. Because original RISE-eval proceeds until all pixels have been sampled, later iterations necessarily operate on regions with negligible attention values. The evaluation then ceases to test the explanation’s declared important regions and instead tests random residual masking, which can dominate the AUC and obscure true differences in attention-map fidelity (Xu et al., 22 Jun 2026).

Modified RISE-eval preserves the perturbation-based spirit of RISE-eval but rejects the assumption that the entire $[0,1]$ mask-ratio interval is always informative. Its core premise is that only the above-threshold support of the map should govern evaluation.

3. Modified algorithm and mathematical formulation

Modified RISE-eval introduces four explicit design changes. Only deletion is retained. Attention values are thresholded so that only pixels with $\widetilde{A}[u,v] \ge t$ are eligible for masking. The x-axis of the masking-performance curve is changed from the desired sampling ratio $r^{\text{samp}}$ to the achieved mask ratio $r^{\text{mask}}$, which may be smaller because of thresholding. Finally, AUC is replaced by a score that weights earlier performance drops more strongly than later ones (Xu et al., 22 Jun 2026).

For a single input $\widetilde{A} \in \mathbb{R}^{h \times w}$0 and attention map $\widetilde{A} \in \mathbb{R}^{h \times w}$1, with threshold $\widetilde{A} \in \mathbb{R}^{h \times w}$2 and desired sampling ratio $\widetilde{A} \in \mathbb{R}^{h \times w}$3, Modified RISE-eval constructs a binary mask $\widetilde{A} \in \mathbb{R}^{h \times w}$4. Pixels satisfying

$\widetilde{A} \in \mathbb{R}^{h \times w}$5

are first marked as candidate pixels. If the number of candidates is at least $\widetilde{A} \in \mathbb{R}^{h \times w}$6, the algorithm resets $\widetilde{A} \in \mathbb{R}^{h \times w}$7 and selects the top $\widetilde{A} \in \mathbb{R}^{h \times w}$8 pixels with highest $\widetilde{A} \in \mathbb{R}^{h \times w}$9, setting

$100\%$0

Otherwise, it uses all candidate pixels and sets

$100\%$1

Deletion masking is then applied as

$100\%$2

The result is a masked input and an achieved mask ratio whose value reflects the actual size of the explanation’s above-threshold support (Xu et al., 22 Jun 2026).

At the dataset level, for each requested sampling ratio in a set $100\%$3, the algorithm averages the achieved mask ratios and computes classification accuracy over the masked inputs. This yields a curve $100\%$4, where $100\%$5 is average achieved mask ratio and $100\%$6 is dataset-level accuracy at that ratio. Because thresholding may exhaust all above-threshold pixels before the requested sampling ratios reach $100\%$7, the resulting curve naturally terminates at an “effective interval” shorter than the full unit interval (Xu et al., 22 Jun 2026).

The score is defined as

$100\%$8

An implementation detail adds a small $100\%$9 to the denominator when $$c_{\mathrm{Petsiuk}}\!\left(\frac{(j+1)\,n_{\mathrm{step}}}{wh}; \widetilde{A}_i\right) = f\big(\widetilde{X}_i[j]\big)[y_i],$$0. The numerator $$c_{\mathrm{Petsiuk}}\!\left(\frac{(j+1)\,n_{\mathrm{step}}}{wh}; \widetilde{A}_i\right) = f\big(\widetilde{X}_i[j]\big)[y_i],$$1 measures the drop in classification accuracy between two consecutive masking levels, while division by $$c_{\mathrm{Petsiuk}}\!\left(\frac{(j+1)\,n_{\mathrm{step}}}{wh}; \widetilde{A}_i\right) = f\big(\widetilde{X}_i[j]\big)[y_i],$$2 amplifies drops occurring at small mask ratios. The intended effect is to prioritize cases where removing only a small fraction of the most highly attended pixels causes the model’s performance to collapse, which the paper treats as a stronger indicator of explanation fidelity than late-stage degradation (Xu et al., 22 Jun 2026).

4. Experimental instantiation in speaker recognition

The experimental study uses a ResNet34-based CNN trained by Chung et al. on VoxCeleb2. The model takes 200-frame mel spectrograms corresponding to 2-second audio clips and had an Equal Error Rate of $$c_{\mathrm{Petsiuk}}\!\left(\frac{(j+1)\,n_{\mathrm{step}}}{wh}; \widetilde{A}_i\right) = f\big(\widetilde{X}_i[j]\big)[y_i],$$3 on VoxCeleb1 for speaker verification with 400-frame inputs. For the XAI study, activations are extracted from the outputs of all four ResNet layers, and gradients of the model’s ground-truth class score with respect to these activations are computed to construct GradCAM and LayerCAM maps (Xu et al., 22 Jun 2026).

For a convolutional layer with activations $$c_{\mathrm{Petsiuk}}\!\left(\frac{(j+1)\,n_{\mathrm{step}}}{wh}; \widetilde{A}_i\right) = f\big(\widetilde{X}_i[j]\big)[y_i],$$4, GradCAM is defined by

$$c_{\mathrm{Petsiuk}}\!\left(\frac{(j+1)\,n_{\mathrm{step}}}{wh}; \widetilde{A}_i\right) = f\big(\widetilde{X}_i[j]\big)[y_i],$$5

with channel weights

$$c_{\mathrm{Petsiuk}}\!\left(\frac{(j+1)\,n_{\mathrm{step}}}{wh}; \widetilde{A}_i\right) = f\big(\widetilde{X}_i[j]\big)[y_i],$$6

where $$c_{\mathrm{Petsiuk}}\!\left(\frac{(j+1)\,n_{\mathrm{step}}}{wh}; \widetilde{A}_i\right) = f\big(\widetilde{X}_i[j]\big)[y_i],$$7. LayerCAM is defined by

$$c_{\mathrm{Petsiuk}}\!\left(\frac{(j+1)\,n_{\mathrm{step}}}{wh}; \widetilde{A}_i\right) = f\big(\widetilde{X}_i[j]\big)[y_i],$$8

In both cases, the maps are resized to spectrogram resolution by bilinear interpolation and min–max normalized to $$c_{\mathrm{Petsiuk}}\!\left(\frac{(j+1)\,n_{\mathrm{step}}}{wh}; \widetilde{A}_i\right) = f\big(\widetilde{X}_i[j]\big)[y_i],$$9. For each input, this yields eight explanations: two CAM methods across four ResNet layers. The target class is always the ground-truth class of the spectrogram (Xu et al., 22 Jun 2026).

Modified RISE-eval is then applied to all method-layer combinations on the VoxCeleb1 test set. The reported threshold is $f(\widetilde{X}_i[j])[y_i]$0. Pixels with attention value below $f(\widetilde{X}_i[j])[y_i]$1 are excluded from masking. The requested sampling ratios are

$f(\widetilde{X}_i[j])[y_i]$2

Only deletion is used. For each combination, the algorithm produces a masking-performance curve of classification accuracy versus achieved mask ratio and then computes a single scalar score using the early-drop–weighted formula above (Xu et al., 22 Jun 2026).

5. Empirical findings

The reported masking-performance curves are layer dependent. In layer 1, within the overlapping region of their effective intervals, LayerCAM’s curve lies below GradCAM’s, meaning that accuracy is lower when masked according to LayerCAM regions at the same average mask ratio. The same pattern appears in layer 2. In layer 3, the two curves largely overlap, suggesting comparable explanation quality under this evaluation. In layer 4, the pattern reverses: GradCAM’s curve lies below LayerCAM’s, so removing regions highlighted by GradCAM at the deepest layer damages performance more strongly (Xu et al., 22 Jun 2026).

The scalar scores reported by Modified RISE-eval are as follows.

Method	Layer 1	Layer 2	Layer 3	Layer 4
LayerCAM	15.50	12.58	10.27	9.52
GradCAM	10.14	9.40	8.90	10.12

Higher scores indicate larger early drops in accuracy at small achieved mask ratios. On this basis, LayerCAM outperforms GradCAM in layers 1–3, whereas GradCAM outperforms LayerCAM in layer 4 (Xu et al., 22 Jun 2026).

The paper also reports a qualitative concatenation experiment in which two 100-frame spectrogram halves from different speakers are combined into a single 200-frame input. When the target class is set to one or the other speaker, last-layer GradCAM is described as strongly class-discriminative: when targeting speaker A in the left half, $f(\widetilde{X}_i[j])[y_i]$3 of attention-map energy lies on the left; when targeting speaker B in the right half, only $f(\widetilde{X}_i[j])[y_i]$4 lies on the left, implying $f(\widetilde{X}_i[j])[y_i]$5 on the right. First-layer maps from both methods are much less discriminative in this sense, with energy distributions near $f(\widetilde{X}_i[j])[y_i]$6–$f(\widetilde{X}_i[j])[y_i]$7 between halves (Xu et al., 22 Jun 2026).

A notable tension follows. Modified RISE-eval scores are highest at layer 1 for both methods, but the qualitative concatenation experiment suggests stronger class-discriminative localization at layer 4, especially for GradCAM. The paper treats this as an important observation rather than a contradiction-free ranking: perturbation-based fidelity under deletion and class-discriminative localization are related but not identical properties.

6. Limitations, biases, and broader applicability

Modified RISE-eval has several stated strengths. Thresholding and effective-range truncation address overmasking by ensuring that evaluation remains tied to regions the attention map actually marks as important. Deletion-only evaluation avoids the empirically non-discriminative insertion curves observed in earlier speaker-recognition experiments. The method is model-centric and annotation-free: it uses changes in the network’s own performance, rather than human region labels, to evaluate explanations (Xu et al., 22 Jun 2026).

The paper also identifies several limitations. First, the metric appears biased toward shallow-layer maps. The highest scores occur at layer 1 even though qualitative evidence indicates better class-discriminative localization in deeper layers. The proposed explanation is that shallow maps are fine-grained and align more strongly with acoustically active regions such as formant-rich structure, whereas deeper maps are coarser and may include silent or broader regions; at the same mask ratio, shallow maps therefore remove more acoustically informative content and induce larger performance drops. Second, the method is sensitive to hyperparameters: the threshold $f(\widetilde{X}_i[j])[y_i]$8 and the sampling-ratio schedule $f(\widetilde{X}_i[j])[y_i]$9 determine the effective interval and therefore affect the score. Third, masking spectrogram pixels can create out-of-distribution inputs, so some part of the measured degradation may reflect distribution shift rather than pure importance. Fourth, the experimental validation is restricted to one ResNet34-based speaker recognizer and one task family, so behavior may differ for transformer-based architectures, raw-waveform systems, or other audio tasks (Xu et al., 22 Jun 2026).

The broader applicability of Modified RISE-eval follows from its treatment of attention maps as generic $$c_{\mathrm{Li}}\!\left(\frac{(j+1)\,n_{\mathrm{step}}}{wh}; \widetilde{\mathbb{A}}\right) = \frac{1}{n}\sum_{i=1}^n \mathbf{1}\!\left[\arg\max f(\widetilde{X}_i[j]) = y_i\right],$$0-D objects. The paper explicitly notes that the method can be applied to other audio tasks using time–frequency representations, including automatic speech recognition, speech emotion recognition, acoustic scene classification, and anomaly detection. It also suggests direct applicability to image CNNs, vision transformers, multimodal models with aligned spatial features, and, by extension of the same principle, to $$c_{\mathrm{Li}}\!\left(\frac{(j+1)\,n_{\mathrm{step}}}{wh}; \widetilde{\mathbb{A}}\right) = \frac{1}{n}\sum_{i=1}^n \mathbf{1}\!\left[\arg\max f(\widetilde{X}_i[j]) = y_i\right],$$1-D time-series explanations, spatiotemporal $$c_{\mathrm{Li}}\!\left(\frac{(j+1)\,n_{\mathrm{step}}}{wh}; \widetilde{\mathbb{A}}\right) = \frac{1}{n}\sum_{i=1}^n \mathbf{1}\!\left[\arg\max f(\widetilde{X}_i[j]) = y_i\right],$$2-D maps for video, and attention matrices in transformers. This suggests that Modified RISE-eval is best understood not as a speaker-specific heuristic but as a thresholded, deletion-only, early-drop–weighted perturbation framework for evaluating explanation maps whenever the explanation support is sparse and late-stage masking would otherwise become effectively random (Xu et al., 22 Jun 2026).