Remove and Retrain (ROAR)

• ROAR is a retraining-based protocol that evaluates feature importance by measuring accuracy drops after the top-ranked features are removed and models are retrained.
• The method applies across domains such as image attribution, tabular data analysis, EEG saliency, and Tsetlin Machine scoring, showcasing its versatility.
• ROAR involves ranking features, iteratively removing them, retraining models, and comparing metrics like accuracy decrease or AUC to assess explainer quality.

Searching arXiv for ROAR-related papers to ground the article. Remove and Retrain (ROAR), often stylized RemOve-And-Retrain, is a retraining-based protocol for evaluating feature-importance and attribution methods. The procedure ranks features or pixels by an explainer, removes the highest-ranked subset from the data, retrains a fresh model on the modified training set, and measures the resulting performance, typically through accuracy or an accuracy-drop curve. Because the model is retrained after removal, ROAR is intended to test whether the removed features carried task-relevant signal rather than merely inducing inference-time ablation artifacts; it has been instantiated in image attribution, tabular feature selection, EEG saliency analysis, and Tsetlin Machine (TM) feature scoring (Song et al., 2023, Halenka et al., 9 Aug 2025).

1. Formalization and intended interpretation

In its generic tabular formulation, let $f$ be a model trained on features $X=\{x_1,\ldots,x_n\}$, let $A(f)$ denote an evaluation metric on held-out data, and let an XAI method rank features as $f_{(1)},f_{(2)},\ldots,f_{(n)}$ in descending order of importance. For $k=1,\ldots,n$, define $S_k=\{f_{(1)},\ldots,f_{(k)}\}$. ROAR removes $S_k$, retrains the model on $X\setminus S_k$, and measures performance after retraining. A standard summary is

$$A_0=A(f),\qquad A_{-S_k}=A\bigl(f\text{ retrained on }X\setminus S_k\bigr),$$

with drop

$\Delta_{\mathrm{ROAR}(k)}=A_0-A_{-S_k}.$

One may also report the relative drop or simply inspect how sharply $X=\{x_1,\ldots,x_n\}$0 declines as $X=\{x_1,\ldots,x_n\}$1 increases (Salih, 31 Jan 2025).

In attribution-map settings, ROAR is often defined per sample rather than through a single global feature ranking. A trained classifier $X=\{x_1,\ldots,x_n\}$2 produces predictions, an attribution method $X=\{x_1,\ldots,x_n\}$3 yields a saliency map, and for each drop-rate $X=\{x_1,\ldots,x_n\}$4 one constructs a mask of the top-$X=\{x_1,\ldots,x_n\}$5 attributed coordinates, zero-imputes those coordinates, retrains a fresh model on the masked training set, and evaluates it on the correspondingly masked test set. In that formulation, a “better” attribution method is expected to yield a steeper collapse of retained accuracy as $X=\{x_1,\ldots,x_n\}$6 grows (Song et al., 2023, Park et al., 2024).

A closely related TM-specific formulation replaces per-sample masks with a global ranking over the $X=\{x_1,\ldots,x_n\}$7 input features. After training one TM on the full feature set, the method computes a scalar score $X=\{x_1,\ldots,x_n\}$8 for each feature, sorts the features into a permutation $X=\{x_1,\ldots,x_n\}$9, defines retained feature sets $A(f)$0, removes the top $A(f)$1 features from both training and test data, retrains a fresh TM with the same hyperparameters, and records the resulting test accuracy $A(f)$2 (Halenka et al., 9 Aug 2025).

2. Procedural instantiations across model classes

The cited literature uses ROAR at materially different granularities. In TM feature selection, the protocol is fully global: one baseline TM is trained on all $A(f)$3 features, the ranking is computed once, and the same ranked prefixes are removed for every sample. In the comparative TM study, this was done on 12 UCI and synthetic datasets; continuous features were binned into 10 thermometer levels; the TM used 500 clauses and 30 training epochs; and for each $A(f)$4, 10 independent TMs were retrained from scratch with the same $A(f)$5, thermometer bins, and epoch count, after which the 10 test accuracies were averaged to obtain $A(f)$6 (Halenka et al., 9 Aug 2025).

In EEG emotion recognition, the masking object is not a per-feature global score but a group-level saliency map. Single-trial relevance maps $A(f)$7 are first computed with methods such as Smooth-Grad Squared, LRP-B, PatternNet, or Pattern-Attribution. These maps are averaged into a group-level map

$A(f)$8

which is then thresholded into a binary keep-mask $A(f)$9 for removal fractions $f_{(1)},f_{(2)},\ldots,f_{(n)}$0. A fresh CNN is retrained under leave-one-trial-out cross-validation on inputs $f_{(1)},f_{(2)},\ldots,f_{(n)}$1, and accuracy is averaged across held-out trials and subjects (Mayor-Torres et al., 2021).

In CNN attribution benchmarking for natural images, ROAR is frequently input-specific. For each image, the top $f_{(1)},f_{(2)},\ldots,f_{(n)}$2 fraction of pixels according to the attribution method is set to zero, yielding a modified training set $f_{(1)},f_{(2)},\ldots,f_{(n)}$3. A fresh network is then trained from random initialization on $f_{(1)},f_{(2)},\ldots,f_{(n)}$4, and its classification accuracy is measured after retraining. Khakzar et al. applied this procedure on CIFAR-10 with a ResNet-8 and on BirdSnap with a ResNet-50, repeating each $f_{(1)},f_{(2)},\ldots,f_{(n)}$5 configuration three times (Khakzar et al., 2019).

These differences matter. ROAR is not a single masking recipe but a family of retraining-based evaluations whose behavior depends on whether ranking is global or per-sample, whether masking is feature deletion or zero-imputation, and whether evaluation is performed on reduced or original test inputs. A plausible implication is that numerical ROAR outcomes are only directly comparable within a fixed operational definition.

3. Performance summaries and empirical behavior

The basic ROAR observable is the retained performance curve after iterative removal. In the TM study, the primary per-step metric is test accuracy $f_{(1)},f_{(2)},\ldots,f_{(n)}$6, plotted as a function of the number of removed features. To summarize a curve by a single number, the paper computes the area under the pruning curve,

$f_{(1)},f_{(2)},\ldots,f_{(n)}$7

Methods are then compared by AUC and by how often they appear among the top-5 AUC scores across datasets (Halenka et al., 9 Aug 2025).

These curves can reveal dataset structure as much as explainer quality. In the TM experiments, the Digits dataset showed that many methods tolerate removal of a large fraction of pixels with only a small loss in accuracy, which the authors interpret as reflecting feature redundancy in the 64-pixel input. On Transfusion, by contrast, accuracy dropped sharply once the top 1–3 features were removed, indicating a small number of highly informative features (Halenka et al., 9 Aug 2025).

In deep-image attribution benchmarks, ROAR is typically interpreted through steepness of accuracy decline. On CIFAR-10 and BirdSnap, PruneGrad and PruneGrad-Mid produced lower retained accuracies than VanillaGrad, Integrated Gradients, GuidedBackProp, RectGrad, and GradCAM across removal fractions from $f_{(1)},f_{(2)},\ldots,f_{(n)}$8 to $f_{(1)},f_{(2)},\ldots,f_{(n)}$9, which in that benchmark was taken as evidence that input-specific pruning identifies features more critical to retrained models (Khakzar et al., 2019).

In EEG emotion recognition, all four tested saliency methods exhibited monotonically decreasing accuracy as the removal rate $k=1,\ldots,n$0 increased, but the steepness differed. Smooth-Grad Squared and Pattern-Attribution caused the earliest and largest drops at modest $k=1,\ldots,n$1, whereas LRP-B showed the largest degradation for ASD subjects at $k=1,\ldots,n$2–$k=1,\ldots,n$3. XAI-guided removals also yielded significantly lower accuracy than two random baselines at every $k=1,\ldots,n$4 in both groups, with statistical validation by one-way ANOVAs with Bonferroni–Holm correction (Mayor-Torres et al., 2021).

4. Computational burden and practical trade-offs

ROAR’s most persistent practical characteristic is retraining cost. The TM study makes this explicit: for each removal level $k=1,\ldots,n$5, one must train 10 full TMs, each for 30 epochs, and repeat this for every feature-selection method. The authors recorded both ranking times and retraining times to analyze speed–quality trade-offs, and concluded that retraining time dominates total cost, especially when all $k=1,\ldots,n$6 are swept (Halenka et al., 9 Aug 2025).

The cost interacts with the choice of scoring method. Under ROAR in the TM setting, filter methods such as Chi2, MutualInfo, and Variance are essentially instantaneous to rank but plateau at moderate AUC; Permutation-Importance achieves slightly better AUC but at 10–100× the ranking time of filters, plus the cost of retraining for each $k=1,\ldots,n$7; and TM-embedded scorers such as Stability-PosNeg and CW-Sum occupy a middle ground, offering AUC close to wrappers at only a fraction of the runtime because they require no external perturbations or surrogate training loops (Halenka et al., 9 Aug 2025).

The same computational limitation appears in vision. Khakzar et al. note that, because each attribution-method-by-removal-fraction pair requires retraining from scratch, ROAR is too expensive to run at ImageNet scale, which is why their experiments are confined to CIFAR-10 and BirdSnap (Khakzar et al., 2019).

The literature therefore treats ROAR as a high-fidelity but high-cost benchmark. In the TM discussion, one recommendation is to use simple filter pre-screening to reduce $k=1,\ldots,n$8 before running ROAR with more expensive scorers; another is to prefer TM-embedded scorers such as CW-Sum or TM-Weight when one wants low-overhead rankings that still capture clause interactions (Halenka et al., 9 Aug 2025).

5. Failure modes, theoretical criticism, and common misconceptions

A common interpretation is that a steeper ROAR curve necessarily indicates a more faithful attribution. The theoretical critique of Song et al. challenges that reading. Building on the observation that, under mild assumptions, ROAR correlates inversely with the mutual information $k=1,\ldots,n$9 between modified inputs and labels, the paper argues that ROAR can reward transformations that destroy more label information even when they contain less information about the original explainer or decision function. In their structural causal account, any degradation $S_k=\{f_{(1)},\ldots,f_{(k)}\}$0 of an attribution map satisfies the Data Processing Inequality

$S_k=\{f_{(1)},\ldots,f_{(k)}\}$1

yet a degraded attribution can still produce lower retained accuracy after retraining and thus a “better” ROAR score (Song et al., 2023).

The empirical consequence is what the paper calls blurriness bias. On CIFAR-10, SVHN, and CUB-200, applying a model-agnostic max-pool filter or Gaussian blur to attribution maps caused a larger accuracy drop under ROAR than the unprocessed attribution for nearly every method. On CUB-200 at $S_k=\{f_{(1)},\ldots,f_{(k)}\}$2, for example, Grad$S_k=\{f_{(1)},\ldots,f_{(k)}\}$3 fell from $S_k=\{f_{(1)},\ldots,f_{(k)}\}$4 to $S_k=\{f_{(1)},\ldots,f_{(k)}\}$5, IG$S_k=\{f_{(1)},\ldots,f_{(k)}\}$6 from $S_k=\{f_{(1)},\ldots,f_{(k)}\}$7 to $S_k=\{f_{(1)},\ldots,f_{(k)}\}$8, and VG from $S_k=\{f_{(1)},\ldots,f_{(k)}\}$9 to $S_k$0 when Gaussian blur was applied to the attribution map before masking. The same study reported a strong positive linear correlation between total variation $S_k$1 and post-ROAR test accuracy, with $S_k$2 on CUB-200 at $S_k$3: sharper maps retained higher accuracy and therefore looked worse under ROAR, while blurrier maps retained lower accuracy and therefore looked better (Song et al., 2023).

A second failure mode concerns redundancy and multicollinearity. The revisit paper argues that ROAR assumes feature independence and monotonic model behavior, assumptions that break down when several features carry overlapping signal or when baseline accuracy is low. Their empirical examples include cases where removing the top feature produced only a mild effect, no effect, or even improved performance. In simulated classification, removing $S_k$4, $S_k$5, or $S_k$6 left F1 unchanged at $S_k$7 for the first three iterations; in the CDC Diabetes example, removing Stroke at iteration 17 increased F1 from $S_k$8 to $S_k$9 (Salih, 31 Jan 2025).

These results do not imply that ROAR is unusable. They do imply that ROAR measures the combined effect of attribution quality, data redundancy, masking geometry, and retraining dynamics. This suggests that ROAR is best interpreted as a perturbation-and-retraining stress test rather than as a standalone ground-truth oracle for faithfulness.

6. Variants, extensions, and complementary metrics

Several later proposals modify or supplement ROAR rather than discard it. ROAD (Remove-And-Debias), discussed alongside ROAR in the blurriness-bias paper, shares the same data-generation DAG and exhibits the same effect under the tested post-processings; in that account, ROAD does not cure blurriness bias (Song et al., 2023).

GOAR, or Geometric Remove-and-Retrain, changes the perturbation model itself. Instead of zeroing coordinates along the ambient basis, GOAR perturbs each sample along its attributed feature direction,

$X\setminus S_k$0

then projects the perturbed point back onto the data manifold with a diffusion-model-based projector,

$X\setminus S_k$1

A fresh classifier is retrained on $X\setminus S_k$2, and the benchmark records the cumulative number of misclassified examples as $X\setminus S_k$3 varies. The central claim is geometric: because the perturbation is aligned with $X\setminus S_k$4 rather than with coordinate axes, the pipeline is coordinate-free and avoids ROAR’s lack of coordinate invariance and its inability to discriminate between attributions that differ only by permutation of large coordinates (Park et al., 2024).

The reported experiments are explicitly framed as a remedy to ROAR’s failure modes. On a synthetic Gaussian-mixture toy problem, ROAR and ROAD produced nearly flat curves across attribution quality levels, whereas GOAR’s misclassification count peaked when the attribution direction aligned with the true discriminative direction. On OpenXAI tabular benchmarks, GOAR achieved the highest positive correlations with six OpenXAI scores, while ROAR/Eval-X correlations were near zero or negative; on MNIST, Fashion-MNIST, CIFAR10, and tabular UCI sets, GOAR separated gradient-based methods that ROAR and ROAD failed to distinguish (Park et al., 2024).

A different line of work proposes a complementary uncertainty-aware statistic rather than a new perturbation geometry. Expected Accuracy Interval (EAI) predicts upper and lower bounds on post-removal accuracy from the initial model accuracy and the XAI-derived proportion of contribution of the Single Most Significant Feature: $X\setminus S_k$5

$X\setminus S_k$6

In the reported examples, EAI brackets post-removal outcomes that appear anomalous under a naive ROAR reading, especially when features are collinear or when many informative features share variance (Salih, 31 Jan 2025).

Taken together, these extensions recast ROAR from a definitive ranking criterion into one component of a broader evaluation toolkit. GOAR alters the perturbation geometry, EAI adds uncertainty bounds, and the critiques around ROAD emphasize that retraining alone does not immunize perturbation-based benchmarks against biases induced by the perturbation mechanism itself.