IRDS: Interpretable RLVR Data Selection via Verifier-Coupled Sparse Autoencoder Coverage

Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a key technique for en- hancing LLM reasoning, yet its data ineffi- ciency remains a major bottleneck. Existing methods address this problem only partially, each missing at least one of subset-level cov- erage, verifier signal use, or interpretability. To address this gap, we present IRDS (Inter- pretable RLVR Data Selection), which selects RLVR training instances on a sparse autoen- coder (SAE) cluster basis so the selection itself is auditable on recognizable problem motifs. To select instances the model both fails on and can still learn from, we introduce a verifier- coupled coverage objective on the SAE basis and solve it by greedy log-determinant max- imization. Experiments on three instruction- tuned models and six math reasoning bench- marks show that IRDS achieves the highest overall accuracy, exceeding the strongest base- line by +3.9/+4.0 pp on the two Qwen models and by +0.5 pp on Llama-3.1-8B, while run- ning an order of magnitude cheaper than the trajectory-based baseline.

• The paper introduces the IRDS pipeline that selects RLVR training data using verifier-coupled sparse autoencoder clusters to balance failure and trainability signals.
• It demonstrates significant accuracy improvements across math reasoning benchmarks with lower computational cost compared to trajectory-based approaches.
• The method enhances interpretability and auditability by disentangling difficulty and trainability via a D-optimal subset selection process.

IRDS: Interpretable RLVR Data Selection via Verifier-Coupled Sparse Autoencoder Coverage

Motivation and Problem Formulation

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as the dominant paradigm for post-training LLMs on mathematical reasoning, leveraging programmatic verifiers to enable scalable reward feedback without human intervention. Despite its empirical successes, RLVR remains data-inefficient due to suboptimal training instance selection: instances where the current policy always fails or already succeeds yield zero or vanishing group-relative gradients, and overlapping weaknesses waste the selection budget on redundant corrections.

Previous approaches typically rely on pointwise scalar utilities, learned instance embeddings without verifier signal, or dense gradient proxies, but none combine subset-level coverage, explicit verifier feedback, and interpretability. The paper "IRDS: Interpretable RLVR Data Selection via Verifier-Coupled Sparse Autoencoder Coverage" (2605.28247) introduces the IRDS pipeline, which selects RLVR training subsets through an interpretable, verifier-coupled coverage objective on a sparse autoencoder (SAE) cluster basis, making both the selection and auditability tractable.

Figure 1: IRDS pipeline—data characterization, SAE-cluster encoding, verifier-coupled geometric construction, and D-optimal coreset selection for RLVR training.

Methodology

SAE Cluster-Based Representation

IRDS leverages BatchTopK-Ortho sparse autoencoders to encode instances into high-dimensional, semantically meaningful latent coordinates. These latents are clustered into $F=256$ semantic groups, each corresponding to recognizable math motifs (e.g., geometry, divisor counting, MCQ templates). This enables selection over interpretable content directions rather than superficial features like length or notation density, with each cluster admitting auditability.

Figure 2: BatchTopK-Ortho SAE training loss profiles across models and sparsities, demonstrating efficient reconstruction and semantic consistency.

Verifier-Coupled Coverage Objective

Each instance is assigned two weights derived from the base policy's verifier rollouts: the difficulty weight ($d_i$), highest where the policy currently fails; and the trainability weight ($r_i$), highest where rollouts yield maximal within-group advantage variance. Covariance matrices $\Sigma_d$ (difficulty) and $\Sigma_r$ (trainability) are computed over SAE clusters, and the ridge-regularized metric $M$ is formed as $$(\Sigma_r+\rho I)^{-1/2} (\Sigma_d+\rho I) (\Sigma_r+\rho I)^{-1/2}$$. The top directions of $M$ concentrate difficulty relative to trainability, enabling selection of non-redundant, improvable, and diverse instances.

D-Optimal Subset Selection

Instances are scored via their design vectors (weighted SAE coordinates and, optionally, residualized gradient block signatures) and selected by greedy maximization of log-determinant information gain—a D-optimal design—which penalizes redundancy (overlapping weaknesses) and maximizes actionable coverage of policy improvement directions. Fast rank-one Sherman–Morrison updates enable practical scaling to large corpora.

Experimental Results

Accuracy and Cost Efficiency

On three instruction-tuned models (Qwen3-4B, Qwen3-1.7B, Llama-3.1-8B-Instruct) and six math reasoning benchmarks, IRDS achieves highest overall mean@$16$ accuracy at every budget, with improvements of $+3.9/+4.0$\,pp for the Qwen models and $d_i$0\,pp for Llama-3.1-8B, compared to the strongest baselines. On competition subsets (AIME/AMC/OlympiadBench) individual lifts are even more pronounced, e.g., $d_i$1\,pp on AIME24 for Qwen3-4B.

IRDS selection is one order of magnitude cheaper than trajectory-based baselines like LIMR, requiring only a single offline probe of the base policy and fast parallelizable operations, rather than expensive RL trajectory replay.

Component Ablation

The method’s gains derive from its structural formulation: ablating the trainability weight leads to catastrophic accuracy drops; the difficulty weight provides fine-grained cluster coverage; whitening and D-optimal set-level objective yield significant margins over pointwise utilities; the gradient block is an optional refinement.

Out-of-Domain Transfer

IRDS maintains comparable performance to other baselines on out-of-domain tasks (HumanEval, LiveCodeBench, ReClor), with minimal regression ($d_i$2\,pp) from random or dense baselines, thus not overfitting to mathematical reasoning at the expense of broader generalization.

Interpretability and Auditability

Selection reflects true content diversity rather than surface artifacts: leading $d_i$3-eigenvectors localize on motifs with low success rates, clusters are semantically stable across base models, and regression on superficial features explains negligible variance in instance selection. A label-shuffle falsification collapses leading metric structure, confirming verifiability dependence.

SAE Spectrum Sensitivity

Figure 3: SAE cluster spectrum amplification: logit-shift magnitudes distributed across all 256 eigenvectors, indicating broad sensitivity to content axes.

A full-spectrum scan reveals that coverage mass is broadly distributed, not concentrated solely in the leading directions—justifying full-basis D-optimal selection rather than pruning to top K.

Practical and Theoretical Implications

IRDS advances RLVR data selection in two previously unattained directions: it achieves subset-level, verifier-aware coverage with interpretable semantic coordinates, and it does so with drastically improved computational efficiency. The method's explicit failure/trainability disentanglement streamlines RLVR subset selection, opening robust avenues for further auditability and semi-online refresh paradigms. Mechanistically, its approach aligns well with interpretability research in LLMs, leveraging monosemantic representations for actionable diagnostics [bricken2023monosemanticity], and activates directions of mathematical content where current policies are weakest yet most trainable.

Practically, IRDS unlocks scalable RLVR post-training for resource-constrained environments and enables transparent oversight of selection criteria, critical for high-stakes domains. Theoretically, its geometric coverage formulation motivates new metrics for subset-level curriculum learning and opens cross-domain applications where verifiable signals are available.

Speculation on Future Directions

Potential future developments include semi-online weight refresh to align selection with evolving failure profiles, expansion to non-mathematical RLVR domains, integration with other programmatic verifier schemes, and modular hybridization with model-dependent activation analyses. The interpretability-first design philosophy may catalyze broader adoption of spectrum-level selection in RL pipelines, especially for specialized skills emergence and auditability assurances in LLM post-training.

Conclusion

IRDS operationalizes interpretable, verifier-aware RLVR data selection via sparse-autoencoder cluster coordinates and a novel coverage metric, achieving superior accuracy, diversity, and auditability at low computational cost. Its architecture enables both practical deployment and theoretical analysis of subset-level RL effectiveness, establishing a new standard for interpretable RLVR selection methods (2605.28247).