Single-Rollout Hidden-State Dynamics for Training-Free RLVR Data Selection

Abstract: Reinforcement learning with verifiable rewards (RLVR) can yield large reasoning gains from very few training instances, yet its strong sensitivity to which instances are used makes data selection a central bottleneck. Most existing selection pipelines rely on training-time optimization signals and/or require access to verifiable rewards or ground-truth answers over large candidate pools, which is costly and often infeasible in specialized domains. We study RLVR data selection in a setting where selection must be performed before any RL training and without labels or reward evaluation on the full pool. We propose SHIFT, a one-shot, training-free selector based solely on inference-time hidden-state dynamics. For each candidate instance, SHIFT runs a single deterministic reasoning rollout and computes a reasoning-induced representation shift (RIRS) as the start-to-end hidden-state delta. SHIFT uses the RIRS magnitude as a lightweight proxy for instance utility and enforces coverage via a quality-weighted farthest-first CoreSet procedure in an RIRS-augmented feature space, producing compact subsets that scale to large unlabeled pools. Across mathematical reasoning and medical QA benchmarks under ultra-low budgets, SHIFT consistently outperforms training-free diversity and difficulty/uncertainty baselines, improving both in-domain accuracy and transfer to harder evaluation settings. Ablations show that RIRS-based coverage and quality-weighting contribute complementary gains, and analyses indicate that RIRS is not explained by simple input/output length statistics. Code is available at github.com/JianghaoWu/SHIFT.

• The paper introduces SHIFT, a one-shot method leveraging hidden-state dynamics to select high-utility examples for training-free RLVR.
• It employs a quality-weighted farthest-first CoreSet algorithm to maximize data diversity and instance utility without reward signals.
• Experimental results demonstrate improved in-domain and OOD performance on mathematical and medical QA tasks while reducing labeling costs.

Training-Free RLVR Data Selection via Single-Rollout Hidden-State Dynamics

Introduction and Motivation

This paper addresses a central challenge in reinforcement learning with verifiable rewards (RLVR) for LLMs: highly efficient data selection under extreme supervision scarcity. Prior work has established that RLVR can yield strong reasoning improvements from just a handful of impactful examples, but the identification of such examples—especially without access to labels, rewards, or training-time signals—remains intractable in large, domain-specific pools. The authors introduce SHIFT, a novel one-shot, training- and label-free data selector, which leverages the inference-time hidden-state dynamics of LLMs to compute a reasoning-induced representation shift (RIRS) and perform subset selection efficiently and scalably.

The core insight is that the magnitude of the start-to-end hidden-state change observed in a single deterministic reasoning rollout acts as a proxy for an instance's utility for subsequent RLVR optimization. SHIFT produces high-utility, diverse subsets using a quality-weighted farthest-first CoreSet algorithm in an RIRS-augmented feature space, operating without any reward estimation or multiple rollouts per candidate.

Figure 1: RLVR selection regimes: prior approaches require label or reward signals, whereas SHIFT enables one-shot, label-free, training-free selection via inference-time representation shifts.

Methodology

Hidden-State Dynamics, RIRS, and Selection Procedure

SHIFT formalizes data selection for RLVR in a pre-training, pre-labeling regime, assuming only access to an unlabeled candidate pool and the base model. For each instance, a single greedy rollout is generated; hidden states at the start and end of the reasoning segment are extracted (e.g., delimited by > and tokens). These are aggregated across layers and the difference defines the reasoning-induced representation shift:

$\Delta(x) = \mathbf{e}(x) - \mathbf{s}(x)$

where $\mathbf{s}(x)$ and $\mathbf{e}(x)$ are the averaged start and end hidden states, respectively, across all transformer layers for instance $x$.

The $\ell_2$ norm, optionally log-transformed, is used as a utility score $q(x) = \log(1 + \|\Delta(x)\|_2)$, hypothesized to reflect the "learning potential" an instance confers during RLVR training. Theoretical underpinning is drawn from recent work interpreting transformer computation as implicit weight updates controlled by context-induced representation changes, validating the use of observable hidden-state shifts as selection proxies.

To ensure diversity and comprehensive coverage of the feature space, SHIFT concatenates $\mathbf{s}(x)$ and $\Delta(x)$, normalizes this vector, and executes a quality-weighted farthest-first CoreSet greedy selection. This balances instance utility and coverage, selecting instances one-by-one to maximize $\tilde{q}(x) \cdot d(x, S)$. All selection computation is completed in one inference-time pass (single rollout per candidate), making the approach highly scalable.

Experimental Results

Mathematical Reasoning: In-Domain and OOD Generalization

In stringent low-budget regimes on MATH-500 (2–4% of pool), SHIFT consistently outperforms all training-free baselines (diversity-based KMeans/CoreSet, difficulty proxies such as perplexity, and uncertainty-based selectors like self-consistency entropy):

• At 2% data, SHIFT achieves 62.67% Pass@1 in-domain and 38.55% OOD (AMC), compared to 53.73% / 25.78% for Random and substantially lower for pure diversity selectors.
• Even with a tiny training set, SHIFT's OOD generalization on AMC surpasses the full-data RLVR reference (38.55% vs 33.73%).

The findings hold as the training budget expands; gains over other methods decrease but remain significant, especially for in-domain accuracy.

Medical QA and Clinical Transfer

On MedQA and associated transfer benchmarks (MedMCQA, PubMedQA, MedXpertQA), SHIFT delivers the best accuracy under 0.1–0.2% selection budgets:

• MedQA: 50.35% (SHIFT) vs 45.28% (Random) at 0.1% budget
• Robust transfer: Outperforms all baselines on MedXpertQA subsets, with gains especially pronounced on clinical reasoning (e.g., 12.22% on MedXpertQA-U nearly matches the full-data RLVR result).

Ablation and Analysis

Ablation studies highlight:

• The synergy of quality-weighted selection and coverage in RIRS-augmented space; neither component alone matches SHIFT's full performance.
• RIRS is not a proxy for simple input/output length statistics.

  Figure 2: RIRS magnitude is weakly correlated with question length and essentially independent from response length, confirming non-trivial alignment with instance utility.

A direct per-instance evaluation on MATH shows RIRS rank correlates strongly with RLVR-induced gain (Spearman's ρ = 0.818).

Extensive cross-architecture evaluation on Llama-3-8B and Olmo-3-7B-Instruct verifies that the RIRS-based selection policy is robust across model families and scales, with consistent accuracy advantages over other training-free selectors.

Theoretical and Practical Implications

SHIFT introduces a principled, scalable criterion for instance selection in RLVR without the prohibitive cost of reward evaluation or multiple rollouts. The proxy is motivated by transformer block dynamics, and empirical results validate its informativeness for downstream adaptation, in both standard and OOD settings.

Important practical consequences include:

• Strong reductions in annotation and compute required to adapt LLMs for complex reasoning tasks.
• Utility for sensitive or specialized domains (e.g., medical QA) where training data and reward signals are costly or unavailable.
• The RIRS signal is lightweight (linear cost) and not explained by superficial input statistics, making it more robust for diverse pools.
• The approach is compatible with multiple model architectures and can be generalized across a range of reasoning domains.

Limitations and Future Work

The main limitations include the gap between block-level theory and trajectory-level metric, reliance on a single static pre-training rollout (potentially brittle in presence of degenerate completions), and the yet unproven generalization to even larger models or novel domains. Addressing these may require layer-aware RIRS aggregation, dynamic or adaptive multi-rollout proxies, and broader cross-domain validation.

Future research could explore layer-specific weighting, richer token-level dynamics, as well as integration with domain-specific reward estimation and safety auditing strategies.

Conclusion

SHIFT establishes a scalable, training- and label-free solution to data selection for RLVR adaptation of LLMs. By operationalizing a theoretically motivated, computation-efficient hidden-state dynamics proxy, it reliably selects impactful examples from large unlabeled pools, outperforming standard diversity and uncertainty heuristics across mathematical and medical domains. The evidence indicates that RIRS-based selection holds promise for effective and accessible LLM adaptation in low-resource, high-value settings.