1-shot RLVR: Data-Efficient Reinforcement Learning

• 1-shot RLVR is a reinforcement learning paradigm that uses a single demonstration and verifiable reward signals to unlock latent reasoning patterns in large models.
• It employs Group Relative Policy Optimization and policy gradient methods on duplicated examples to achieve substantial accuracy gains while controlling overfitting.
• The approach demonstrates significant improvements in mathematical and vision-language tasks, boosting accuracy by up to 26 percentage points in low-data regimes.

1-shot RLVR is a reinforcement learning paradigm for enhancing the reasoning abilities of pretrained LLMs and vision–LLMs (VLMs) using a single, carefully chosen training example in conjunction with verifiable, rule-based reward signals. In contrast to traditional supervised approaches demanding large annotated datasets, 1-shot RLVR leverages policy-gradient methods—often Group Relative Policy Optimization (GRPO)—to achieve substantial, generalizable performance gains from extreme low-data regimes. This methodology has demonstrated efficacy across mathematical reasoning, vision-language tasks (e.g., satellite imagery), and varied multimodal benchmarks, offering an efficient and pragmatic alternative for specialized domains where labeled data are scarce (Wang et al., 29 Apr 2025, Koksal et al., 29 Jul 2025).

1. Core Principles and Definitions

1-shot RLVR is defined by three distinguishing elements:

• Verifiable Reward: Rewards are assigned through automatic, lightweight binary or structured checks. For closed tasks, correctness is binary (e.g., answer equality). For grounding, metrics like Intersection over Union (IoU) provide quantized or continuous rewards.
• Single Example Supervision: Only one (x*, y*) pair serves as the demonstration. During training, all RLVR updates use this instance, duplicated as needed to match desired batch and group sizes. No ground-truth annotations are required for other data (Wang et al., 29 Apr 2025, Koksal et al., 29 Jul 2025, Shao et al., 12 Jun 2025).
• Policy-Gradient Optimization: The RL objective maximizes expected reward via policy gradients, using group normalization to stabilize sparse binary feedback. Empirically, the effectiveness arises not from coverage of the single example, but from surfacing dormant reasoning patterns within the pretrained model (Wang et al., 29 Apr 2025, Shao et al., 12 Jun 2025).

In some variants, especially for mathematical LLMs, “one-shot RLVR” includes rewarding any rollout matching the single demonstration answer, regardless of question (Shao et al., 12 Jun 2025). For VLMs, the framework extends to image–text pairs along with corresponding task-specific structured rewards (Koksal et al., 29 Jul 2025).

2. Algorithmic Framework and Optimization

The typical 1-shot RLVR system can be summarized as follows:

• Batch Construction: The one (image, prompt, answer) triple is duplicated to create an RL batch (e.g., batch size 128).
• Rollout Generation: For each input in the batch, multiple completions (group size K, e.g., K=4 or 8) are sampled from the current model policy.
• Reward Evaluation: Each rollout y is scored:
  • Classification, VQA: $$R_\text{acc}(y, y^*) = 1 \text{ if answer in } y = y^*, 0 \text{ otherwise}$$
  • Grounding: $R_\text{iou}(b, b^*) = 1$ if $IoU \geq 0.7$, $IoU$ if $0.4 \leq IoU < 0.7$, $0$ else
  • Format Consistency: $R_\text{fmt}(y)$ enforces output in <reasoning>...</reasoning><answer>...</answer> tags (Koksal et al., 29 Jul 2025)
• Loss and Gradient Update: A group-normalized advantage is computed per group:

$A_i = R_i - \bar{R}$

The GRPO loss—

$$\mathcal{L}_\text{PG}(\theta) = - \frac{1}{G K} \sum_{g=1}^G \sum_{i=1}^K A_{g,i} \log \pi_\theta(x_{g,i})$$

—is supplemented with a KL-divergence penalty to prevent drift from the reference model. In all published work, Adam is used for optimization. For VLMs, default hyperparameters are: learning rate $10^{-6}$, batch size 128, KL coefficient $\beta = 0.001$, temperature 0.9 (train), 1.0 (eval) (Koksal et al., 29 Jul 2025).

• Entropy Regularization: Including an entropy bonus promotes exploration and enhances generalization in the low-data regime, yielding up to 4–5 percentage points accuracy improvement (Wang et al., 29 Apr 2025).

Ablation confirms that the policy-gradient loss is the main driver of test performance gains, with regularizers (KL, weight decay) contributing marginally. Training typically proceeds for 1,000–2,000 steps; prompt design and low KL weight are essential for stable convergence (Koksal et al., 29 Jul 2025).

3. Empirical Results and Generalization

Results across mathematical, language, and vision–language domains establish several key points:

• Substantial improvements are achieved with a single example: for instance, 1-shot RLVR elevates Qwen2.5-Math-1.5B’s MATH500 score from 36.0% to 73.6%, nearly matching 1.2k-shot RLVR (73.6%) and full-supervised baselines (75.4%). On remote sensing vision–language tasks, π₁V (one VQA) increases classification accuracy by +10 points, RSVQA by +24.5 points, visual grounding by +8.9 points (Wang et al., 29 Apr 2025, Koksal et al., 29 Jul 2025).
• By 32–128 shots, the model matches or exceeds the performance of RLVR trained with thousands of labeled samples.
• Mild, task-local overfitting can occur in the 1-shot regime, but generalization to other tasks or datasets is not negatively affected.
• In long training, test accuracy continues to rise after training accuracy on the one-shot instance saturates (“post-saturation generalization”), indicating the policy-gradient mechanism drives emergence of general reasoning patterns rather than memorization (Wang et al., 29 Apr 2025).
• The approach confers large cross-domain gains: for example, improvement on algebraic tasks cascades to geometry and number theory; one VQA instance boosts classification and grounding benchmarks even for unseen datasets.
• Inclusion of an entropy penalty ($\alpha = -0.001$) further improves generalization, especially during the post-saturation phase (Wang et al., 29 Apr 2025).

4. Mechanistic Insights and Interpretations

A central observation, repeatedly validated, is that 1-shot RLVR does not teach fundamentally new algorithms. Rather, it amplifies latent reasoning modes—e.g., code-style mathematical problem solving or chain-of-thought formats—already acquired during pretraining (Wang et al., 29 Apr 2025, Shao et al., 12 Jun 2025).

Empirical studies show that for Qwen2.5-Math, any RLVR signal (even spurious rewards) drives output frequency of code-form answers from 65% up to over 90%, and 1-shot RL boosts accuracy by 26 percentage points, close to the 29 points achieved by full ground-truth RLVR (Shao et al., 12 Jun 2025). This suggests that the policy gradient with group normalization biases the model toward its highest prior, reward-compatible reasoning style. As a result, the apparent “generalization” is rooted in surfacing robust pretrained structures, not in learning from the specifics of the single example.

Notably, the “grokking” phenomenon (where test accuracy surges after long flat phases under heavy regularization) is absent: the main test gain occurs immediately with the policy-gradient update, and entropy regularization, not weight decay, explains post-saturation improvement (Wang et al., 29 Apr 2025).

5. Extensions to Vision–Language and Multimodal Reasoning

The transition of 1-shot RLVR to the vision–language domain introduces additional considerations:

• RLVR directly leverages rule-based binary or IoU-style rewards for closed-answer, VQA, and grounding tasks, with no caption or manual labeling required (Koksal et al., 29 Jul 2025).
• The base architecture (e.g., Qwen2-VL-2B) combines a vision transformer encoder, minimal VL adapter, and a LLM backbone; no additional pre-finetuning is used.
• Extensive duplication of the single example yields a stable RL batch suitable for policy-gradient updates; even with this duplication, out-of-set performance is significantly improved.
• Overfitting on the training dataset is generally mild; increasing to a 2–8 shot regime mitigates this entirely and brings performance to within a few points of fully supervised baselines.
• Prompting with a concise, format-enforcing template is essential; longer or more detailed prompts reduce test accuracy by up to 15 points.

Empirical efficiency is high: training 1,000–2,000 steps requires only a few GPU-hours, a reduction by orders of magnitude compared to standard finetuning (Koksal et al., 29 Jul 2025).

6. Practical Implementation and Limitations

Guidelines for deploying 1-shot RLVR:

• Start from a compact but competent base model (e.g., Qwen2-VL-2B or Qwen2.5-Math-1.5B).
• Curate 1–32 examples that are verifiable using simple binary or structured rules.
• Use short system prompts that precisely specify reasoning and answer format, avoiding verbose or overly task-specific instructions.
• Set KL regularization low (β ≈ 0.001); over-regularization causes instability or loss of diversity.
• Sample a small number of completions per input (K=4–8) for manageable memory usage and stable group-normalized policy gradients.
• Monitor test accuracy throughout training; if improvements plateau or single-task overfitting occurs, add a few more diverse examples.

Limitations:

• Strong reliance on pretrained competencies means true new-skill acquisition is not observable; failure occurs if the pretrained model lacks a reasoning pattern compatible with the reward.
• Some overfitting may appear on the dataset or task of the single example; proper few-shot or prompt mixing addresses this.
• In highly multimodal or sparse reward problems, group-based normalization is essential for training stability.
• Spurious rewards can induce misleading improvements in select model families (e.g., Qwen), highlighting the need for multi-family benchmarking and evaluation (Shao et al., 12 Jun 2025).

7. Outlook and Research Directions

1-shot RLVR has rapidly advanced data-efficient reasoning alignment for LLMs and VLMs across mathematical and remote sensing tasks. Ongoing research focuses on:

• Mechanistic explanations: understanding why and when one-shot signals suffice to unlock pretrained reasoning capabilities, and the limits of this effect for tasks without strong priors.
• New algorithms for further stabilizing low-data RLVR (e.g., Weighted GRPO with rare event amplification, positive–negative prompt pairing) (Sheng et al., 3 Feb 2026).
• Broader benchmarking: assessing generality across architectures (Qwen, Llama3, OLMo2) and problem domains to rule out model-specific idiosyncrasies.
• Applying the paradigm to dynamic system identification, where latent-space adaptation from a single trajectory enables rapid model-based policy optimization (Farid et al., 2021).
• Extending to more complex reward regimes (non-binary, hierarchical) and integrating richer prompts or task specifications.

A plausible implication is that “1-shot” RLVR will remain a highly attractive approach for steering LLMs and VLMs in data-constrained specialist applications, where high-quality pretrained models and well-designed verifiable reward signals are available. The framework sets a new standard for empirical efficiency and generalization in extreme low-shot alignment of large models.