Context Bootstrapped Reinforcement Learning

Abstract: Reinforcement Learning from Verifiable Rewards (RLVR) suffers from exploration inefficiency, where models struggle to generate successful rollouts, resulting in minimal learning signal. This challenge is particularly severe for tasks that require the acquisition of novel reasoning patterns or domain-specific knowledge. To address this, we propose Context Bootstrapped Reinforcement Learning (CBRL), which augments RLVR training by stochastically prepending few-shot demonstrations to training prompts. The injection probability follows a curriculum that starts high to bootstrap early exploration, then anneals to zero so the model must ultimately succeed without assistance. This forces the policy to internalize reasoning patterns from the demonstrations rather than relying on them at test time. We validate CBRL across two model families and five Reasoning Gym tasks. Our results demonstrate that CBRL consistently improves success rate, provides better exploration efficiency, and is algorithm-agnostic. We further demonstrate CBRL's practical applicability on Q, a domain-specific programming language that diverges significantly from mainstream language conventions.

• The paper introduces CBRL, an in-context augmentation strategy with an annealed injection schedule that efficiently overcomes exploration bottlenecks in sparse reward environments.
• It employs a curriculum-driven approach by gradually reducing few-shot demonstrations, fostering robust policy learning across various reasoning and code generation tasks.
• Empirical results show up to 47-point accuracy improvements over baselines and a pass@1 increase from 5% to 26.3% on challenging real-world benchmarks.

Context Bootstrapped Reinforcement Learning: A Formal Analysis

Introduction

Context Bootstrapped Reinforcement Learning (CBRL) addresses a central inefficiency in Reinforcement Learning from Verifiable Rewards (RLVR) for LLMs: exploration bottlenecks that result from sparse reward signals in structured, programmatic tasks requiring compositional reasoning or acquisition of domain-specific conventions. RLVR depends on models independently discovering successful rollouts to receive binary rewards, a challenge exacerbated in tasks underrepresented in pretraining datasets or requiring nontrivial reasoning patterns. CBRL proposes an in-context augmentation regime: during RLVR training, few-shot solved demonstrations are injected into training prompts with a decaying probability, effectively serving as curriculum-driven scaffolding for early capability acquisition.

Figure 1: Overview of the CBRL method. Left: Demonstrates the annealing schedule for injection probability $p_i$. Right: Training prompts stochastically include few-shot demonstrations that are phased out as learning progresses.

The CBRL Framework

CBRL modifies standard policy-gradient RL pipelines, such as GRPO and RLOO, exclusively at the input distribution level. For each task, a few-shot bank $\mathcal{B}$ is constructed (either using expert demonstrations, SFT solutions, or strong model rollouts) and, at training step $t$, a subset of examples is stochastically prepended to prompts with probability $p_i$. This probability initiates at a moderate value (typically 0.5), linearly annealing to 0 by the end of training, ensuring that the transition from scaffolded learning to autonomous rollout is gradual and seamless.

The impact of this method is illustrated clearly: early in training, most prompts include demonstrations, greatly increasing the likelihood of successful rollouts and thus improving the reward signal. As the model internalizes task-specific policies, reliance on context is diminished, ultimately forcing generalization with no run-time computational overhead.

Empirical Results

CBRL demonstrates improvements over baselines—both vanilla GRPO and RLOO—in a suite of compositional tasks from Reasoning Gym (ARC-1D, Word Sorting, Puzzle24, Spell Backward, and Manipulate Matrix) and on a real-world code generation challenge in the Q programming language (syntactically alien to most pretrained models). The improvements are algorithm-agnostic, persisting across model families (Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct, QQwen-7B-Pretrain).

Ablation over injection rates reveals that moderate initial injection ($p_i = 0.5$) maximizes success—high $p_i$ inhibits independent exploration, while low $p_i$ provides insufficient bootstrapping.

Figure 3: (Left) Tabulated accuracy gains of CBRL-RLOO over RLOO on Reasoning Gym tasks—up to +47 points in Word Sorting and +43 in Puzzle24; (Right) Success-rate sensitivity to initial $p_i$ for ARC-1D: performance peaks at moderate injection.

Performance gains from CBRL are most significant on tasks whose solution requires systematic, compositional, or otherwise unfamiliar reasoning (e.g., ASCII sorting, game solving). For Q programming, CBRL raises pass@1 from 5% to 26.3%—a substantial gain on challenging code generation benchmarks. The effect extends to learning dynamics: reward curves show faster acquisition of reward under CBRL, with the gap to baseline persisting after injection is annealed (that is, gains do not revert post-scaffolding removal).

Figure 2: Reward curves for CBRL (blue) versus baseline (red) for Q Programming, Word Sorting (GRPO), and Word Sorting (RLOO). Higher reward acquisition under CBRL is evident during and after context injection.

Analysis of Policy Learning

Qualitative analysis elucidates that CBRL-trained models explicitly learn reasoning algorithms reflected in demonstrations, rather than mimicking superficial patterns. For example, in ASCII word sorting, CBRL-trained models output stepwise computation of ASCII values and their application (mirroring the demonstration traces), while vanilla RLVR models produce generic, often incorrect, answers without structured justification.

CBRL’s learning is robust against demonstration removal post-training. Training trajectories show that early guidance suffices for policy acquisition and models retain the learned strategies without demonstration dependence.

Theoretical and Practical Implications

CBRL generalizes curriculum learning and in-context learning by integrating them into policy-gradient training under reward sparsity. Unlike mixed-policy and partial-supervision methods, CBRL is strictly on-policy, does not require reward model shaping, and does not yield the model into off-policy distribution drift. The approach leverages LLMs’ innate ICL abilities as “transient parametrization” for exploration, but requires no overhead or modification to objective, optimizer, or run-time inference calls.

The results suggest that curriculum-driven context augmentation is particularly advantageous for bootstrapping exploration in domains with significant distributional shift from pretraining (e.g., Q language) or in settings with reward sparsity.

Limitations and Future Directions

CBRL can reduce performance if context selection is poorly matched to the task or if injected demonstrations dominate rollout diversity (as seen in reduced gains for ARC-1D and Manipulate Matrix under RLOO). The method’s effectiveness depends on the fidelity and diversity of the example bank and the specifics of the injection schedule.

Potential advances include adaptive schedules that leverage online reward trends, automated retrieval and augmentation pipelines for example selection, and extension to hierarchical/sequential RL for long-horizon tasks. There is also scope to analyze its interplay with large-scale off-policy RL, goal-directed exploration, and non-stationary environments.

Conclusion

CBRL defines an effective, algorithm-agnostic augmentation for RLVR training that leverages in-context learning to overcome exploration inefficiency and reward sparsity. Through stochastic, annealed few-shot prompting, models acquire reasoning strategies rapidly and retain them post-scaffolding. The empirical results across reasoning and code generation benchmarks, with robust ablations and qualitative analysis, validate CBRL as an important methodological advance for RL in models with strong in-context adaptation capabilities (2603.18953).