R$^3$L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification

Abstract: Reinforcement learning drives recent advances in LLM reasoning and agentic capabilities, yet current approaches struggle with both exploration and exploitation. Exploration suffers from low success rates on difficult tasks and high costs of repeated rollouts from scratch. Exploitation suffers from coarse credit assignment and training instability: Trajectory-level rewards penalize valid prefixes for later errors, and failure-dominated groups overwhelm the few positive signals, leaving optimization without constructive direction. To this end, we propose R$^3$L, Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification. To synthesize high-quality trajectories, R$^3$L shifts from stochastic sampling to active synthesis via reflect-then-retry, leveraging language feedback to diagnose errors, transform failed attempts into successful ones, and reduce rollout costs by restarting from identified failure points. With errors diagnosed and localized, Pivotal Credit Assignment updates only the diverging suffix where contrastive signals exist, excluding the shared prefix from gradient update. Since failures dominate on difficult tasks and reflect-then-retry produces off-policy data, risking training instability, Positive Amplification upweights successful trajectories to ensure positive signals guide the optimization process. Experiments on agentic and reasoning tasks demonstrate 5\% to 52\% relative improvements over baselines while maintaining training stability. Our code is released at https://github.com/shiweijiezero/R3L.

• The paper introduces a structured three-phase framework combining language-guided reflect-then-retry exploration, pivotal credit assignment, and positive amplification to overcome RL fine-tuning challenges for LLMs.
• It demonstrates robust performance improvements with average reward gains up to 52% over baselines across agentic and mathematical reasoning tasks.
• Empirical and theoretical analyses confirm that isolating gradient updates and amplifying positive signals stabilize training and reduce computational overhead.

Reflect-then-Retry RL: R$^3$L with Language-Guided Exploration, Pivotal Credit, and Positive Amplification

Motivation and Challenges in RL Fine-Tuning for LLMs

Reinforcement learning for LLMs in agentic and reasoning domains faces significant challenges. Inefficient exploration arises when stochastic sampling yields predominantly failed trajectories, particularly in sparse-reward, long-horizon environments. This leads to insufficient gradient signals, high computational cost, and suboptimal learning. Exploitation, on the other hand, suffers from coarse trajectory-level credit assignment, penalizing valid prefixes for later errors. Additionally, in regimes dominated by failures, optimization becomes unstable; gradients suppress errors without reinforcing correct solutions, resulting in entropy collapse.

R$^3$L Methodology: A Structured Three-Phase Approach

R$^3$L introduces a cohesive framework to overcome the aforementioned bottlenecks via three integral mechanisms: language-guided reflect-then-retry exploration, pivotal credit assignment, and positive amplification. The system alternates between actively synthesizing high-quality trajectories and robust optimization, integrating detailed contextual analysis and credit assignment.

Figure 1: The R$^3$L framework integrates language-guided reflect-then-retry for efficient trajectory synthesis, pivotal credit assignment to isolate critical learning regions, and positive amplification to stabilize off-policy optimization.

Language-Guided Reflect-Then-Retry: R$^3$L leverages natural language feedback from the environment or verifier to diagnose errors and identify specific failure points (pivots) within base trajectories. For failed trajectories, it restarts generation from the identified pivot using corrective guidance, significantly increasing the likelihood of successful trajectory synthesis. Four trajectory types are constructed: base, reflection, retry, and context-distilled trajectories combining corrected suffix with the original prefix. Auxiliary meta-tasks explicitly train the model to maintain reflection and retry capabilities.

Figure 2: Four trajectory types in R$^3$L: base exploration, error diagnosis (reflection), guided retry, and context-distilled correction, with explicit separation of guidance dependencies for robust learning.

Pivotal Credit Assignment: Standard trajectory-level credit penalizes all tokens equally, suppressing valid reasoning even when errors occur late. R$^3$L exploits the contrastive structure between base and retry trajectories, masking the shared prefix before the pivot from gradient updates. This localizes learning only to diverging suffixes, acting as a control variate that reduces gradient variance and ensures optimization focuses on actionable decision points.

Positive Amplification: In failure-dominated groups, negative gradients from failed samples can dilute or overpower constructive signals. R$^3$L applies an amplification factor $\alpha$ to successful trajectories, ensuring constructive gradients structurally dominate, which drives convergence and prevents entropy collapse. Empirical sensitivity analysis (Appendix) and theoretical analysis confirm the efficacy and robustness of $\alpha = 3.0$ across tasks.

Empirical Evaluation and Numerical Results

Extensive experiments on embodied environments (ALFWorld, WebShop, ScienceWorld) and multiple mathematical reasoning benchmarks validate R$^3$L’s efficacy. Relative improvements ranging from 5\% to 52\% over strong baselines are observed. For instance, with Qwen2.5-1.5B-Instruct, R$^3$L achieves average reward scores of 0.928 on ALFWorld, 0.663 on WebShop, and 0.385 on ScienceWorld—surpassing the best baseline by 8.0\% and 5.2\% on the latter two.

Figure 3: Standard RL (GRPO) penalizes prefixes and suffers from gradient asymmetry, whereas R$^3$L actively synthesizes corrections, precisely isolates credit, and amplifies constructive gradients.

On mathematical tasks (GSM8K, Math500, OlympiadBench, AMC23), R$^3$L consistently outperforms both group-relative sampling and reflect-or-critique baselines, with gains most pronounced on harder benchmarks where deliberate trajectory synthesis and fine-grained credit assignment are essential.

Ablation studies demonstrate the contribution of each module. Removal of reflect-then-retry yields the most severe degradation, showing that stochastic sampling alone is insufficient for sparse-reward landscapes. Exclusion of positive amplification results in gradients focusing on suppressing errors rather than reinforcing solutions, while omitting pivotal credit reduces performance on long-horizon tasks.

Stability, Training Dynamics, and Theoretical Guarantees

R$^3$L exhibits robust learning dynamics free from the collapse observed in baseline approaches. Reward curves show rapid phase transitions and higher asymptotic rewards. Policy drift, gradient spikes, and high KL divergence—critical failure modes in GRPO—are effectively eliminated in R$^3$L.

Figure 4: R$^3$L outperforms GRPO with superior reward curves, stable KL divergence, smooth gradient norms, and consistent policy loss, confirming theoretical stability analysis.

Theoretical decomposition clarifies that, absent amplification, the gradient is dominated by destructive signals that merely suppress high-probability errors and disperse probability mass, increasing entropy. Positive amplification enforces gradient dominance for constructive updates. Masking via pivotal credit further acts as a variance reducer, ensuring the effective signal-to-noise ratio increases as models mature.

Empirically, R$^3$L maintains off-policy stability even without importance sampling or KL regularization, because only verified high-reward corrections are distilled, and amplified gradients consistently channel learning toward successful behaviors.

Analysis of Exploration Efficiency

Quantitative metrics demonstrate that language-guided reflection reliably improves base trajectories throughout training, with reward improvement rates exceeding 70\% on agentic tasks for Qwen2.5-7B. The mechanism exhibits a “warm-up” phase for smaller models but sustains improvement as training progresses, supported by auxiliary meta-task supervision.

Practical and Theoretical Implications

By transforming sparse-reward exploration into a high-yield guided synthesis process, R$^3$L reconfigures RL fine-tuning for LLMs. The explicit use of contrastive structure for credit assignment obviates the need for step-level reward models or costly annotation, and the simplicity of amplification delivers robust convergence. Removal of externally imposed KL constraints and importance sampling ratios streamlines training and reduces computational overhead, without sacrificing stability.

R$^3$L gives a blueprint for how language feedback and trajectory structure can be leveraged not only to guide exploration but also to propagate credit precisely and optimize learning signals in RL settings with high dimensions and sparse supervision.

Future Directions

Several avenues emerge:

• Extending context distillation and the core pivotal mechanism to open-ended environments with subjective criteria.
• Integrating richer error analysis (e.g., multimodal feedback) for more complex agentic behaviors.
• Exploring meta-RL and higher-order supervision where reflection and retry themselves are subject to learning and competition.

Conclusion

R$^3$L introduces a principled and empirically validated framework for RL fine-tuning of LLMs, overcoming key barriers in exploration efficiency, credit assignment precision, and optimization stability. By synthesizing corrections through language feedback, isolating learning to critical regions, and structurally amplifying constructive signals, R$^3$L achieves state-of-the-art performance, robust training, and new theoretical insight into the optimization of agentic and reasoning behaviors in large models.