RLTL;DR: Self-improvement by Internalizing Self-generated Feedback

Abstract: The common paradigm of reinforcement learning with verifiable rewards (RLVR) is to let agents make multiple attempts at a task, and optimize towards the successful ones. This becomes problematic in the realms of self-improvement, where tasks are so difficult that the agent has a low or even no chance of success, and where there are no teacher models or example solutions to distill from. In this paper, we introduce RLTL;DR. After each failed attempt, we show the policy the verifier outputs and let it write its own feedback, in the form of a single TL;DR insight. The next rollout is conditioned on all previous insights, and we sequentially sample rollouts until a solution is found. Moreover, we enable backpropagation on the in-context insights to internalize a direct task to insight mapping. On challenging tool-calling and coding datasets (filtered to Pass@128=0), standard GRPO training of a Qwen 3.5 9B Thinking policy stays flat at a Pass@1 of 0% to 1%. RLTL;DR breaks through this learning barrier, achieving a Pass@1 of 14-31% with insights in context during training and, crucially, 12-13% when no insight is in context at eval time. We identify that the key is the task to insight internalization. To study this further, we reduce our approach to SFTL;DR, training only on (task, insight) tuples, without showing or backpropagating on any rollouts. Training on only 4k of these tuples recovers almost the full performance of RLTL;DR and classical SFT on full rollouts. This demonstrates a promising compacted training paradigm of the form "on this sort of task, keep this sort of thing in mind", which we hope to inspire future research on.

• RLTL;DR advances reinforcement learning (RL) by introducing sequential tasks with self-generated insights that help overcome learning failures, achieving 12-18% Pass@1 in tasks ML models struggle with.
• The methodology enhances RL cognitive ability by creating and internalizing insights from task rollouts, significantly improving success rates (up to 57% Pass@1).
• Its approach distills compact, actionable feedback into the policy, demonstrating sustained learning beyond initial training data.

Problem setting and central claim

"RLTL;DR: Self-improvement by Internalizing Self-generated Feedback" (2609.37633) addresses a structural failure mode of RL with verifiable rewards (RLVR). In standard GRPO-style training, a task is sampled repeatedly and the resulting trajectories are scored by a binary verifier. When all rollouts fail, the group-relative advantage is uninformative and the policy receives no useful update. This is especially consequential in self-improvement settings, where the policy is assumed to operate near its capability frontier and no stronger teacher or reference solution is available.

The paper proposes RLTL;DR, which combines two interventions. First, it replaces independent rollouts with sequential attempts on the same task. After each failed attempt, the policy analyzes the trajectory and verifier output and generates a short, high-level insight describing what should change. Subsequent attempts are conditioned on accumulated insights. Second, the training objective backpropagates through the insight tokens, despite those tokens being inserted as user-context messages and never being generated at evaluation time. This auxiliary SFT objective trains a task-to-insight association that is intended to internalize the information extracted during exploration.

The paper's strongest claim is that the insight-internalization objective, rather than the GRPO objective, is the principal source of learning on frontier-difficult tasks. In the authors' experiments, training solely on task–insight pairs nearly recovers the performance of the complete RLTL;DR system, even though the model is never trained on the corresponding solution trajectories during the update.

Sequential exploration with self-generated insights

The exploration procedure retains the same task while generating up to $K$ attempts sequentially. The first attempt is unconditioned. If it fails, a separate feedback-generation prompt receives the trajectory, environment observations, and failed unit-test outputs. The policy then produces a structured analysis but only the final one-sentence TL;DR is retained as an insight. Subsequent attempts receive the task and previously generated insights when the running success rate remains below a threshold, set by default to $50\%$.

This design transforms failed rollouts from purely negative samples into a source of conditional information. The verifier supplies precise failure evidence, while the policy compresses that evidence into a reusable procedural correction, such as using the correct enum value, persisting a state change, or removing an invalid argument. The method does not expose the complete previous trajectory to the next attempt, thereby avoiding contextual drag and limiting the context overhead. Insights average approximately 17 tokens.

The exploration effect is substantial before any parameter update. On tasks with approximately zero success probability under 128 independent attempts, sequential insight conditioning finds at least one solution for approximately 57% of tasks, compared with approximately 38% for repeatedly using one insight and approximately 6% for independent sampling.

Figure 1: Sequential insights substantially increase Pass@k over independent sampling on tasks with Pass@128 approximately zero.

On a less difficult split, sequential insights solve approximately 66% of tasks, compared with approximately 54% using a single insight and approximately 13% with independent sampling.

Figure 2: Sequential feedback remains effective on a less difficult task split, although the relative advantage decreases as independent exploration becomes more capable.

These results establish that sequential feedback produces a substantially richer training distribution. They do not, however, establish that context-conditioned success will transfer to ordinary one-shot inference. The paper explicitly observes that prompting alone improves the conditioned rollout but leaves unaided performance close to the original model.

Insight internalization

To transfer the value of sequential exploration to evaluation, RLTL;DR changes the gradient mask applied to the insight tokens. Let $g$ denote the task and $f$ the generated insight. The additional objective trains the model to increase the likelihood of $f$ conditioned on $g$, rather than only conditioning on the failed trajectory from which $f$ was produced. The complete objective is the GRPO loss plus a weighted insight-SFT loss,

$$\mathcal{L} = \mathcal{L}_{\mathrm{GRPO}} + \lambda \mathcal{L}_{\mathrm{SFT}}.$$

At test time, the model receives neither the insight nor multiple sequential attempts. The intended mechanism is therefore representational: gradient updates on task-to-insight mappings alter the policy so that it applies the relevant procedural rule while generating a normal solution. The paper characterizes this as a form of context distillation, but differs from full-rollout distillation by supervising only the compact feedback tokens.

The implementation is computationally economical during policy updates. The insights already occur in the rollout context, so internalization requires changing the backpropagation mask rather than generating a separate training sequence. The authors use $\lambda=0.5$ as a default and report relative robustness to the precise mixture, although the GRPO component affects convergence speed.

Experimental design

The experiments use Qwen 3.5 9B Thinking on three environments:

• Synthetic API (SAPI): a proprietary tool-use benchmark with approximately 16,000 tasks.
• Appworld: simulated device and application tasks involving multi-step tool calls.
• Leetcode: code-generation problems.

The principal difficulty split contains tasks for which the base model achieves no successes in 128 attempts. This yields 458 SAPI tasks, 34 Appworld tasks, and 123 Leetcode tasks. Evaluation is deliberately deconfounded: reported Pass@1 values are computed on rollouts without insights in context, rather than on the easier conditioned trajectories. The authors also macro-average over tasks to avoid bias from asynchronous worker throughput.

The baseline comparison includes standard GRPO, Strategy-guided Exploration (SGE), and RLTF-SD, a self-distillation approach that transfers insight-conditioned rollouts to an unconditioned policy. The GRPO implementation includes several stabilization modifications, including removal of advantage standard-deviation normalization and positive-ratio filtering. Consequently, the comparison is against a tuned GRPO system rather than an unmodified textbook baseline.

Breaking the RLVR learning barrier

On the Pass@128=0 splits, standard GRPO remains essentially flat at 0–1% Pass@1. SGE behaves similarly because it conditions on previous failed attempts without explicitly extracting failure-specific information from the verifier. RLTL;DR reaches approximately 12–13% Pass@1 on unaided evaluation rollouts. RLTF-SD also learns, but the authors do not claim a decisive superiority of RLTL;DR over that method.

The result is important because the improvement is measured without the mechanism that produced the additional training signal. The method does not merely show that a model can solve a task after being given a hint; it shows that training on self-generated hints can improve first-attempt behavior after the hints are removed.

Figure 3: RLTL;DR alternates failed attempts, verifier-based insight generation, insight-conditioned retries, and backpropagation through the inserted insight tokens.

Held-out results show that the improvement can generalize beyond the hard training subset, although the evaluation composition requires caution. Training is performed on filtered subsets, and the Appworld training mixture includes some non-challenge data that would ordinarily be part of the broader dataset. The reported values therefore should not be interpreted as standard benchmark scores.

Method	SAPI	Appworld	Leetcode
GRPO	57.8%	33.7%	55.1%
SGE	57.7%	33.5%	56.4%
RLTF-SD	53.0%	38.9%	42.2%
RLTL;DR	91.1%	61.7%	49.1%

The held-out results suggest strong transfer on SAPI and Appworld, but not on Leetcode. The paper reports that Leetcode likely exhibits overfitting and does not provide evidence that compact insight training is uniformly effective across domains. On normal-difficulty splits, where most tasks are solvable within 128 attempts, RLTL;DR generally does not outperform GRPO. This is consistent with the method's purpose: it is intended to restore learning signal when ordinary RLVR is exploration-limited, not to replace GRPO in already solvable regimes.

The surprising reduction to SFTL;DR

The most consequential ablation removes the GRPO contribution entirely. On a SAPI split with 642 difficult tasks, the complete RLTL;DR system reaches 21.5% train Pass@1 and 18.9% held-out Pass@1. The variant using only the insight-SFT loss reaches 20.0% and 17.0%, respectively. By contrast, reducing the insight weight to $0.01$ or removing it entirely substantially degrades performance.

Training method	Train Pass@1	Held-out Pass@1	Backward tokens
RLTL;DR	21.5%	18.9%	12M
RLTL;DR, $50\%$0	9.3%	14.0%	12M
RLTL;DR, $50\%$1	6.1%	13.2%	11M
Insight SFT only, no GRPO	20.0%	17.0%	839k
SFT on all full rollouts	28.0%	20.9%	6.6M
SFT on 100 full rollouts	25.9%	20.8%	217k
SFTL;DR on insights	17.8%	16.8%	292k
Deduplicated SFTL;DR	19.1%	16.9%	68k

The reduction is especially notable because SFTL;DR does not backpropagate through the agent's solution trajectory. It trains only on $50\%$2 pairs, where $50\%$3 is the task and $50\%$4 is the one-sentence insight. The deduplicated variant trains on approximately 4,000 unique task–insight pairs and achieves 16.9% held-out Pass@1 while requiring only 68,000 backward tokens. Full rollout collection remains necessary to generate the insights, accounting for approximately 1 billion tokens, but the policy-update computation is sharply reduced.

The implication is specific and technically significant: the useful learning target may be a compact representation of the failure correction rather than the successful trajectory itself. The result also challenges the assumption that the output format used during supervision must match the output format required at inference. The model is trained to predict a natural-language correction but evaluated by generating tool calls or code. The paper attributes this transfer to semantic smoothness in the pretrained representation, while acknowledging that the mechanism is not established.

Figure 4: Insight-only backpropagation recovers most of RLTL;DR performance, whereas removing the insight-SFT signal substantially weakens learning.

GRPO remains useful despite not being necessary for the final asymptotic result. Across loss-mixture experiments, any nonzero GRPO contribution accelerates early learning relative to pure insight SFT. Pure insight SFT requires approximately 25% more environment interactions to reach comparable performance. Thus, the paper's evidence supports a division of labor: sequential feedback supplies compact supervision, insight SFT drives internalization, and GRPO accelerates optimization when successful trajectories become available.

What makes an insight useful

The paper's ablations distinguish immediate task assistance from transferable supervision. More detailed feedback can improve the conditioned rollout while harming unaided learning. Replacing the one-sentence TL;DR with a diagnostic paragraph reduces train Pass@1 from 21.4% to 20.3%; adding a summary reduces it to 13.8%, and adding corrected code yields 15.7%. By contrast, detailed feedback can provide a larger improvement when retained in context, reaching a reported conditioned gain of +41.1%, compared with +36.0% for the TL;DR format.

This contrast supports the paper's distinction between useful context and useful training labels. Detailed diagnostics often contain episode-specific identifiers, values, or implementation details that help solve the current task but are difficult to predict from the task description and unlikely to transfer. Short procedural rules suppress these details and provide a lower-variance target for task-conditioned internalization.

The source of the insight matters less than its access to failure information. A stronger GLM 5.2 generator modestly improves results, reaching 26.2% train Pass@1 with one insight, compared with 20.6% for the student in the corresponding setting. However, withholding failed unit-test outputs causes a much larger decline: student-generated insights without those outputs yield only 3.4% train Pass@1 and 10.0% evaluation Pass@1. The verifier therefore functions as privileged supervision for the feedback generator.

The implication is that RLTL;DR depends on accurate failure localization, not merely on additional language-model reasoning. If the insight does not identify the actionable error, sequential retries may not enter a productive exploration regime and the internalization loss may reinforce irrelevant text.

Additional behavioral analyses

The paper reports several analyses consistent with genuine capability improvement rather than simple reliance on the inserted text. On revisited tasks, unaided success increases from 13.8% to 25.3%, with $50\%$5 across 85 tasks. The improvement is comparable to the gain on tasks that are not revisited, suggesting that the policy is not merely memorizing a fixed insight attached to an individual task.

Insights also evolve during training. Within a single visit, the average word-level dissimilarity between insights is 0.19, whereas dissimilarity between the first and last visit to the same task is 0.82. This indicates that the policy and verifier identify different failure modes as the policy changes, rather than repeatedly emitting a fixed textual correction.

Figure 5: Insights change substantially across training visits to the same task, consistent with evolving strategies and failure modes.

Insight reliance remains comparatively low. The reported difference in per-token log-likelihood between successful trajectories with and without insights rises from approximately +0.009 in the first half of training to +0.034 in the second half. The authors interpret this as evidence that successful trajectories benefit from insights without becoming fully dependent on them, which is consistent with the observed transfer to unaided evaluation.

Figure 6: Insight-conditioned trajectories become somewhat more likely under the trained policy, but the measured reliance remains modest.

Limitations and open questions

The evidence is concentrated in simulated tool-use environments and a selected set of coding tasks. All tool-use experiments use research-only environments, and the SAPI benchmark is proprietary. The frontier subsets are intentionally filtered for Pass@128=0, which is appropriate for testing the learning barrier but limits conclusions about ordinary training distributions. On normal-difficulty datasets, RLTL;DR generally matches rather than exceeds GRPO, and the Leetcode results are mixed.

The method also assumes access to informative verifier outputs. Removing failed unit-test information causes a severe degradation, so the approach is not demonstrated when failure feedback is weak, delayed, or unavailable. The paper acknowledges a further unresolved issue: in domains where identifying the correct insight is as difficult as solving the original task, sequential feedback may not generate useful supervision.

The proposed generalization mechanism remains speculative. The authors hypothesize that backpropagation on task–insight pairs updates semantically related representations and thereby affects different output formats, but they do not isolate this mechanism from ordinary fine-tuning effects. It is also unclear how performance scales with insight noise, task specificity, model size, or distribution shift. The failure of Leetcode transfer and the dependence on verifier-derived feedback provide concrete reasons not to assume that insight-only learning will generalize to arbitrary reasoning domains.

Finally, the wall-clock cost of sequential sampling is nontrivial. With eight attempts per task, asynchronous collection is approximately 4.5 times slower than parallel sampling, although fixed update costs reduce the overall wall-time increase to approximately 1.5 times in the reported setup. The authors preserve equal data volume for fairness rather than maximizing GPU utilization, so the systems tradeoff remains incompletely characterized.

Conclusion

RLTL;DR modifies RLVR at both the exploration and update stages. Sequential retries conditioned on verifier-grounded self-generated insights recover successes on tasks for which independent sampling provides no useful reward signal. Backpropagation through the compact insight tokens then transfers part of that training signal to unaided inference.

The central empirical result is that the insight-SFT objective explains most of the improvement: removing GRPO still retains nearly full performance, while removing insight internalization sharply reduces it. Training solely on task–insight pairs is therefore a viable simplification, although it still depends on collecting rollouts to produce accurate feedback. The paper establishes a narrowly defined but important result: compact, verifier-grounded self-generated feedback can serve as a more efficient supervision target than full trajectories when RLVR is blocked by exploration failure.