RLTL;DR: Reinforcement-Language Method

• RLTL;DR is a reinforcement-learning method designed for sparse-reward environments, such as simulated APIs and tool-use tasks, by converting failed attempts into compact, verifier-grounded insights;
• The method leverages a sequential feedback-conditioned exploration process and utilizes a supervised objective on self-generated insight tokens, improving the success rate on target tasks drastically;

RLTL;DR is a reinforcement-learning method for language-model agents that converts failed attempts into compact, verifier-grounded insights and trains the policy to internalize those insights. It targets sparse-reward settings in which standard reinforcement learning with verifiable rewards (RLVR) cannot learn because independent rollouts almost never succeed. The method combines sequential feedback-conditioned exploration with a masked supervised objective on self-generated insight tokens, enabling improvements to transfer from insight-assisted training contexts to evaluation on the task alone (Kirchhof et al., 29 Sep 2026).

1. Problem setting and learning barrier

RLTL;DR is designed for environments with verifiable outcomes, including simulated APIs, tool-use tasks, and code execution. A task $g$ is presented to a language-model policy operating in a partially observable Markov decision process with goal space $\mathcal G$, observation space $\mathcal O$, action space $\mathcal A$, and binary reward function $R$. At each step, the policy generates an action containing reasoning or code, receives an environment observation, and eventually obtains a verifier reward $r\in\{0,1\}$.

A trajectory can be represented as

$\tau=(h_T,a_T,o_T,r),$

where $h_T$ is the interaction history, $a_T$ is the final action, $o_T$ is the final verifier observation, and $\mathcal G$0 is the binary outcome. The conventional objective is to maximize expected outcome reward,

$\mathcal G$1

Standard GRPO samples $\mathcal G$2 independent trajectories for the same task and computes group-relative advantages from their rewards. When every trajectory fails,

$\mathcal G$3

the group provides no successful behavior against which unsuccessful behavior can be contrasted. The resulting advantage signal vanishes or becomes uninformative. RLTL;DR refers to this failure mode as advantage collapse, the learning cliff, or the learning barrier.

The barrier is particularly severe when the base model has negligible success probability, when no rollout among 128 attempts succeeds, when no teacher model or gold solution is available, and when the task lies near the model’s capability frontier. Under these conditions, increasing the number of independent samples does not necessarily create a useful learning signal because all samples may come from the same unsuccessful distribution.

The method therefore changes the sampling process before changing the optimization objective. Instead of treating all attempts as independent, it uses verifier feedback from failed attempts to guide subsequent attempts at the same task.

2. Sequential feedback-conditioned exploration

For a task $\mathcal G$4, RLTL;DR first generates an ordinary trajectory. If the verifier reports failure, the policy is invoked in a new conversation to analyze the failed attempt and produce a compact insight.

Formally, after a failed trajectory $\mathcal G$5, the feedback process generates

$\mathcal G$6

The feedback-generation prompt includes the original task, the full agent rollout, environment observations, verifier output, and failed tests. The model is asked to summarize the attempt, explain what went wrong, identify the incorrect step, propose corrected code, and produce a one-sentence TL;DR insight. Although the feedback generator may use up to 4096 thinking tokens and 4096 action or output tokens, only the final short insight is retained. Insights average approximately 17 tokens.

Examples of retained insights include:

• “You need to mark the article as read rather than just viewing it.”
• “Remove the answer parameter from the complete task call.”
• “Use the uppercase Orange color enum instead of lowercase orange.”
• “Include the access token when making the search request.”

The next task attempt is conditioned on the original task and the accumulated insights:

$\mathcal G$7

The previous complete rollout is not inserted into the next attempt. Only the compressed feedback is preserved, avoiding contextual drag and repetition associated with carrying the entire failed conversation forward.

Insights are inserted when the running success rate among previous attempts is at most $\mathcal G$8:

$\mathcal G$9

This threshold is intended to maintain a “Goldilocks zone.” Easy tasks do not receive unnecessary hints, while difficult tasks generate corrective guidance. If insights make every subsequent attempt successful, the group may again lose the relative contrast needed for GRPO.

The sequential attempts are treated as one GRPO group, although they are not independent because later attempts depend on earlier verifier feedback. The default maximum number of accumulated insights is 16.

On the frozen base Qwen 3.5 9B Thinking model, for tasks with approximately zero success under independent sampling, the paper reports that independent sampling solves roughly 6% of tasks at the studied budget, one insight increases this to roughly 38%, and sequentially accumulating insights increases it to roughly 57%. In another Pass@32-style analysis, the corresponding figures are approximately 13%, 54%, and 66%.

Sequential exploration therefore creates successful insight-conditioned trajectories that ordinary independent sampling does not find. However, contextual success alone does not solve the transfer problem: evaluation requires the model to act from the task prompt without the previously generated insights.

3. Insight internalization

3.1 The transfer problem

Sequential exploration produces useful feedback in context, but the evaluation policy must operate as

$\mathcal O$0

rather than

$\mathcal O$1

Training only on insight-conditioned successful trajectories can therefore teach the model to use hints without teaching it to recover the relevant principle from the task itself. RLTL;DR addresses this by applying a supervised loss directly to the insight tokens.

The model is trained to assign high probability to the insights given the task:

$\mathcal O$2

This process is called task-to-insight internalization. The model is not required to generate the insights during evaluation. Instead, the update is intended to make the parameters absorb their useful semantic content, producing the mapping

$\mathcal O$3

The paper characterizes this as a form of context distillation that is more compact than distilling an entire rollout.

3.2 Masked insight loss

The insight objective is a next-token negative log-likelihood:

$\mathcal O$4

In implementation, the insight messages remain user-context tokens, but their token masks are activated for backpropagation. The full rollout tokens retain the GRPO treatment. When multiple insights are present, all insight tokens receive the supervised loss simultaneously, with later insights conditioned on earlier ones.

The combined objective is

$\mathcal O$5

with default internalization weight

$\mathcal O$6

The reported experiments use PPO/GRPO-style clipped policy optimization, no KL regularization, no advantage standard-deviation normalization, token-loss averaging outside rollout sums as in DAPO, and positive-ratio filtering for stability.

The training loop alternates among rollout collection, verifier execution, feedback generation for failed attempts, and GRPO plus insight-SFT updates. At evaluation, the sequential process and all insights are discarded, and the model generates a single ordinary rollout from the task alone.

4. Algorithmic workflow

For each training batch, RLTL;DR executes the following procedure:

1. Task selection: choose a task $\mathcal O$7 and initialize an empty insight set $\mathcal O$8.
2. Sequential rollout: for $\mathcal O$9, construct a prompt from $\mathcal A$0 and the accumulated insights, then sample a trajectory

$\mathcal A$1

1. Environment execution: execute the trajectory and obtain verifier reward $\mathcal A$2 together with failure observations.
2. Feedback generation: if the attempt fails, provide the task, rollout, observations, verifier output, and failed tests to the feedback generator.
3. Insight extraction: generate a detailed diagnostic and retain its one-sentence TL;DR insight $\mathcal A$3.
4. Insight accumulation: append $\mathcal A$4 to the current insight set when the insertion condition is satisfied.
5. Group formation: treat the sequential attempts as a single GRPO group.
6. Policy optimization: compute the GRPO advantages and clipped policy loss.
7. Insight optimization: activate the gradient mask on all insight tokens and compute the autoregressive supervised loss.
8. Parameter update: optimize

   $\mathcal A$5

9. Evaluation: remove all insights and sequential retries, then measure single-attempt performance from $\mathcal A$6 alone.

The method depends critically on informative verifier feedback. In an ablation in which failed unit tests were hidden from the insight generator, training Pass@1 fell to 3.4%, compared with 21.4% for the student feedback generator with access to the relevant failure information on the SAPI hard split.

The collection process is slower than parallel GRPO because attempts for the same task are sequentially dependent. At $\mathcal A$7, sequential sampling is approximately 4.5 times slower in the sampling phase, while total wall-clock time increases by approximately 1.5 times because update and fixed costs remain constant.

5. Empirical evaluation

5.1 Models, environments, and task filtering

The primary policy is Qwen 3.5 9B with thinking. The method is evaluated on:

• Appworld, a multi-step tool-calling environment with 90 training, 57 development, 168 test-normal, and 417 test-challenge tasks.
• Synthetic-API (SAPI), an Appworld-like API environment with approximately 16,000 training tasks and evaluation on 1,624 tasks across four unseen applications.
• Leetcode, containing 2,641 coding problems and 228 unseen evaluation problems.

The main experiments filter tasks for which the base model has no success in 128 attempts:

$\mathcal A$8

This produces 458 SAPI tasks, 34 Appworld tasks after combining train, development, and test-normal tasks while retaining test-challenge tasks as unseen data, and 123 Leetcode tasks. An easier SAPI split contains 642 tasks, including the 458 frontier tasks and 184 additional difficult tasks; the base model achieves approximately 4% Pass@1 on this split.

Baselines include standard GRPO, Strategy-guided Exploration (SGE), RLTF-SD, and classical offline SFT on successful full rollouts.

5.2 Frontier-difficulty results

On tasks filtered to Pass@128 $\mathcal A$9, standard GRPO remains approximately at 0–1% Pass@1. SGE behaves similarly. RLTL;DR reaches approximately 12–13% Pass@1 without insights at evaluation, while insight-conditioned training measurements reach approximately 14–31% Pass@1 and approximately 14–59% Pass@$R$0.

The paper summarizes the main improvement as approximately 11–14% Pass@1 depending on the benchmark and run. The key distinction is that the no-insight evaluation score measures transfer from internalized feedback rather than direct use of hints.

On the 642-task SAPI split, RLTL;DR achieves 18.9% held-out Pass@1, compared with 17.0% for insight-only SFT without GRPO in the corresponding experiment. The method therefore receives additional benefit from GRPO, but the principal transfer mechanism is the insight loss.

5.3 Held-out results after hard-split training

Method	SAPI	Appworld	Leetcode
GRPO	57.8%	33.7%	55.1%
SGE	57.7%	33.5%	56.4%
RLTF-SD	53.0%	38.9%	42.2%
RLTL;DR	91.1%	61.7%	49.1%

These are held-out evaluations after training on filtered difficult data rather than ordinary leaderboard scores. The paper cautions that some Appworld test-normal tasks were included in the hard training pool.

On ordinary, unfiltered datasets, where 95–97% of tasks are solvable within 128 attempts, standard GRPO eventually learns and RLTL;DR generally provides no major advantage. For example, SAPI training yields approximately 52% for RLTL;DR and 53% for GRPO on held-out evaluation; Appworld train-to-test-challenge performance is 72.1% for RLTL;DR and 72.2% for GRPO.

These results position RLTL;DR primarily as a method for sparse-reward frontier tasks rather than as a universal replacement for conventional RLVR.

6. SFTL;DR and mechanism analysis

A major finding is that much of RLTL;DR’s benefit can be recovered by training only on task–insight pairs, without backpropagating through complete rollouts. This reduced method is called SFTL;DR.

An SFTL;DR example consists of

$R$1

where $R$2 is the task and $R$3 is a verifier-grounded insight. Its objective is

$R$4

The tuples are still obtained through online interaction: a full rollout is executed, the verifier identifies failure, a feedback generator produces an insight, and the pair $R$5 is retained. Once collected, however, the policy update does not require the original rollout, code, tool interactions, or observations.

The deduplicated SFTL;DR variant uses approximately 4,000 unique task–insight tuples. On the 642-task SAPI split, it achieves 16.9% evaluation Pass@1 with 4.6 million forward tokens and 68,000 backward tokens. RLTL;DR with the combined GRPO and insight-SFT objective achieves 18.9% evaluation Pass@1, while insight-only training achieves 17.0%.

Method	Forward tokens	Backward tokens	Train Pass@1	Eval Pass@1
RLTL;DR	~720M	12M	21.5%	18.9%
RLTL;DR, $R$6	~720M	12M	9.3%	14.0%
RLTL;DR, $R$7	~720M	11M	6.1%	13.2%
RLTL;DR, no GRPO; insight SFT only	9.2M	839k	20.0%	17.0%
SFT on all 3.5k full rollouts	400M	6.6M	28.0%	20.9%
SFT on 1k rollouts	107M	1.9M	27.1%	21.0%
SFT on 100 rollouts	9.5M	217k	25.9%	20.8%
SFTL;DR on insights	2.9M	292k	17.8%	16.8%
SFTL;DR, deduplicated	4.6M	68k	19.1%	16.9%

The experiments support several conclusions.

Internalization is the principal transfer mechanism. With default $R$8, evaluation Pass@1 is 18.9% on the SAPI split; reducing $R$9 to 0.01 lowers it to 14.0%, and setting $r\in\{0,1\}$0 lowers it to 13.2%.

Context alone is insufficient. Insight-conditioned attempts can succeed without producing comparable no-insight performance if the insight tokens are not included in the learning signal.

Short insights generalize better than detailed diagnostics. On the SAPI hard split, TL;DR sentences obtain 19.2% evaluation Pass@1, diagnostic paragraphs 18.6%, summary plus diagnostic paragraphs 16.0%, and summary plus diagnostic plus corrected code 17.3%.

Verifier information is more important than feedback-generator strength. When failed-test information is withheld, evaluation performance falls sharply. The paper reports evaluation Pass@1 of 19.2% for the student thinking generator, 19.8% for the student non-thinking generator, 10.0% for the student without failed-test information, 22.7% for GLM 5.2 non-thinking with one insight, and 20.3% for GLM 5.2 thinking with one insight.

More insights are not uniformly better. On the held-out SAPI split, maximum insight counts of 1, 2, 4, 8, and 16 yield evaluation Pass@1 values of 18.9%, 19.3%, 17.8%, 20.3%, and 19.2%, respectively.

Insight quality must remain discriminative. If insights make all attempts trivially successful, the group can again lose relative learning signal. If they are too weak, they do not improve exploration; if they are too specific, they transfer poorly to new tasks.

7. Limitations and research directions

RLTL;DR assumes that the verifier exposes informative failure information. A verifier that returns only “incorrect” without failed tests, assertions, tracebacks, or final-state discrepancies may not support useful insight generation. The method is therefore not purely reward-only: it uses privileged verifier observations during training.

Insight generation may itself be difficult. In domains where identifying the correction is as hard as solving the task, the feedback model may produce vague, incorrect, or episode-specific guidance. The method also depends on task similarity and semantic overlap. It may fail when every task requires a unique trick, when insights contain exact hidden values or credentials, or when tasks have little reusable structure.

The reported Leetcode results are weaker than those on SAPI and Appworld, which the authors associate with highly task-specific solutions. This suggests that the compact task-to-insight mapping is more effective when procedural principles recur across tasks.

Training remains expensive because full rollouts are needed to discover verifier-grounded feedback. The relevant SAPI experiments require roughly 1 billion rollout tokens, and training runs use 8 B200 GPUs for approximately 1–4 days. Sequential collection increases sampling latency even though it reduces the need for successful independent exploration.

Incorrect insights can reinforce the wrong behavior, encourage repeated failure, or overfit to an individual episode. The method must therefore balance insight specificity and generality. Larger models may help by generating better diagnostics, recognizing verifier failures more accurately, and generalizing compact principles across related tasks, but the paper does not establish whether the mechanism scales consistently with model size.

The evaluation environments are simulated research environments. The results do not establish direct transfer to deployed products, real-world APIs, scientific research, mathematical proofs, or other environments in which verifier feedback may be less diagnostic.

Future directions include understanding the mechanism of task-to-insight generalization, testing larger models, improving insight generation with stronger teachers or structured verifier messages, combining RLTL;DR with other self-distillation methods, storing insights in external retrieval systems for frozen models, and using compact task–insight pairs for federated or low-bandwidth learning.

RLTL;DR’s central contribution is the separation of two problems that standard RLVR conflates. Sequential insight conditioning addresses exploration by turning failed attempts into guided retries. Backpropagation through the compact insights addresses transfer by internalizing the guidance so that the model can improve when the insights are absent. The resulting learning pathway is

$r\in\{0,1\}$1