Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning

Abstract: Long-context reasoning is an essential capability for LLMs, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories. Reinforcement learning (RL) has recently emerged as a dominant paradigm for improving this ability, yet existing work largely focuses on reward engineering while diverse training data remains scarce. We revisit this problem from a data-centric perspective and show that a simple yet effective data recipe alone, paired with a minimal outcome-based GRPO setup, suffices to substantially improve long-context reasoning. Our recipe targets three complementary task families -- retrieval, multi-evidence synthesis, and reasoning -- for which we construct and curate eight datasets totaling ~14K examples. Experiments on three models (Qwen3-4B/8B/30B-A3B) yield average gains of +7.2/+3.2/+6.4 points across seven long-context benchmarks, surpassing prior RL training sets. We further demonstrate that these gains transfer to agentic tasks, where continuing RL training on an agent-tuned model with our data recipe improves GAIA by +4.8 and BrowseComp by +7.0 points. We will release our datasets to facilitate future research.

• The paper introduces a data recipe that leverages a task-diverse dataset with minimal outcome-based rewards to substantially improve long-context reasoning in LLMs.
• It employs task-balanced sampling and per-dataset advantage normalization to integrate retrieval, multi-evidence synthesis, and stepwise reasoning effectively.
• Experimental results show robust benchmark gains, with improvements up to +10.5 points in reasoning tasks and transferability to extended context lengths.

Data-Centric Training for Long-Context Reinforcement Learning in LLMs

Motivation and Problem Statement

The paper "Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning" (2606.18831) addresses a core limitation facing LLMs as autonomous agents: robust reasoning over lengthy input trajectories. Despite significant advances in RL post-training, most prior work remains fixated on reward engineering to steer long-context behavior, while the diversity and scale of training data are often neglected. Sparse rewards alone have been shown to result in superficial contextual recall and shortcut reasoning, especially in long-input scenarios. This work introduces a data-centric perspective, proposing that judicious construction of task-diverse datasets, when combined with minimal outcome-based RL, can substantially improve reasoning capabilities across extended contexts.

Methodological Contributions

Task Taxonomy and Dataset Design

The authors posit that long-context reasoning hinges on three complementary capabilities:

• Retrieval: Locating relevant evidence in large and noisy contexts, overcoming shortcut strategies (e.g., lexical pattern matching) and robustly distinguishing semantically indirect or scattered information.
• Multi-evidence Synthesis: Integrating disparate, often incomplete or distractor-laden textual fragments to generate coherent answers unattainable from any single span.
• Reasoning: Performing multi-step, often mathematical or logical derivations grounded in evidence distributed throughout lengthy narratives.

Eight datasets (~14K examples) were synthesized or extended for these three categories, including FuzzyNeedle, MultiNeedle, CrossEntity, WebSearch, MultiQuery, KeyChain, LongDocQA, and LongMath, targeting fine-grained weaknesses previously documented in real-world and synthetic benchmarks.

RL Training Protocol

Training leveraged Group Relative Policy Optimization (GRPO), with task-balanced sampling and per-dataset advantage normalization to accommodate heterogeneity in reward scales and variance. Crucially, reward design was kept minimal: word-level recall for most, and an LLM-judge for equivariant answer forms or derived math, avoiding auxiliary reward shaping or step-level annotations. All RL was conducted on instances capped at 64K input tokens, but evaluation extended to 230K, testing generalization.

Experimental Results

Benchmark Performance

Three Qwen3 models (4B/8B/30B-A3B) were trained and evaluated on seven established benchmarks—spanning multi-hop QA, holistic reasoning, and synthetic long-context tasks (LongBench v1/v2, FRAMES, AA-LCR, DocFinQA, LongReason, GraphWalks). The proposed data recipe yielded consistent improvements:

• Qwen3-4B-Thinking-2507: +7.2 points average increase
• Qwen3-8B-128K: +3.2 points average increase
• Qwen3-30B-A3B-Thinking-2507: +6.4 points average increase

On reasoning-heavy benchmarks (AA-LCR, DocFinQA, LongReason), gains were particularly pronounced (+10.5, +8.6, +5.6 points respectively), outperforming prior RL data sets such as DocQA-RL-1.6K (Wan et al., 23 May 2025) and KeyChain-15K (Su et al., 21 May 2026).

Ablation Studies

Isolated and pairwise training on task categories showed each contributed to benchmark improvement, confirming their complementarity. Reasoning tasks provided the strongest single-category lift, but only the full mixture maximized performance. Task-balanced sampling and advantage normalization were necessary to prevent inter-task competition, while adding generic process rewards (LLM-as-judge for thinking trajectories) did not further improve results, indicating sufficiency of outcome-based rewards given the diverse data.

Length-Generalization

Despite training capped at 64K tokens, improvements transferred to contexts up to 512K tokens in LongBench-v2 and HELMET (Whitecross et al., 10 Apr 2026), indicating the solution is not length-anchored but instills robust, generic long-context abilities.

Agentic Task Transfer

Long-context RL-trained models were further evaluated on agent benchmarks (GAIA (OpenAI et al., 2024), BrowseComp (Wei et al., 16 Apr 2025)), both requiring tool use across web search and document fetching. Continued RL yielded +4.8 and +7.0 points improvement on GAIA Pass@3 and BrowseComp, respectively. Gains were most pronounced on difficult cases (GAIA L3, BrowseComp hard), reinforcing the practical impact of enhanced long-context reasoning for autonomous agents.

Practical and Theoretical Implications

The findings demonstrate that, contrary to recent reward engineering approaches, a high-quality, diverse data recipe for long-context RL suffices to substantially elevate LLM reasoning performance. This underscores the primacy of data curation over reward definition in long-context agentic scenarios. The robust length-generalization and transferability to agent tasks suggest that training mixtures exercising retrieval, synthesis, and reasoning should be foundational to post-training pipelines.

Practically, this allows for strong, scalable long-context RL improvements without dependence on step-wise evidence annotations or reward design complexities. The approach is agnostic to model size (4B–30B) and robust to context extension, offering reproducible gains across both reasoning benchmarks and real-world agent benchmarks.

Theoretically, this advances understanding of core abilities underlying long-context performance and challenges assumptions that reward engineering is necessary for deep reasoning in extended contexts. It also calls for further investigation into curriculum construction, synthetic-real distribution alignment, and scaling beyond 30B parameters or other model families.

Future Directions

• Extension to larger models and alternative architectures is warranted.
• Scaling RL training to >64K context lengths to test limits of length-generalization.
• Incorporation of real-world, non-synthetic long-context data for improving domain robustness.
• Deeper exploration of the correlation between long-context reasoning gains and agentic performance across more complex tool use and interaction scenarios.

Conclusion

A unified, task-diverse data recipe for long-context RL enables consistent, substantial improvements in reasoning capabilities of LLMs. This data-centric approach is demonstrably superior to prior reward engineering methods and provides both theoretical insight and practical utility for strengthening agentic LLMs operating over lengthy, complex input trajectories. The datasets and protocol released in this work facilitate reproducibility and further research in scalable, long-context RL.