DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research (2511.19399v2)

Abstract: Deep research models perform multi-step research to produce long-form, well-attributed answers. However, most open deep research models are trained on easily verifiable short-form QA tasks via reinforcement learning with verifiable rewards (RLVR), which does not extend to realistic long-form tasks. We address this with Reinforcement Learning with Evolving Rubrics (RLER), in which we construct and maintain rubrics that co-evolve with the policy model during training; this allows the rubrics to incorporate information that the model has newly explored and to provide discriminative, on-policy feedback. Using RLER, we develop Deep Research Tulu (DR Tulu-8B), the first open model that is directly trained for open-ended, long-form deep research. Across four long-form deep research benchmarks in science, healthcare and general domains, DR Tulu substantially outperforms existing open deep research models, and matches or exceeds proprietary deep research systems, while being significantly smaller and cheaper per query. To facilitate future research, we release all data, models, and code, including our new MCP-based agent infrastructure for deep research systems.

• The paper reveals that DR Tulu-8B, using RL with evolving rubrics, significantly outperforms prior models with gains up to 53.4 points while maintaining cost efficiency.
• The methodology employs instance-specific, search-grounded rubrics and dynamic tool selection to adaptively guide deep research and evidence attribution.
• Experimental benchmarks across datasets like GeneticDiseasesQA and ScholarQA-CS2 confirm the agent’s robust generalization and scalable performance.

DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

Overview and Motivation

DR Tulu-8B addresses the challenge of training open-domain LLMs for "deep research" tasks, defined as those requiring multi-step planning, external tool use, and synthesis of well-cited, comprehensive responses across complex domains such as science, healthcare, and general knowledge. Prior open-source solutions focused either on heuristic prompting or reinforcement learning with static, verifiable reward signals suitable for short-form tasks, which fail to transfer to the demands of long-form research and evidence attribution. The paper proposes Reinforcement Learning with Evolving Rubrics (RLER) as an extensible protocol wherein instance-specific rubrics dynamically co-evolve with the policy LM, grounding evaluation in newly searched and retrieved knowledge that is germane to the model's current decision space.

Reinforcement Learning with Evolving Rubrics (RLER)

RLER is formulated to overcome the key limitations of static rubric rewards and closed-book verification in RL for deep research:

• Instance-specific, search-grounded rubrics are generated before and during training via an external LM, incorporating retrieved context to evaluate factuality and comprehensiveness.
• Rubric buffer management filters rubric candidates by their discriminative variance over recent rollout generations, ensuring adaptive supervision while capping evaluation costs.
• Online rubric evolution treats positive and negative grading criteria as first-class citizens, penalizing reward hacking and emergent undesirable behavior by incorporating negative rubrics responsive to new policy behaviors.

This protocol explicitly closes the loop between policy improvement and reward evolution, mitigating the train-test mismatch inherent in short-form RLVR recipes.

Figure 1: The RLER training workflow: model rollouts generate search traces, which serve as the basis for new rubric synthesis, maintaining alignment of verification criteria with on-policy exploration.

Model and Infrastructure: DR Tulu-8B Agent

DR Tulu-8B comprises supervised fine-tuning (SFT) for agentic tool use and citation format, followed by online RL using RLER. SFT leverages trajectories generated by strong teacher models (e.g., GPT-5, Ai2 ScholarQA) over diverse long-form and short-form prompts, constructed from filtered public data (e.g. SearchArena, OpenScholar) to curate robust research skills. RL optimization uses the GRPO algorithm with asynchronous rollout and tool invocation, supporting flexible orchestration of search, browsing, and paper-retrieval tools.

The agent infrastructure (dr-agent-lib) is MCP-compliant, facilitating seamless integration of heterogeneous tools, caching, and lock-based concurrency, thereby enabling efficient high-throughput training and scalable inference. DR Tulu adaptively selects tools at inference, maximizing both evidence quality and cost-efficiency.

Figure 2: DR Tulu-8B achieves superior performance-to-cost tradeoff, outperforming larger open models and matching proprietary systems along the Pareto frontier of research agent efficiency.

Experimental Benchmarks and Ablation Studies

Evaluation spans four major research agent benchmarks: AstaBench-ScholarQA-CS2, HealthBench, ResearchQA, and DeepResearchBench, as well as a new out-of-domain, clinically relevant dataset, GeneticDiseasesQA.

Key results include:

• Performance: DR Tulu-8B significantly surpasses previous best open models (e.g., Tongyi DeepResearch-30B) by 13.7–53.4 points average gain and is competitive with proprietary systems such as OpenAI Deep Research and Gemini3 Pro + Search, despite being 4×–8× smaller.
• Cost-efficiency: On ScholarQA-CS2, DR Tulu-8B achieves an inference cost nearly three orders of magnitude lower than OpenAI’s solution (USD $0.0019$ vs. $1.8$ per query), mainly owing to judicious tool selection and free resource utilization.
• Generalization: On GeneticDiseasesQA, a clinical variant analysis dataset, DR Tulu-8B matches proprietary agents on evidence synthesis and citation support, demonstrating domain robustness.

  Figure 3: Evaluation on GeneticDiseasesQA: DR Tulu is competitive with closed research agents and delivers robust evidence synthesis in a clinical setting.

• RLER Impact: Ablation studies reveal that search-based and evolving rubrics deliver consistent performance improvement compared to static, closed-book rubrics. Negative evolving rubrics rapidly suppress spurious behaviors (e.g., code emission in answers) not anticipated by static verification criteria.

  Figure 4: Negative evolving rubrics penalize undesirable outputs, e.g., unsolicited Python code, curbing reward hacking during RL.

• SFT Mixture Scaling: Including both long-form and short-form tasks in SFT yields the strongest generalization to both types of queries. RL training benefits increasingly from stronger SFT initialization, but continual RL steps furnish additional steady gains across all datasets.

  Figure 5: SFT ablation: mixture scaling shows diminishing returns for long-form tasks, but steady improvement on short-form QA.

  

  Figure 6: RL training trajectories for various SFT base models: marginal RL gains decrease with stronger SFT initialization, but final RL performance scales consistently.

Adaptive Tool Usage and Retrieval Behavior

DR Tulu-8B displays task-adaptive orchestration of tools and resource domains, selecting paper search for science tasks, web search for healthcare and general queries, and minimizing browsing unless required, aligning with the specific demands inferred from query context.

Figure 7: Tool call distribution across benchmarks: DR Tulu dynamically adjusts its tool usage profile based on domain.

Figure 8: DR Tulu’s targeted domain retrieval: scientific and medical datasets result in retrieval from authoritative, niche sources, while general QA leans on open web platforms.

Implications and Future Directions

The DR Tulu framework introduces a scalable paradigm for reward evolution in RL for open research agents, with several theoretical and practical implications:

• Scalable verification: RLER permits scaling verifier compute in terms of knowledge context, rather than merely input length or reasoning depth, offering a new axis for robust reward modeling in language agent training.
• Generalized agentic behavior: Dynamic integration of heterogeneous tools and rubric types supports cross-domain adaptation, crucial for expert-level synthesis in specialized domains.
• Foundation for open research: Full open-source release of data, code, models, and agent libraries establishes an extensible foundation for future research agent development and benchmarking.
• Train-test mismatch and reward hacking: Observed divergence between training reward and benchmark metrics highlights the need for continued research into rubric alignment, judge model variance, and evaluation protocol standardization.

Conclusion

DR Tulu-8B represents a substantial advancement in open deep research agent training, demonstrating that instance-specific, evolving, and search-grounded rubrics in RL can unlock robust long-form synthesis and evidence attribution in domains previously dominated by closed models. The cost-efficiency, extensibility, and generality of this approach provide a promising template for future agentic AI research, with implications spanning scalable evaluation, tool-rich reasoning, and specialized automation in science and medicine.