Training AI Co-Scientists Using Rubric Rewards (2512.23707v1)

Abstract: AI co-scientists are emerging as a tool to assist human researchers in achieving their research goals. A crucial feature of these AI co-scientists is the ability to generate a research plan given a set of aims and constraints. The plan may be used by researchers for brainstorming, or may even be implemented after further refinement. However, LLMs currently struggle to generate research plans that follow all constraints and implicit requirements. In this work, we study how to leverage the vast corpus of existing research papers to train LLMs that generate better research plans. We build a scalable, diverse training corpus by automatically extracting research goals and goal-specific grading rubrics from papers across several domains. We then train models for research plan generation via reinforcement learning with self-grading. A frozen copy of the initial policy acts as the grader during training, with the rubrics creating a generator-verifier gap that enables improvements without external human supervision. To validate this approach, we conduct a study with human experts for machine learning research goals, spanning 225 hours. The experts prefer plans generated by our finetuned Qwen3-30B-A3B model over the initial model for 70% of research goals, and approve 84% of the automatically extracted goal-specific grading rubrics. To assess generality, we also extend our approach to research goals from medical papers, and new arXiv preprints, evaluating with a jury of frontier models. Our finetuning yields 12-22% relative improvements and significant cross-domain generalization, proving effective even in problem settings like medical research where execution feedback is infeasible. Together, these findings demonstrate the potential of a scalable, automated training recipe as a step towards improving general AI co-scientists.

• The paper introduces a novel reinforcement learning framework that uses rubric rewards for training AI co-scientists to generate research plans.
• Key methodology includes automated extraction of research goals and grading rubrics from scientific papers to provide fine-grained feedback.
• Empirical evaluations demonstrate 12–22% improvements in performance across domains, validated through both human and LLM assessments.

Training AI Co-Scientists with Rubric Rewards: An Expert Analysis

Motivation and Scope

This work addresses the training of LLMs as AI co-scientists capable of generating rigorous, constraint-satisfying research plans for open-ended scientific goals. Existing paradigms in AI for Science have advanced end-to-end scientific discovery within programmatically simulable domains, but remain fundamentally limited where digital execution is infeasible or real-world experimentation is costly (notably in medicine and many subfields of basic science). There is an acute need for scalable approaches that extract supervision beyond end-to-end execution or sparse, expensive human annotation, especially for complex planning tasks central to scientific inquiry.

Methodological Framework

The central methodological innovation is the scalable extraction of research goals and associated grading rubrics from a large corpus of scientific papers, enabling reinforcement learning (RL) with rubric-based self-grading to train plan-generating models.

Figure 1: Summary of the methodology—automated extraction of research goals and rubrics, self-grading with a frozen grader model, and RL signal from rubric satisfaction.

The process elides direct experiment execution as the primary RL signal, substituting it with structured, goal-specific rubric evaluation. Key methodological components include:

• Automated Task and Rubric Extraction: LLMs are prompted to extract from papers both explicit research goals (providing context, constraints, and uncertainties) and goal-specific grading rubrics that codify must-have requirements for a valid plan. These are filtered and assessed for quality by a distinct LLM serving as a sample selector.
• Self-Grading RL Loop: The plan generation model is trained using Group Relative Policy Optimization (GRPO), with a frozen copy of the initial model serving as the reward model (grader) using privileged access to goal-specific rubrics. The rubric-based reward measures compliance at a fine-grained level, avoiding noisy scalar feedback.
• Integration of General Guidelines: In addition to instance-specific rubrics, general guidelines are systematically incorporated for holistic plan evaluation, controlling for gaps in coverage and common failure modes not specific to the research goal.

This protocol is domain-agnostic and scalable, requiring only access to paper text and a strong LLM for data extraction.

Evaluation and Experimental Results

Human Expert Assessment

A robust human study was conducted in the ML domain, leveraging 225 total expert hours to evaluate the effect of fine-tuning Qwen-3-30B-A3B using the proposed rubric-RL approach. Key findings:

• The finetuned model's plans were preferred in 70% of research goals ($p < 0.01$), with annotators rating rubric quality highly (mean 4.3/5; 84% of rubric items deemed essential).
• Experts found the finetuned plans significantly more sound, constraint-abiding, and more likely to lead to successful research outcomes, albeit sometimes more complex and less minimal than the base model’s outputs.

Automated, Cross-Domain, and Model-Family Generalization

Performance was benchmarked by rubric-graded evaluations using a jury of strong frontier LLMs (GPT-5-Thinking, Claude-4-Sonnet, Gemini-2.5-Pro) across the ML, medical, and arXiv domains.

Figure 3: Research plan generation scores across domains and models. Finetuned variants yield 12–22% improvements over initial policies and achieve notable cross-domain improvements.

Findings include:

• 12–22% relative improvements in rubric satisfaction across ML, medical, and arXiv datasets for domain-specific finetunes over the Qwen-3-30B baseline, with competitive performance to stronger closed models (e.g., Grok-4-Thinking).
• Strong cross-domain generalization: Medical-domain finetunes substantially improved performance on ML and ArXiv goals, exceeding ArXiv-specific finetuning in some transfer evaluations.
• The approach is robust to underlying backbone architecture: comparable gains were obtained for Llama-3.1 and Gemma-3 models.

  Figure 2: Finetuning improvements across multiple backbone families, based on rubric grading from a diverse LLM jury.

  

  Figure 4: Comparable validation performance between instruct and thinking-variant models; increased compute for "thinking" did not yield gains in this regime.

Ablation and Sensitivity Analyses

Detailed ablations reveal several strong and somewhat unexpected findings:

• Supervised fine-tuning (SFT) degrades long-form research planning performance versus RL with rubric rewards (see quantitative and qualitative breakdowns). The RL recipe is crucial for generalization and task alignment.
• Reward model scaling and rubric informativeness are both critical: Larger, more capable graders as reward models yield better alignment with strong external evaluators. Removing either instance-specific rubrics or general guidelines results in large degradation (Figure 6, subplots 1 & 3).
• Disabling KL regularization in the RL objective further increases performance for long-form planning, in contrast to established practice, as plans benefit from less-constrained exploration under strict rubric-based rewards (Figure 6, subplot 2).
• Best-of-3 LLM extraction and reference solution augmentations contribute less, suggesting diminishing returns from data selection once strong rubrics are available.

Figure 5: Ablation analyses—reward model scaling, KL penalty, rubric specificity/generality, filtering, and reference solution provision; only rubric-based and general-guideline rewards are critical drivers of performance.

Decomposing guideline-wise performance across domains exposes cost/effort efficiency as a persistent failure point, suggesting the self-grading protocol enables verbosity or unnecessary complexity, a limitation of current reward models.

Figure 6: Finer-grained breakdown by evaluation guideline across ML, ArXiv, and PubMed benchmarks, highlighting strengths and remaining gaps of the finetuned models.

Theoretical and Practical Implications

Theoretical Contributions

This work advances our understanding of leveraging static scientific text (papers) as a scalable training resource for open-ended, high-expertise tasks. The instance-specific rubric mechanism provides a robust framework for aligning LLMs with nuanced, domain-appropriate desiderata that cannot be distilled from fixed-format, synthetic data or passive imitation learning. Additionally, the generator-verifier separation establishes a practical instantiation of the generator-verifier gap, offering a training signal aligned with expert verification without resorting to intractable experiment execution. The demonstrable cross-domain generalization supports the hypothesis that planning and scientific reasoning skills are transferable and can be abstracted across diverse scientific domains.

Practical Impact

Pragmatically, the scalable data extraction pipeline, ResearchPlanGen, and rubric-RL framework substantially lower the cost and barrier for training AI co-scientists that generate high-quality plans for arbitrary research goals. Notably, fine-tuned models offer significant improvements under both human and strong LLM evaluation, with empirical evidence of usefulness for graduate-level research planning and as structured brainstorming partners. The approach is feasible in high-stakes, slow-feedback environments such as medicine, circumventing the impracticality of trial-and-error RL.

Sensitivity analyses emphasize the importance of careful reward model/grade design: overoptimization for weak graders or naively applied SFT leads to degraded performance, supporting broader caution around reward model overoptimization and data contamination in RLHF settings.

Limitations and Future Directions

The study explicitly recognizes key limitations:

• The evaluation is restricted to static expert and LLM jury preferences; real-world research outcomes from executing the generated plans remain largely untested, especially outside ML.
• Rubric extraction and grading are inherently noisy; further improvements await stronger and more reliable graders/reward models.
• Plans produced under the current protocol frequently favor completeness over parsimony, with the reward model less sensitive to unnecessary complexity.
• LLM-generated rubrics may bias training toward explicit, paper-reported criteria, potentially underweighting innovative or unconventional strategy planning.

Future work will require extending automated evaluation paradigms, improving rubric and grader capabilities, and investigating the efficacy of rubric-guided RL with integrated tool-use and experiment-execution feedback, as well as more direct and scalable human-in-the-loop data curation and plan validation. Developing frameworks for leveraging structured language feedback in RL objectives and expanding cross-domain capability to incorporate reasoning under uncertainty, creative abduction, and open-ended ideation are promising avenues.

Conclusion

This paper presents a principled and empirically validated recipe for learning generalist AI co-scientists via RL with self-rewarding, instance-specific rubrics extracted from the collective corpus of scientific literature. Models trained with this recipe achieve statistically significant gains in plan quality, demonstrate robust cross-domain generalization, and set a practical standard for scalable, automated co-scientist development. The findings highlight both the tractability and the limits of rubric-based training; while substantial improvements have been realized, modeling true scientific creativity and judgment remains an open challenge necessitating further coordination of rubric extraction, reward model development, and evaluation methods. The ResearchPlanGen pipeline and associated training regime constitute an important step toward robust, transferable, and scalable LLM planning capabilities for scientific research.