Agentic-Ideation: Sample Efficient Agentic Trajectories Synthesis for Scientific Ideation Agents

Abstract: Ideation plays a pivotal role in scientific discovery. Recent LLM, especially AI Scientist systems, show promising potential for automated ideation. However, existing approaches predominantly rely on pre-defined agentic workflows. This constraint severely limits the flexibility required to navigate the vast search space of scientific literature and the complex action space of research reasoning. Recently, training Agentic LLMs has emerged as a promising direction, offering flexible reasoning frameworks and the capability for autonomous tool utilization. However, there remains a non-trivial challenge: applying previous agentic data synthesis methods to scientific ideation suffers from prohibitively high data synthesis cost. To bridge this gap, we propose Agentic-Ideation, a novel framework comprising an automated trajectory synthesis pipeline and a specialized agentic LLM trained for scientific ideation. Specifically, we first define a comprehensive tool space incorporating three external tools and three cognitive tools. Then we introduce an Oracle-Guided Data Synthesis strategy. By leveraging a reference idea as oracle guidance, this approach steers the multi-agent system to efficiently reconstruct the logical reasoning and tool invocation paths, transforming aimless trial-and-error into directed trajectory generation. Finally, we train the agent on these synthesized trajectories, employing a masking strategy on tool execution results. This ensures the model focuses on decision-making logic without interference from external feedback. Experimental results demonstrate that our method outperforms the SOTA workflow-based baseline by \textbf{11.91\%} in overall quality. Furthermore, our approach improves the sample efficiency of high-quality data synthesis by \textbf{over 10$\times$}.

• The paper introduces a novel oracle-guided multi-agent framework that significantly improves sample efficiency in generating research ideation trajectories.
• It leverages a hybrid tool space combining external retrieval and internal cognitive utilities, achieving over 10× efficiency compared to traditional rejection sampling.
• Empirical results across major conferences demonstrate substantial improvements in novelty, significance, and feasibility, validated by rigorous human and automated evaluations.

Agentic-Ideation: Sample Efficient Agentic Trajectories Synthesis for Scientific Ideation Agents

Motivation and Challenges in Scientific Ideation Automation

Automated scientific ideation is central to the broader vision of AI-driven scientific discovery. Despite substantial advances in LLM-based systems for downstream tasks, existing ideation solutions remain constrained by rigid, workflow-centric paradigms. These systems rely heavily on pre-defined action sequences and prompt engineering, resulting in limited adaptability across the dynamic and open-ended research landscape. Training Agentic LLMs—models capable of autonomous tool orchestration and reasoning—presents a promising avenue, yet suffers from prohibitive data synthesis costs due to inefficient exploration and lack of ground-truth signals.

Agentic-Ideation Framework Design

Agentic-Ideation addresses sample inefficiency and rigidity in scientific ideation via an integrated, oracle-guided multi-agent architecture. The core contributions are threefold: (i) hybrid tool space construction combining retrieval and cognitive utilities, (ii) oracle-guided hierarchical multi-agent data synthesis, and (iii) agentic supervised fine-tuning with masking on environmental feedback.

The tool space, $\mathcal{T}$, unifies external retrieval tools (Search, Get_References, Get_Cited) for active literature exploration and cognitive tools (Analyse_Gap, Ideation, Reflection) for abstract reasoning, gap detection, proposal generation, and iterative critique.

Figure 1: The AgenticIdeation framework comprises Oracle-Guided Agentic Data Synthesis and Agentic Supervised Fine-Tuning, utilizing oracle guidance for sample-efficient synthesis and masking for robust agentic training.

Hierarchically, a Planner establishes macro-level ideation strategy, while a Controller executes stepwise reasoning and tool invocations. Oracle guidance is implemented by infusing a reference idea as $I_{ref}$, enabling directed trajectory reconstruction in lieu of aimless trial-and-error. The Controller generates dual cognitive streams: private (oracle-informed) for optimal gap bridging and public (history-limited) for naturalistic agentic reasoning.

Through this mechanism, each generated trajectory encapsulates logical reasoning, tool exploits, and self-reflection, terminating upon successful recovery of the reference idea. Masking during SFT prohibits memorization of tool outputs, optimizing strictly decision logic and facilitating generalization.

Figure 2: Oracle guidance eliminates inefficiency endemic to rejection sampling, yielding a >10$\times$ improvement in synthesis efficiency versus traditional methods.

Empirical Evaluation and Results

Extensive experiments were conducted on datasets derived from ICLR, ICML, and NeurIPS 2025 papers. Ideas were generated based solely on anchor references, forcing models to operate under minimal context. Baselines were selected from prominent workflow and multi-agent systems (ResearchAgent, SciPIP, SciMON, VirSci) and standard prompting on Qwen3-8B.

Quantitatively, Agentic-Ideation achieved a mean Overall score of 6.67, outperforming all baselines by 11.91%, with statistically significant gains in Novelty (+11.40%), Significance (+11.14%), and Feasibility (+8.70%). Cross-domain robustness was confirmed across NLP, CV, and other fields. The backbone-only agent (Qwen3-8B) suffered in both creativity and groundedness, underscoring the necessity of active tool integration for high-fidelity ideation.

Data synthesis efficiency was measured by rollout attempts per valid sample; Agentic-Ideation averaged 1, compared to 12 for rejection sampling, representing a computational breakthrough in agentic data aggregation.

Ablation confirmed indispensability of both external and internal tools, with Search and Analyse_Gap contributing most critically—removal disproportionately impacted knowledge boundary expansion and innovation, respectively.

Human panel evaluation, adhering to a double-blind protocol, corroborated automated results, with Agentic-Ideation scoring highest in every metric. Theoretical depth and feasibility were most valued by expert judges.

Implications, Limitations, and Future Directions

Agentic-Ideation's agentic trajectory synthesis enables scalable training for LLMs capable of flexible research reasoning, effectively overcoming longstanding bottlenecks in ideation data scarcity. The oracle-guided methodology transforms the open-ended search space into a tractable trajectory reconstruction landscape, unlocking high-quality, sample-efficient supervision. Trained agents demonstrate strong generalization, producing scientifically novel, significant, and feasible proposals validated by both human and automated evaluators.

Practically, this paradigm supports the development of autonomous AI scientists, democratizing access to high-level ideation and accelerating multidisciplinary research cycles. Theoretical implications concern the decoupling of reasoning logic from deterministic environmental feedback, which may improve robustness and transferability across domains.

Limitations include dependency on the backbone scale, which bounds reasoning depth, and the current focus on retrieval-centric tools, lacking active execution or simulation capabilities. Future work should expand agentic tool spaces to allow for direct hypothesis testing and code execution, further bridging the gap between ideation and experimental validation.

Conclusion

Agentic-Ideation introduces a novel oracle-guided framework for agentic trajectory synthesis and supervised fine-tuning, resolving both flexibility and data efficiency challenges in scientific ideation for LLM agents. Empirical and human validation confirm state-of-the-art performance, sample efficiency, and practical innovation potential. This approach establishes a new foundation for robust, adaptive AI scientist systems and advances automated research ideation across computational domains.