MedAI: Evaluating TxAgent's Therapeutic Agentic Reasoning in the NeurIPS CURE-Bench Competition

Abstract: Therapeutic decision-making in clinical medicine constitutes a high-stakes domain in which AI guidance interacts with complex interactions among patient characteristics, disease processes, and pharmacological agents. Tasks such as drug recommendation, treatment planning, and adverse-effect prediction demand robust, multi-step reasoning grounded in reliable biomedical knowledge. Agentic AI methods, exemplified by TxAgent, address these challenges through iterative retrieval-augmented generation (RAG). TxAgent employs a fine-tuned Llama-3.1-8B model that dynamically generates and executes function calls to a unified biomedical tool suite (ToolUniverse), integrating FDA Drug API, OpenTargets, and Monarch resources to ensure access to current therapeutic information. In contrast to general-purpose RAG systems, medical applications impose stringent safety constraints, rendering the accuracy of both the reasoning trace and the sequence of tool invocations critical. These considerations motivate evaluation protocols treating token-level reasoning and tool-usage behaviors as explicit supervision signals. This work presents insights derived from our participation in the CURE-Bench NeurIPS 2025 Challenge, which benchmarks therapeutic-reasoning systems using metrics that assess correctness, tool utilization, and reasoning quality. We analyze how retrieval quality for function (tool) calls influences overall model performance and demonstrate performance gains achieved through improved tool-retrieval strategies. Our work was awarded the Excellence Award in Open Science. Complete information can be found at https://curebench.ai/.

• The paper presents a rigorous evaluation of TxAgent's iterative retrieval-augmented tool-calling method in clinical settings, showcasing improved pharmacological reasoning.
• It integrates DailyMed with a unified biomedical tool suite, enhancing retrieval accuracy of drug information and reducing error propagation in multi-hop reasoning.
• The study underscores the importance of fine-tuning and context-aware retrieval in achieving reliable agentic therapeutic reasoning for safe clinical decision-making.

Comprehensive Evaluation of TxAgent's Therapeutic Agentic Reasoning in CURE-Bench

Introduction

This paper presents a rigorous analysis and enhancement of TxAgent—a therapeutic reasoning agent leveraging iterative Retrieval-Augmented Generation (RAG) and agentic function-calling—evaluated within the CURE-Bench NeurIPS 2025 competition framework (2512.11682). The clinical setting addressed by the competition demands robust multi-hop reasoning, correctness, and safety, scrutinizing not only final outcomes but also the intermediate reasoning trace and sequence of tool invocations. This environment requires that agentic AI systems integrate heterogeneous clinical information, prioritize up-to-date pharmacological data, and optimize each tool call to minimize both error propagation and unnecessary context expansion.

Agentic and Tool-Augmented Reasoning in Clinical Contexts

TxAgent is built atop a fine-tuned Llama-3.1-8B, augmented by a Qwen2-1.5B retriever, and is centered around ToolUniverse—a unified biomedical tool suite combining FDA Drug API, OpenTargets, Monarch, and, as extended here, DailyMed. The system operationalizes agentic workflows by decomposing therapeutic questions, reformulating them for retrieval, and orchestrating sequence-to-sequence tool invocations. Critically, medical agentic systems face unique demands for verifiability and error minimization at each reasoning step, with suboptimal tool selection or retrieval frequently resulting in clinical irrelevance.

The TxAgent workflow involves several iterative ToolRAG cycles: Llama-3.1-8B reformulates the intention-centric query, Qwen2-1.5B encodes this against ToolUniverse function descriptions, and the top-$k$ (here, $k=10$) candidate functions are surfaced using cosine similarity. TxAgent then autonomously selects, parametrizes, invokes, and processes tool outputs, continuing cyclically until sufficient information has been aggregated for decision-making or until a terminating condition is met.

Enhancement of Pharmacological Retrieval: DailyMed Integration

One core focus of the work is the augmentation of TxAgent's tool-calling framework with DailyMed, a resource offering structured, version-controlled drug labeling information. DailyMed complements the granular, metadata-centric design of openFDA. While openFDA permits focused, token-efficient queries, its output lacks narrative depth necessary for broader reasoning, often necessitating multiple calls for a complete context. The integration of DailyMed via ToolUniverse allows direct access to comprehensive clinical drug narratives, mitigating prior limitations and enabling the agent to autonomously invoke this additional resource.

Experimental Analysis

Retriever Comparison and Tool-Calling Performance

The evaluation benchmarks sparse (e.g., BM25) and dense retrievers for tool selection, as well as the effect of DailyMed integration on retrieval accuracy.

Figure 1: Performance comparison across dense retrievers, sparse BM25, and TxAgent’s fine-tuned Qwen2-1.5B retriever with and without DailyMed integration.

Sparse retrievers like BM25 underperform due to reliance on lexical matching, unable to contend with the semantically complex function descriptions in ToolUniverse (typically capped at two sentences per tool). Dense retrievers, albeit variable in runtime, generally improve selection but do not surpass the Qwen2-1.5B retriever, especially when fine-tuned on therapeutic tool-calling data. The introduction of DailyMed enables further performance gains, as its concise function description aligns well with the retriever and supports retrieval of clinically authoritative outputs.

Impact of Retrieval Quality and Model Parametric Knowledge

The experiments leverage fixed-retrieval setups across LLMs (e.g., open-source GPT-OSS, Llama variants, Gemini), examining performance on open-ended multiple-choice (OE-MC) and standard multiple-choice (MC) datasets, both with and without retrieval aiding.

Figure 2: OE-MC question accuracy across LLMs under RAG and non-RAG (frozen) retrieval settings, with option permutation and baseline TxAgent reference.

Figure 3: MC question accuracy across LLMs under analogous settings to Figure 2.

Results indicate marked performance degradation across all models when deprived of retrieval-augmented context, affirming that parametric knowledge alone is insufficient for the majority of CURE-Bench queries. Option permutation further exacerbates this effect—a drop much steeper for OE-MC questions due to their intrinsically higher reasoning complexity. Fine-tuned Llama-3.1-8B outperforms its vanilla counterpart when given identical retrieved context, demonstrating the value of supervised instruction on biomedical tool workflows. Notably, competitive performance is achievable by smaller models (e.g., Gemma3-4B, Qwen3-4B) in RAG settings, underscoring both the transferability of the retrieval framework and the relative sufficiency of smaller architectures under well-structured context injection.

Implications and Theoretical Insights

The study underscores the necessity of fine-grained, context-aware tool selection in therapeutic AI systems, where even minor retrieval errors materially affect both reasoning trajectories and answer correctness. The integration of DailyMed demonstrates that access to comprehensive, versioned drug information is a critical differentiator in therapeutic agentic reasoning, particularly in contrast to parametric model limitations.

From a practical standpoint, these insights suggest that well-engineered agentic frameworks can elevate open-source or compact models to parity or near-parity with larger or proprietary systems, provided high-quality, authoritative retrieval tools are available. Theoretically, the work reinforces the importance of explicit supervision not only for final answers but also for intermediate token-level reasoning and function-call behavior in safely deploying AI in high-stakes biomedical environments.

Future Directions

The results indicate several promising directions:

• Adaptive Retrieval Expansion: Balancing fine-grained tool-calling (which optimizes for context size) against comprehensive sources such as DailyMed (which may increase context window requirements) could be framed as an explicit optimization problem, particularly as context window sizes and computational budgets scale.
• Semantic Reasoning Trace Supervision: More granular supervision of intermediate reasoning steps and execution traces could further reduce tool-calling failures and increase verifiability.
• Generalization to Additional Medical Workflows: The demonstrated framework is applicable beyond pharmacology (e.g., genomics, imaging), suggesting agentic architectures should focus on extensible, modular tool universes.
• Lightweight Models with Robust Retrieval: Investment in retrieval integration and tool curation may obviate the need for excessively large LLMs for medical reasoning tasks.

Conclusion

The evaluation and enhancement of TxAgent within the CURE-Bench framework provide critical evidence for the role of targeted retrieval strategies—particularly the integration of authoritative and comprehensive biomedical resources—in advancing agentic AI applied to therapeutic decision-making. The experiments affirm that function-level retrieval quality, combined with fine-tuning for specific tool-calling patterns, is essential for robust, clinically relevant automated reasoning. These findings set a clear research agenda prioritizing retrieval and agentic workflow engineering, with implications for both the safe deployment of AI in clinical practice and the design of future multimodal biomedical agents.