---
title: 'Doctor-R1: AI Agent for Clinical Inquiry'
url: https://www.emergentmind.com/topics/docr1
type: topic
---

# Doctor-R1: AI Agent for Clinical Inquiry

DocR1 most directly denotes **Doctor-R1**, an AI doctor agent for outpatient clinical inquiry trained with **experiential agentic reinforcement learning** to combine strategic multi-turn inquiry, empathetic communication, and clinical decision-making [2510.04284]. It is designed for clinical interactions in which the agent must ask high-yield questions under uncertainty rather than solve a static benchmark item in a single turn. In the supplied literature, similar shorthand also appears for unrelated systems, which suggests a degree of bibliographic ambiguity; in the medical setting, however, the referent is the Doctor-R1 framework built around a multi-agent interactive environment, a two-tiered reward architecture, and an experience repository that retrieves high-quality prior trajectories during policy learning [2510.04284].

## 1. Clinical inquiry as a sequential decision problem

Doctor-R1 is motivated by a specific gap in medical LLMs: frontier and specialized models can achieve strong performance on static decision benchmarks such as **USMLE**, **MedQA**, and **MMLU**, yet often degrade in open-ended, multi-turn clinical scenarios that require strategic information gathering, hypothesis revision, and patient-facing communication [2510.04284]. The failure modes identified in the framework are concrete. Such models may follow generic scripts instead of adapting to patient responses, miss high-yield questions that rapidly surface red flags, and communicate in a non-empathic, overly technical style.

The framework formalizes real outpatient interaction as **“Dynamic Inquiry”**, a sequential and partially observable process in which the clinician iteratively forms differential diagnoses, asks high-yield, hypothesis-driven questions, reassesses after new evidence, and communicates clearly and compassionately. “High-yield questions” are defined operationally as questions optimized for diagnostic utility and safety or triage value, especially those targeting the most uncertain or high-risk branches of the differential rather than low-yield demographic or generic symptom prompts [2510.04284].

A recurring misconception in the surrounding literature is that better static medical QA performance should directly transfer to consultation quality. Doctor-R1 is explicitly structured against that assumption. Its objectives are not limited to final diagnosis or recommendation accuracy, but include strategic multi-turn inquiry, empathetic communication, and learning from prior trajectories. This suggests that Doctor-R1 treats consultation not as answer generation alone, but as a policy-learning problem over interaction histories.

## 2. POMDP formulation and multi-agent environment

Doctor-R1 models outpatient interaction as a POMDP with tuple $\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, \mathcal{R}\rangle$ [2510.04284]. The latent patient scenario is the state $s \in \mathcal{S}$; doctor utterances are actions $a_t \in \mathcal{A}$; dialogue history is the observation $o_t \in \mathcal{O}$; and rewards are composed of turn-level and terminal components. The doctor policy is written as $\pi_\theta(a_t \mid o_t)$, and the optimization target is

$$
J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\Big[\sum_{t=0}^{T} \gamma^{t}\, r_t\Big].
$$

Three agents define the environment: a **Doctor Agent** as the learnable policy, a **Patient Agent** as an LLM simulator conditioned on a case scenario, and a **Consultation Evaluator** as an LLM judge that returns both process and outcome rewards. The doctor sees only the dialogue history and not the latent patient state, which makes information gathering central rather than incidental.

| Component | Role | Instantiation |
|---|---|---|
| Doctor Agent | Generates questions, diagnoses, and recommendations | Qwen3-8B |
| Patient Agent | Simulates patient responses and environment dynamics | Qwen3-8B |
| Consultation Evaluator | Produces turn-level and terminal rewards | Qwen3-8B |

Each turn contains a reasoned **“think”** block and an **“answer”** block, with a strict template. If information is insufficient, the answer is a single question: `<think>[reasoning]</think><answer>Question: …</answer>`. Otherwise, the model outputs a recommendation: `<think>[reasoning]</think><answer>Recommendation: …</answer>`. A trajectory is represented as $\tau=\{(o_0,a_0,r_0),\dots,(o_T,a_T,r_T)\}$, with dialogue history updated by the patient response after each doctor action [2510.04284].

This design has two consequences. First, the framework is intrinsically interactive rather than prompt-only. Second, the output format itself becomes verifiable and trainable, since question-asking and recommendation issuance are distinct action types within the same policy.

## 3. Experiential agentic reinforcement learning

Doctor-R1 is positioned against **RLHF** and **SFT** by training the model as an interactive policy in a closed-loop multi-agent environment rather than as a single-turn preference-aligned generator [2510.04284]. The policy optimization method is **GRPO**, whose listwise loss contrasts one chosen response against a group of rejected responses:

$$
\mathcal{L}_{\mathrm{GRPO}} = -\mathbb{E}\left[\log \frac{\exp(R_\psi(x,y_c))}{\sum_{y \in \{y_c\}\cup\{y_r\}} \exp(R_\psi(x,y))}\right].
$$

The reward architecture is two-tiered. The **process reward** is dense and turn-level, spanning eight dimensions: **Safety, Reasoning, Medical Accuracy, Completeness, Information Gathering, Faithfulness to ground truth, Empathy & Clarity,** and **Humility**. Each dimension receives a score $S_i \in [-5,+5]$, then a safety-first hierarchical veto is applied:

$$
R_{\text{turn}} =
\begin{cases}
-1.0 & \text{if } S_{\text{safety}} < 0, \\
-0.75 & \text{if } (S_{\text{reasoning}} < 0)\ \text{or}\ (S_{\text{accuracy}} < 0), \\
\mathrm{clip}\!\left(\frac{\sum_{i=1}^{8} w_i S_i}{S_{\max}\! \sum_{i=1}^{8} w_i},\, -1,\, +1\right) & \text{otherwise}.
\end{cases}
$$

The weights are explicitly specified: $w_{\text{safety}}=1.0$, $w_{\text{reasoning}}=1.0$, $w_{\text{accuracy}}=1.0$, $w_{\text{info}}=0.8$, $w_{\text{faithfulness}}=0.7$, $w_{\text{completeness}}=0.7$, $w_{\text{empathy}}=0.5$, and $w_{\text{humility}}=0.5$. The **outcome reward** is sparse and terminal, with $R_{\text{final}} \in \{0.0, 0.5, 1.0\}$ depending on correctness of the primary diagnosis or recommendation [2510.04284].

A distinctive element is the **experience repository**. After each batch, only “high-reward” tuples are stored. Retrieval proceeds in three stages: candidate selection with **jina-embeddings-v3**, reranking with **bge-reranker-base**, and novelty-plus-reward filtering using a dynamic threshold. The combined retrieval score is

$$
S_{\mathrm{combined}}(Q, E_i) = f_{\mathrm{sim}}\!\big(f_{\mathrm{emb}}(Q),\, f_{\mathrm{emb}}(E_i^{\mathrm{state}})\big) + \alpha\, R(E_i),
$$

with $\alpha = 0.5$, top-$\mathcal{N}=30$ candidates, and final top-$k=2$ experiences prepended into the doctor prompt [2510.04284]. This makes prior trajectories an in-context support mechanism for policy improvement rather than a static memory bank.

Implementation is correspondingly explicit: training uses **7× NVIDIA A100 80GB GPUs**, the **VeRL (HybridFlow)** framework, **NCCL**, and **SGLang**; the training corpus contains **100,000 simulated diagnostic dialogues**, with **max 10 turns**, **max prompt length 1,024 tokens**, **max response length 3,072 tokens**, **batch size 448**, **micro-batch 8/GPU**, **actor LR $1\times10^{-6}$**, **critic/reward-model LR $1\times10^{-5}$**, and **1 epoch** [2510.04284].

## 4. Inquiry tactics, communication policy, and safety stance

The learned strategy emphasized by Doctor-R1 is not generic elicitation, but a disciplined policy for asking the most informative next question given current evidence [2510.04284]. In the reported behavior, GRPO with dense process rewards teaches the policy to prioritize high-yield questions that disambiguate urgent risks early, sequence questions to narrow differentials stepwise, avoid repetition, maintain clarity and empathy, and calibrate uncertainty through humility.

The framework’s illustrative hemoptysis dialogue is representative. A weaker baseline asks questions such as “Do you have fever? When was your last meal?” and misdiagnoses viral infection. Doctor-R1 instead asks whether the cough is dry or productive, the color and amount of blood, anticoagulant or trauma history, and TB history, then issues an immediate emergency recommendation with supportive language after learning “Bright red, ~500 mL” and prior TB with stopped treatment. In that example, the evaluator score is reported as **0.90** for Doctor-R1, versus **−0.80** for GPT-4.1 and **0.40** for Baichuan-M2 [2510.04284].

The safety position is encoded both in prompt structure and in rewards. The **think/answer** template enforces reason-first behavior. The hierarchical veto makes unsafe, illogical, or inaccurate content immediately costly. Patient-agent prompts enforce gradual disclosure and human-like behavior, while rule-based filters and adherence scoring reject low-fidelity dialogues. The framework also states that there is **no medication prescribing in unsafe contexts** [2510.04284].

This architecture supports a particular interpretation of “medical professionalism” in model design: strategic inquiry, safety-first triage, and communicative conduct are optimized as first-class targets rather than treated as secondary style attributes.

## 5. Evaluation, benchmark results, and ablations

Doctor-R1 is evaluated on **HealthBench**, **MAQuE**, **MedQA**, and the **MMLU medical subset**, with additional human evaluation on coherence, adherence, clarity, and empathy [2510.04284]. On **HealthBench Main**, Doctor-R1 reports **Avg 36.29**, **Accuracy 37.84**, **Communication Quality 64.15**, **Instruction Following 54.39**, **Context Awareness 49.24**, and **Completeness 40.93**. It exceeds **UltraMedical-70B** by **9.91** points in average score, exceeds **Baichuan-M2-32B** by **3.13**, and exceeds **GPT-4.1** by **5.11**.

| Benchmark | Doctor-R1 result | Comparison stated in the paper |
|---|---|---|
| HealthBench Main | Avg 36.29 | Higher than GPT-4.1 at 31.18 |
| MAQuE | Accuracy 60.00; Empathy 93.80 | Ties GPT-4.1 on accuracy; higher empathy than GPT-4.1 at 75.20 |
| HealthBench Hard | Avg 18.73; Communication Quality 63.86 | Higher average than GPT-4.1 at 16.92 |
| MedQA | 83.50% | Base Qwen3-8B: 63.50% |
| MMLU-medical | 85.00% | Base Qwen3-8B: 70.00% |

On **MAQuE**, Doctor-R1 reports **Accuracy 60.00**, **Robustness 77.03**, **Coverage 38.52**, **Relevance 87.50**, **Adherence 70.20**, **Coherence 76.20**, **Clarity 69.80**, and **Empathy 93.80**. It ties GPT-4.1 on accuracy and exceeds it strongly on empathy. On **HealthBench Hard**, it reports **Avg 18.73** and the highest **Communication Quality 63.86** among the compared models. On static QA, Doctor-R1 reaches **83.50%** on MedQA and **85.00%** on MMLU-medical, improving over the base Qwen3-8B by **20** and **15** points respectively [2510.04284].

The ablations are central to the framework’s claims. Removing the experience retrieval mechanism degrades HealthBench Main from **36.29** to **31.69**. Removing process reward yields **32.61**, with **Accuracy 34.43** and **Communication Quality 59.99**. Experience-retrieval scaling also shows monotonic gains: Communication Quality improves **38.49 → 39.40 → 47.16** for **No-Experience → Similarity-only → Full**, and Context Awareness improves **45.31 → 46.86 → 49.24** [2510.04284].

Dialogue-turn scaling further supports the inquiry-centric thesis. On MAQuE-like metrics, Doctor-R1 accuracy rises from **0.36** at turn 1 to **0.58** at turn 5 and **0.60** at turn 10, while empathy rises from **0.76** to **0.94**. Human evaluation ranks Doctor-R1 first across all four qualitative metrics, although inter-rater reliability statistics are not reported [2510.04284].

## 6. Limitations, safety debates, and related systems

Doctor-R1’s limitations are explicit. Performance still drops on difficult and ambiguous cases in **HealthBench Hard**. The reward model is itself an **LLM judge**, specifically Qwen3-8B, which introduces proxy bias and inconsistency risk. Human clinician verification is not included in the reward model. Reward hacking is stated to be mitigated but not eliminated, even with hierarchical vetoes and multi-objective scoring. Real-world generalization may also be challenged by patient variability in language, culture, and co-morbidities, while multimodal signals such as imaging, labs, and longitudinal monitoring are خارج the framework’s present scope [2510.04284].

These limitations matter because Doctor-R1 occupies a specific design space within medical AI. It differs from **DoctorRAG**, which addresses clinical reasoning through retrieval over explicit medical knowledge and similar patient cases, then constrains answer generation with **Med-TextGrad**; that system is retrieval-augmented and knowledge-fusion-centric rather than an outpatient POMDP optimized through experiential GRPO [2505.19538]. It also differs from the unrelated multi-page document-understanding model **DocR1**, which uses **Evidence Page-Guided GRPO** for multimodal document reasoning rather than clinical dialogue [2508.07313].

A plausible implication is that the Doctor-R1 line of work should be read as an attempt to optimize **consultation policy** rather than merely medical answer quality. Its future directions in the paper are consistent with that interpretation: clinician-in-the-loop judging, multimodal capability, explicit uncertainty quantification, calibrated triage policies, better off-policy corrections and replay, and formal KL or entropy regularization for policy stabilization [2510.04284]. Within that framing, DocR1 is significant less as a conventional medical QA model than as a system that treats inquiry, empathy, and decision-making as jointly trainable components of clinical interaction.

Source: https://www.emergentmind.com/topics/docr1