Doctor-R1: AI Agent for Clinical Inquiry

• DocR1 is an AI doctor agent that models outpatient clinical interactions as a POMDP to optimize strategic multi-turn questioning and empathetic communication.
• The system employs experiential agentic reinforcement learning with a two-tier reward architecture to enhance diagnostic accuracy and patient safety.
• An integrated experience repository and multi-agent environment enable dynamic retrieval of high-yield inquiry trajectories for improved policy learning in clinical settings.

DocR1 most directly denotes Doctor-R1, an AI doctor agent for outpatient clinical inquiry trained with experiential agentic reinforcement learning to combine strategic multi-turn inquiry, empathetic communication, and clinical decision-making (Lai et al., 5 Oct 2025). It is designed for clinical interactions in which the agent must ask high-yield questions under uncertainty rather than solve a static benchmark item in a single turn. In the supplied literature, similar shorthand also appears for unrelated systems, which suggests a degree of bibliographic ambiguity; in the medical setting, however, the referent is the Doctor-R1 framework built around a multi-agent interactive environment, a two-tiered reward architecture, and an experience repository that retrieves high-quality prior trajectories during policy learning (Lai et al., 5 Oct 2025).

1. Clinical inquiry as a sequential decision problem

Doctor-R1 is motivated by a specific gap in medical LLMs: frontier and specialized models can achieve strong performance on static decision benchmarks such as USMLE, MedQA, and MMLU, yet often degrade in open-ended, multi-turn clinical scenarios that require strategic information gathering, hypothesis revision, and patient-facing communication (Lai et al., 5 Oct 2025). The failure modes identified in the framework are concrete. Such models may follow generic scripts instead of adapting to patient responses, miss high-yield questions that rapidly surface red flags, and communicate in a non-empathic, overly technical style.

The framework formalizes real outpatient interaction as “Dynamic Inquiry”, a sequential and partially observable process in which the clinician iteratively forms differential diagnoses, asks high-yield, hypothesis-driven questions, reassesses after new evidence, and communicates clearly and compassionately. “High-yield questions” are defined operationally as questions optimized for diagnostic utility and safety or triage value, especially those targeting the most uncertain or high-risk branches of the differential rather than low-yield demographic or generic symptom prompts (Lai et al., 5 Oct 2025).

A recurring misconception in the surrounding literature is that better static medical QA performance should directly transfer to consultation quality. Doctor-R1 is explicitly structured against that assumption. Its objectives are not limited to final diagnosis or recommendation accuracy, but include strategic multi-turn inquiry, empathetic communication, and learning from prior trajectories. This suggests that Doctor-R1 treats consultation not as answer generation alone, but as a policy-learning problem over interaction histories.

2. POMDP formulation and multi-agent environment

Doctor-R1 models outpatient interaction as a POMDP with tuple $$\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, \mathcal{R}\rangle$$ (Lai et al., 5 Oct 2025). The latent patient scenario is the state $s \in \mathcal{S}$; doctor utterances are actions $a_t \in \mathcal{A}$; dialogue history is the observation $o_t \in \mathcal{O}$; and rewards are composed of turn-level and terminal components. The doctor policy is written as $\pi_\theta(a_t \mid o_t)$, and the optimization target is

$$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\Big[\sum_{t=0}^{T} \gamma^{t}\, r_t\Big].$$

Three agents define the environment: a Doctor Agent as the learnable policy, a Patient Agent as an LLM simulator conditioned on a case scenario, and a Consultation Evaluator as an LLM judge that returns both process and outcome rewards. The doctor sees only the dialogue history and not the latent patient state, which makes information gathering central rather than incidental.

Component	Role	Instantiation
Doctor Agent	Generates questions, diagnoses, and recommendations	Qwen3-8B
Patient Agent	Simulates patient responses and environment dynamics	Qwen3-8B
Consultation Evaluator	Produces turn-level and terminal rewards	Qwen3-8B

Each turn contains a reasoned “think” block and an “answer” block, with a strict template. If information is insufficient, the answer is a single question: > [reasoning]</think><answer>Question: …</answer>. Otherwise, the model outputs a recommendation: <think>[reasoning]<answer>Recommendation: …</answer>. A trajectory is represented as $\tau=\{(o_0,a_0,r_0),\dots,(o_T,a_T,r_T)\}$, with dialogue history updated by the patient response after each doctor action (Lai et al., 5 Oct 2025).

This design has two consequences. First, the framework is intrinsically interactive rather than prompt-only. Second, the output format itself becomes verifiable and trainable, since question-asking and recommendation issuance are distinct action types within the same policy.

3. Experiential agentic reinforcement learning

Doctor-R1 is positioned against RLHF and SFT by training the model as an interactive policy in a closed-loop multi-agent environment rather than as a single-turn preference-aligned generator (Lai et al., 5 Oct 2025). The policy optimization method is GRPO, whose listwise loss contrasts one chosen response against a group of rejected responses:

$$\mathcal{L}_{\mathrm{GRPO}} = -\mathbb{E}\left[\log \frac{\exp(R_\psi(x,y_c))}{\sum_{y \in \{y_c\}\cup\{y_r\}} \exp(R_\psi(x,y))}\right].$$

The reward architecture is two-tiered. The process reward is dense and turn-level, spanning eight dimensions: Safety, Reasoning, Medical Accuracy, Completeness, Information Gathering, Faithfulness to ground truth, Empathy & Clarity, and Humility. Each dimension receives a score $S_i \in [-5,+5]$, then a safety-first hierarchical veto is applied:

$$R_{\text{turn}} = \begin{cases} -1.0 & \text{if } S_{\text{safety}} < 0, \ -0.75 & \text{if } (S_{\text{reasoning}} < 0)\ \text{or}\ (S_{\text{accuracy}} < 0), \ \mathrm{clip}\!\left(\frac{\sum_{i=1}^{8} w_i S_i}{S_{\max}\! \sum_{i=1}^{8} w_i},\, -1,\, +1\right) & \text{otherwise}. \end{cases}$$

The weights are explicitly specified: $s \in \mathcal{S}$0, $s \in \mathcal{S}$1, $s \in \mathcal{S}$2, $s \in \mathcal{S}$3, $s \in \mathcal{S}$4, $s \in \mathcal{S}$5, $s \in \mathcal{S}$6, and $s \in \mathcal{S}$7. The outcome reward is sparse and terminal, with $s \in \mathcal{S}$8 depending on correctness of the primary diagnosis or recommendation (Lai et al., 5 Oct 2025).

A distinctive element is the experience repository. After each batch, only “high-reward” tuples are stored. Retrieval proceeds in three stages: candidate selection with jina-embeddings-v3, reranking with bge-reranker-base, and novelty-plus-reward filtering using a dynamic threshold. The combined retrieval score is

$s \in \mathcal{S}$9

with $a_t \in \mathcal{A}$0, top-$a_t \in \mathcal{A}$1 candidates, and final top-$a_t \in \mathcal{A}$2 experiences prepended into the doctor prompt (Lai et al., 5 Oct 2025). This makes prior trajectories an in-context support mechanism for policy improvement rather than a static memory bank.

Implementation is correspondingly explicit: training uses 7× NVIDIA A100 80GB GPUs, the VeRL (HybridFlow) framework, NCCL, and SGLang; the training corpus contains 100,000 simulated diagnostic dialogues, with max 10 turns, max prompt length 1,024 tokens, max response length 3,072 tokens, batch size 448, micro-batch 8/GPU, actor LR $a_t \in \mathcal{A}$3, critic/reward-model LR $a_t \in \mathcal{A}$4, and 1 epoch (Lai et al., 5 Oct 2025).

4. Inquiry tactics, communication policy, and safety stance

The learned strategy emphasized by Doctor-R1 is not generic elicitation, but a disciplined policy for asking the most informative next question given current evidence (Lai et al., 5 Oct 2025). In the reported behavior, GRPO with dense process rewards teaches the policy to prioritize high-yield questions that disambiguate urgent risks early, sequence questions to narrow differentials stepwise, avoid repetition, maintain clarity and empathy, and calibrate uncertainty through humility.

The framework’s illustrative hemoptysis dialogue is representative. A weaker baseline asks questions such as “Do you have fever? When was your last meal?” and misdiagnoses viral infection. Doctor-R1 instead asks whether the cough is dry or productive, the color and amount of blood, anticoagulant or trauma history, and TB history, then issues an immediate emergency recommendation with supportive language after learning “Bright red, ~500 mL” and prior TB with stopped treatment. In that example, the evaluator score is reported as 0.90 for Doctor-R1, versus −0.80 for GPT-4.1 and 0.40 for Baichuan-M2 (Lai et al., 5 Oct 2025).

The safety position is encoded both in prompt structure and in rewards. The think/answer template enforces reason-first behavior. The hierarchical veto makes unsafe, illogical, or inaccurate content immediately costly. Patient-agent prompts enforce gradual disclosure and human-like behavior, while rule-based filters and adherence scoring reject low-fidelity dialogues. The framework also states that there is no medication prescribing in unsafe contexts (Lai et al., 5 Oct 2025).

This architecture supports a particular interpretation of “medical professionalism” in model design: strategic inquiry, safety-first triage, and communicative conduct are optimized as first-class targets rather than treated as secondary style attributes.

5. Evaluation, benchmark results, and ablations

Doctor-R1 is evaluated on HealthBench, MAQuE, MedQA, and the MMLU medical subset, with additional human evaluation on coherence, adherence, clarity, and empathy (Lai et al., 5 Oct 2025). On HealthBench Main, Doctor-R1 reports Avg 36.29, Accuracy 37.84, Communication Quality 64.15, Instruction Following 54.39, Context Awareness 49.24, and Completeness 40.93. It exceeds UltraMedical-70B by 9.91 points in average score, exceeds Baichuan-M2-32B by 3.13, and exceeds GPT-4.1 by 5.11.

Benchmark	Doctor-R1 result	Comparison stated in the paper
HealthBench Main	Avg 36.29	Higher than GPT-4.1 at 31.18
MAQuE	Accuracy 60.00; Empathy 93.80	Ties GPT-4.1 on accuracy; higher empathy than GPT-4.1 at 75.20
HealthBench Hard	Avg 18.73; Communication Quality 63.86	Higher average than GPT-4.1 at 16.92
MedQA	83.50%	Base Qwen3-8B: 63.50%
MMLU-medical	85.00%	Base Qwen3-8B: 70.00%

On MAQuE, Doctor-R1 reports Accuracy 60.00, Robustness 77.03, Coverage 38.52, Relevance 87.50, Adherence 70.20, Coherence 76.20, Clarity 69.80, and Empathy 93.80. It ties GPT-4.1 on accuracy and exceeds it strongly on empathy. On HealthBench Hard, it reports Avg 18.73 and the highest Communication Quality 63.86 among the compared models. On static QA, Doctor-R1 reaches 83.50% on MedQA and 85.00% on MMLU-medical, improving over the base Qwen3-8B by 20 and 15 points respectively (Lai et al., 5 Oct 2025).

The ablations are central to the framework’s claims. Removing the experience retrieval mechanism degrades HealthBench Main from 36.29 to 31.69. Removing process reward yields 32.61, with Accuracy 34.43 and Communication Quality 59.99. Experience-retrieval scaling also shows monotonic gains: Communication Quality improves 38.49 → 39.40 → 47.16 for No-Experience → Similarity-only → Full, and Context Awareness improves 45.31 → 46.86 → 49.24 (Lai et al., 5 Oct 2025).

Dialogue-turn scaling further supports the inquiry-centric thesis. On MAQuE-like metrics, Doctor-R1 accuracy rises from 0.36 at turn 1 to 0.58 at turn 5 and 0.60 at turn 10, while empathy rises from 0.76 to 0.94. Human evaluation ranks Doctor-R1 first across all four qualitative metrics, although inter-rater reliability statistics are not reported (Lai et al., 5 Oct 2025).

6. Limitations, safety debates, and related systems

Doctor-R1’s limitations are explicit. Performance still drops on difficult and ambiguous cases in HealthBench Hard. The reward model is itself an LLM judge, specifically Qwen3-8B, which introduces proxy bias and inconsistency risk. Human clinician verification is not included in the reward model. Reward hacking is stated to be mitigated but not eliminated, even with hierarchical vetoes and multi-objective scoring. Real-world generalization may also be challenged by patient variability in language, culture, and co-morbidities, while multimodal signals such as imaging, labs, and longitudinal monitoring are خارج the framework’s present scope (Lai et al., 5 Oct 2025).

These limitations matter because Doctor-R1 occupies a specific design space within medical AI. It differs from DoctorRAG, which addresses clinical reasoning through retrieval over explicit medical knowledge and similar patient cases, then constrains answer generation with Med-TextGrad; that system is retrieval-augmented and knowledge-fusion-centric rather than an outpatient POMDP optimized through experiential GRPO (Lu et al., 26 May 2025). It also differs from the unrelated multi-page document-understanding model DocR1, which uses Evidence Page-Guided GRPO for multimodal document reasoning rather than clinical dialogue (Xiong et al., 10 Aug 2025).

A plausible implication is that the Doctor-R1 line of work should be read as an attempt to optimize consultation policy rather than merely medical answer quality. Its future directions in the paper are consistent with that interpretation: clinician-in-the-loop judging, multimodal capability, explicit uncertainty quantification, calibrated triage policies, better off-policy corrections and replay, and formal KL or entropy regularization for policy stabilization (Lai et al., 5 Oct 2025). Within that framing, DocR1 is significant less as a conventional medical QA model than as a system that treats inquiry, empathy, and decision-making as jointly trainable components of clinical interaction.