World Feedback for Clinical Agents: Diagnosing RL in FHIR Environments

Abstract: Clinical protocol-execution tasks -- checking a lab value, applying a threshold, placing a correctly structured FHIR order -- are natural candidates for RL from world feedback: once clinical SMEs encode decision logic into a verifier, that verifier grades unlimited rollouts without per-episode annotation. But applying RL requires a sound feedback channel and sufficient base capability. We audit MedAgentBench v1/v2, find a 41.7\% silent-finish ceiling that makes inaction the RL dominant strategy, and construct \textbf{MedAgentBench-v3 (MAB-v3)} (508 tasks, 8.9\% ceiling). Training Qwen3-8B exposes two structural barriers: a \emph{capability ceiling} (10/20 task types have 0\% base performance, zero gradient) and a \emph{format-knowledge barrier} (3/20 types require exact clinical codes undiscoverable by exploration). Pure RL reaches 18.2\% pass@1 vs.\ 34.1\% for rule-based SFT; the 15.9~pp gap is attributable entirely to these barriers. A decision/format-knowledge/lookup taxonomy predicts RL learnability and prescribes the fix: SFT to inject codes, RL to learn conditionals.

• The paper identifies and corrects a flaw in MedAgentBench that reduced silent-finish rates from 41.7% to 8.9%, ensuring meaningful RL feedback.
• It benchmarks RL using a deterministic FHIR environment with Qwen3-8B+LoRA, comparing pure RL with supervised fine-tuning methods.
• The study advocates a hybrid SFT+RL approach to overcome RL’s limitations on tasks requiring discrete clinical codes and format compliance.

Diagnosing Reinforcement Learning for Clinical Agents via World Feedback in FHIR Environments

Background and Motivation

Reinforcement learning with world feedback—where agents interact with an environment and receive feedback solely from programmatically encoded verifiers—offers a promising paradigm for automating large-scale protocol-execution in clinical domains. The MedAgentBench (MAB) benchmarks have defined task suites for clinical workflow automation grounded in FHIR tool usage. Yet, prevailing benchmarks overlooked structural pitfalls that can undermine the utility of RL. This paper systematically audits these RL failure modes, constructs a corrected benchmark (MAB-v3), and analyzes the conditional applicability of RL from world feedback for clinical agents (2607.01470).

Benchmark Audit and Environment Redesign

A critical flaw identified in MedAgentBench v1/v2 is the branch imbalance and "silent-finish" ceiling: 41.7% of tasks permitted agents to succeed by taking no action, rendering abstention the RL dominant strategy (Figure 1).

Figure 1: Branch imbalance before (hatched) and after (solid) 1:1 cap in MAB-v3, reducing the silent-finish ceiling from 41.7% to 8.9% and eliminating the inaction loophole as an RL solution.

The authors curated MedAgentBench-v3 to address these issues: capping the action/no-action branch ratio at 1:1, correcting cohort composition, and ensuring label integrity. Additional fixes included explicit task context documentation for grader-required formats and freezing time-dependent logic for temporal consistency. The resulting MAB-v3 benchmark features 508 tasks spanning 20 types across ~100 anonymized patients, reducing the silent-finish ceiling to 8.9% and eliminating reward-hacking pathways.

The FHIR RL environment is offline, deterministic, and supports thousands of reproducible rollouts. A rule-based verifier, encoding clinical protocol specifications, assigns all rewards, thus allowing for scalable, fully auditable agent evaluation with no per-episode human intervention.

Reinforcement Learning and RL-specific Failure Modes

RL experiments were conducted using Qwen3-8B with LoRA adaptation and GRPO as the learning algorithm, trained from base initialization (no SFT warmstart). The agent's policy generates tool calls in the <tool_call> interface; the FHIR environment executes these, returns deterministic responses, and the verifier grades each episode's outcome (Figure 2).

Figure 2: World feedback loop for clinical FHIR agent training, with the Qwen3-8B+LoRA model generating <tool\_call> actions, deterministic world execution, and reward assignment by a rule-based verifier.

Two structural failure modes emerged:

1. Capability ceiling: For many task types, especially those requiring format-knowledge (exact codes) or lookups, the base model has 0% pass@1; rollouts produce no gradient, precluding RL improvement.
2. Format-knowledge barrier: Certain tasks require injecting discrete identifiers (like SNOMED/NDC codes) that cannot be discovered via exploratory reward optimization in a flat landscape. The environment cannot convey the correct code for a failed response, making world feedback structurally insufficient for these tasks.

Empirical results show base Qwen3-8B performance at 16.6% pass@1; RL increases this marginally to 18.2%. Supervised fine-tuning (SFT) with rule-based demonstrations yields 34.1%. The 15.9pp SFT/RL gap is attributable solely to the aforementioned structural barriers: SFT injects both format and code knowledge, which RL alone cannot acquire given the environment and base capabilities.

Task Taxonomy and Analysis

The paper introduces a taxonomy to predict RL learnability by task structure:

• Decision: Tasks requiring reading and thresholding (e.g., potassium replacement) exhibit reward variance and are RL-learnable; RL improves performance due to shaped, informative rewards.
• Lookup: Tasks requiring extraction and surface-form copying (e.g., patient MRN) provide only weak RL signal; supervised demonstrations are more efficient.
• Format-knowledge: Tasks dependent on external identifiers or formatting—undiscoverable via exploration—yield flat reward landscapes; RL offers no improvement over base.

Per-type analysis confirms this taxonomy: RL provides gains only on decision tasks with informative gradients, makes no progress on format-knowledge or lookup tasks without prior base capability, and can even harm performance by overwriting rare successful outcomes from pretraining.

Implications, Recommendations, and Future Directions

The evidence underscores that RL from world feedback is not universally effective in clinical environments: learnability is conditional on the reward landscape and base model priors. For protocol-execution tasks, pure RL is sharply limited by dead-zones (0% gradient, 0% base accuracy) and flat-landscape (code-knowledge) failures. SFT overcomes these by demonstrative injection of codes and payload structures, which world feedback alone cannot provide.

Key numerical highlights include:

• MAB-v3 now quantifies the effect of structural flaws: branch rebalance and silent-finish correction reduced inaction success from 41.7% to 8.9%.
• SFT significantly outperforms pure RL (34.1% vs. 18.2% pass@1), with the entire gap attributed to format/code-knowledge and base capability dead-zones.
• Frontier models (e.g., GPT-5.5, Gemini 3.1 Pro) deliver >75% pass@1, but even these fail completely on previously unseen codes in format-knowledge tasks.

The empirical prescription is clear: SFT is essential for injecting clinical codes and format knowledge, while RL (from world feedback) is well suited for refining decision conditionals and calibrating action/no-action policies. This hybrid (SFT+RL) curriculum emerges as the rational next step for clinical RL agent development.

Theoretical implications extend to RL environment design in other scientific and industrial domains: any RL-from-verifier training regime must audit for reward-hacking paths (e.g., silent-finish loopholes) and validate that task structures actually create learnable gradients—benchmarks must be constructed to guarantee the possibility of meaningful policy improvement. The methodology and checklist outlined here apply to any setting where RL is deployed atop verifiable, rule-based feedback.

Conclusion

This work demonstrates that successful RL for clinical protocol execution hinges on a clean, well-balanced benchmark and an explicit understanding of which task structures permit agent learning from world feedback. Addressing structural failures in benchmark design is prerequisite to meaningfully evaluating RL approaches. Furthermore, RL is decisively limited when base model priors are insufficient for code discovery or format compliance; these must be bootstrapped via supervised methods. The union of programmatic SFT demonstration and RL from deterministic verifier feedback—systematically evaluated in a corrected environment—offers the most reliable path forward for scalable, accurate clinical AI agents.