- The paper introduces MedCUA-Bench, a 432-instance benchmark spanning 18 clinical scenarios, 10 medical domains, synthetic and real GUIs, paired intent and step goals, and deterministic safety checks.
- The paper finds GPT-5.4 leads with 54.2% strict success, while all models score below 9% on real OpenEMR and open-source agents average just 2.5%, revealing major deployment gaps.
- The paper shows that timeouts, visual grounding failures, action-format errors, and looping dominate failures, while the safety checker recorded 53 major and 15 minor violations but too few harmful completions to validate clinical safety.
Motivation and positioning
Computer-use agents (CUAs) that operate software through screenshots have been evaluated extensively on general web and desktop tasks, but their reliability in clinical graphical user interfaces remains largely unvalidated. The authors identify four properties that distinguish clinical GUIs from the settings covered by prior benchmarks: tasks require domain knowledge; medical software exhibits legacy, form-dense UI designs unlike mainstream applications; real clinical systems are regulated and proprietary, precluding public test environments or API/DOM access; and clinical use demands safety validation beyond binary task completion—an agent may nominally finish a task while acting on the wrong patient or violating a required workflow (2606.03203).
MedCUA-Bench is presented as the first benchmark to evaluate CUAs across complete clinical care workflows in executable GUIs. Relative to related work—AgentClinic (text-based dialogue), MedAgentBench (FHIR APIs rather than GUIs), MedSPOT (grounding only, no end-to-end execution), and HealthAdminBench (administrative rather than care workflows)—it combines medical domain coverage, real clinical software, pixel-level actions, paired goal granularities, safety-aware evaluation, and deterministic checking in a single testbed.
Benchmark design
The benchmark comprises 216 base tasks across 18 clinical scenarios spanning 10 medical domains (Outpatient, Inpatient, ICU, Nursing, PACS/Pathology, Imaging, RadTherapy, ECG, Endoscopy, Ultrasound), organized into three page-fidelity tiers:
- Synthetic shadow GUIs (15 scenarios): HTML reconstructions of deployed systems such as VistA/CPRS triage, Cerner PowerChart-style nurse stations, Philips IntelliVue central monitoring, Varian Eclipse/RayStation treatment planning, and GE imaging consoles, built from product manuals to preserve layout, terminology, and interaction patterns without shipping proprietary code.
- Real EHR (1 scenario): OpenEMR v7.0.2 in Docker, seeded with five synthetic demonstration patients chosen to surface allergy-conflict, renal-safety, and multi-medication tasks.
- Real imaging (2 scenarios): OHIF Viewer connected to DICOMweb endpoints for radiology and whole-slide pathology studies.
Each base task carries two natural-language goals over an identical environment and checker: an intent goal stating the clinical objective at the level a senior clinician would delegate, and a step goal specifying the procedure click by click. Registering both granularities yields 432 evaluated instances, allowing failures of workflow inference to be separated from failures of pixel-level execution.
Evaluation is deterministic. At episode end, a checker compares final page state, agent messages, and hidden navigation/HTTP-request traces against expected values, returning strict completion, a progress vector, and violations across five dimensions: patient identity, data accuracy, information fidelity, record integrity, and workflow safety. Violations are weighted critical/major/minor at 1.0/0.3/0.05, and the episode reward clamps task completion minus penalties to [−1,+1], so a harmful completion can score below inaction; critical violations terminate the episode immediately. Scenario construction was guided by two practising physicians who jointly defined the safety dimensions.
Experimental setup
Twenty-three vision-capable agents were run under a uniform screenshot-only AgentLab harness with DOM, accessibility tree, HTML, and set-of-marks inputs disabled; each model executed all 432 instances once with a 30-step budget. Success requires terminal cumulative reward at the completion threshold with no safety penalty.
Headline results
Current CUAs are not clinically reliable on this benchmark. The leading results:
| Model |
Strict success (%) |
Reward |
Timeout (%) |
| GPT-5.4 |
54.2 |
0.563 |
43.3 |
| Claude-Opus-4.7 |
52.6 |
0.551 |
43.3 |
| Claude-Sonnet-4.6 |
45.4 |
0.472 |
53.0 |
| Kimi-K2.6 |
30.8 |
0.316 |
66.2 |
| Qwen2.5-VL-32B (best open) |
16.2 |
0.173 |
82.2 |
| Gemini-3.1-Pro |
4.6 |
0.046 |
95.1 |
The closed/open gap is stark: mean success is 33.7% for closed-source versus 2.5% for open-source agents. Notably, scale does not close the gap—Qwen2.5-VL-72B-AWQ scores 6.2%, well below the 32B variant's 16.2%, and newer Qwen3.5 models (1.2–2.3%) fall far below Qwen2.5-VL-32B. The authors interpret this as evidence that MedCUA rewards the visual grounding and interface-control skills emphasized in Qwen2.5-VL more than generational improvements in language backbones. CUA-specialized OpenCUA models also remain near the floor (0.5–0.9%), indicating the bottleneck is reliable control from clinical screenshots, not model scale alone.
A partial human pilot reinforces the headroom: a single trained operator solved 83.3% of a stratified 24-run subsample (95% Wilson CI [64.1, 93.3]), 29.1 pp above the strongest agent, with 75% of episodes within the same 30-step budget—confirming the budget is not the binding constraint on agent failure. This pilot is admittedly compact (one annotator), so it serves as a sanity check rather than a normalized human-relative score.
Page fidelity
Aggregate scores conceal large tier differences. On real OpenEMR, every model falls into a narrow 0–8.3% band, with the three strongest closed models tied at the 8.3% ceiling and Gemini-3.1-Pro and Kimi-K2.6 solving none. General screenshot-based GUI skill therefore transfers poorly to a deployed EHR characterized by nested menus, focus-stealing modal dialogs, strict server-side validation, and dense forms with small click targets. Because synthetic scenarios reproduce workflow but not full visual chrome, the paper notes synthetic-tier scores should be read as upper bounds; the OpenEMR result is the most dependable readout of production-EHR readiness.
OHIF shows the opposite pattern: it is the only tier where several open-source agents approach closed-model performance (Qwen2.5-VL-72B-AWQ 45.8%, InternVL3.5-14B 35.4%). The sparse imaging interface—large viewport, compact toolbar, few text fields—separates clinical-software navigation from visual measurement, two abilities a single aggregate would blur.
Goal granularity
Paired goals reveal two regimes. The four strongest closed models all improve under step goals (+3.3 to +5.1 pp), though procedural guidance does not close cross-model gaps—much residual error lies in executing a known plan. Most open-source models degrade under step prompts (e.g., Qwen2.5-VL-32B drops from 18.5% to 13.9%), plausibly because reading a longer procedure while grounding actions in the screenshot adds burden. Stronger models often know the plan but fail to execute it; weaker models are strained by the plan itself.
Failure modes and safety profile
Truncation dominates: even the top two models exhaust the budget on ~43% of episodes, and all main-text open-source models on at least 82.2%. Step-budget analysis shows 99.0% of successful episodes finish well under 30 actions (median 7), and unsolved tasks are no longer than solved ones—truncation reflects inability to converge on the correct UI element, not procedures requiring more steps.
A five-bucket taxonomy over 8,877 failures exposes qualitatively different regimes. Closed-source failures are dominated by exploration timeouts (53.5%) with modest looping (17.2%); open-source failures split between format lockouts (41.3%, including four models at ~100% zero-action due to reverting to proprietary action schemas) and heavy loops (45.4%, e.g., Qwen2.5-VL variants above 83%). Gemini-3.1-Pro exemplifies grounding-limited failure among closed models: well-formed action calls at every step, yet 4.6% success—roughly an order of magnitude below GPT-5.4, mirroring a comparable gap reported by HealthAdminBench under screenshot-only operation.
On safety, across 9,936 episodes the checker recorded zero critical, 53 major, and 15 minor violations, concentrated in open-source runs and in the OpenEMR and OHIF tiers. The authors are explicit that this is not evidence stronger agents are safe: most agents stall before reaching safety-critical screens, so the checker has limited discriminative signal today. A sensitivity analysis over ten alternative severity-weight schemes and a grid sweep confirms rank stability (ρ≥0.957, unchanged top-1), though the authors concede this robustness is mechanical given how few violations occur—the weighting becomes consequential only once confident wrong completions replace timeouts as the dominant failure mode.
Limitations and open questions
The paper concedes several constraints directly. Synthetic scenarios reproduce workflow and terminology but not full vendor visual chrome, so their scores bound rather than equal real-system performance. The human baseline involves one operator on 24 runs. Each agent receives a single run per task, so results are point estimates without run-to-run variance. Most fundamentally, the deterministic safety checker is currently under-exercised: the zero-critical-violation finding is a floor measurement under present capability, not a positive safety claim, and the authors caution against citing it to justify autonomous clinical deployment. Open questions include whether agents can be trained to close the OpenEMR gap specifically, whether longer horizons and multi-seed evaluation change rankings, and whether the safety dimensions become discriminative as completion rates rise.
Conclusion
MedCUA-Bench makes clinical GUI reliability measurable for screenshot-only agents through realistic reconstructed environments, paired intent/step goals, and a deterministic five-dimension safety checker. Its central findings—that the best closed-source model reaches only 54.2% strict success, that all models fall below 9% on real OpenEMR, and that open-source agents average 2.5%—quantify a substantial gap between current CUA capability and clinically dependable software operation, and provide a reproducible testbed whose safety machinery is designed to become informative precisely as that gap begins to close.