AutoMedBench: Medical AI Workflow Benchmark

• AutoMedBench is a diagnostic benchmark that evaluates autonomous medical AI systems through a structured five-stage workflow.
• It decomposes medical research tasks into planning, setup, validation, inference, and submission to isolate process-level errors.
• It provides quantitative metrics on segmentation, VQA, report generation, and lesion detection, informing robust error recovery strategies.

AutoMedBench is a workflow-aware, diagnostics-driven benchmark designed to systematically evaluate autonomous agentic AI systems across end-to-end medical-AI research workflows. Distinct from prior benchmarks that focus on final output quality for narrow tasks, AutoMedBench offers transparency into agent behaviors within complex, long-horizon tasks typical of modern medical AI pipelines. Through decomposition into five explicit workflow stages and analysis spanning multiple medical imaging and multimodal inference tracks, it enables fine-grained, stage-level error assessment and thus facilitates the development and evaluation of robust, research-capable agentic models in medicine (Liu et al., 1 Jun 2026).

1. Motivation and Benchmark Scope

Traditional medical AI benchmarks, including MedQA, PubMedQA, and HealthBench, focus on single-step prediction (e.g., answering a clinical question, classifying an image) and assess only final output correctness. This approach obscures critical failure points arising from issues in planning, environment setup, intermediate validation, or deliverable formatting—each of which can individually or cumulatively degrade the reliability of autonomous medical research systems.

AutoMedBench was introduced to fill this diagnostic gap by explicitly modeling the multi-stage structure of plausible medical-AI research workflows. It focuses on agentized automation of tasks such as segmentation, image enhancement, visual question answering, report generation, and lesion detection, providing both stage-level and final artifact performance metrics. This focus reflects the emerging need for autonomous agents to conduct entire experiments, not just atomic predictions (Liu et al., 1 Jun 2026).

2. Unified Workflow Stages

AutoMedBench formalizes agent execution into a five-stage workflow, mirroring standard medical-AI experimentation:

Stage	Objective (summary)
S1 Plan	Read the task brief, research methods, compose a plan (plan.md)
S2 Setup	Resolve dependencies, verify model loading and environment
S3 Validate	Pilot inputs, check intermediate outputs, perform debugging
S4 Inference	Execute full-dataset inference, save predictions
S5 Submit	Ensure submission artifact format compliance, deliverable handoff

Each stage is evaluated and scored. Stages S1–S3 are assessed by an LLM judge on artifacts and traces; S4–S5 employ deterministic checks on output existence, formatting, and compliance.

The five-track design supports segmentation (macro Dice), enhancement (SSIM), VQA (accuracy), report generation (BLEU, METEOR, ROUGE-L, RadGraph-F1), and lesion detection ([email protected]) tasks. These are held constant for both Lite (scaffolded) and Standard (minimally hinted) difficulty tiers, isolating the effects of autonomy and prior knowledge (Liu et al., 1 Jun 2026).

3. Scoring, Evaluation Metrics, and Error Analysis

AutoMedBench produces two complementary scores per run:

• Agentic (Process) Score: Weighted sum of S1–S5, $$\text{Agentic} = 0.25 S_1 + 0.15 S_2 + 0.35 S_3 + 0.15 S_4 + 0.10 S_5$$ with $S_i \in [0,1]$.
• Task Score: Standard metric for each track (e.g., Dice for segmentation, accuracy for VQA).
• Overall Benchmark Score:

$$\text{Overall} = 0.5 \ \text{Agentic} + 0.5 \ \text{Task}$$

Stage-level diagnostics are central. Across thousands of runs and 33 average conversational turns per run, S2 (Setup) is consistently strongest, indicating competency in environment engineering (dependencies, model readiness), while S3 (Validate) is weakest (35% weight), revealing a pervasive absence of systematic pilot validation and pipeline checking.

Error analysis is explicit and quantitative:

• Validation (E3) and Submission (E5) failures comprise 37.7% and 38.1% of fired error codes, respectively.
• Task-understanding errors (E1) are rare at 0.9%.
• Runs with a single error code have a 48% lower overall score, and multiple errors almost never recover to high performance.
• High-performing agents demonstrate robust error recovery (continuing after failures) (Liu et al., 1 Jun 2026).

4. Task Diversity, Difficulty Tiers, and Experimental Findings

AutoMedBench covers 24 tasks spanning CT/MRI segmentation, low-dose CT denoising, MRI super-resolution, radiology/pathology VQA, report generation, and blood-smear/dental/wrist anomaly detection using identical held-out datasets, wall-time limits, submission schemas, and evaluation protocols.

Difficulty tiers are governed by task-brief scaffolding:

• Lite: Concretely specifies a method or model, pins dependencies, and offers stage-specific hints.
• Standard: Lists candidate methods but all pipeline design and validation checks are left to the agent.

These distinctions isolate pure agentic planning and design capacity from reliance on supplied workflows.

Key quantitative findings:

• Detection tasks yield the highest mean track scores; VQA and report generation are more challenging; segmentation lies in between.
• Stage-level scoring consistently accentuates agents’ weaknesses in validation.
• Engineering-oriented errors outweigh domain understanding failures, highlighting the necessity for more advanced error-recovery, validation routines, and submission checks (Liu et al., 1 Jun 2026).

5. Significance, Lessons Learned, and Comparison to Related Efforts

AutoMedBench’s process-level observability uniquely qualifies it to reveal where autonomous agents are bottlenecked—whether at pipeline engineering, pilot checking, or output compliance. It demonstrates that domain knowledge alone is insufficient for robust research automation; systematic engineering and validation are equal prerequisites.

Contrasting with prior work, AutoMedBench enables:

• Stage-specific diagnosis rather than monolithic evaluation.
• Comparison of agent autonomy under identical data and metrics.
• Rich, process-aware error classification facilitating iterative model and system improvement.

Insights from EHRBench (Xie et al., 28 May 2026) reinforce the need for robust quality control and knowledge-base verification in benchmark construction. AutoMedBench can in principle extend EHRBench’s pipeline—LLM-driven, KB-verified, template-based generation—with multimodal data, longer longitudinal context, and richer clinical reasoning objectives to further enhance clinical fidelity.

A plausible implication is that future agentic benchmarks will increasingly require systematic pipeline observability, knowledge-base grounding, and multi-stage artifact evaluation to foster generalizable and reliable autonomous research agents.

6. Future Directions and Extensions

Potential extensions for AutoMedBench include:

• Training-time adaptation for agents to learn from workflow errors and success cases.
• Multi-agent collaboration, enabling division of labor and peer verification within complex workflows.
• Human-in-the-loop or subjective clinical decision stages, bridging benchmarking with real-world clinical research practices.
• Incorporation of extended multi-modal EHR and imaging data, calibrated event forecasting, and integration of chain-of-thought or intermediate supervision.

The unifying principle remains the elevation from output-centric to process-transparent, workflow-competent medical AI, positioning AutoMedBench as a foundation for developing genuinely autonomous, diagnosable, and robust medical research agents (Liu et al., 1 Jun 2026).