MedCTA: Benchmark & Imaging Applications

• MedCTA is a benchmark providing 107 real-world clinical tasks to evaluate AI agents on tool retrieval, evidence acquisition, and integration using five deployed tools.
• It measures both process fidelity and clinical reasoning through step-by-step metrics like tool-selection and argument validity, highlighting controller instability challenges.
• The term also denotes dynamic CTA in imaging, where temporal information aids artery–vein separation and automated vessel segmentation in cerebrovascular analyses.

MedCTA is principally the name of a 2026 benchmark for evaluating medical AI agents that must perform tool retrieval, evidence acquisition, and evidence integration on clinician-validated, step-implicit tasks grounded in realistic multimodal clinical inputs, including radiology images, pathology slides, and reports (Ashraf et al., 10 Jun 2026). In current literature, however, the term is not entirely univocal: a separate dynamic CTA study uses “MedCTA” to denote medical CTA enhanced by temporal information for bone removal, artery–vein separation, and automated vessel analysis in cerebrovascular imaging (Yadav et al., 14 Feb 2025). The term therefore has a benchmark-centered meaning in medical AI evaluation and a distinct imaging-centered meaning in some CTA research contexts.

1. Terminology and referents

The most formally defined use of the term is the benchmark introduced as “MedCTA: A Benchmark for Clinical Tool Agents,” which targets autonomous multi-step clinical tool use rather than isolated perception or one-shot question answering (Ashraf et al., 10 Jun 2026). A separate usage appears in dynamic CTA segmentation work, where “MedCTA” denotes medical CTA enhanced by temporal information to facilitate automated vessel analysis (Yadav et al., 14 Feb 2025). The name also appears informally in coronary CT writing, including the phrases “routine MedCTA reporting” and “Recommendations for MedCTA Practice” (Gerbasi et al., 2023, Wu et al., 2023).

Usage	Description
MedCTA benchmark	107 real-world clinical tasks with clinician-verified executable trajectories over 5 deployed tools
MedCTA in dynamic CTA	Medical CTA enhanced by temporal information for bone removal, artery–vein separation, and automated vessel analysis
Informal coronary CT usage	Appears in coronary CTA discussions without a separate formal benchmark definition

This terminological overlap matters because the benchmark and the imaging usage address different technical problems. The former evaluates agentic orchestration over external tools; the latter concerns vessel segmentation and vascular template construction from 4D-CTA.

2. Motivation and formal problem setting

MedCTA was introduced to address a gap in medical AI evaluation: clinically grounded decision-making often requires agents to go beyond simple recognition and to perform tool retrieval, evidence acquisition, and integration (Ashraf et al., 10 Jun 2026). Existing medical benchmarks were described as focusing largely on isolated perception or single-turn question answering, with limited visibility into failures of planning, tool recruitment, and rollout reliability. MedCTA therefore evaluates the full agentic loop in a safety-critical domain: perception, planning, tool invocation, evidence integration, and conclusion.

The benchmark is organized around a tool library

$\mathcal{D}=\{\tau_1,\dots,\tau_5\}$

and a task collection

$$\{(\mathcal{X},\mathcal{Q},\mathcal{U},\pi,\mathcal{A})\},$$

where $\mathcal{X}$ is the clinical context, $\mathcal{Q}$ is a step-implicit clinical query, $\mathcal{U}\subseteq\mathcal{D}$ is the hidden subset of tools sufficient to solve the query, $\pi=\{(t_i,\alpha_i,\rho_i)\}_{i=1}^m$ is a clinician-verified reference trajectory of tool calls with arguments and observations, and $\mathcal{A}$ is the final answer (Ashraf et al., 10 Jun 2026). The benchmark’s design deliberately withholds the tool sequence at test time. Agents must infer the workflow implicitly from the clinical goal rather than follow an exposed execution plan.

This formalization is important because it shifts evaluation from answer-only correctness to trajectory quality. A clinically correct final answer obtained through an unstable or non-reproducible sequence of tool calls is treated differently from a faithful, executable, and evidence-grounded rollout.

3. Dataset composition and tool environment

MedCTA comprises 107 real-world clinical tasks with an average of 3.1 steps and a range of 2–4 steps (Ashraf et al., 10 Jun 2026). The tasks span 34 anatomical regions across body systems including brain, chest, abdomen, ocular, and cellular pathology. Modalities include CT (20%), reports (20%), histopathology (18%), X-ray (10%), MRI (9%), fundus (7%), gross pathology (5%), plus dermoscopy, PET, mammography, and ultrastructure. This multimodal composition is central to the benchmark’s purpose: clinical tool agents are evaluated under the same kind of heterogeneous evidence conditions that appear in practice.

The five deployed tools are fixed and executable:

Tool	Output
OCR(image)	text + optional bounding boxes
ImageDescription(image)	holistic image caption
RegionAttributeDescription(image, bbox, attribute?)	localized attribute descriptions
GoogleSearch(query, k)	top-$k$ frozen snippets
Calculator(expression)	numeric result

Each tool is exposed through a Lagent-based wrapper (Ashraf et al., 10 Jun 2026). The frozen GoogleSearch snippets are specifically intended to preserve reproducibility. Agents never see the gold trajectory $\pi$ during testing; they must decide whether to caption an image, localize a region, retrieve external evidence, or compute a numerical quantity.

A representative task in the benchmark is phrased only as a clinical objective, such as “Based on the CT image, what type of venous thrombosis is present?” (Ashraf et al., 10 Jun 2026). The implied solution may require a chain such as ImageDescription, then RegionAttributeDescription over a portal-vein bounding box, followed by a summary or calculation step. The benchmark’s step-implicit formulation is meant to expose failures of tool recruitment and stopping behavior that would be hidden in pipelines where the correct tool order is prescribed.

4. Evaluation methodology and experimental protocols

MedCTA uses process-aware evaluation. If a model rollout is

$$\hat\pi=\{\hat s_i=(\hat t_i,\hat\alpha_i,\hat\rho_i)\}_{i=1}^{\hat m}$$

and the gold trajectory is

$$\{(\mathcal{X},\mathcal{Q},\mathcal{U},\pi,\mathcal{A})\},$$0

the benchmark scores step execution, clinical reasoning, and final outcomes separately (Ashraf et al., 10 Jun 2026).

In step-by-step mode, the reported metrics are instruction-format accuracy, tool-selection accuracy, argument validity, and per-step summarization accuracy. The first three are defined as:

$$\{(\mathcal{X},\mathcal{Q},\mathcal{U},\pi,\mathcal{A})\},$$1

$$\{(\mathcal{X},\mathcal{Q},\mathcal{U},\pi,\mathcal{A})\},$$2

$$\{(\mathcal{X},\mathcal{Q},\mathcal{U},\pi,\mathcal{A})\},$$3

SummAcc evaluates whether the agent correctly integrates the observation $$\{(\mathcal{X},\mathcal{Q},\mathcal{U},\pi,\mathcal{A})\},$$4 into the expected intermediate conclusion (Ashraf et al., 10 Jun 2026).

Clinical reasoning mode adds three scored dimensions: Clinical Faithfulness $$\{(\mathcal{X},\mathcal{Q},\mathcal{U},\pi,\mathcal{A})\},$$5, Context Integration $$\{(\mathcal{X},\mathcal{Q},\mathcal{U},\pi,\mathcal{A})\},$$6, and Semantic Completeness $$\{(\mathcal{X},\mathcal{Q},\mathcal{U},\pi,\mathcal{A})\},$$7, each scored from 0 to 1 and linearly mapped to $$\{(\mathcal{X},\mathcal{Q},\mathcal{U},\pi,\mathcal{A})\},$$8. Outcome mode reports Goal Accuracy $$\{(\mathcal{X},\mathcal{Q},\mathcal{U},\pi,\mathcal{A})\},$$9, defined against a clinician-validated whitelist for closed-form answers or by rubric-based semantic matching for open answers. All metrics are macro-averaged across tasks, and missing steps due to premature stopping or API/format failures receive zeros (Ashraf et al., 10 Jun 2026).

The experimental setup benchmarks 18 open- and closed-source multimodal models under two protocols: fully autonomous rollout and gold-standard next-tool routing (Ashraf et al., 10 Jun 2026). The latter forces the correct tool identity at each step while leaving argument composition and summary generation to the model. The model suite includes OpenAI, Anthropic, Google, Qwen, DeepSeek, Meta, Mistral, and Microsoft systems. Infrastructure is based on a Lagent and OpenCompass harness, ReAct-style prompting, and AgentLego APIs, with 1,926 autonomous rollouts across the 107 tasks.

5. Empirical findings and failure structure

The central empirical result is that strong backbone perception does not translate into reliable agentic behavior in clinical settings (Ashraf et al., 10 Jun 2026). Even the best closed-source agent barely exceeds 31.5% autonomous outcome accuracy. For GPT-5.4, the reported metrics are InstAcc 35.3, ToolAcc 23.5, ArgAcc 12.6, SummAcc 35.5, $\mathcal{X}$0 17.5, $\mathcal{X}$1 14.2, $\mathcal{X}$2 18.6, and $\mathcal{X}$3 31.5. Claude-opus-4-6 attains $\mathcal{X}$4, and Qwen3-8B attains $\mathcal{X}$5 (Ashraf et al., 10 Jun 2026). No model achieves non-zero strict trajectory success under full step-by-step metrics.

The failure profile is dominated by controller instability. Global rollout diagnostics report an API/protocol error rate of 64.2%, an under-call rate of 99.2%, protocol failure first in 58.3% of cases, and tool-selection failure first in 41.6% of cases (Ashraf et al., 10 Jun 2026). The benchmark therefore reveals that many systems fail before substantive clinical reasoning is even reached. Step-by-step and reasoning metrics only partially correlate with final accuracy; the reported Pearson correlation is approximately $\mathcal{X}$6 for SummAcc versus $\mathcal{X}$7, with stronger association for $\mathcal{X}$8 and $\mathcal{X}$9.

Gold-standard tool routing produces large but incomplete gains:

Model	Auto $\mathcal{Q}$0	Gold $\mathcal{Q}$1
GPT-5.4	31.5	49.5
Claude-opus-4-6	31.3	66.4
Qwen3.5-9B	21.6	49.5

These deltas expose a controller gap: when the next tool is supplied by an oracle, accuracy rises sharply, but long-horizon grounding and faithful evidence use remain incomplete (Ashraf et al., 10 Jun 2026). Qualitative examples reinforce this interpretation. Reported failure cases include protocol breakage via ARGS_ERROR, looping behavior, drift toward generic or unrelated answer priors instead of grounding in OCR and GoogleSearch evidence, and fluent but biologically inverted answers produced without tool use.

A common misconception challenged by MedCTA is that high-quality single-turn multimodal performance is a sufficient proxy for safe clinical autonomy. The benchmark explicitly reports that strong single-turn VLM performance does not translate into robust agentic behavior (Ashraf et al., 10 Jun 2026).

6. Interpretive significance, limitations, and extension path

MedCTA’s main significance lies in relocating evaluation from end answers to process fidelity. It provides visibility into tool selection, argument validity, execution stability, trajectory fidelity, and outcome quality within the same benchmark (Ashraf et al., 10 Jun 2026). This is consequential for clinical deployment because brittle rollout control, premature stopping, and incorrect tool recruitment are failure modes that answer-only benchmarks tend to conceal.

The benchmark also delineates a technical bottleneck. The dominant failure layer is controller reliability rather than raw medical knowledge (Ashraf et al., 10 Jun 2026). This suggests that future progress in clinical agents may depend not only on stronger multimodal encoders or larger LLMs, but on improved protocol adherence, argument construction, stopping policies, and explicit trajectory control.

Several limitations are stated. MedCTA focuses on five core tools: OCR, captioning, region description, search, and calculation (Ashraf et al., 10 Jun 2026). The 107 tasks cover many modalities, but institutional and demographic diversity are not examined, and fairness and bias remain unexamined. Automated clinical-reasoning metrics are validated by a human audit with Spearman $\mathcal{Q}$2, but they cannot fully replace expert review. External retrieval is frozen for reproducibility, which improves comparability but sacrifices live-web freshness.

The proposed extension path includes adding segmentation, structured EHR retrieval, clinical-score calculators, and interactive clarification tools (Ashraf et al., 10 Jun 2026). In effect, MedCTA functions not merely as a leaderboard but as a diagnostic framework for auditing why medical tool agents fail.

7. MedCTA in imaging literature beyond the benchmark

A distinct usage of the term appears in cerebrovascular imaging, where dynamic CTA is described as providing both arterial- and venous-phase images with near-perfect subtraction of bone and soft tissue, forming the basis for “MedCTA—medical CTA enhanced by temporal information to facilitate bone removal, artery–vein separation, and automated vessel analysis” (Yadav et al., 14 Feb 2025). In that work, dynamic CTA refers to time-resolved acquisitions, exemplified by 19 volumes over 60 s in a 320-slice 4D-CTA protocol. The technical program is unrelated to agent benchmarking: it develops angiographic CT templates and vessel segmentation methods.

The imaging pipeline has two branches. The atlas-based branch extracts arterial-phase and venous-phase volumes, builds four unbiased population-average templates with ANTs, tracks convergence by Dice Similarity Coefficient and root-mean-squared error, thresholds vessel-separated templates at 30 HU, retains the largest connected component, and applies neuroradiologist correction to arterial and venous ROIs (Yadav et al., 14 Feb 2025). The deep-learning branch trains nnU-Net and NexToU on bone-in CT phases, with loss

$\mathcal{Q}$3

patch size $\mathcal{Q}$4, batch size 2, SGD with initial learning rate 0.01 linearly decaying over 1000 epochs, and multi-phase augmentation by treating multiple 4D-CTA phases as independent training samples (Yadav et al., 14 Feb 2025).

Its quantitative findings differ correspondingly from the benchmark setting. Atlas-based segmentation attains overall amDC 0.324 for arteries and 0.495 for veins, whereas deep learning attains 0.856 for arteries and 0.743 for veins (Yadav et al., 14 Feb 2025). For the internal carotid arteries, vertebral arteries, and basilar artery, atlas-based segmentation has amDC 0.402 and deep learning 0.913; for MCA-M1, PCA-P1, and ACA-A1, the values are 0.474 and 0.932. The paper’s conclusion is that dynamic 4D-CTA can transform conventional CTA into a rich platform for automated vessel segmentation, with atlas methods as rapid initializations and deep learning as the more accurate fully automated route (Yadav et al., 14 Feb 2025).

This separate imaging usage does not redefine the benchmark. Instead, it illustrates that “MedCTA” is currently polysemous across arXiv-adjacent medical AI and CTA literature. In benchmark discussions, it denotes a testbed for clinical tool agents; in dynamic CTA work, it denotes CTA workflows enhanced by temporal information for automated vascular analysis (Ashraf et al., 10 Jun 2026, Yadav et al., 14 Feb 2025).