RadAgent: A tool-using AI agent for stepwise interpretation of chest computed tomography
Abstract: Vision-LLMs (VLM) have markedly advanced AI-driven interpretation and reporting of complex medical imaging, such as computed tomography (CT). Yet, existing methods largely relegate clinicians to passive observers of final outputs, offering no interpretable reasoning trace for them to inspect, validate, or refine. To address this, we introduce RadAgent, a tool-using AI agent that generates CT reports through a stepwise and interpretable process. Each resulting report is accompanied by a fully inspectable trace of intermediate decisions and tool interactions, allowing clinicians to examine how the reported findings are derived. In our experiments, we observe that RadAgent improves Chest CT report generation over its 3D VLM counterpart, CT-Chat, across three dimensions. Clinical accuracy improves by 6.0 points (36.4% relative) in macro-F1 and 5.4 points (19.6% relative) in micro-F1. Robustness under adversarial conditions improves by 24.7 points (41.9% relative). Furthermore, RadAgent achieves 37.0% in faithfulness, a new capability entirely absent in its 3D VLM counterpart. By structuring the interpretation of chest CT as an explicit, tool-augmented and iterative reasoning trace, RadAgent brings us closer toward transparent and reliable AI for radiology.
Summary
- The paper introduces an RL-optimized AI agent that leverages a curated toolbox to generate interpretable, sequential diagnostic reports for chest CT scans.
- The paper details a novel methodology that integrates multiple open-source models with a radiologist-curated checklist to enhance report fidelity and tool efficiency.
- The paper demonstrates significant improvements over baseline systems in clinical accuracy, robustness, and faithfulness, validated through extensive ablation studies and real-world datasets.
RadAgent: Reinforcement-Learned Tool-Using Agent for Stepwise Chest CT Interpretation
Introduction and Motivation
RadAgent represents a significant methodological advance in the automation of chest computed tomography (CT) analysis and report generation by introducing explicit, dynamic tool use supervised by reinforcement learning (RL). Distinct from monolithic vision-LLMs (VLMs), which generate reports as opaque outputs, RadAgent exposes its full diagnostic trajectory—planning, tool selection, evidence accumulation, and refinement—thereby offering a transparent, clinician-inspectable process. This interpretability is critical for high-stakes clinical settings where validation and traceability are prerequisites for trust and adoption.
Architecture: Agentic Tool-Oriented Reasoning
RadAgent is built atop a 14B-parameter instruction-tuned Qwen3-based LLM, orchestrating a toolbox of ten open-source models for slice-based VLM question answering, segmentation, classification, windowing, report generation, and other 3D chest CT analysis sub-tasks. The agent process is governed by a radiologist-curated nine-point diagnostic checklist, which structures its exploration of the CT volume in a manner analogous to expert radiology practice.
Given a 3D CT volume and user query, RadAgent first drafts a preliminary report, then iteratively revisits and verifies findings, using the checklist to guide sequential tool calls. At each step, the agent plans actions, reasons about new information, and updates its persistent scratchpad, ensuring every inference is anchored in identifiable tool outputs and decisions. This creates a lineage of evidence connecting low-level visual features to high-level report findings.
Figure 1: Schematic of RadAgent's agentic reasoning loop, evidentiary scratchpad, and composite RL training pipeline.
Training Paradigm: RL-Optimized Tool-Use Policy
Where prior agentic systems in medical imaging generally rely on rule-based or prompt-programmed tool selection, RadAgent’s agentic policy is optimized end-to-end via RL. The training is conducted using the GRPO algorithm with LoRA finetuning, maximizing a carefully balanced composite reward over report accuracy, tool use coherence, diversity, checklist adherence, and sequence efficiency. The reward function includes macro and micro F1 agreement with reference reports (for 18 benchmark pathologies), matching of abnormal findings, successful tool execution, tool usage diversity, graph-theoretic coherence of tool call sequences, and LLM-judged checklist adherence. The reward schedule is curriculum-based, with early emphasis on exploration and later prioritization on adherence and efficiency.
Ablation experiments confirm the necessity of the curriculum and composite reward: lack of sequence coherence in reward rapidly degrades adherence to the diagnostic checklist and increases incoherent or wasteful tool calls, while training with sequence constraints from the outset curtails report quality due to premature exploitation.
Figure 3: Reward ablation study demonstrating the necessity of staged curriculum for both report quality and tool sequence coherence.
Toolbox and Modular Tool Framework
The RadAgent toolbox, accessible via the Model Context Protocol (MCP), comprises the following core tool types:
- 3D/2D Visual Question Answering: Whole-volume and slice-level VLMs for fine-grained or region-specific evidence gathering.
- Disease Classifier: Pathology-wide screening via contrastive latent representations.
- Report Generation: Fast-drafting of study-level narratives.
- Segmentation Tools: Anatomical and effusion segmentors supporting region-specific analyses.
- Slice Extraction and Windowing: Facilitating targeted visual queries and enhanced interpretability.
The system is deployed on an eight-GPU cluster to efficiently parallelize inference across the computational pipeline.
Figure 5: Overview of specialized tools in the RadAgent toolbox, covering the entire 3D chest CT diagnostic workflow.
Experimental Protocol: Datasets and Metrics
Evaluation is conducted on the CT-RATE (25,692 3D chest CTs with structured reports and 18 labeled pathologies) and RadChestCT (external, multi-institutional, 84 abnormality-labeled) datasets. The primary metrics are macro and micro F1-scores over automatic report-derived pathology labels, consistent with emerging chest CT evaluation standards. Robustness is quantified by the conditional probability of correct label retention following adversarial prompt perturbation, while faithfulness, following Chen et al., measures whether a model’s output acknowledges if its decision was influenced by a prompt hint.
Quantitative Results: Clinical Accuracy, Robustness, and Faithfulness
RadAgent demonstrates statistically significant and practically meaningful improvements over the baseline 3D VLM (CT-Chat):
- Clinical Accuracy: On CT-RATE test, RadAgent achieves macro F1 +6.0 (36.4% rel.), micro F1 +5.4 (19.6% rel.) over CT-Chat. On RadChestCT, RadAgent outperforms both before and after RL fine-tuning, with the RL-trained variant overcoming initial generalization deficits observed in training-free tool use.
Figure 4: RadAgent versus CT-Chat on report generation F1 scores across multiple internal and external test sets.
- Per-Pathology Performance: Gains are concentrated in challenging pathologies poorly detected by baselines, as evident in pathology-specific F1 breakdowns.
Figure 6: Per-pathology F1-scores illustrating robust improvements by RadAgent, especially on difficult findings.
- Robustness and Faithfulness: Under manual hint injection, RadAgent maintains an 83.7% robustness (vs. 58.9% for CT-Chat) and 37.0% faithfulness (vs. 0.00%). RadAgent acknowledges when its outputs are influenced by external hints, revealing its reasoning trace, while the baseline remains vulnerable to being surreptitiously steered without disclosure.
Figure 7: RadAgent's agentic trace confers substantial robustness and faithfulness versus direct VLM inference under adversarial hints.
- Tool-Use Policy and Scalability: Post-training, RadAgent discovers efficient, non-redundant tool calling strategies, emphasizes diversity without unnecessary computational overhead, and remains extensible to new tools and evolving checklists.
(Figure 8; Figure 9)
Figure 10: Learned tool-use policy visualized via Sankey plot; training improves average tool call success rates and effective resource allocation.
Interpretability and Workflow Integration
RadAgent’s agentic trace allows inspection at each reasoning step: which findings are supported by which tool outputs, how discrepancies are resolved, and where evidence is accumulated or refined. This enables actionable human-AI collaboration. For instance, after report generation, a radiologist can query RadAgent’s scratchpad to directly verify supporting segmentations or question/answer pairs for each reported abnormality.
(Figure 1b; Figure 1c; Figure 11)
Figure 12: End-to-end illustration of RadAgent’s stepwise plan, tool calls, scratchpad updates, and final evidence-anchored report formation, including error recovery mechanisms for tool or data failures.
Discussion and Broader Implications
RadAgent's RL-trained tool orchestration distinguishes it from prior prompt-based or training-free agentic systems, which are brittle in the face of the evolving tool ecosystem and require significant manual engineering. By formalizing the chest CT diagnostic workflow as a partially observable Markov decision process with RL-optimized tool sequencing, RadAgent automates the discovery of performant diagnostic policies—an approach scalable to expanded toolsets, new imaging modalities, and institution-local reporting guidelines.
This paradigm begins to close the specialization-versus-breadth tradeoff observed in clinical AI: an adaptive agentic interface can marshal state-of-the-art narrow expert models for sub-tasks, while retaining the flexibility and inspection capability necessary for safe and auditable real-world deployment.
However, the deployment of RadAgent currently requires multi-GPU infrastructure due to the compute demands of multiple models. Further work in model compression, distillation, or workflow distillation into fixed pipelines could reduce this burden and increase practicality. Another open challenge is dynamically adapting tool policies as toolboxes change, but the RL-finetunable agent architecture is uniquely suited for continual policy improvement.
Conclusion
RadAgent establishes a new standard for interpretable, robust, and faithful automated chest CT analysis and report generation by integrating RL-optimized tool use within a transparent agentic reasoning framework. Its strong quantitative performance, traceable diagnostic process, and human-in-the-loop compatibility signal a scalable design pattern for future medical AI systems—bridging the gap between generalist foundation models and reliable clinical practice.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Easy-to-understand summary of “RadAgent: A tool-using AI agent for stepwise interpretation of chest computed tomography”
What is this paper about?
This paper introduces RadAgent, a new kind of AI that helps doctors write reports for chest CT scans. Unlike many AI systems that give a final answer without showing how they got there, RadAgent works like a careful detective: it uses a checklist, calls different “tools” (small specialized programs), and keeps a step-by-step record of what it checked and why. This makes the process more transparent and easier for doctors to trust and verify.
What questions were the researchers trying to answer?
The researchers mainly asked:
- Can an AI that thinks and works step by step (using tools and a checklist) write better chest CT reports than a regular “all-in-one” AI model?
- Can this step-by-step AI be more robust (harder to fool) and more faithful (honest about what influenced its decisions)?
- Can training the AI to use tools wisely (instead of just telling it what to do) lead to better results?
How did they do it? (Methods explained simply)
Think of RadAgent as a smart assistant with a toolkit and a to-do list:
- The toolkit includes specialized helpers, for example:
- A “question-answering” tool that answers questions about the 3D CT scan.
- A “segmentation” tool that outlines organs or fluid to show exactly where something is.
- A “slice picker” that selects the most useful 2D images from a big 3D scan.
- A “windowing” tool that adjusts brightness/contrast to make certain tissues easier to see.
- A “report draft” tool to write an initial version of the report.
- The to-do list is a radiology checklist (like “check lungs for nodules,” “check heart size,” “look for fluid,” etc.). This keeps the process organized.
How RadAgent works on a scan:
- It first asks a 3D vision-LLM (called CT-Chat) to draft a basic report.
- Then it goes through the checklist, step by step. At each step, it decides:
- What question to investigate next.
- Which tool to call to gather evidence.
- It keeps a “scratchpad” (a transparent memory) of findings and the evidence behind them.
- When it has enough evidence, it writes the final report—linked to the steps it took and the tools it used.
How RadAgent learned:
- The team used a training method called “reinforcement learning.” You can think of it like training a student by rewarding good behavior:
- Good behavior = a report that matches what’s in the scan, uses tools sensibly, follows the checklist, and avoids wasted steps.
- Over time, RadAgent learned not just “what to say,” but “how to work”—which tools to use and when.
How they tested it:
- They tested RadAgent on large chest CT datasets (one it was trained on, and another from a different hospital) and compared it against the base 3D model (CT-Chat).
- They measured:
- Accuracy: How often the report correctly mentions common diseases.
- Robustness: If you try to mislead the system with a wrong hint, does it still get the right answer?
- Faithfulness: If a hint actually changes the AI’s answer, does the AI admit that the hint influenced it?
A quick word on metrics:
- F1 score: A score from 0 to 100 that balances “catching true problems” and “avoiding false alarms.” Macro-F1 treats each disease equally; micro-F1 looks at overall correctness across all cases.
- Robustness: Staying correct even when misled.
- Faithfulness: Being honest about what influenced the decision.
What did they find, and why is it important?
Main results:
- Higher accuracy: RadAgent made more correct findings than the base model (CT-Chat).
- On a main test set, RadAgent improved macro-F1 by about 6 points (a 36% relative improvement) and micro-F1 by about 5.4 points (about 20% relative).
- Better robustness: When given misleading hints (“I think there is calcification,” even if there isn’t), RadAgent resisted being fooled much more often—about 84% vs. 59% for the baseline.
- More faithfulness: If a hint did affect its answer, RadAgent was far more likely to say so (about 37% of the time), while the baseline model never did (0%). In other words, RadAgent is more open about what shaped its decision.
- Training mattered: Even a “training-free” version (just the toolkit and checklist without reinforcement learning) already helped somewhat, but training RadAgent to use tools effectively led to bigger gains and better generalization to the external dataset.
Why this matters:
- Medical AI is high-stakes. Doctors need to see not just the final answer, but also how it was reached.
- RadAgent’s step-by-step, tool-using process creates an audit trail doctors can inspect—helping build trust and catch mistakes.
- The AI didn’t just get better at writing reports; it became harder to mislead and more honest about its influences.
What could this change in the future?
- More transparent AI in healthcare: RadAgent shows a path toward AI systems that explain their reasoning and tie findings to concrete evidence (like segmented regions or selected slices).
- Better teamwork between doctors and AI: With a clear trace, doctors can quickly check, confirm, or challenge specific steps (e.g., “Show me the slices where you saw the effusion”).
- Smarter use of specialized tools: The AI learns effective strategies for when and how to use different tools—something that’s hard to hand-design.
- Room to grow: Right now, RadAgent focuses on chest CTs and needs significant computing power. Also, while faithfulness improved, there’s still a lot of space to get better. But the approach can be expanded to more tools, more body regions, and more hospital settings.
In short: RadAgent turns CT reporting from a single black-box step into a clear, step-by-step investigation. It not only improves accuracy but also strengthens trust by showing its work—an important step toward safer, more reliable AI in radiology.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a consolidated list of concrete gaps and open questions that remain unresolved and could guide future research on RadAgent and tool-using radiology agents.
- Clinical validation is absent: no reader studies, no time-to-report measurements, and no assessment of error detection/clarification rates when radiologists use the agent’s trace, leaving real-world utility and workflow impact unquantified.
- Scope restricted to non-contrast chest CT: generalization to contrast-enhanced CT (e.g., CTPA), high-resolution CT, trauma protocols, abdominal/pelvic CT, or other modalities (XR, MR, US) is untested.
- Evaluation limited to 18 CT-RATE pathologies: performance on the full RadChestCT label set (82 abnormalities) and rarer findings is unexplored; detection of critical-but-infrequent entities remains unknown.
- External evaluation uses the same CT-RATE text classifier on RadChestCT: classifier transferability and reliability across institutions/domains are unvalidated, risking biased estimates of generalization.
- Potential reward and evaluation “gaming”: RL optimizes against the CT-RATE label extractor and an LLM report judge; there is no audit for stylistic artifacts or phrasing that inflate scores without improving clinical correctness.
- Lack of radiologist-adjudicated benchmarks: no human gold-standard scoring of report correctness, abnormality localization, or clinical adequacy to complement automated metrics.
- GREEN metric critique without a replacement: despite length-bias concerns with GREEN, the work does not propose or validate a CT-specific, radiologist-calibrated metric that balances abnormality detection, location, and severity without length bias.
- Faithfulness metric depends on LLM-based hint-admission judging: no human validation that “admission” reflects true reasoning transparency; vulnerability to gaming via generic disclaimers is unassessed.
- Hint robustness tested only on single-pathology, single-hint prompts: behavior under multiple, conflicting, or chain-of-thought hints; adversarial jailbreaks; or subtle instruction-style perturbations is not explored.
- No analysis of tool accuracy and error propagation: segmentation, classification, and VQA components are not independently benchmarked on the same datasets; how upstream tool failures bias downstream conclusions is unknown.
- Per-tool ablations are missing: contribution of each tool (e.g., segmentation, windowing, 2D VQA) to performance and faithfulness is not quantified; the minimal effective toolbox is unclear.
- Orchestrator/model choice not explored: only Qwen3-14B was used; scaling laws, model size trade-offs, and comparisons with other open-source and closed-source LLMs are not provided.
- RL design space underexplored: comparisons to supervised fine-tuning, other RL algorithms (e.g., PPO, DPO variants, ToT-optimized policies), credit assignment strategies for long tool sequences, and sensitivity to hyperparameters/seeds are missing.
- Training is short and small-scale (150 steps, 8 rollouts): stability, convergence, and reproducibility across seeds/runs are unreported; how performance scales with more steps/rollouts remains open.
- Reward design trade-offs not fully quantified: the diversity reward may still incentivize unnecessary tool calls; no explicit analysis of compute cost vs. accuracy/faithfulness gains per reward component.
- Runtime and resource demands are unspecified: latency per study, throughput, GPU-hours per case, and memory footprints are not reported; feasibility for routine clinical deployment is unknown.
- Robustness to image-level domain shifts untested: performance under variations in scanner vendor, kernel, slice thickness, dose, motion, metal artifacts, limited field-of-view, and reconstruction algorithms is not measured.
- Subgroup performance and fairness are unassessed: no stratified analyses by site, age, sex, BMI, comorbidities, or scanner characteristics; potential performance disparities are unknown.
- Checklist portability and validity: the AI-initialized, single-radiologist–revised checklist is not validated across sites or subspecialties; how different checklists affect policy and outcomes is unknown.
- Human-in-the-loop design is untested: how clinicians interact with the trace (e.g., evidence slices, overlays), what UI affordances are needed, and whether the trace reduces cognitive load or increases trust remain open.
- Safety and guardrails: no mechanisms to prevent or detect hallucinated tool outputs, misinterpretation of tool results, or risky action sequences; no “tool-grounded citation” requirement linking report statements to specific evidence slices/masks.
- Evidence-use auditing is missing: causal analyses linking specific tool outputs to final sentence-level findings are absent; whether the agent actually relies on collected evidence vs. prior is unclear.
- Generalization beyond fixed tools: the policy is tailored to a fixed toolbox; methods for continual learning, plug-and-play tool updates, or zero-shot incorporation of new tools are not proposed or evaluated.
- Distillation to a fixed, auditable workflow is not demonstrated: the hypothesized benefit for regulatory stability and efficiency is untested; the fidelity of distilled policies and their safety need study.
- Baseline comparisons are narrow: no head-to-head evaluations against recent agentic systems (e.g., CT-Agent, CTPA-Agent) or strong non-agentic 3D/2D VLM baselines under identical conditions/datasets.
- Dual use of CT-Chat (as both baseline and toolbox component) may confound comparisons: isolating gains attributable to orchestration vs. underlying model capability is not straightforward.
- Report completeness and structure: adherence to standardized reporting templates (e.g., Fleischner/Society guidelines), inclusion of required measurements, and impression quality are not evaluated.
- Uncertainty quantification is absent: no calibrated confidence for findings or whole-report uncertainty is provided for downstream clinician decision-making.
- Longitudinal/temporal reasoning is unsupported: comparing current and prior studies to assess progression/regression or treatment response is left unexplored.
- Data privacy and security implications of MCP-based multi-node tool orchestration are unaddressed: PHI handling, logging policies, and auditability need specification.
- Reproducibility details are incomplete: sensitivity to random seeds, full prompts for all judges/tools, exact splits, and environment configuration for faithful replication require release/verification.
- Extension to multi-organ or pan-thoracic findings: how to scale the checklist, tools, and reward design to broader anatomic coverage (e.g., mediastinal nodes, musculoskeletal findings) is unclear.
- Explainability artifacts not provided: automatic generation of slice overlays and direct linkage between report sentences and visual evidence for clinician review needs development and evaluation.
Practical Applications
Immediate Applications
Below are actionable, sector-linked use cases that can be deployed with the paper’s current methods and performance, assuming appropriate human oversight and institutional integration.
- Radiology report drafting with traceable reasoning — Sectors: healthcare, medical imaging IT
- What it looks like: Deploy RadAgent as a “first-pass” or “co-pilot” that drafts chest CT reports, records an inspectable stepwise trace (tool calls, intermediate findings), and surfaces evidence (e.g., segmented effusions, selected slices) for the radiologist to verify before sign-off.
- Potential tools/products/workflows: PACS/RIS plug‑in or web app integrating MCP servers for CT-Chat (report/VQA), TotalSegmentator (anatomy/effusion), CT-CLIP (18-pathology screening), windowing and slice-picking utilities; checklist-driven UI that lists completed vs. pending checks.
- Assumptions/dependencies: Non-contrast chest CT domain; multi‑GPU availability; DICOM ingestion/PHI protection; clinician-in-the-loop; local checklist alignment; not a stand‑alone diagnostic device (requires regulatory clearance for autonomous use).
- Quality assurance (QA) and consistency checks for existing reports — Sectors: healthcare, quality/safety
- What it looks like: Run RadAgent on finalized or dictated reports to detect likely omissions or contradictions (e.g., segmentation suggests effusion but report is silent), prompting targeted review.
- Potential tools/products/workflows: “QA pass” button that replays tool trace against a fixed checklist and highlights mismatches; dashboards showing macro/micro-F1 vs. departmental benchmarks.
- Assumptions/dependencies: Integration with reporting systems; acceptance of mild latency; agreed-upon escalation rules for flagged cases.
- Triage and worklist prioritization — Sectors: healthcare operations
- What it looks like: Use the disease classifier and quick agentic checks to flag likely urgent findings (e.g., large effusions, suspected consolidation) and push them up the queue.
- Potential tools/products/workflows: Lightweight pre‑read pipeline (classification → targeted VQA/segmentation) that assigns a triage score; worklist integration.
- Assumptions/dependencies: Sensitivity/PPV thresholds tuned per site; domain limited to supported pathologies; not a substitute for clinician judgment.
- Radiology education and competency training — Sectors: education (medical schools, residency programs)
- What it looks like: Interactive modules that expose the agent’s reasoning trace, show how tools support each finding, and allow residents to compare their workflow against an expert-guided checklist.
- Potential tools/products/workflows: “Explain my call” mode; side-by-side comparison of trainee vs. RadAgent step sequences; curated cases from CT-RATE or de‑identified institutional archives.
- Assumptions/dependencies: De-identified data; faculty review of content; institution IT support.
- Robustness and faithfulness auditing of imaging AI — Sectors: software/ML governance, healthcare QA
- What it looks like: Adopt the paper’s hint-injection protocol and faithfulness/robustness metrics to evaluate in‑house report generators or vendor models; keep audit logs of agent traces.
- Potential tools/products/workflows: Model governance dashboards tracking robustness to misleading prompts and admission of hint influence; periodic red‑team tests.
- Assumptions/dependencies: Access to representative test sets; acceptance of trace logging as part of AI QA; workforce to review audits.
- Vendor/enterprise tool orchestration via MCP — Sectors: health IT, software infrastructure
- What it looks like: Standardize multimodal tool connections (VLMs, segmenters, windowing) with MCP servers to make internal pipelines easier to maintain and extend.
- Potential tools/products/workflows: Containerized MCP-packaged tools; GPU resource pooling across nodes; observability for per-tool performance and cost.
- Assumptions/dependencies: DevOps maturity; GPU scheduling; security hardening of MCP endpoints.
- Research reproducibility and benchmarking — Sectors: academia, R&D labs
- What it looks like: Use the released RadAgent, GRPO reward design, and tool suite to benchmark new tools or reward functions on CT‑RATE and RadChestCT; ablate tool policies.
- Potential tools/products/workflows: Training scripts (LoRA + GRPO), composite reward curriculum (report quality + tool-sequence coherence), public repo for agent traces.
- Assumptions/dependencies: Access to datasets under license; 8×GPU-class compute; consistent evaluation (CT-RATE pathologies, F1 extraction).
- Fixed-workflow distillation for cost control — Sectors: healthcare IT, operations
- What it looks like: Distill the learned policy into a fixed, rule‑based tool sequence for common cases to reduce inference cost/latency while retaining high-yield steps.
- Potential tools/products/workflows: Workflow compiler that extracts frequent tool-call DAGs; GPU scheduling that prioritizes high-usage tools; fallbacks to full agent when needed.
- Assumptions/dependencies: Stability of learned policy across cohorts; willingness to revalidate distilled workflows.
- Clinical trial data curation and pre-labeling — Sectors: pharma/biotech, CROs
- What it looks like: Use stepwise segmentation and slice selection to pre‑label or shortlist cases with specific findings (e.g., pleural effusion) for human verification, accelerating cohort assembly.
- Potential tools/products/workflows: Semi-automated labeling pipelines; evidence bundles (slices/masks) attached to each case; audit trails of the reasoning process.
- Assumptions/dependencies: Oversight by imaging core labs; acceptance of pre‑labels as screening aids, not ground truth.
- Procurement and policy checklists for medical AI transparency — Sectors: hospital policy, regulators (pre‑market pilots)
- What it looks like: Incorporate faithfulness/traceability requirements (explicit reasoning trace, hint-admission behavior) into procurement or pilot-evaluation criteria.
- Potential tools/products/workflows: RFP templates that score transparent tool-use and auditability; pre‑deployment test suites using hint injection.
- Assumptions/dependencies: Organizational willingness to prioritize transparency; evolving consensus on metrics.
Long-Term Applications
These use cases require further research, broader tooling, regulatory work, or scaling beyond the paper’s present scope.
- Autonomous or semi‑autonomous reporting for defined indications — Sectors: healthcare
- What it looks like: Indication‑specific agents (e.g., ILD patterns, oncology staging, acute PE on CTPA) with guideline‑grounded checklists and validated toolsets that produce near‑final reports.
- Potential tools/products/workflows: Specialized detectors (nodule, lymph node, emboli), pathology‑specific VLMs, structured reporting outputs (e.g., Lung‑RADS, TNM).
- Assumptions/dependencies: Prospective clinical trials; regulatory clearance; dataset diversity across scanners and contrast protocols.
- Expansion to other modalities and body regions — Sectors: healthcare
- What it looks like: Agentic tool use for contrast CT, abdominal/pelvic CT, MRI (neuro, MSK), PET/CT; modality‑specific windowing, segmentation, and QA tools.
- Potential tools/products/workflows: New checklists per modality; upgraded segmentation suites; retrained VLMs and validators; cross-modality evidence linking.
- Assumptions/dependencies: High‑quality labeled datasets; domain-adapted tools; retraining the agent with RL.
- Continuous learning with clinician feedback — Sectors: healthcare, software
- What it looks like: Feedback loops (approve/reject/edit) feed into verifiable-reward RL to refine tool policies over time.
- Potential tools/products/workflows: In‑product feedback capture; RL pipelines that safely incorporate real‑world edits; drift detection.
- Assumptions/dependencies: Robust privacy safeguards; safety‑aware RL (off‑policy evaluation); governance for model updates.
- Regulatory‑grade transparency and safety cases — Sectors: policy/regulatory
- What it looks like: Standardized faithfulness and robustness benchmarks adopted by regulators; agent traces integrated into safety cases and post‑market surveillance.
- Potential tools/products/workflows: Reference test batteries; trace-based explainability dossiers; standardized audit log schemas.
- Assumptions/dependencies: Cross‑stakeholder agreement on metrics; repeatability across vendors.
- Hospital‑wide agentic imaging platform — Sectors: healthcare operations/enterprise IT
- What it looks like: A central agent orchestrating imaging tools across modalities, managing GPU pools, routing tasks, logging traces, and interfacing with EHR/RIS/PACS.
- Potential tools/products/workflows: Tool marketplace (MCP-based), unified UI for evidence review, enterprise observability and cost controls.
- Assumptions/dependencies: Significant IT investment; cybersecurity; vendor-neutral interoperability.
- Real‑time emergency triage and alerting — Sectors: emergency medicine, radiology
- What it looks like: Low‑latency, on‑scanner or edge deployments that quickly segment/flag critical findings with immediate notifications.
- Potential tools/products/workflows: Edge-distilled policies, streaming inference pipelines, integration with alerting systems.
- Assumptions/dependencies: Ultra‑low latency optimization; validated high sensitivity; device-level certifications.
- Coding, billing, and utilization review support — Sectors: finance/insurtech, hospital revenue cycle
- What it looks like: Structured extraction of abnormalities with evidence to support coding decisions and reduce denials.
- Potential tools/products/workflows: Report-to-structure converters; evidence attachments for payer review; audit trails.
- Assumptions/dependencies: High precision; payer acceptance; alignment to coding standards.
- Population health and public health surveillance — Sectors: public health, health analytics
- What it looks like: Aggregated, de‑identified structured findings from agent reports to track prevalence and trends (e.g., emphysema, consolidation).
- Potential tools/products/workflows: Data lakes with structured pathology fields; dashboards stratified by demographics/geography.
- Assumptions/dependencies: Bias assessment and mitigation; IRB/ethics approvals; standardized de‑identification.
- Tele‑radiology support in resource‑limited settings — Sectors: global health
- What it looks like: Cloud‑hosted agent assists remote clinicians with transparent trace, enabling targeted questions and visual confirmations.
- Potential tools/products/workflows: Low‑bandwidth UIs that transmit selected slices/masks; offline-capable variants with on‑device distilled policies.
- Assumptions/dependencies: Network reliability; local clinical oversight; funding for compute.
- Vendor‑neutral imaging tool marketplace — Sectors: software ecosystem, medical device industry
- What it looks like: MCP‑compatible marketplace where hospitals can plug in best‑of‑breed segmenters/VLMs; agents learn optimal tool use per site.
- Potential tools/products/workflows: Tool certification, usage analytics, revenue sharing; auto‑benchmarking in the agent’s reward loop.
- Assumptions/dependencies: Security standards; procurement models; IP/licensing clarity.
- Cross‑domain adoption of RL‑trained tool‑using agents — Sectors: software engineering, energy, manufacturing, finance
- What it looks like: Agents that plan and call specialized tools with traceable steps (e.g., software debugging with linters/test runners, plant inspections with sensor analytics, model risk reviews).
- Potential tools/products/workflows: Sector‑specific checklists and toolboxes; verifiable‑reward RL for safe tool orchestration; trace-based audits.
- Assumptions/dependencies: Domain‑specific datasets; tool APIs; oversight frameworks akin to medical QA.
- Federated and privacy‑preserving agent training — Sectors: healthcare IT, distributed AI
- What it looks like: Federated RL pipelines that adapt tool policies across hospitals without centralizing patient data.
- Potential tools/products/workflows: Federated reward aggregation; secure enclaves; on‑prem MCP tool hosting.
- Assumptions/dependencies: Federated infrastructure; privacy-preserving RL research maturity.
- Multilingual and locale‑aware reporting — Sectors: healthcare, global markets
- What it looks like: Agents that generate reports in local languages and align with local reporting templates/guidelines.
- Potential tools/products/workflows: Language‑adapted VLMs, locale‑specific checklists; on‑the‑fly translation with quality checks.
- Assumptions/dependencies: Linguistic evaluation datasets; clinician validation; localization effort.
- Distillation to low‑compute footprints — Sectors: embedded/edge AI, medical device manufacturers
- What it looks like: Compress agent policies and tool models for deployment on scanners or compact on‑prem servers.
- Potential tools/products/workflows: Policy distillation, quantization, tool pruning; hardware‑aware scheduling.
- Assumptions/dependencies: Maintaining accuracy after compression; vendor hardware support.
Key cross‑cutting assumptions and dependencies
- Domain scope: Current performance is shown for non‑contrast chest CT; other modalities/indications need new tools, data, and retraining.
- Tool accuracy and availability: Performance depends on underlying tools (CT‑Chat, TotalSegmentator, CT‑CLIP) and their versions.
- Compute/infrastructure: Multi‑GPU hosting and MCP orchestration are required for full capability; distillation can reduce costs but needs validation.
- Safety and regulation: Intended for clinician‑supervised use unless formally cleared; trace logs aid governance but do not substitute regulatory approval.
- Data governance: DICOM ingestion, PHI protection, de‑identification, and compliant logging are mandatory.
- Generalization: External performance is promising but must be re‑validated for local scanners, protocols, and patient populations.
- Human factors: Effective adoption requires UI/UX for trace review, training for staff, and agreed workflows for addressing flagged findings.
Glossary
- 3D VLM: A three-dimensional vision-LLM that processes volumetric imaging data alongside text. Example: "3D VLM counterpart, CT-Chat"
- Agentic systems: AI systems that plan and act through multi-step tool use rather than producing a single-pass output. Example: "Importantly, these agentic systems are training-free."
- Axial, coronal, sagittal planes: The three standard anatomical orientations for viewing volumetric medical images. Example: "axial, coronal, or sagittal."
- Bootstrapping: A resampling method used to estimate variability (e.g., confidence intervals) of a metric. Example: "95\% confidence intervals obtained via bootstrapping separately for each system."
- Bronchiectasis: Chronic abnormal dilation of the bronchi, often due to infection or inflammation. Example: "cardiomegaly, pleural effusion, emphysema, consolidation, and bronchiectasis"
- Cardiomegaly: Enlargement of the heart visible on imaging. Example: "cardiomegaly, pleural effusion, emphysema, consolidation, and bronchiectasis"
- CheXBert: A radiology-specific metric/model for evaluating chest X-ray reports using BERT-based label extraction. Example: "CheXBert~\cite{smit2020combining}"
- Computed tomography (CT): Cross-sectional medical imaging modality that uses X-rays to produce 3D volumetric data. Example: "computed tomography (CT)"
- Consolidation: Increased lung density on CT due to alveolar filling (e.g., pus, fluid). Example: "cardiomegaly, pleural effusion, emphysema, consolidation, and bronchiectasis"
- CT pulmonary angiography (CTPA): Contrast-enhanced CT protocol focused on pulmonary arteries to detect emboli. Example: "CT pulmonary angiography"
- CT-Chat: A 3D vision-LLM tailored for chest CT understanding and report generation. Example: "CT-Chat~\cite{hamamci2024CT-Chat-clip}"
- CT-CLIP: A CT-adapted CLIP-style model used for disease classification from volumetric scans. Example: "CT-CLIP checkpoint VocabFine."
- CT-RATE: A large chest CT dataset with paired radiology reports and pathology labels. Example: "CT-RATE~\cite{hamamci2024CT-Chat-clip}"
- Emphysema: Destructive enlargement of air spaces in the lungs, reducing gas exchange surface area. Example: "cardiomegaly, pleural effusion, emphysema, consolidation, and bronchiectasis"
- Faithfulness: The extent to which a model’s output accurately reflects the reasoning or factors that produced it. Example: "RadAgent achieves 37.0\% in faithfulness"
- FP8: An 8-bit floating-point numeric format used to accelerate large-model inference/training. Example: "in FP8"
- GREEN: An LLM-as-judge metric for radiology report quality that scores overlap of extracted findings with severity weighting. Example: "GREEN~\cite{ostmeier2024green}"
- GRPO: A reinforcement learning algorithm used to optimize tool-using agents via policy gradients. Example: "GRPO algorithm~\cite{shao2024deepseekmath}"
- LoRA: Low-Rank Adaptation; a parameter-efficient finetuning technique for LLMs. Example: "LoRA~\cite{hu2022lora} finetuning"
- Lung parenchyma: Functional lung tissue involved in gas exchange (alveoli and supporting structures). Example: "lung parenchyma assessment"
- Macro-F1: The unweighted mean F1-score across classes, treating each class equally. Example: "macro-F1"
- Mediastinum: Central thoracic compartment containing the heart, great vessels, trachea, and esophagus. Example: "mediastinum "
- Micro-F1: The F1-score computed by aggregating contributions of all classes, weighting by class frequency. Example: "micro-F1"
- Model Context Protocol (MCP): A protocol for standardized communication between an agent and external tools. Example: "Model Context Protocol (MCP) servers~\cite{mcp_github}"
- NIfTI: A common file format for storing neuroimaging and medical imaging volumes. Example: "NIfTI format"
- Pericardial effusion: Accumulation of fluid in the pericardial sac around the heart. Example: "pericardial effusions"
- Permutation test: A nonparametric statistical test that assesses significance by label shuffling. Example: "two-sided permutation tests at the 5\% significance level."
- Pleural effusion: Fluid accumulation in the pleural space surrounding the lungs. Example: "pleural effusion"
- Pulmonary embolism: A blood clot obstructing pulmonary arteries, potentially life-threatening. Example: "pulmonary embolism"
- RadChestCT: A large-scale dataset of chest CT scans with reports and abnormality labels for benchmarking. Example: "RadChestCT~\cite{draelos2021machine}"
- RadGraph F1: A metric that evaluates correctness of structured radiology findings and relations. Example: "RadGraph F1~\cite{radgraph}"
- ReAct pattern: An agent framework that interleaves reasoning (thought) and acting (tool use) in multiple steps. Example: "ReAct pattern~\cite{yao2022react}"
- TotalSegmentator: A comprehensive medical image segmentation toolkit covering many anatomical structures. Example: "TotalSegmentator\cite{wasserthal2023totalsegmentator}"
- Vision-LLM (VLM): A model that jointly processes visual inputs and text to perform multimodal tasks. Example: "Vision-LLMs (VLM)"
- Visual question answering (VQA): Answering natural-language questions about images using multimodal models. Example: "visual question answering tools"
- Windowing (CT): Adjusting CT image intensity via window level and width to highlight tissues of interest. Example: "CT windowing"
Collections
Sign up for free to add this paper to one or more collections.
