BAAI Cardiac Agent for CMR Interpretation

• The paper demonstrates an end-to-end multimodal agent that orchestrates segmentation, quantification, and diagnostic routing in CMR imaging with high internal AUCs exceeding 0.93.
• It employs a hierarchical, multi-tool architecture that decomposes complex CMR tasks into sequence-specific modules for cine and LGE analysis.
• Practically, the system reduces report generation time from 1800 to 90 seconds, highlighting improved clinical workflow efficiency despite current multimodal data limitations.

BAAI Cardiac Agent is a multimodal intelligent system for end-to-end interpretation of cardiac magnetic resonance imaging (CMR). It is designed to integrate automated segmentation of cardiac structures, ventricular functional quantification, tissue characterization, disease diagnosis, visual question answering, retrieval-augmented knowledge access, and structured report generation within a unified workflow. The system is presented as the first end-to-end agent framework specifically designed for CMR imaging analysis, and was evaluated on CMR datasets from two hospitals comprising 2413 patients spanning 7-types of major cardiovascular diseases; in that evaluation, the area under the receiver-operating-characteristic curve exceeded 0.93 internally and 0.81 externally, while Pearson correlation coefficients for core left ventricular function indices all exceeded 0.90 (Qu et al., 5 Apr 2026).

1. Clinical scope and target domain

The system is motivated by the observation that CMR is a cornerstone for diagnosing cardiovascular disease but remains underutilized because interpretation is complex, time-consuming, and dependent on specialized expertise. A standard study involves multiple sequences, multiple views, and multiple cardiac phases, and therefore requires joint reasoning over anatomy, motion, tissue signal, and quantitative indices. The paper contrasts a conventional workflow requiring approximately 1800 seconds with agent-assisted reporting in approximately 90 seconds (Qu et al., 5 Apr 2026).

The baseline CMR study considered by the system includes short-axis cine, two-chamber cine, four-chamber cine, short-axis late gadolinium enhancement, and rest myocardial perfusion imaging. Cine imaging is performed with retrospective ECG gating; short-axis cine covers the entire left ventricle from the mitral annulus to the apex and contains 25–30 phases per cardiac cycle; short-axis cine slice thickness is 8 mm; long-axis cine includes 2CH, 4CH, and 3CH views, with long-axis slice thickness of 5 mm. LGE is acquired 10–15 minutes after intravenous gadolinium-DTPA at a dose of 0.2 mmol/kg using phase-sensitive inversion recovery segmented gradient-echo (Qu et al., 5 Apr 2026).

Its intended functions are broad but CMR-centered. The system is designed for automatic cardiac structure segmentation from cine and LGE sequences, ventricular function and chamber measurement, tissue characterization, screening and diagnosis of major cardiovascular disease categories and cardiomyopathy subtypes, automatic generation of structured CMR reports, and interactive querying through visual question answering. The current framework does not yet incorporate non-imaging clinical data such as ECG or laboratory tests, which the paper identifies as a limitation rather than a capability (Qu et al., 5 Apr 2026).

2. Agent architecture and orchestration

The defining architectural feature is that a large multimodal model functions simultaneously as an action planner or router and as a results aggregator. Rather than using one monolithic predictor, the system decomposes interpretation into sequence-specific and clinically meaningful subtasks, invokes specialized expert tools, and synthesizes their outputs into a final clinical response (Qu et al., 5 Apr 2026).

The workflow has four stages. First, the user provides a contextual image and a natural-language instruction. Second, the large multimodal model analyzes the instruction and image, selects the appropriate tool from the skill repository, prepares parameters and API action, and emits a structured tool-use command. Third, the designated expert model executes and returns a tool result. Fourth, the multimodal model integrates the tool result with the original instruction and image and generates the final answer. The paper represents this interaction as a dialogue-style tool-use process rather than as a static inference graph (Qu et al., 5 Apr 2026).

The integrated components include SAX cine segmentation, 2CH cine segmentation, 4CH cine segmentation, SAX LGE segmentation, cardiac disease screening, non-ischemic cardiomyopathy subclassification, retrieval-augmented generation, medical report generation, and native VQA capability. In the Discussion, the system is described as integrating eight expert models—four segmentation experts plus CDS, NICMS, MRG, and RAG—with VQA requiring no extra tool because the backbone already supports it (Qu et al., 5 Apr 2026).

A concise view of the tool inventory is as follows.

Component	Primary role
SAXCS / 2CHCS / 4CHCS	Sequence-specific cine segmentation
SAXLGES	LGE segmentation and tissue characterization
CDS	NH vs IHD vs NICM screening
NICMS	HCM / DCM / RCM / ACM / Myocarditis classification
RAG	Guideline and literature grounding
MRG	Structured CMR report generation
VQA	Interactive question answering

This orchestration is hierarchical. CDS first separates normal heart, ischemic heart disease, and non-ischemic cardiomyopathy. If NICM is predicted, the workflow proceeds to NICMS, which adds short-axis LGE to cine-based evidence. Report generation then integrates measurements, disease predictions, and imaging findings. This suggests a clinically structured dependency graph rather than flat tool calling.

3. Segmentation, quantification, and tissue characterization

The segmentation subsystem uses a two-stage coarse-to-fine architecture. Stage 1 performs ROI localization on coarsely processed images. Stage 2 performs fine segmentation within the localized cardiac region using a dual-path pyramid feature extraction design with a global branch and a local branch. The global branch has a receptive field twice that of the local branch and a resolution half that of the local branch; global features are upsampled by sub-pixel sampling and fused with local crops before refined segmentation. The shared backbone is a ResUNet-style U-shaped network with an initial double-convolution block, three residual layers, stride-2 convolutions for downsampling, a decoder with three upsampling steps, and skip connections (Qu et al., 5 Apr 2026).

Patch sizes are sequence-specific: $32 \times 64 \times 64$ for 2CH and 4CH cine, $64 \times 64 \times 64$ for SAX cine, and $3 \times 64 \times 64$ for SAX LGE. Because LGE has large slice thickness and limited inter-slice continuity, the LGE model is adapted from 3D to 2D. Training uses Adam, an initial learning rate of $5\times10^{-4}$, weight decay of $5\times10^{-4}$, a piecewise schedule with linear warm-up, and augmentations including random flipping, scaling, translation, elastic deformation, and random field perturbation (Qu et al., 5 Apr 2026).

The labels are sequence dependent. SAX cine is labeled for LV cavity, LV myocardium, and RV cavity; 2CH cine for LV cavity and LV myocardium; 4CH cine for LV cavity, LV myocardium, RV cavity, RV myocardium, LA cavity, and RA cavity; SAX LGE for LV cavity, LV myocardium, and LGE region. Pixel-level annotations were created by two radiologists with at least 10 years of CMR experience and validated by a senior radiologist with over 20 years’ experience (Qu et al., 5 Apr 2026).

Quantification is derived from cine segmentation rather than from a separate learned regressor. The reported indices include LVEDV, LVESV, LVEF, SV, CO, LVM, LVEDD, LVEDWT under the AHA 17-segment model, LV apex thickness, LAT4CHD, and RAT4CHD. Tissue characterization is centered on SAX LGE, with localization of LGE regions, assessment of LGE involvement extent, and cumulative 17-segment analysis visualized with bullseye maps (Qu et al., 5 Apr 2026).

The paper evaluates segmentation with DSC, HD, and ASD. At the sequence level, the reported average DSC values for the proposed method are $90.21\pm0.02$ for SAX cine, $88.75\pm0.03$ for 2CH cine, $86.92\pm0.03$ for 4CH cine, and $75.07\pm0.07$ for SAX LGE. For SAX cine, the reported DSC values are $92.44\pm0.03$ for LV myocardium, $64 \times 64 \times 64$0 for LV cavity, and $64 \times 64 \times 64$1 for RV cavity (Qu et al., 5 Apr 2026).

4. Diagnostic tasks, datasets, and performance

The full cohort comprises 2413 patients from two hospitals. The internal dataset from Beijing Anzhen Hospital contains 2134 patients; the external dataset from the First Affiliated Hospital of Xinxiang Medical University contains 279 patients. Disease labels are organized hierarchically. The screening task, CDS, is a tri-class classification among NH, IHD, and NICM. The second-stage task, NICMS, is a five-way classification among HCM, DCM, RCM, ACM, and Myocarditis for studies routed to the NICM branch (Qu et al., 5 Apr 2026).

For segmentation, 150 samples were randomly selected for annotation and split 7:1:2 into train, validation, and test after quality control. For diagnosis, the internal dataset was split 7:1:2 with stratification according to disease category distribution, and the external dataset served as independent external validation. Reported test sizes are $64 \times 64 \times 64$2 for internal CDS, $64 \times 64 \times 64$3 for external CDS, $64 \times 64 \times 64$4 for internal NICMS, and $64 \times 64 \times 64$5 for external NICMS (Qu et al., 5 Apr 2026).

The headline diagnostic results are summarized below.

Task	Internal result	External result
CDS	weighted F1 0.858, accuracy 0.858	weighted F1 0.728, accuracy 0.724
CDS AUC range	0.938–0.980 across NH/IHD/NICM	0.817–0.933 across NH/IHD/NICM
NICMS	weighted F1 0.834, accuracy 0.832	weighted F1 0.754, accuracy 0.730
NICMS AUC range	class-weighted mean AUC 0.9650	0.862–0.907 across subtypes

For CDS on the internal test set, the reported AUCs are 0.980 for NH, 0.938 for IHD, and 0.960 for NICM; the corresponding F1 scores are 0.856, 0.860, and 0.857. On the external set, the reported AUCs are 0.933 for NH, 0.827 for IHD, and 0.817 for NICM (Qu et al., 5 Apr 2026).

For NICMS on the internal test set, the class-weighted mean AUC is 0.9650 and the weighted F1 is 0.834. Class-wise internal AUCs are 0.975 for HCM, 0.971 for DCM, 0.961 for RCM, 0.958 for ACM, and 0.938 for Myocarditis. On the external set, the reported AUCs are 0.907 for HCM, 0.867 for DCM, 0.877 for RCM, 0.902 for ACM, and 0.862 for Myocarditis, with lower external F1 for rare categories, especially RCM and ACM (Qu et al., 5 Apr 2026).

Quantitative functional analysis is reported to be highly consistent with clinical reports. The Pearson correlations are 0.968 for LVEDV, 0.979 for LVESV, 0.925 for LVEF, 0.906 for SV, 0.937 for LVM, and 0.874 for LVEDD. The paper also states that thickness prediction error remained within 1 mm in each segment across different cohorts (Qu et al., 5 Apr 2026).

5. Report generation, VQA, and clinician-facing behavior

The reporting subsystem uses the agent outputs to generate structured clinical reports that combine cardiac function measurements, disease conclusions, and imaging findings. To improve factual grounding, the RAG module retrieves knowledge from the cardiovascular section of ChatCAD+, clinical guidelines, and heart-related literature from PubMed; explicitly named sources include chronic coronary disease and prevention guidelines, as well as the 2023 ESC cardiomyopathy guidelines (Qu et al., 5 Apr 2026).

Instruction tuning uses 32k samples, including 8k augmented VQA instructions and 3k samples each for SAXCS, 2CHCS, 4CHCS, SAXLGES, CDS, NICMS, RAG, and MRG. The loss is computed only for reasoning text related to tool invocation, tool invocation actions, and the final natural-language answer, while redundant text irrelevant to tool use is excluded from loss (Qu et al., 5 Apr 2026).

Report generation is evaluated both automatically and by human readers. On 200 internal reports, BERTScore is reported at precision 0.903, recall 0.894, and F1 0.898. In a reader study with six radiologists across junior, mid, and senior experience levels, the proposed system achieved mean report scores of $64 \times 64 \times 64$6, $64 \times 64 \times 64$7, and $64 \times 64 \times 64$8, respectively, substantially higher than the compared multimodal baselines (Qu et al., 5 Apr 2026).

The VQA subsystem supports direct querying about structure, function, valves, pericardium, perfusion, and disease-relevant findings. Accuracy exceeds 0.965 for low-abnormality-rate categories such as RVS, RVWM, RVSF, and RVDF, while more difficult categories such as LVS, LVWM, LVSF, LVDF, MV, TV, pericardial abnormalities, and Rest MPI abnormalities fall in the range 0.610–0.860. Sequence recognition accuracy is reported as 0.980. Tool invocation success rates are 99.82% on an internal validation subset and 99.46% on the external validation set, which is central to the paper’s claim that the LMM can reliably orchestrate specialized expert models (Qu et al., 5 Apr 2026).

6. Interpretation, limitations, and relation to adjacent cardiac agents

BAAI Cardiac Agent is an imaging-centric agent rather than a general cardiology agent. Its present form is centered on CMR and does not yet incorporate ECG, laboratory tests, CCTA, echocardiography, or other structured clinical data. The authors explicitly identify this as a limitation and describe future work as extending toward a CMR-centered multimodal cardiovascular diagnostic framework (Qu et al., 5 Apr 2026).

Several constraints are also clear in the evaluation. Rare disease categories such as ACM and RCM have limited sample sizes, which restricts the strength of subtype-specific validation. Most patients come from East Asian populations, so the paper calls for validation in Europe, the Americas, and Africa. External validation exists, but it is limited to one additional hospital, and the paper does not report prospective deployment, PACS or RIS integration, regulatory clearance, uncertainty estimation, or formal ablation studies (Qu et al., 5 Apr 2026).

The system should therefore be distinguished from other cardiac agent paradigms. HeartAgent is a cardiology-specific autonomous system for explainable differential diagnosis from heterogeneous patient data and curated knowledge resources (Zhou et al., 11 Mar 2026), while ATRIA is an iterative multi-agent ECG reporting system built around traceable staged artifacts and clinician revision (Hong et al., 23 Jun 2026). BAAI Cardiac Agent instead targets end-to-end CMR interpretation, using a large multimodal model to orchestrate specialized imaging experts within a diagnostic and reporting workflow (Qu et al., 5 Apr 2026). This suggests that the term “cardiac agent” now spans several distinct technical lineages: imaging interpretation, ECG reporting, and broader differential-diagnostic reasoning.

Within that landscape, BAAI Cardiac Agent’s specific contribution is to show that an agent framework can coordinate sequence-specific segmentation, functional quantification, tissue characterization, diagnostic routing, knowledge retrieval, and report generation inside one CMR workflow. Its strongest evidence lies in high internal discrimination, credible external validation, high agreement with clinical functional measurements, and strong reader-rated report quality. Its principal unresolved questions concern broader multimodal integration, rare-disease robustness, cross-population generalization, and real-world deployment.