MICCAI CMRxMotion Challenge

• The MICCAI CMRxMotion Challenge is a benchmarking initiative designed to assess deep learning algorithms for cardiac MRI quality and segmentation under systematic respiratory motion artifacts.
• It features two tasks: automated image quality assessment using metrics like Cohen’s Kappa and robust segmentation evaluated via Dice Similarity Coefficient and Hausdorff Distance.
• The challenge provides a controlled dataset with open-source resources and standardized benchmarks, fostering reproducible research and advancements in clinically robust CMR analysis.

The MICCAI CMRxMotion Challenge is an international benchmarking initiative launched to systematically evaluate and improve the robustness of deep learning algorithms for cardiac magnetic resonance (CMR) analysis—specifically under conditions of respiratory motion degradation. Hosted as part of MICCAI, the challenge addresses two primary tasks: (1) automated assessment of cardiac cine MRI quality in the presence of motion artifacts, and (2) robust segmentation of cardiac structures when respiratory motion is present. The challenge dataset comprises prospectively acquired short-axis cine CMR volumes from 40 healthy volunteers, each scanned under a controlled spectrum of breath-hold behaviors to induce graded artifact severity. The CMRxMotion Challenge has established a new paradigm for model development and evaluation in the face of clinically realistic motion corruption, offering fully cross-validated benchmarks, open-source resources, and comprehensive performance reporting for state-of-the-art algorithms (Wang et al., 2022, Wang et al., 25 Jul 2025).

1. Motivation and Objectives

Respiratory motion is a pervasive challenge in cardiac MRI, leading to significant artifacts—ghosting, blurring, and contour distortion—that degrade both image diagnostic quality and the accuracy of downstream quantitative measurements such as ventricular volumes and ejection fraction. Previous segmentation benchmarks have focused on scanner heterogeneity, protocol variation, or pathological cases, but none have explicitly “stressed” algorithms across the spectrum of motion artifacts encountered in routine clinical scenarios (e.g., pediatric, elderly, or heart failure populations with suboptimal breath-hold compliance).

The primary objectives of CMRxMotion are:

• To provide a publicly available, prospectively collected cine MRI dataset with systematically controlled respiratory artifacts.
• To benchmark and compare automated methods for (i) image quality assessment (IQA) regarding motion severity and (ii) robust segmentation under artifact degradation (Wang et al., 2022, Wang et al., 25 Jul 2025).

2. Dataset Acquisition and Labeling Protocol

Forty healthy adult volunteers (24 males, 16 females, ages 19–48) were scanned on a single Siemens 3T MAGNETOM Vida system using a balanced steady-state free precession (TrueFISP) short-axis cine protocol. For each subject, four distinct breath-hold instructions were used: full compliance (no motion), half-duration, free breathing, and intensive breathing. Each breathing condition was performed at both end-diastolic (ED) and end-systolic (ES) phases, yielding eight 3D volumes per subject (total 320 volumes) (Wang et al., 2022, Wang et al., 25 Jul 2025).

Image quality was rated in consensus by two experienced radiologists using a standardized 5-point Likert scale. Scores were collapsed into three severity levels:

• Label 1: mild motion (Likert 4-5)
• Label 2: intermediate motion (Likert 3)
• Label 3: severe motion (Likert 1-2)

For all volumes with diagnostic image quality (labels 1–2), the left ventricular (LV) blood pool, LV myocardium (MYO), and right ventricular (RV) blood pool were manually segmented in 3D Slicer and verified by two radiologists. The annotation protocol yielded four exclusive labels: 0 (background), 1 (LV), 2 (MYO), 3 (RV).

3. Benchmark Tasks and Evaluation Metrics

Automated Image Quality Assessment (Task 1)

Task 1 required classification of CMR volumes into mild, intermediate, or severe motion artifact classes. The primary evaluation metric was Cohen’s Kappa statistic:

$\kappa = \frac{p_0 - p_e}{1 - p_e}$

where $p_0$ is observed agreement and $p_e$ is expected agreement by chance. This statistic robustly quantifies the consistency of predicted versus ground-truth artifact grades, accounting for imbalanced class distributions (Wang et al., 2022, Wang et al., 25 Jul 2025).

Robust CMR Segmentation (Task 2)

Task 2 required voxel-wise segmentation of LV, MYO, and RV structures for all diagnostic-quality cases. Segmentation accuracy was evaluated using the Dice Similarity Coefficient (DSC) and 95th-percentile Hausdorff Distance (HD95):

$$\mathrm{DSC}(X, Y) = \frac{2|X \cap Y|}{|X| + |Y|} \qquad \mathrm{HD}_{95}(X, Y) = \max\left\{P_{95}(d(x, Y)), P_{95}(d(y, X))\right\}$$

where $X$ and $Y$ are the predicted and ground-truth sets, and $d(x, Y)$ is the minimum Euclidean distance. Ranking used a “rank-then-aggregate” strategy across all structures and both metrics, with Wilcoxon signed-rank tests to assess significance (Wang et al., 2022, Wang et al., 25 Jul 2025).

4. Data Splits, Challenge Workflow, and Submission Infrastructure

The 40-subject cohort was partitioned as follows:

• Training set: 20 subjects (160 volumes; images and full annotations released)
• Validation set: 5 subjects (40 volumes; images only; annotations withheld for online evaluation via leaderboard)
• Test set: 15 subjects (120 volumes; both images and annotations withheld; offline evaluation using submitted Docker containers)

Participants developed models on the training set, tuned on the unlabeled validation set, and submitted containerized solutions for final testing. This framework supported blind assessment and robust cross-team comparison (Wang et al., 2022, Wang et al., 25 Jul 2025).

5. Representative Methods and Challenge Results

Task 1 (IQA Classification)

• The winning entry (UON_IMA) used a patch-based, ensemble transfer learning approach combining ResNet-18, EfficientNet-B0, and Vision Transformer backbones, each trained on both intensity and gradient image patches. Hierarchical voting and ROI localization produced the top-ranked performance (accuracy 72.5%, $\kappa$ = 0.6309) (Li et al., 2022, Wang et al., 25 Jul 2025).
• Other high-performing methods leveraged self-supervised pretraining, radiomics features, and autoencoding/denoising strategies. Notably, the best ensemble classifiers consistently outperformed single-architecture baselines, particularly in handling the challenge’s class imbalance and small data regime.

Task 2 (Robust Cardiac Segmentation)

• The best segmentation performances were achieved by large-scale ensembles of nnU-Net architectures (2D and 3D), Swin UNETR transformer hybrids, and adversarially augmented pipelines. Top teams exploited transfer learning from external datasets (e.g., ACDC, M&Ms) and heavy data augmentation including simulated k-space motion artifacts (Wang et al., 25 Jul 2025).
• On the test set, DSC for LV/MYO/RV ranged from 87–94%, with HD95 between 2–4 mm for the best models. Performance degraded as artifact severity increased, with DSC dropping by 3–6% and HD95 roughly doubling in severe motion cases.

6. Impact of Motion Artifacts and Clinical Implications

Comprehensive analysis of segmentation accuracy stratified by IQA score revealed that mild motion artifacts permitted near-human-level segmentation overlap—LV DSC ≈ 94% for top AI versus 95% for human observers—while severe artifacts markedly degraded both overlap and boundary accuracy. Errors in key clinical biomarkers (ventricular volumes and ejection fraction) increased with artifact severity, highlighting the necessity of robust IQA filtering and artifact-aware segmentation in clinical pipelines (Wang et al., 25 Jul 2025).

A key insight is that segmentation-only solutions, without prior artifact classification or quality filtering, are suboptimal when applied blindly to non-diagnostic scans. Quality-aware workflows—where scans are filtered before segmentation—reduce downstream biomarker error and clinical misinterpretation risk (Wang et al., 2022, Li et al., 2022, Ranem et al., 2022).

7. Technical Innovations and Future Research Directions

The CMRxMotion challenge stimulated several technical advances:

• ROI-focused, patch-based learning to address data scarcity and input heterogeneity.
• Gradient magnitude and multi-view image features for increased artifact sensitivity.
• Transformer-based segmentation (e.g., Swin UNETR) and multi-task architectures capable of simultaneous artifact grading and structure delineation (Grzeszczyk et al., 2022).
• Adversarial and k-space motion simulation for data augmentation.

Anticipated future research directions include: domain adaptation to pathological cohorts, integration of motion-correction reconstruction with analysis (e.g., Compressed Sensing Plus Motion—CS+M framework (Aviles-Rivero et al., 2018)), and fully end-to-end networks jointly optimizing artifact detection and segmentation (Wang et al., 2022, Wang et al., 25 Jul 2025). A plausible implication is that combining multi-view motion estimation networks (e.g., MulViMotion (Meng et al., 2022)) with challenge-style evaluation could further improve robustness to complex 3D motion.

Table: Summary of Key Results (Task 1 and Task 2, Test Set) (Wang et al., 25 Jul 2025)

Task	Metric	Best Team(s)	Value(s)
1	Accuracy (%)	UON_IMA	72.5
1	Cohen’s Kappa	UON_IMA	0.631
2	LV Dice (%)	UA-SVCC/Med-Air	93.9 / 93.7
2	MYO Dice (%)	UA-SVCC/Med-Air	87.4 / 87.4
2	RV Dice (%)	UA-SVCC/Med-Air	92.2 / 92.4
2	LV HD95 (mm)	UA-SVCC	3.05
2	MYO HD95 (mm)	UA-SVCC	2.20
2	RV HD95 (mm)	UA-SVCC	3.53

8. Public Resources and Ongoing Impact

All data, labels, code, and leaderboard evaluation scripts remain publicly accessible (GitHub: https://github.com/CMRxMotion; Docker Hub: https://hub.docker.com/r/cmrxmotion), under a CC-BY non-commercial license (Wang et al., 25 Jul 2025). The CMRxMotion challenge has set a new standard for reproducible, artifact-aware cardiac MRI benchmarking and continues to support advances in clinically robust CMR analysis.