AtomiMed: Hierarchical Atomic Fact-Checking for Universal Clinical-Aware Medical Report Evaluation

Abstract: Traditional metrics for Medical Report Generation (MRG) predominantly rely on surface-level n-gram overlap, which fails to capture clinical factual accuracy and often overlooks catastrophic diagnostic errors. We address this fundamental limitation by proposing \textbf{AtomiMed}, a universal, modality-agnostic evaluation framework that decomposes complex medical narratives into a standardized, multi-level hierarchy of Atomic Clinical Facts, encompassing Disease-level entities and Attribute-level descriptors, including location, morphology, and severity. By implementing an Agentic Cross-Verification loop between ground-truth and predicted reports, AtomiMed simulates a multi-radiologist peer-review process to verify clinical consistency, thus enabling the decoupled assessment of diagnostic detection and descriptive accuracy. To facilitate standardized evaluation, we introduce \textbf{MRGEvalKit}, an open-source toolkit for automated hierarchical extraction, and curate \textbf{OmniMRG-Bench}, a comprehensive multi-modal benchmark covering X-ray, CT, MRI, and Ultrasound. Extensive experiments on multiple expert-annotated reader studies demonstrate that AtomiMed achieves significantly higher correlation with human radiologist judgment compared to traditional and model-based metrics. Our code are release at https://github.com/Venn2336/MRGEvalkit

• The paper introduces a novel framework that decomposes clinical narratives into atomic facts for detailed, hierarchical evaluation.
• It employs instruction-tuned LLMs and bidirectional agentic cross-verification to assess both diagnostic presence and descriptive accuracy.
• The method outperforms traditional metrics with superior radiologist alignment (Spearman’s ρ=0.806) and robust cross-modality performance.

AtomiMed: Hierarchical Atomic Fact-Checking for Universal Clinical-Aware Medical Report Evaluation

Introduction and Motivation

The evaluation of automated Medical Report Generation (MRG) remains a critical unsolved problem, directly impacting model development, deployment, and clinical integration. Most existing metrics—n-gram overlap (BLEU, ROUGE, METEOR), domain-specific classifiers (CheXbert, RadGraph, SembScore, RaTEScore), and LLM-based holistic judges (e.g., GREEN)—exhibit significant blind spots. They either lack clinical grounding, fail outside narrow domains, cannot trace errors to specific findings, or are computationally expensive. These weaknesses persist because current metrics do not reflect the compositional and hierarchical structure of medical narratives, nor do they provide interpretable, actionable error attributions required for clinical audit and regulatory scrutiny.

AtomiMed addresses this by decomposing reports into a standardized hierarchy of Atomic Clinical Facts (ACFs) and implementing a bidirectional Agentic Cross-Verification process. This architecture enables universal, interpretable, and fine-grained assessment of both diagnostic detection and descriptive accuracy, decoupling disease presence from attributes such as location, severity, or morphology.

Figure 1: Schematic of the AtomiMed pipeline, showing hierarchical atomic decomposition followed by agentic cross-verification for clinical consistency assessment.

Methodology

Hierarchical Atomic Decomposition

AtomiMed leverages instruction-tuned LLMs to transform unstructured clinical narratives into a two-level hierarchy of ACFs:

• Disease-level QA: Presence or absence of normalized clinical entities, extracted as binary question-answer pairs $(q_i, a_i)$.
• Attribute-level QA: For each detected disease, extraction of structured descriptors over facets such as location, morphology, size, quantity, severity, and temporal change.

ACFs are enforced through a strict prompt structure with emission in valid JSON, promoting modularity and robustness across modalities and datasets.

Agentic Cross-Verification

The evidential consistency between reference (ground-truth) and generated reports is quantified via a bidirectional evaluation:

• Recall: All disease and attribute questions from the GT report are posed as queries against the generated report.
• Precision: All disease and attribute questions from the generated report are posed as queries against the GT.

Precision, recall, and F1 are computed at both levels, with final aggregation via equal weighting. Critically, attribute-level assessment is conditioned on disease-level alignment, ensuring attributes are not scored unless the underlying clinical finding is detected in both reports.

This protocol operationalizes radiological peer review as a computational process—each report serves as both questioner and evidence provider.

OmniMRG-Bench and MRGEvalKit

AtomiMed is accompanied by the OmniMRG-Bench, which establishes a new standard for multi-modal, universal evaluation in MRG. Spanning four imaging modalities and nine anatomical systems, OmniMRG-Bench provides over 178,000 expert-verified ACF pairs, establishing breadth and depth far beyond previous benchmarks.

MRGEvalKit, the open-source toolkit, facilitates hierarchical extraction, scoring, and audit tracing for any report, enabling transparent and reproducible evaluation pipelines.

Figure 2: Overview of OmniMRG-Bench and MRGEvalKit, highlighting its coverage of anatomical regions, modality diversity, and attribute categories.

Experimental Validation

Radiologist Correlation

AtomiMed demonstrates consistently superior alignment with expert annotations across four established and diverse testbeds. On ReXVal, the framework achieves Spearman's $\rho=$ 0.806 and Kendall's $\tau=$ 0.642, outperforming not only n-gram and medical-specialized metrics but also the state-of-the-art GREEN judge by non-trivial margins. Notably, all traditional metrics stagnate with substantially lower correlations, confirming their inability to capture clinically-relevant errors such as finding omissions, false positives, or attribute inversions.

Pairwise Preference and Modality Robustness

In pairwise preference studies covering X-ray, CT, MRI, and Ultrasound, AtomiMed achieves up to 95.71% ranking accuracy and 0.0214 MAE on X-ray, with performance gracefully degrading but remaining stable (ACC 49.86% on Ultrasound) in out-of-domain settings. In contrast, GREEN achieves only 33.83% on Ultrasound, and no lexical or specialist score approaches this cross-domain robustness.

Figure 3: Scatter plots comparing metric scores to human radiologist rankings in MRI. AtomiMed achieves tight alignment, while GREEN exhibits substantial scatter and lower correlation as measured by Kendall's $\tau$.

Fine-Grained Failure Audit

A distinctive strength of AtomiMed is its capacity to localize model performance deficiencies. The granular audit reveals that popular models show a consistent performance gap between recognizing presence/absence of findings and correctly describing attributes like severity or size—a deficiency masked by single scalar metrics. Disease-level auditing also exposes heavy pretraining biases (e.g., chest X-ray, respiratory system dominance), with subpar performance for reproductive, urinary, and digestive anatomical regions.

Figure 4: Granular attribute-level and disease-level performance heatmaps reveal non-uniform, systematic model deficiencies invisible to holistic scorers.

Implications and Future Directions

AtomiMed’s hierarchical, interpretable assessment protocol closes the gap between operational clinical review processes and automated evaluation. It unlocks new directions:

• Clinical compliance and regulatory audit: The explicit audit trail for each finding-attribute enables compliance with regulatory standards and systematic error analysis.
• Model diagnostics and debiasing: Exposure of systematic weaknesses (e.g., anatomical under-coverage, attribute blindness) directly informs dataset curation, pretraining strategies, and targeted fine-tuning.
• Universal QA benchmarking: The structured ACF corpus enables broader use in universal clinical QA tasks, including longitudinal and cross-report reasoning.

Future work will focus on backbone model distillation for cost-efficiency, dynamic attribute-hierarchy expansions (e.g., temporal change for longitudinal exams), and extension to specialties beyond radiology.

Conclusion

AtomiMed introduces a robust, universal, and interpretable framework for clinical-aware MRG evaluation—addressing longstanding limitations of traditional and holistic metrics by leveraging hierarchical atomic fact decomposition and agentic cross-verification. Its performance, interpretability, and extensibility set a new standard for evaluating medical NLG systems and facilitating rigorous model development for real-world clinical deployment.