AtomiMed: A Clinical Fact-Checking Framework

• AtomiMed is a modality-agnostic framework that structures radiology reports into a hierarchical set of atomic clinical facts for accurate clinical evaluation.
• It employs a two-stage process—hierarchical extraction and agentic cross-verification—to separate diagnostic detection from descriptive accuracy, mirroring radiologist peer review.
• The framework is supported by MRGEvalKit and OmniMRG-Bench, which standardize evaluation processes across multiple imaging modalities.

Searching arXiv for the AtomiMed paper and closely related medical report evaluation work. AtomiMed is a modality-agnostic evaluation framework for medical report generation (MRG) that judges whether a generated radiology report is clinically correct rather than merely textually similar to a reference report. It does so by decomposing reports into a standardized, multi-level hierarchy of Atomic Clinical Facts and then cross-verifying those facts between the ground-truth and generated reports in a manner explicitly framed as simulating multi-radiologist peer review. The framework is introduced together with MRGEvalKit, an open-source toolkit for automated hierarchical extraction, and OmniMRG-Bench, a multi-modal benchmark covering X-ray, CT, MRI, and Ultrasound (Wang et al., 30 Jun 2026).

1. Clinical motivation and problem formulation

AtomiMed was proposed in response to a limitation the paper identifies in traditional MRG evaluation: predominant reliance on surface-level n-gram overlap. Metrics such as BLEU, ROUGE, METEOR, and CIDEr compare reports as flat text, yet radiology reports are described as structured, hierarchical, and fact-critical. Under this formulation, textual overlap can fail to capture clinical factual accuracy and can overlook catastrophic diagnostic errors, including missing findings, wrong laterality, wrong severity, wrong location, and wrong morphology (Wang et al., 30 Jun 2026).

The paper further situates AtomiMed against structure-based medical metrics such as CheXbert, RadGraph, SembScore, and RaTEScore. These methods are presented as improvements over lexical overlap metrics because they attempt to extract clinical entities or relations, but the paper characterizes them as narrow and not universal. CheXbert is stated to cover only 14 chest X-ray labels, and RadGraph extractors are described as mainly trained on chest radiography, making them weak for CT, MRI, and ultrasound. LLM-as-a-judge methods such as GREEN are described as better aligned with radiologist judgment than lexical metrics, but still limited because they produce holistic judgments rather than per-finding audit traces, are costly, are not naturally decomposed into diagnostic versus descriptive error types, and can be brittle outside their dominant modality or domain (Wang et al., 30 Jun 2026).

The core diagnosis advanced by the paper is a mismatch between holistic report comparison and the compositional structure of clinical narratives. A report is treated not as a monolithic statement but as a collection of disease-level presence or absence claims together with attribute-level descriptors such as location, size, morphology, severity, quantity, and temporal change. AtomiMed is therefore designed to evaluate MRG at the level of clinically meaningful atomic facts rather than at the level of surface-form similarity (Wang et al., 30 Jun 2026).

2. Hierarchical Atomic Clinical Facts

The central representational unit in AtomiMed is the Atomic Clinical Fact (ACF). The framework decomposes a report into a two-level hierarchy consisting of disease-level ACFs and attribute-level ACFs. This decomposition is intended to convert an unstructured report into a structured set of verifiable question-answer pairs (Wang et al., 30 Jun 2026).

At the disease level, a report $R$ is represented as

$$\mathcal{Q}^{\mathrm{dis}(R)} = \{(q_i, a_i)\}_{i=1}^{N}, \quad a_i \in \{yes, no\}.$$

Each $q_i$ is a normalized clinical entity question such as whether a particular finding is present, and each answer is binary. This level captures the diagnostic detection problem: whether the finding is there or not (Wang et al., 30 Jun 2026).

At the attribute level, for each detected finding $d_k$, the report is represented as

$$\mathcal{Q}^{\mathrm{attr}(R)} = \bigl\{(d_k,\, \{(q_{k,j}, a_{k,j})\}_{j=1}^{M_k}) \bigr\}_{k=1}^{K}.$$

Here, each finding receives a set of attribute question-answer pairs. The attributes listed in the paper are location, size, morphology, severity, quantity, and temporal change. This second level is essential because two reports may agree that a disease is present while disagreeing on clinically important descriptive details (Wang et al., 30 Jun 2026).

This hierarchical formulation is the defining feature of AtomiMed. It enables the framework to separate diagnostic detection from descriptive accuracy, rather than collapsing both into a single opaque similarity score. A plausible implication is that the method can expose different error modes that are confounded under holistic metrics.

3. Pipeline and agentic cross-verification

AtomiMed operates in two main stages: Hierarchical Atomic Decomposition and Agentic Cross-Verification. In the first stage, given a report $R$, AtomiMed prompts an instruction-tuned LLM $\mathcal{M}$ to extract the ACF hierarchy. The model is constrained to emit valid JSON, a fixed prompt template is used, and a robust parsing pipeline ensures structured output. The paper states that this extraction is applied to any medical report across modalities (Wang et al., 30 Jun 2026).

The second stage is the core mechanism. AtomiMed compares the ground-truth report $R_{\mathrm{gt}}$ and the generated report $R_{\mathrm{inf}}$ through a bidirectional verification loop. The paper calls this process “agentic” because the LLM acts like an evidence reader: it answers questions from one report using the other report as evidence. It is “cross-verification” because the process runs in both directions. GT $\rightarrow$ INF measures recall, while INF $$\mathcal{Q}^{\mathrm{dis}(R)} = \{(q_i, a_i)\}_{i=1}^{N}, \quad a_i \in \{yes, no\}.$$0 GT measures precision. This design is explicitly said to resemble radiology peer review, where a second radiologist independently re-reads a study and discrepancies are tracked (Wang et al., 30 Jun 2026).

At the disease level, precision, recall, and F1 are defined as

$$\mathcal{Q}^{\mathrm{dis}(R)} = \{(q_i, a_i)\}_{i=1}^{N}, \quad a_i \in \{yes, no\}.$$1

The special case in which both reports contain no disease statements is handled by setting the scores to 1, so that a truly normal study is not penalized (Wang et al., 30 Jun 2026).

At the attribute level, evaluation is conditioned on disease-level agreement. Attributes are matched only for findings correctly aligned in both directions. Disease names are extracted from surface forms, and matching uses fuzzy string similarity with threshold

$$\mathcal{Q}^{\mathrm{dis}(R)} = \{(q_i, a_i)\}_{i=1}^{N}, \quad a_i \in \{yes, no\}.$$2

Precision, recall, and F1 are then computed analogously over aligned findings. The paper emphasizes that if a finding is not correctly detected, comparing its attributes is clinically meaningless; this conditional design therefore enforces a strict distinction between diagnostic detection and descriptive accuracy (Wang et al., 30 Jun 2026).

The final aggregation uses equal-weight averaging: $$\mathcal{Q}^{\mathrm{dis}(R)} = \{(q_i, a_i)\}_{i=1}^{N}, \quad a_i \in \{yes, no\}.$$3 The overall score thus reflects both whether the report identified the correct findings and whether it described those findings correctly (Wang et al., 30 Jun 2026).

4. Interpretability, auditability, and error taxonomy

A major feature of AtomiMed is that its bidirectional hierarchical procedure yields not only a scalar score but also question-level audit traces. The paper argues that this makes the method interpretable in a way that lexical metrics and holistic LLM judges are not. Instead of only reporting that a generated report is better or worse, AtomiMed can attribute failure to concrete disagreements in disease presence, location, size, morphology, severity, quantity, or temporal change (Wang et al., 30 Jun 2026).

The paper presents the separation between disease-level and attribute-level assessment as conceptually central. Disease-level evaluation corresponds to diagnostic detection: it answers whether a condition is present, whether an abnormality was omitted, or whether a false positive was introduced. Attribute-level evaluation corresponds to descriptive accuracy: it answers where a finding is located, what it looks like, how severe it is, how large it is, and how many instances are present. This division is meant to expose the fact that a model can correctly identify a disease but describe it badly, or describe a disease well while missing the detection itself (Wang et al., 30 Jun 2026).

This structure also addresses a common misconception in MRG evaluation: that better wording necessarily implies better clinical correctness. In the AtomiMed formulation, reports with similar wording can differ critically in factual content, while reports with different wording can express the same clinically correct finding. The framework is therefore designed to be less dependent on surface form and more aligned with the way radiologists judge reports. This suggests that AtomiMed is best understood not as a general-purpose text metric, but as a clinically structured fact-checking system specialized for radiological narrative assessment.

5. MRGEvalKit and OmniMRG-Bench

AtomiMed is accompanied by two infrastructure components intended to standardize evaluation. MRGEvalKit is the open-source toolkit supporting automatic hierarchical extraction and scoring. According to the paper, it provides standardized prompts, parsing for LLM-generated JSON ACFs, hierarchical atomic scoring, and reproducible evaluation infrastructure. Its function is to operationalize the AtomiMed pipeline so that others can apply the same evaluation procedure consistently (Wang et al., 30 Jun 2026).

OmniMRG-Bench is the benchmark curated to support universal, modality-agnostic MRG evaluation. The paper states that it spans X-ray, CT, MRI, and Ultrasound, covers 9 anatomical systems and 6 attribute categories, and contains over 178K expert-verified hierarchical ACF pairs. These ACFs are extracted from ground-truth reports and verified by board-certified radiologists. The benchmark is described as suitable not only for MRG evaluation but also as a broader clinical QA benchmark (Wang et al., 30 Jun 2026).

The role of OmniMRG-Bench is partly methodological and partly corrective. The paper explicitly frames it as a response to chest-X-ray bias in prior evaluation datasets. A plausible implication is that benchmark design is inseparable from metric design in this setting: a modality-agnostic evaluator requires a modality-diverse fact ontology and verification corpus.

6. Experimental findings and comparative behavior

The paper evaluates AtomiMed using two complementary protocols. The first measures correlation with radiologist error counts on ReXVal, ReFiSco-v0, RadEvalX, and RaTE-Eval, reporting Kendall’s $$\mathcal{Q}^{\mathrm{dis}(R)} = \{(q_i, a_i)\}_{i=1}^{N}, \quad a_i \in \{yes, no\}.$$4 and Spearman’s $$\mathcal{Q}^{\mathrm{dis}(R)} = \{(q_i, a_i)\}_{i=1}^{N}, \quad a_i \in \{yes, no\}.$$5. The second measures pairwise preference and ranking consistency using 20 sampled cases each from IU-Xray, AMOS (CT), RadGenome (MRI), and KMVE (Ultrasound), comparing 10 models pairwise with a human preference gold standard and reporting MAE, ACC, and Kendall’s $$\mathcal{Q}^{\mathrm{dis}(R)} = \{(q_i, a_i)\}_{i=1}^{N}, \quad a_i \in \{yes, no\}.$$6 (Wang et al., 30 Jun 2026).

On correlation with radiologist judgment, the paper reports the following Spearman values: on ReXVal, AtomiMed achieves $$\mathcal{Q}^{\mathrm{dis}(R)} = \{(q_i, a_i)\}_{i=1}^{N}, \quad a_i \in \{yes, no\}.$$7 versus GREEN at $$\mathcal{Q}^{\mathrm{dis}(R)} = \{(q_i, a_i)\}_{i=1}^{N}, \quad a_i \in \{yes, no\}.$$8; on ReFiSco-v0, AtomiMed achieves $$\mathcal{Q}^{\mathrm{dis}(R)} = \{(q_i, a_i)\}_{i=1}^{N}, \quad a_i \in \{yes, no\}.$$9 versus GREEN at $q_i$0; on RadEvalX, AtomiMed achieves $q_i$1 versus GREEN at $q_i$2. On RaTE-Eval, AtomiMed is described as competitive, although GREEN has a slightly higher $q_i$3 in the table. The paper’s summary claim is that AtomiMed shows the strongest or near-strongest alignment with human clinical judgment (Wang et al., 30 Jun 2026).

In pairwise preference prediction, the paper reports especially strong performance on X-ray:

• MAE = 0.0214
• ACC = 95.71%
• Kendall’s $q_i$4

It further states that AtomiMed remains meaningful across CT, MRI, and Ultrasound, whereas other metrics often degrade heavily outside chest X-ray. GREEN is characterized as strong on chest X-rays but dropping sharply on CT and MRI, and AtomiMed’s relative robustness is attributed to being hierarchy-based, not tied to a fixed label set, modality-agnostic, and grounded in report structure rather than a chest-specific extractor (Wang et al., 30 Jun 2026).

The paper also reports granular analytical findings enabled by the hierarchical framework. At the attribute level, models are generally better at Morphology but worse at Severity and Size. At the disease level, performance varies heavily by anatomical system, with respiratory or chest-related systems strongest and digestive, reproductive, and urinary systems weaker. The authors interpret these patterns as exposing dataset or pretraining bias. These patterns are presented as difficult or impossible to see with holistic metrics (Wang et al., 30 Jun 2026).

7. Significance, scope, and limitations

AtomiMed’s significance lies in reframing medical report evaluation as structured clinical fact-checking. The framework is presented as reducing reliance on misleading lexical metrics, detecting clinically dangerous errors more reliably, working across multiple imaging modalities, separating missed diagnoses from poor descriptions, and offering interpretable evidence traces for auditing. In that sense, its primary contribution is not a new surface metric but a formalization of report comparison around atomic clinical consistency (Wang et al., 30 Jun 2026).

The framework’s universality claim is specifically “modality-agnostic,” not modality-free in the abstract sense. Its intended scope is radiology report evaluation across X-ray, CT, MRI, and Ultrasound. It is therefore distinct from chest-X-ray-centric systems whose ontology or extractor is restricted to a narrow label space (Wang et al., 30 Jun 2026).

At the same time, the paper acknowledges the context from which AtomiMed emerges. It is designed to address the mismatch between holistic report comparison and the compositional structure of clinical narratives. A plausible implication is that its effectiveness depends on the quality of the hierarchical extraction stage and on the stability of LLM-mediated cross-verification. The paper addresses standardization and reproducibility through MRGEvalKit and through JSON-constrained extraction, but the broader methodological debate over LLM-based evaluators remains relevant.

Within the paper’s framing, the principal conceptual advance is the decoupled assessment of diagnostic detection and descriptive accuracy. By explicitly treating disease presence and attribute specification as separate yet jointly aggregated objectives, AtomiMed aligns evaluation with the structure of radiological interpretation rather than the structure of natural-language overlap. That alignment underpins its reported correlation with radiologist judgment and its role as a universal clinical-aware medical report evaluation framework (Wang et al., 30 Jun 2026).