---
title: 'CheXmix: Unified Fusion for Chest X-ray Reports'
url: https://www.emergentmind.com/topics/chexmix
type: topic
---

# CheXmix: Unified Fusion for Chest X-ray Reports

Searching arXiv for the cited CheXmix-related papers to ground the article in the relevant preprints.
CheXmix is a unified early-fusion generative model for chest X-ray interpretation and report generation that tokenizes chest X-ray images into discrete VQ-GAN codes, interleaves these image tokens with radiology report tokens into a single unified sequence, and processes that sequence with a decoder-only transformer initialized from RadPhi-2 [2604.22989]. In contrast to the prevalent medical multimodal foundation model pattern based on a CLIP- or SigLIP-pretrained vision encoder connected to a large language model through a projection MLP and LLaVA-style finetuning, CheXmix eliminates the projection bottleneck by treating image tokens as native discrete symbols in the model vocabulary [2604.22989]. The broader methodological lineage includes earlier multimodal chest X-ray pretraining work that did not use the name “CheXmix” but evaluated a closely related paradigm in which chest X-ray images and radiology report text are jointly exploited for transfer to novel tasks and healthcare systems via multimodal self-supervised learning, domain-adaptive pretraining, and linear probing then finetuning [2305.08017].

## 1. Definition and conceptual position

CheXmix is presented as a unified early-fusion multimodal generative model trained on a large corpus of chest X-rays paired with radiology reports [2604.22989]. Its central architectural commitment is early fusion: rather than connecting a separate vision encoder to a language model through an adapter, it uses a single decoder that jointly attends to image and text tokens in one stream [2604.22989]. This design is motivated by the claim that the adapter in decoupled multimodal large language models must map continuous visual embeddings to the language-model token space and can distort or lose visual attributes, which is especially concerning in medical imaging where subtle, fine-grained cues are diagnostically important [2604.22989].

The 2026 formulation positions CheXmix against two distinct baselines. The first is the LLaVA-style medical multimodal large language model exemplified by CheXagent, whose discriminative component relies on a SigLIP encoder and whose generative component uses a projection-connected large language model with instruction tuning [2604.22989]. The second is the early-fusion autoregressive line represented by Chameleon; CheXmix adopts the general idea of unified token-space modeling but specializes it to chest radiographs and radiology reports and extends it with a two-stage multimodal generative pretraining strategy [2604.22989].

A broader interpretation of the term emerges from prior chest X-ray multimodal pretraining work. The 2023 study “How to Train Your CheXDragon” does not mention “CheXmix” by name, but it is explicitly interpreted as evaluating a “CheXmix-style approach” in which chest X-ray images and report text are mixed during pretraining to learn generalizable representations and then adapted to new institutions and tasks through multimodal domain-adaptive pretraining and LP-FT [2305.08017]. This suggests that “CheXmix” can denote both a specific 2026 early-fusion generative architecture and, more broadly, a family of multimodal chest X-ray pretraining strategies that exploit paired image-report corpora.

## 2. Architecture, tokenization, and unified sequence modeling

The backbone is a decoder-only transformer initialized from RadPhi-2, which is adapted from Phi-2 with 2.7B parameters [2604.22989]. CheXmix extends the tokenizer to include image tokens. In the analyses reported in the paper, the model uses 32 transformer layers; the embedding dimensionality used for classification probes is 2560-D, while the retrieval pipeline reports 2,056-D for both modalities [2604.22989].

Image tokenization is performed by Chameleon’s VQ-GAN tokenizer. Each 512×512 chest X-ray is encoded into 1,024 discrete tokens $z_1,\dots,z_{1024}$ drawn from a codebook of size 8,192, and the VQ-GAN is not retrained [2604.22989]. Text tokenization uses the RadPhi-2 tokenizer with vocabulary size $|V_{\text{text}}| = 50{,}368$; the reports include Findings and Impression sections [2604.22989].

The joint vocabulary includes modality-boundary tokens and has size $|V| = 58{,}592 = |V_{\text{text}}| + |V_{\text{img}}|$ plus special tokens [2604.22989]. The unified sequence is formatted either as
$$
S=(I_S, z_1,\dots,z_{1024}, I_E, T_S, y_1,\dots,y_m)
$$
or as
$$
S=(T_S, y_1,\dots,y_m, I_S, z_1,\dots,z_{1024}, I_E),
$$
with randomized order so that image precedes text in 50% of cases [2604.22989]. Context length is capped at 1,300 tokens; more than 99% of sequences fit within that bound, text may be cropped, and image tokens are kept intact [2604.22989].

The main-paper models use a causal mask for both modalities [2604.22989]. An ablation examines bidirectional attention over image tokens while keeping text causal, but causal modeling of both modalities yields better overall performance, especially in report generation, and therefore remains the primary configuration [2604.22989]. The paper argues that this setup allows image tokens to benefit from the inductive priors of language decoders, including autoregressive modeling, compositional semantics, and long-range dependency modeling [2604.22989].

## 3. Training strategy and pretraining data

CheXmix uses a two-stage multimodal generative pretraining strategy [2604.22989]. The pretraining corpus aggregates five public chest X-ray datasets with paired reports: MIMIC-CXR, CheXpert, PadChest, BIMCV-COVID19+, and OpenI [2604.22989]. The paper reports 550,395 image-text training pairs and 14,111 test pairs, for a total of 627,809,814 tokens, including 577,054,144 image tokens and 49,755,670 text tokens [2604.22989].

Stage 1 is standard autoregressive pretraining over the unified mixed-modality sequence, using the objective
$$
L_{AR} = -\sum_{t=1}^{T}\log p(s_t \mid s_{<t}; \theta),
$$
which the paper also writes as
$$
\mathcal{L}_{\text{NTP}} = - \sum_{i=1}^{N} \log p_{\theta}(s_i \mid s_1, \dots, s_{i-1}).
$$
[2604.22989] Stage 1 is trained for 703,671 steps with AdamW $(\beta_1=0.9, \beta_2=0.98, \epsilon=10^{-6})$, weight decay 0.1, learning rate $10^{-5}$ with cosine decay, global batch size 8, gradient accumulation 4 for effective batch size 32, bfloat16 precision, and 4 NVIDIA A100 (80GB) GPUs [2604.22989].

Stage 2 is masked image-language autoregressive pretraining [2604.22989]. The model randomly replaces 50% of image and text tokens with a special [MASK] token; the mask token ID used in generation experiments is 58,560 [2604.22989]. The loss is computed only on masked positions:
$$
\mathcal{L}_{\text{MIL}} = - \sum_{i \in \mathcal{M}} \log p_{\theta}(s_i \mid \mathbf{S}'_{<i}).
$$
[2604.22989] Stage 2 is trained for 513,993 steps with the same optimizer and schedule as Stage 1 [2604.22989]. The training is sequential rather than jointly weighted: Stage 1 optimizes $L_{AR}$ and Stage 2 optimizes $L_{MIL}$; there is no simultaneous combined loss in the main training [2604.22989].

A masking-ratio ablation evaluates 25%, 50%, 75%, and 90% masking for Stage 2, trained for 100K steps, and reports that CheXpert classification achieves the best AUROC and AUPRC at 50% masking, with AUROC 0.676 and AUPRC 0.341 [2604.22989]. This suggests that the model’s robustness gains are not merely a consequence of masking per se, but of a particular masking regime that balances reconstruction difficulty against retention of informative context.

Earlier multimodal chest X-ray work provides a different but related pretraining template. The 2023 study uses GLoRIA, a multimodal global-local image-text self-supervised learning method with DenseNet-121 for images and BioClinicalBERT for the report impression section, and combines global image-text and local patch-token contrastive alignment objectives [2305.08017]. That earlier framework is discriminative-contrastive rather than unified autoregressive, yet it similarly treats paired image-report corpora as the basis for transferable chest X-ray representation learning [2305.08017].

## 4. Tasks, inference modes, and evaluation protocol

CheXmix is explicitly designed to support both discriminative and generative tasks without separate encoders or instruction tuning [2604.22989]. For discriminative classification on CheXpert’s 14 findings, embeddings are extracted from model layers, mean-pooled across image tokens per image, and used to train multi-head masked linear probes evaluated with AUROC and AUPRC [2604.22989]. Classification robustness is assessed under varying image masking ratios [2604.22989].

For radiology report generation, the model conditions on image tokens, begins text decoding with $T_S$, and generates findings and impression content [2604.22989]. Maximum generated length is set equal to the original reference length for fair comparison [2604.22989]. The paper also introduces a test-time augmentation procedure without additional training: image tokens are partitioned into five disjoint subsets; under 20% masking one subset is masked per pass, under 80% masking only one subset is kept per pass, five reports are generated, and Gemini 2.5 Pro consolidates them into a single synthesized report [2604.22989]. The reported effect is an approximately 10–13% gain in GREEN and CheXbert [2604.22989].

For image inpainting, selected image-token indices are masked, the model autoregressively fills the masked positions by argmax over token probabilities, and the completed discrete token sequence is decoded back to pixels through VQ-GAN [2604.22989]. Evaluation uses PSNR, MS-SSIM, and FID [2604.22989]. For retrieval, the model supports both image-to-text and text-to-image retrieval using cosine similarity in embedding space, with Top-8 and Top-16 recall measured across candidate pools of size 32, 64, and 128 [2604.22989].

The 2023 precursor study addresses a different evaluation setting: transfer across institutions and tasks [2305.08017]. Pretraining occurs on CheXpert, and transfer is evaluated on Intermountain Health outpatient centers and Dunedin Hospital through downstream tasks including pneumonia variants, pneumothorax, acute rib fracture, and intercostal chest tube [2305.08017]. Metrics include macro-AUROC, micro-AUROC, macro-AUPRC, micro-AUPRC, macro-F1, and micro-F1; each label-fraction setup is run with five random seeds and 95% confidence intervals, with significance testing by paired samples $t$-test and Bonferroni correction [2305.08017]. This earlier evaluation emphasis situates the later CheXmix model within a broader research program concerned with label efficiency, robustness under distribution shift, and adaptation to novel healthcare systems.

## 5. Empirical performance and comparative results

On CheXpert classification, CheXmix (S1+S2) reports AUROC values of $0.712 \pm 0.001$, $0.705 \pm 0.001$, $0.702 \pm 0.001$, $0.689 \pm 0.002$, and $0.656 \pm 0.002$ at masking ratios 0%, 20%, 40%, 60%, and 80%, respectively [2604.22989]. The corresponding values for CheXagent (SigLIP) are $0.831 \pm 0.001$, $0.756 \pm 0.002$, $0.683 \pm 0.005$, $0.634 \pm 0.008$, and $0.569 \pm 0.003$ [2604.22989]. The paper emphasizes that AUROC at 0–20% masking favors SigLIP, whereas CheXmix exceeds CheXagent at high masking, specifically 0.702 versus 0.683 at 40%, 0.689 versus 0.634 at 60%, and 0.656 versus 0.569 at 80% [2604.22989]. Against generative baselines, CheXmix (S1+S2) is reported to outperform Chameleon, MAE, and M3AE across all evaluated masking ratios [2604.22989].

For AUPRC on CheXpert classification, the paper highlights that at 40%, 60%, and 80% masking, CheXmix (S1+S2) achieves 0.368, 0.358, and 0.337, respectively, exceeding many generative baselines under occlusion [2604.22989]. This supports the paper’s claim that masked image-language pretraining improves fine-grained visual representations for discriminative use cases under degraded visual context [2604.22989].

For image inpainting on 5,000 images, the PSNR values for CheXmix (S1+S2) are 24.13, 21.25, 19.25, and 16.96 at 20%, 40%, 60%, and 80% masking, respectively [2604.22989]. The corresponding RadPhi-2 text-only generative baseline values are 18.98, 13.47, 11.08, and 10.38 [2604.22989]. The paper states that CheXmix (S1+S2) improves PSNR over RadPhi-2 by 26% at 20% mask, 58% at 40%, 74% at 60%, and 63% at 80%, averaging approximately 51.1% [2604.22989].

For radiology report generation on 1,000 samples at 0% masking, CheXmix (S1+S2) reports GREEN $0.221 \pm 0.015$ and CheXbert-F1 $0.390 \pm 0.023$, compared with CheXagent’s GREEN $0.152 \pm 0.011$ and CheXbert-F1 $0.383 \pm 0.022$, and Chameleon’s GREEN $0.019 \pm 0.005$ and CheXbert-F1 $0.202 \pm 0.018$ [2604.22989]. Under 80% masking, CheXmix (S1+S2) records GREEN 0.176 versus 0.068 for CheXagent, and CheXbert 0.422 versus 0.185 [2604.22989]. The paper characterizes the degradation in GREEN from 0.221 at 0% masking to 0.176 at 80% masking as modest, approximately a 25% drop, whereas CheXmix (S1) drops by approximately 329% [2604.22989].

External validation further supports the model’s domain performance. On NIH ChestX-ray14 without masking, CheXmix (S1+S2) achieves AUROC $0.773 \pm 0.000$, compared with Chameleon’s $0.770 \pm 0.001$, M3AE’s $0.752 \pm 0.001$, and HealthGPT’s $0.716 \pm 0.000$ [2604.22989]. On ReXGradient report generation without masking, CheXmix (S1+S2) reports GREEN $0.217 \pm 0.016$ and CheXbert $0.413 \pm 0.027$, compared with CheXagent’s GREEN $0.135 \pm 0.011$ and CheXbert $0.310 \pm 0.024$ [2604.22989].

Earlier multimodal chest X-ray transfer experiments offer a complementary empirical perspective. In the 2023 study, multimodal SSL without DAPT consistently outperforms unimodal SSL and often matches supervised pretraining at larger label fractions [2305.08017]. With DAPT and LP-FT, the combined strategy yields the strongest transfer; for example, CheXpert-GLoRIA DAPT+LP-FT reaches macro-AUROC 0.930 at 10% labels and 0.946 at 100% labels, while CheXpert-Sup DAPT+LP-FT reaches 0.925 and 0.936, respectively [2305.08017]. These results indicate that multimodal pretraining on paired chest X-ray images and reports was already empirically competitive for cross-institution transfer before the unified early-fusion generative design of CheXmix proper [2305.08017].

## 6. Relation to prior multimodal chest X-ray research

The immediate antecedent to CheXmix is not a single model but a line of work on chest X-ray and radiology-report co-training. The 2023 “CheXDragon” study evaluates three principal pretraining modes on CheXpert: supervised pretraining over 14 observations, unimodal image-only SSL via MoCo-CXR, and multimodal SSL via GLoRIA [2305.08017]. GLoRIA uses DenseNet-121 as the image encoder, BioClinicalBERT as the text encoder, global image-report contrast, and local patch-token alignment [2305.08017]. Transfer is then improved by multimodal domain-adaptive pretraining on the target institution’s paired images and reports and by linear probing followed by end-to-end finetuning [2305.08017].

That earlier framework differs from CheXmix in several essential respects. First, it remains encoder-based and contrastive rather than generative and unified [2305.08017; 2604.22989]. Second, its adaptation strategy is explicitly organized around domain shift between institutions and label spaces, whereas the 2026 CheXmix paper emphasizes broad task flexibility within a single generative backbone and robustness to masking and occlusion [2305.08017; 2604.22989]. Third, the 2023 method processes report text primarily through the impression section, while CheXmix uses Findings and Impression sections in a unified token stream [2305.08017; 2604.22989].

Even so, the 2023 results supply an important conceptual backdrop. They show that multimodal pretraining gives substantial gains over unimodal SSL across new healthcare systems and tasks, that target-domain unlabeled image-report pairs can be exploited through multimodal DAPT, and that LP-FT improves label efficiency and robustness [2305.08017]. A plausible implication is that CheXmix should be understood not as an isolated architectural novelty, but as a more integrated realization of a research trajectory in which paired chest X-rays and reports are the core substrate for transferable medical multimodal learning.

## 7. Limitations, robustness, and clinical implications

The 2026 paper identifies several limitations [2604.22989]. Domain specificity is the first: training and evaluation focus on chest radiographs, and generalization to CT, MRI, or ultrasound is untested [2604.22989]. Computational cost is the second: long unified sequences of up to 1,300 tokens in a decoder-only transformer increase training and inference cost relative to compact encoders [2604.22989]. The third is masking sensitivity at low occlusion, since classification performance at 0–20% masking still trails specialized encoders such as SigLIP [2604.22989]. The fourth concerns possible failure modes in generation: hallucinations in report generation under extreme occlusion, clinically misleading errors in inpainting, and the possibility that VQ-GAN tokenization may miss ultra-fine radiographic patterns if codebook resolution is insufficient [2604.22989].

The same paper frames robustness primarily through masking-based stress tests [2604.22989]. Systematic evaluation from 20% to 80% masking in classification and report generation is used to demonstrate that Stage 2 masked image-language training improves performance under degraded visual input [2604.22989]. Test-time augmentation by sampling disjoint masked views is presented as a way to improve GREEN and CheXbert without retraining, and the paper notes that this hints at ways to probe epistemic uncertainty [2604.22989].

Clinical implications are described cautiously. A single model capable of both classification and generative reporting could simplify deployment workflows, but careful validation and bias audits are required before clinical integration [2604.22989]. The paper also notes that unified sequence modeling offers a coherent path to integrate textual rationales and image reconstructions, although explicit saliency or explanation mechanisms are not provided [2604.22989]. Earlier chest X-ray transfer work adds related deployment concerns: report noise and label variability, subgroup shifts and hidden out-of-distribution variants, the need for strict de-identification and compliance when using radiology reports, and the importance of calibration, clinician oversight, and periodic adaptation to drift [2305.08017].

Taken together, these two papers locate CheXmix at the intersection of multimodal medical foundation modeling, chest X-ray representation learning, and clinically oriented transfer learning. In its narrow sense, CheXmix denotes a unified early-fusion, two-stage multimodal generative model for chest X-ray images and radiology reports [2604.22989]. In a broader sense, the term also names a research direction in which chest X-ray image-report pairing is used not merely for retrieval or captioning, but as the basis for robust discriminative and generative modeling across tasks, institutions, and levels of visual degradation [2305.08017; 2604.22989].

Source: https://www.emergentmind.com/topics/chexmix