CheXmix: Unified Fusion for Chest X-ray Reports

• CheXmix is a unified early-fusion generative model that tokenizes chest X-ray images into discrete codes and interleaves them with report text into a single sequence.
• It employs a two-stage multimodal pretraining strategy—autoregressive followed by masked image-language training—to enhance robustness under various levels of occlusion.
• CheXmix demonstrates improved performance in both classification and report generation, outperforming traditional projection-based models with higher AUROC and PSNR metrics.

Searching arXiv for the cited CheXmix-related papers to ground the article in the relevant preprints. CheXmix is a unified early-fusion generative model for chest X-ray interpretation and report generation that tokenizes chest X-ray images into discrete VQ-GAN codes, interleaves these image tokens with radiology report tokens into a single unified sequence, and processes that sequence with a decoder-only transformer initialized from RadPhi-2 (Kumar et al., 24 Apr 2026). In contrast to the prevalent medical multimodal foundation model pattern based on a CLIP- or SigLIP-pretrained vision encoder connected to a LLM through a projection MLP and LLaVA-style finetuning, CheXmix eliminates the projection bottleneck by treating image tokens as native discrete symbols in the model vocabulary (Kumar et al., 24 Apr 2026). The broader methodological lineage includes earlier multimodal chest X-ray pretraining work that did not use the name “CheXmix” but evaluated a closely related paradigm in which chest X-ray images and radiology report text are jointly exploited for transfer to novel tasks and healthcare systems via multimodal self-supervised learning, domain-adaptive pretraining, and linear probing then finetuning (Uden et al., 2023).

1. Definition and conceptual position

CheXmix is presented as a unified early-fusion multimodal generative model trained on a large corpus of chest X-rays paired with radiology reports (Kumar et al., 24 Apr 2026). Its central architectural commitment is early fusion: rather than connecting a separate vision encoder to a LLM through an adapter, it uses a single decoder that jointly attends to image and text tokens in one stream (Kumar et al., 24 Apr 2026). This design is motivated by the claim that the adapter in decoupled multimodal LLMs must map continuous visual embeddings to the language-model token space and can distort or lose visual attributes, which is especially concerning in medical imaging where subtle, fine-grained cues are diagnostically important (Kumar et al., 24 Apr 2026).

The 2026 formulation positions CheXmix against two distinct baselines. The first is the LLaVA-style medical multimodal LLM exemplified by CheXagent, whose discriminative component relies on a SigLIP encoder and whose generative component uses a projection-connected LLM with instruction tuning (Kumar et al., 24 Apr 2026). The second is the early-fusion autoregressive line represented by Chameleon; CheXmix adopts the general idea of unified token-space modeling but specializes it to chest radiographs and radiology reports and extends it with a two-stage multimodal generative pretraining strategy (Kumar et al., 24 Apr 2026).

A broader interpretation of the term emerges from prior chest X-ray multimodal pretraining work. The 2023 study “How to Train Your CheXDragon” does not mention “CheXmix” by name, but it is explicitly interpreted as evaluating a “CheXmix-style approach” in which chest X-ray images and report text are mixed during pretraining to learn generalizable representations and then adapted to new institutions and tasks through multimodal domain-adaptive pretraining and LP-FT (Uden et al., 2023). This suggests that “CheXmix” can denote both a specific 2026 early-fusion generative architecture and, more broadly, a family of multimodal chest X-ray pretraining strategies that exploit paired image-report corpora.

2. Architecture, tokenization, and unified sequence modeling

The backbone is a decoder-only transformer initialized from RadPhi-2, which is adapted from Phi-2 with 2.7B parameters (Kumar et al., 24 Apr 2026). CheXmix extends the tokenizer to include image tokens. In the analyses reported in the paper, the model uses 32 transformer layers; the embedding dimensionality used for classification probes is 2560-D, while the retrieval pipeline reports 2,056-D for both modalities (Kumar et al., 24 Apr 2026).

Image tokenization is performed by Chameleon’s VQ-GAN tokenizer. Each 512×512 chest X-ray is encoded into 1,024 discrete tokens $z_1,\dots,z_{1024}$ drawn from a codebook of size 8,192, and the VQ-GAN is not retrained (Kumar et al., 24 Apr 2026). Text tokenization uses the RadPhi-2 tokenizer with vocabulary size $|V_{\text{text}}| = 50{,}368$; the reports include Findings and Impression sections (Kumar et al., 24 Apr 2026).

The joint vocabulary includes modality-boundary tokens and has size $$|V| = 58{,}592 = |V_{\text{text}}| + |V_{\text{img}}|$$ plus special tokens (Kumar et al., 24 Apr 2026). The unified sequence is formatted either as

$$S=(I_S, z_1,\dots,z_{1024}, I_E, T_S, y_1,\dots,y_m)$$

or as

$$S=(T_S, y_1,\dots,y_m, I_S, z_1,\dots,z_{1024}, I_E),$$

with randomized order so that image precedes text in 50% of cases (Kumar et al., 24 Apr 2026). Context length is capped at 1,300 tokens; more than 99% of sequences fit within that bound, text may be cropped, and image tokens are kept intact (Kumar et al., 24 Apr 2026).

The main-paper models use a causal mask for both modalities (Kumar et al., 24 Apr 2026). An ablation examines bidirectional attention over image tokens while keeping text causal, but causal modeling of both modalities yields better overall performance, especially in report generation, and therefore remains the primary configuration (Kumar et al., 24 Apr 2026). The paper argues that this setup allows image tokens to benefit from the inductive priors of language decoders, including autoregressive modeling, compositional semantics, and long-range dependency modeling (Kumar et al., 24 Apr 2026).

3. Training strategy and pretraining data

CheXmix uses a two-stage multimodal generative pretraining strategy (Kumar et al., 24 Apr 2026). The pretraining corpus aggregates five public chest X-ray datasets with paired reports: MIMIC-CXR, CheXpert, PadChest, BIMCV-COVID19+, and OpenI (Kumar et al., 24 Apr 2026). The paper reports 550,395 image-text training pairs and 14,111 test pairs, for a total of 627,809,814 tokens, including 577,054,144 image tokens and 49,755,670 text tokens (Kumar et al., 24 Apr 2026).

Stage 1 is standard autoregressive pretraining over the unified mixed-modality sequence, using the objective

$$L_{AR} = -\sum_{t=1}^{T}\log p(s_t \mid s_{<t}; \theta),$$

which the paper also writes as

$$\mathcal{L}_{\text{NTP}} = - \sum_{i=1}^{N} \log p_{\theta}(s_i \mid s_1, \dots, s_{i-1}).$$

(Kumar et al., 24 Apr 2026) Stage 1 is trained for 703,671 steps with AdamW $(\beta_1=0.9, \beta_2=0.98, \epsilon=10^{-6})$, weight decay 0.1, learning rate $10^{-5}$ with cosine decay, global batch size 8, gradient accumulation 4 for effective batch size 32, bfloat16 precision, and 4 NVIDIA A100 (80GB) GPUs (Kumar et al., 24 Apr 2026).

Stage 2 is masked image-language autoregressive pretraining (Kumar et al., 24 Apr 2026). The model randomly replaces 50% of image and text tokens with a special [MASK] token; the mask token ID used in generation experiments is 58,560 (Kumar et al., 24 Apr 2026). The loss is computed only on masked positions:

$$\mathcal{L}_{\text{MIL}} = - \sum_{i \in \mathcal{M}} \log p_{\theta}(s_i \mid \mathbf{S}'_{<i}).$$

(Kumar et al., 24 Apr 2026) Stage 2 is trained for 513,993 steps with the same optimizer and schedule as Stage 1 (Kumar et al., 24 Apr 2026). The training is sequential rather than jointly weighted: Stage 1 optimizes $|V_{\text{text}}| = 50{,}368$0 and Stage 2 optimizes $|V_{\text{text}}| = 50{,}368$1; there is no simultaneous combined loss in the main training (Kumar et al., 24 Apr 2026).

A masking-ratio ablation evaluates 25%, 50%, 75%, and 90% masking for Stage 2, trained for 100K steps, and reports that CheXpert classification achieves the best AUROC and AUPRC at 50% masking, with AUROC 0.676 and AUPRC 0.341 (Kumar et al., 24 Apr 2026). This suggests that the model’s robustness gains are not merely a consequence of masking per se, but of a particular masking regime that balances reconstruction difficulty against retention of informative context.

Earlier multimodal chest X-ray work provides a different but related pretraining template. The 2023 study uses GLoRIA, a multimodal global-local image-text self-supervised learning method with DenseNet-121 for images and BioClinicalBERT for the report impression section, and combines global image-text and local patch-token contrastive alignment objectives (Uden et al., 2023). That earlier framework is discriminative-contrastive rather than unified autoregressive, yet it similarly treats paired image-report corpora as the basis for transferable chest X-ray representation learning (Uden et al., 2023).

4. Tasks, inference modes, and evaluation protocol

CheXmix is explicitly designed to support both discriminative and generative tasks without separate encoders or instruction tuning (Kumar et al., 24 Apr 2026). For discriminative classification on CheXpert’s 14 findings, embeddings are extracted from model layers, mean-pooled across image tokens per image, and used to train multi-head masked linear probes evaluated with AUROC and AUPRC (Kumar et al., 24 Apr 2026). Classification robustness is assessed under varying image masking ratios (Kumar et al., 24 Apr 2026).

For radiology report generation, the model conditions on image tokens, begins text decoding with $|V_{\text{text}}| = 50{,}368$2, and generates findings and impression content (Kumar et al., 24 Apr 2026). Maximum generated length is set equal to the original reference length for fair comparison (Kumar et al., 24 Apr 2026). The paper also introduces a test-time augmentation procedure without additional training: image tokens are partitioned into five disjoint subsets; under 20% masking one subset is masked per pass, under 80% masking only one subset is kept per pass, five reports are generated, and Gemini 2.5 Pro consolidates them into a single synthesized report (Kumar et al., 24 Apr 2026). The reported effect is an approximately 10–13% gain in GREEN and CheXbert (Kumar et al., 24 Apr 2026).

For image inpainting, selected image-token indices are masked, the model autoregressively fills the masked positions by argmax over token probabilities, and the completed discrete token sequence is decoded back to pixels through VQ-GAN (Kumar et al., 24 Apr 2026). Evaluation uses PSNR, MS-SSIM, and FID (Kumar et al., 24 Apr 2026). For retrieval, the model supports both image-to-text and text-to-image retrieval using cosine similarity in embedding space, with Top-8 and Top-16 recall measured across candidate pools of size 32, 64, and 128 (Kumar et al., 24 Apr 2026).

The 2023 precursor study addresses a different evaluation setting: transfer across institutions and tasks (Uden et al., 2023). Pretraining occurs on CheXpert, and transfer is evaluated on Intermountain Health outpatient centers and Dunedin Hospital through downstream tasks including pneumonia variants, pneumothorax, acute rib fracture, and intercostal chest tube (Uden et al., 2023). Metrics include macro-AUROC, micro-AUROC, macro-AUPRC, micro-AUPRC, macro-F1, and micro-F1; each label-fraction setup is run with five random seeds and 95% confidence intervals, with significance testing by paired samples $|V_{\text{text}}| = 50{,}368$3-test and Bonferroni correction (Uden et al., 2023). This earlier evaluation emphasis situates the later CheXmix model within a broader research program concerned with label efficiency, robustness under distribution shift, and adaptation to novel healthcare systems.

5. Empirical performance and comparative results

On CheXpert classification, CheXmix (S1+S2) reports AUROC values of $|V_{\text{text}}| = 50{,}368$4, $|V_{\text{text}}| = 50{,}368$5, $|V_{\text{text}}| = 50{,}368$6, $|V_{\text{text}}| = 50{,}368$7, and $|V_{\text{text}}| = 50{,}368$8 at masking ratios 0%, 20%, 40%, 60%, and 80%, respectively (Kumar et al., 24 Apr 2026). The corresponding values for CheXagent (SigLIP) are $|V_{\text{text}}| = 50{,}368$9, $$|V| = 58{,}592 = |V_{\text{text}}| + |V_{\text{img}}|$$0, $$|V| = 58{,}592 = |V_{\text{text}}| + |V_{\text{img}}|$$1, $$|V| = 58{,}592 = |V_{\text{text}}| + |V_{\text{img}}|$$2, and $$|V| = 58{,}592 = |V_{\text{text}}| + |V_{\text{img}}|$$3 (Kumar et al., 24 Apr 2026). The paper emphasizes that AUROC at 0–20% masking favors SigLIP, whereas CheXmix exceeds CheXagent at high masking, specifically 0.702 versus 0.683 at 40%, 0.689 versus 0.634 at 60%, and 0.656 versus 0.569 at 80% (Kumar et al., 24 Apr 2026). Against generative baselines, CheXmix (S1+S2) is reported to outperform Chameleon, MAE, and M3AE across all evaluated masking ratios (Kumar et al., 24 Apr 2026).

For AUPRC on CheXpert classification, the paper highlights that at 40%, 60%, and 80% masking, CheXmix (S1+S2) achieves 0.368, 0.358, and 0.337, respectively, exceeding many generative baselines under occlusion (Kumar et al., 24 Apr 2026). This supports the paper’s claim that masked image-language pretraining improves fine-grained visual representations for discriminative use cases under degraded visual context (Kumar et al., 24 Apr 2026).

For image inpainting on 5,000 images, the PSNR values for CheXmix (S1+S2) are 24.13, 21.25, 19.25, and 16.96 at 20%, 40%, 60%, and 80% masking, respectively (Kumar et al., 24 Apr 2026). The corresponding RadPhi-2 text-only generative baseline values are 18.98, 13.47, 11.08, and 10.38 (Kumar et al., 24 Apr 2026). The paper states that CheXmix (S1+S2) improves PSNR over RadPhi-2 by 26% at 20% mask, 58% at 40%, 74% at 60%, and 63% at 80%, averaging approximately 51.1% (Kumar et al., 24 Apr 2026).

For radiology report generation on 1,000 samples at 0% masking, CheXmix (S1+S2) reports GREEN $$|V| = 58{,}592 = |V_{\text{text}}| + |V_{\text{img}}|$$4 and CheXbert-F1 $$|V| = 58{,}592 = |V_{\text{text}}| + |V_{\text{img}}|$$5, compared with CheXagent’s GREEN $$|V| = 58{,}592 = |V_{\text{text}}| + |V_{\text{img}}|$$6 and CheXbert-F1 $|V| = 58{,}592 = |V_{\text{text}}| + |V_{\text{img}}|$7, and Chameleon’s GREEN $$|V| = 58{,}592 = |V_{\text{text}}| + |V_{\text{img}}|$$8 and CheXbert-F1 $$|V| = 58{,}592 = |V_{\text{text}}| + |V_{\text{img}}|$$9 (Kumar et al., 24 Apr 2026). Under 80% masking, CheXmix (S1+S2) records GREEN 0.176 versus 0.068 for CheXagent, and CheXbert 0.422 versus 0.185 (Kumar et al., 24 Apr 2026). The paper characterizes the degradation in GREEN from 0.221 at 0% masking to 0.176 at 80% masking as modest, approximately a 25% drop, whereas CheXmix (S1) drops by approximately 329% (Kumar et al., 24 Apr 2026).

External validation further supports the model’s domain performance. On NIH ChestX-ray14 without masking, CheXmix (S1+S2) achieves AUROC $$S=(I_S, z_1,\dots,z_{1024}, I_E, T_S, y_1,\dots,y_m)$$0, compared with Chameleon’s $$S=(I_S, z_1,\dots,z_{1024}, I_E, T_S, y_1,\dots,y_m)$$1, M3AE’s $$S=(I_S, z_1,\dots,z_{1024}, I_E, T_S, y_1,\dots,y_m)$$2, and HealthGPT’s $$S=(I_S, z_1,\dots,z_{1024}, I_E, T_S, y_1,\dots,y_m)$$3 (Kumar et al., 24 Apr 2026). On ReXGradient report generation without masking, CheXmix (S1+S2) reports GREEN $$S=(I_S, z_1,\dots,z_{1024}, I_E, T_S, y_1,\dots,y_m)$$4 and CheXbert $$S=(I_S, z_1,\dots,z_{1024}, I_E, T_S, y_1,\dots,y_m)$$5, compared with CheXagent’s GREEN $$S=(I_S, z_1,\dots,z_{1024}, I_E, T_S, y_1,\dots,y_m)$$6 and CheXbert $$S=(I_S, z_1,\dots,z_{1024}, I_E, T_S, y_1,\dots,y_m)$$7 (Kumar et al., 24 Apr 2026).

Earlier multimodal chest X-ray transfer experiments offer a complementary empirical perspective. In the 2023 study, multimodal SSL without DAPT consistently outperforms unimodal SSL and often matches supervised pretraining at larger label fractions (Uden et al., 2023). With DAPT and LP-FT, the combined strategy yields the strongest transfer; for example, CheXpert-GLoRIA DAPT+LP-FT reaches macro-AUROC 0.930 at 10% labels and 0.946 at 100% labels, while CheXpert-Sup DAPT+LP-FT reaches 0.925 and 0.936, respectively (Uden et al., 2023). These results indicate that multimodal pretraining on paired chest X-ray images and reports was already empirically competitive for cross-institution transfer before the unified early-fusion generative design of CheXmix proper (Uden et al., 2023).

6. Relation to prior multimodal chest X-ray research

The immediate antecedent to CheXmix is not a single model but a line of work on chest X-ray and radiology-report co-training. The 2023 “CheXDragon” study evaluates three principal pretraining modes on CheXpert: supervised pretraining over 14 observations, unimodal image-only SSL via MoCo-CXR, and multimodal SSL via GLoRIA (Uden et al., 2023). GLoRIA uses DenseNet-121 as the image encoder, BioClinicalBERT as the text encoder, global image-report contrast, and local patch-token alignment (Uden et al., 2023). Transfer is then improved by multimodal domain-adaptive pretraining on the target institution’s paired images and reports and by linear probing followed by end-to-end finetuning (Uden et al., 2023).

That earlier framework differs from CheXmix in several essential respects. First, it remains encoder-based and contrastive rather than generative and unified (Uden et al., 2023, Kumar et al., 24 Apr 2026). Second, its adaptation strategy is explicitly organized around domain shift between institutions and label spaces, whereas the 2026 CheXmix paper emphasizes broad task flexibility within a single generative backbone and robustness to masking and occlusion (Uden et al., 2023, Kumar et al., 24 Apr 2026). Third, the 2023 method processes report text primarily through the impression section, while CheXmix uses Findings and Impression sections in a unified token stream (Uden et al., 2023, Kumar et al., 24 Apr 2026).

Even so, the 2023 results supply an important conceptual backdrop. They show that multimodal pretraining gives substantial gains over unimodal SSL across new healthcare systems and tasks, that target-domain unlabeled image-report pairs can be exploited through multimodal DAPT, and that LP-FT improves label efficiency and robustness (Uden et al., 2023). A plausible implication is that CheXmix should be understood not as an isolated architectural novelty, but as a more integrated realization of a research trajectory in which paired chest X-rays and reports are the core substrate for transferable medical multimodal learning.

7. Limitations, robustness, and clinical implications

The 2026 paper identifies several limitations (Kumar et al., 24 Apr 2026). Domain specificity is the first: training and evaluation focus on chest radiographs, and generalization to CT, MRI, or ultrasound is untested (Kumar et al., 24 Apr 2026). Computational cost is the second: long unified sequences of up to 1,300 tokens in a decoder-only transformer increase training and inference cost relative to compact encoders (Kumar et al., 24 Apr 2026). The third is masking sensitivity at low occlusion, since classification performance at 0–20% masking still trails specialized encoders such as SigLIP (Kumar et al., 24 Apr 2026). The fourth concerns possible failure modes in generation: hallucinations in report generation under extreme occlusion, clinically misleading errors in inpainting, and the possibility that VQ-GAN tokenization may miss ultra-fine radiographic patterns if codebook resolution is insufficient (Kumar et al., 24 Apr 2026).

The same paper frames robustness primarily through masking-based stress tests (Kumar et al., 24 Apr 2026). Systematic evaluation from 20% to 80% masking in classification and report generation is used to demonstrate that Stage 2 masked image-language training improves performance under degraded visual input (Kumar et al., 24 Apr 2026). Test-time augmentation by sampling disjoint masked views is presented as a way to improve GREEN and CheXbert without retraining, and the paper notes that this hints at ways to probe epistemic uncertainty (Kumar et al., 24 Apr 2026).

Clinical implications are described cautiously. A single model capable of both classification and generative reporting could simplify deployment workflows, but careful validation and bias audits are required before clinical integration (Kumar et al., 24 Apr 2026). The paper also notes that unified sequence modeling offers a coherent path to integrate textual rationales and image reconstructions, although explicit saliency or explanation mechanisms are not provided (Kumar et al., 24 Apr 2026). Earlier chest X-ray transfer work adds related deployment concerns: report noise and label variability, subgroup shifts and hidden out-of-distribution variants, the need for strict de-identification and compliance when using radiology reports, and the importance of calibration, clinician oversight, and periodic adaptation to drift (Uden et al., 2023).

Taken together, these two papers locate CheXmix at the intersection of multimodal medical foundation modeling, chest X-ray representation learning, and clinically oriented transfer learning. In its narrow sense, CheXmix denotes a unified early-fusion, two-stage multimodal generative model for chest X-ray images and radiology reports (Kumar et al., 24 Apr 2026). In a broader sense, the term also names a research direction in which chest X-ray image-report pairing is used not merely for retrieval or captioning, but as the basis for robust discriminative and generative modeling across tasks, institutions, and levels of visual degradation (Uden et al., 2023, Kumar et al., 24 Apr 2026).