RoentGen: Synthetic Chest X-ray Diffusion Model

• RoentGen is a domain-adapted, text-conditioned latent diffusion model that synthesizes diverse, anatomically realistic chest X-ray images from radiology prompts.
• Its architecture integrates a fixed VAE, a conditional denoising U-Net, and a fine-tuned text encoder to ensure clinically controllable image generation.
• Quantitative evaluations demonstrate improved AUROC, enhanced diversity, and bias mitigation, supporting data augmentation and counterfactual clinical studies.

RoentGen is a domain-adapted, text-conditioned latent diffusion model designed to generate anatomically realistic, diverse synthetic chest X-ray (CXR) images from radiology-specific prompts. Developed as an adaptation of the Stable Diffusion framework, its core purpose is to overcome the distributional divergence between natural and medical image-text pairs, enabling clinically controllable, high-fidelity medical imaging synthesis and facilitating a suite of downstream applications in clinical AI, dataset augmentation, fairness, and interpretability.

1. Architecture and Technical Design

RoentGen employs a three-component latent diffusion pipeline: a pretrained variational autoencoder (VAE) for encoding CXRs into a compact latent space and decoding back to pixel space, a conditional denoising U-Net for iterative refinement of noisy latent codes, and a text encoder (commonly CLIP or fine-tuned variants such as RadBERT or SapBERT) that embeds arbitrarily complex radiology prompts into a conditioning vector. Inputs comprise (i) a CXR image from real datasets such as MIMIC-CXR, and (ii) a free-form or radiology report-derived text prompt.

Training follows the canonical denoising diffusion objective. Let $y_{pixel}$ be a real CXR and $x_{text}$ its associated prompt. The VAE encodes $y_{pixel}$ as $z = \mathrm{VAE}(y_{pixel})$. Gaussian noise $N \sim \mathcal{N}(0,I)$ is added to $z$ at timestep $t$; the text encoder $Enc_{text}(x_{text})$ generates a conditioning embedding. The U-Net predicts the original noise:

$$\hat{N} = \mathrm{UNet}(Enc_{text}(x_{text}), z \oplus_t N, t)$$

with loss

$$\mathcal{L} = \frac{1}{h \cdot w} \sum_{i,j} (\hat{N}_{ij} - N_{ij})^2$$

where $h, w$ are latent dimensions. The VAE is typically held fixed, while the U-Net and text encoder are fine-tuned, optimally with in-domain radiological corpora to distill medical knowledge into the representation.

2. Domain Adaptation and Conditioning

A key advance of RoentGen over general multimodal models is rigorous adaptation to the distinct grayscale texture and pathology semantics of CXRs. Its prompt space is radiologically narrow and semantically rich, supporting detailed anatomical attributes (side, location, size) and nuanced findings (perihilar haziness, device presence). Unlike class-conditional generative adversarial networks (GANs), RoentGen leverages free-form prompt control, supporting generative diversity and compositional imaging not restricted to pre-defined categories.

Fine-tuning the text encoder jointly with the U-Net yields in-domain knowledge distillation. Evaluations show up to a 25% improvement in disease embedding quality for difficult cases such as pneumothorax when both components are adapted, directly enhancing prompt fidelity and synthesis accuracy.

3. Quantitative and Qualitative Evaluation

Evaluation of RoentGen incorporates metrics on fidelity (Fréchet Inception Distance, FID), diversity (Multi-Scale Structural Similarity, MS-SSIM), clinical downstream utility, and expert assessment:

• FID is computed via XRV DenseNet-121 (CXR-trained) and InceptionV3; lower scores indicate greater domain fidelity.
• MS-SSIM evaluates intra-prompt diversity; lower values correspond to higher structural variability across synthesized samples.
• Downstream classifiers trained with RoentGen augmentation obtain a 5% AUROC increase over real-only training, while purely synthetic sets yield a 3% improvement.
• Radiologist review finds synthetic images visually convincing in the majority of cases, with high text-image alignment but noted limitations in reproducing certain device artifacts.

Report generation and retrieval tasks further validate the model: metrics such as BLEU, ROUGE, BERTScore, fact_ENT, fact_ENTNLI, and RadGraph reveal that generated images and reports maintain semantic consistency with clinical expectations.

4. Bias, Hallucinations, and Validity

Recent investigations dissect validity and bias in RoentGen outputs (Bhardwaj et al., 2023). Diagnostic accuracy across subgroups (Asian, White, Hispanic, etc.) uncovered significant disparities; for example, the Hispanic female subgroup had a lower true positive rate for Atelectasis. Explicit inclusion of racial and gender attributes in prompts exacerbated these disparities, reducing selection rates by 16–26% relative to reference categories.

Latent hallucinations—spurious features not present or intended by prompts—were detected using disease classifiers outside the training domain (e.g., COVID detection on non-COVID prompts). Notably, 42% of synthetic images exhibited false COVID markers, illuminating the risk of model-induced artefacts. Diagnostic classifier confidence for synthetic images was lower and false negatives more frequent, particularly near class boundaries.

The studies advocate for new interpretability and validity metrics beyond traditional image similarity, such as out-of-training-class validation and per-head cross-attention tracing. The challenge of demographic bias underlines the need for fairness-aware prompt engineering and synthetic data balancing.

5. Clinical Impact and Data Augmentation

RoentGen's synthetic images have directly advanced medical imaging AI through controlled data augmentation and counterfactual editing:

• Using mixed real/synthetic training or pure synthetic datasets, downstream classifier performance improves by 3–5% AUROC, and disease text encoder representation by up to 25%.
• In RoentMod, RoentGen forms the core generator in an image-to-image modification pipeline enabling counterfactual CXR synthesis (Cooke et al., 10 Sep 2025). By perturbing only the prompted pathology while preserving native anatomy, RoentMod helps reveal and correct shortcut learning in diagnostic networks, as measured by 3–19% AUC improvements on internal and 1–11% AUC gains on external validation.
• Clinical reader studies report 93% realism of modified images and correct incorporation of prompted findings in 89–99% of cases, comparable to real follow-up scans.

RoentGen-v2 (Moroianu et al., 22 Aug 2025) extends these capabilities to demographic conditioning, producing over 565,000 synthetic CXRs with explicit control over sex, age, and race/ethnicity. Synthetic pretraining using this data led to a 6.5% accuracy increase and a 19.3% reduction in underdiagnosis fairness gaps compared to naïve mixing, demonstrably supporting equitable and generalizable clinical deep learning.

6. Limitations and Future Directions

Limitations include residual bias, hallucination risk, and the current restriction to single-pathology prompts in RoentMod. The modular design allows future extension to multi-pathology editing, region-specific modifications, and integration with causal model-based interventions. Rigorous external validation, expansion beyond single-institution data, and the development of finer-grained interpretability and fairness metrics are currently active research areas. Open access to model weights, code, and datasets (https://github.com/StanfordMIMI/RoentGen-v2) fosters transparent evaluation and collaborative advancement.

7. Significance

RoentGen establishes text-conditioned diffusion modeling as a central technology for clinical AI in radiography—enabling the synthesis of domain-faithful images, improvement of classifier robustness, augmentation of scant datasets, and the direct paper and remediation of shortcut learning, bias, and interpretability challenges. Its adaptation, evaluation, and downstream impact set a benchmark for generative modeling in medical imaging research, with broad extension potential across modalities and clinical domains.