Foundation VAEs for 3D CT Reconstruction, Augmentation, and Generation

Abstract: Variational autoencoders (VAEs) compress high resolution CT volumes into compact latents while preserving clinically relevant structure. However, training CT-specific VAEs from scratch or heavily fine-tuning them incurs substantial computational and engineering cost, and often degrades under heterogeneous scanners, protocols, and diseases. This paper makes a progressive stride toward training-free medical VAEs by leveraging a critical observation: a single Foundation VAE, pretrained at scale on natural images and videos, can serve as a unified interface for CT Reconstruction, Augmentation, and Generation. With both encoder and decoder frozen, the Foundation VAE reconstructs CT volumes with preserved anatomy while suppressing acquisition noise; training segmentation models on these reconstructions improves surface accuracy by 3.9% NSD on average for pancreatic tumor and lung tumor. Within the same Foundation VAE latent space, a conditional latent diffusion model achieves 3.9% lower average FVD with 36.2% higher CT CLIP score, and improves multi-disease generation faithfulness across 18 types by 2.76% AUC. These results demonstrate Foundation VAEs as a practical interface for scalable CT representation reuse and faithful CT generation. Our code and demo are available at https://github.com/qic999/Foundation-VAE.

• The paper introduces a unified framework using foundation VAEs to perform 3D CT reconstruction, augmentation, and conditional generation without domain-specific fine-tuning.
• It leverages pretrained natural image/video models to preserve anatomical fidelity and improve segmentation metrics by up to 3.9% in tumor detection tasks.
• The integration of conditional latent diffusion with 3D axial attention ensures volumetric consistency and controllable synthesis of both healthy and pathological CT volumes.

Foundation VAEs for Unified CT Reconstruction, Augmentation, and Generation

Introduction

The paper "Foundation VAEs for 3D CT Reconstruction, Augmentation, and Generation" (2605.30893) proposes a paradigm shift in medical imaging workflows by leveraging large-scale VAEs pretrained on natural images and videos as a unified interface for 3D CT reconstruction, augmentation, and conditional generation. The key insight is that such foundation VAEs—in contrast to CT-specific models—can be applied to CT data without medical-domain fine-tuning, retaining anatomical fidelity while minimizing engineering and computational overhead. This approach directly addresses the limitations of traditional, domain-specific pipeline designs, especially under heterogeneous acquisition protocols and variable disease distributions.

Foundation VAE as a CT Reconstruction and Augmentation Interface

Traditional CT workflows demand bespoke VAE architectures and rigorous medical fine-tuning, resulting in compute-intensive pipelines and degraded transferability. The authors demonstrate that a single foundation VAE, trained solely on natural images and videos, can be repurposed for CT reconstruction via its frozen encoder ($E$) and decoder ($D$). The reconstruction operator $T(x) = D(E(x))$ delivers denoised volumes where anatomical boundaries and clinical structure are preserved. The discrepancy between original and reconstructed CT manifests as high-frequency attenuation or scanner-dependent artifacts, with negligible impact on spatial boundary alignment.

Segmentation models trained on reconstructed volumes achieve equal or improved accuracy compared to training on raw CT data, particularly on surface-sensitive metrics such as NSD, with observed gains up to 3.9% for pancreatic and lung tumor tasks. This finding is formalized via a theoretical bound on excess segmentation risk, showing that boundary-stable reconstructions induce only a small increase in task-relevant error if anatomical geometry is preserved. Empirical evaluations across seven video VAEs confirm high segmentation stability and low distortion.

Figure 1: Schematic overview: foundation VAE enables CT reconstruction, augmentation, and generation via zero-shot transfer and fixed latent space.

Figure 2: Visual comparison of CT reconstructions using off-the-shelf video VAEs (unmodified) against medical VAEs, showing high fidelity for anatomical boundaries and attenuation of acquisition noise.

Conditional Latent Diffusion for Controllable CT Generation

Within the fixed latent space of the foundation VAE, the authors introduce a conditional latent diffusion model for 3D CT generation. The model is grounded via spatial organ masks and textual radiology reports, enabling synthesis of both healthy and pathological volumes. Conditioning signals are integrated through VAE-encoded mask embeddings concatenated with the latent noise at each denoising block, and text embeddings injected via cross-attention.

To enforce volumetric consistency and minimize inter-slice artifacts, a lightweight 3D axial attention module aggregates information across consecutive slices, preserving physiological and pathological coherence throughout the stack.

Figure 3: The architecture for conditional CT generation: latent diffusion model operates in foundation VAE space, conditioned on organ/disease masks and radiology reports, with 3D attention for slice consistency.

Quantitatively, the model achieves:

• 3.9% lower average Fréchet Video Distance (FVD),
• 36.2% higher CT-CLIP score,
• 2.76% absolute AUC improvement in multi-disease generation faithfulness (across 18 categories), relative to prior state-of-the-art CT generators. FID and FVD metrics are balanced across axial, sagittal, and coronal views, indicating robust cross-view geometry and absence of slice-dependent artifacts.

Figure 4: Multi-view comparison of disease-conditioned CT generation: spatially consistent and anatomically precise volumes align with disease-mask guidance, outperforming baselines.

Figure 5: Demonstration of controllable CT generation across diverse disease types; abnormalities are localized to specified regions and preserve anatomical coherence.

Anatomical and Pathology Grounding

Organ-mask adherence, evaluated by Dice and IoU scores using pretrained segmentation models (VISTA3D and TotalSegmentator), is substantially higher than mask-conditioned baselines, especially for thin structures (e.g., vessels, ribs) where Dice increases from 13.5 to 63.38 and from 15.2 to 70.23, respectively. This fidelity enables precise spatial control over disease manifestations and is critical for downstream tasks requiring geometric consistency.

Failure analysis shows that text-only generation models are prone to hallucinated or inconsistent pathology, whereas explicit spatial conditioning minimizes off-target generation and improves multi-view localization. Mild artifacts persist in synthesis of rare or small findings, suggesting opportunities for future refinement in mask accuracy and alignment.

Downstream Clinical Utility and Ablations

Classifiers trained and fine-tuned on synthetic volumes generated by the proposed pipeline outperform those augmented with data from prior CT generators, with the mean AUC raising from 67.95 to 70.71 (+2.76). Label-dependent gains reflect the utility of precise spatial conditioning, particularly for pathologies requiring explicit localization.

A controlled ablation replacing the foundation VAE with MedVAE confirms that the observed performance gains derive from the pretrained VAE latent space: FID drops from 11.28 to 2.19 and CT-CLIP increases from 20.76 to 59.35 in the normal split, validating the efficacy of reuse without medical fine-tuning.

Theoretical and Practical Implications

The methodology generalizes well across heterogeneous clinical CT data, mitigating the need for retrained, domain-specific encoders and accelerating system development for medical imaging tasks. The zero-shot transfer property decouples representation learning from medical supervision, providing a scalable solution for annotation-sensitive tasks such as segmentation, augmentation, and multi-condition synthesis.

The approach sets a foundation for unifying generative workflows in radiology, potentially extending to multimodal and longitudinal contexts. Future directions include uncertainty-aware mask conditioning, class-balanced objectives for rare disease synthesis, and integration of EMR features for richer conditioning.

Conclusion

The paper establishes that foundation VAEs pretrained on large natural datasets can serve as robust, training-free interfaces for 3D CT reconstruction, augmentation, and high-fidelity, controllable generation. This paradigm offers strong numerical improvements in volumetric coherence, segmentation robustness, and downstream classification, with practical advantages in scalability and data efficiency. The unified framework supersedes domain-specific encoder architectures, substantially lowering compute and engineering requirements while improving generalization across clinical distributions.