Adaptive Clinical-Aware Latent Diffusion for Multimodal Brain Image Generation and Missing Modality Imputation

Abstract: Multimodal neuroimaging provides complementary insights for Alzheimer's disease diagnosis, yet clinical datasets frequently suffer from missing modalities. We propose ACADiff, a framework that synthesizes missing brain imaging modalities through adaptive clinical-aware diffusion. ACADiff learns mappings between incomplete multimodal observations and target modalities by progressively denoising latent representations while attending to available imaging data and clinical metadata. The framework employs adaptive fusion that dynamically reconfigures based on input availability, coupled with semantic clinical guidance via GPT-4o-encoded prompts. Three specialized generators enable bidirectional synthesis among sMRI, FDG-PET, and AV45-PET. Evaluated on ADNI subjects, ACADiff achieves superior generation quality and maintains robust diagnostic performance even under extreme 80\% missing scenarios, outperforming all existing baselines. To promote reproducibility, code is available at https://github.com/rongzhou7/ACADiff

• The paper introduces ACADiff, a clinical-aware latent diffusion framework that adaptively synthesizes missing sMRI, FDG-PET, or AV45-PET using available images, clinical metadata, and modality-specific generators.
• ACADiff achieved average PSNR of 27.9, SSIM of 0.911, NMI of 0.859, and MAE of 0.014, outperforming the evaluated generative baselines in multimodal brain image reconstruction.
• The method preserved diagnostic utility under simulated missingness, reaching 77.5% AD-versus-healthy-control accuracy at 80% missing data and recovering 97.2% of oracle accuracy at 20% missingness, though real-world missingness and biomarker validity remain untested.

Overview

ACADiff (Adaptive Clinical-Aware Diffusion) is a latent diffusion framework for synthesizing missing neuroimaging modalities in Alzheimer's disease (AD) analysis (2603.09931). The paper targets a well-known practical problem: multimodal AD studies rely on structural MRI (sMRI), FDG-PET, and AV45-PET, but real-world cohorts such as ADNI contain substantial modality incompleteness due to cost, protocol variability, and dropout. ACADiff generates any missing target modality conditioned on whatever imaging data and clinical metadata are available, supporting both 2→1 and 1→1 translation through three specialized generators covering all six bidirectional directions among the three modalities.

Method

The framework operates in a compact latent space. Modality-specific 3D VAEs compress $160{\times}180{\times}160$ brain volumes to $20{\times}22{\times}20$ latents using pretrained Autoencoder-KL encoders with frozen decoders. Denoising diffusion with $T{=}1000$ steps is performed in this latent space via a volumetric U-Net with GroupNorm and FiLM modulation, trained with AdamW at learning rate $10^{-4}$.

Three hierarchical conditioning mechanisms drive generation:

• Adaptive image conditioning: when two modalities are available, spatially pooled features are fused via multi-head cross-attention; with one available modality, a learnable 3D convolution projection is used instead. The availability vector $z_{\text{avail}} \in \{0,1\}^3$ selects the fusion path, allowing a single model to handle heterogeneous input configurations.
• Semantic clinical guidance: diagnosis labels plus continuous cognitive scores (MMSE, ADAS13, CDR-SOB) are composed into structured natural-language prompts ("Generate [TARGET] from [AVAILABLE] for AD patient with MMSE=22, ...") encoded by GPT-4o's text encoder and injected into decoder layers through cross-attention. An ablation variant, ACADiff-emb, replaces GPT-4o encoding with 512-dimensional learnable embeddings.
• Temporal modulation: FiLM-style $\gamma(t)$/$\beta(t)$ scaling adapts denoising strength across diffusion steps.

Training uses noise-prediction loss plus a consistency regularizer on the predicted clean latent, with modality dropout to expose the model to both 2→1 and 1→1 scenarios; clinical information is also randomly dropped so generation can proceed without it. Inference uses 10-fold Monte Carlo sampling.

Experimental setup

The study uses 1,028 ADNI subjects (198 AD, 495 MCI, 335 HC) with all three modalities. To prevent leakage, 600 subjects train the generators and 428 are reserved for classification (300 training, 128 held-out test). Missingness is simulated by removing 1–2 modalities from 20–80% of classifier-training subjects and imputing them with each generator. Baselines include Pix2Pix, DS-GAN, LDM, PASTA, and FICD, alongside non-generative Drop and Mean imputation. Downstream classification uses a 3D DenseNet-121 on completed volumes; generation quality is measured with MAE, PSNR, SSIM, and NMI within a brain mask.

Results

Generation quality: ACADiff achieves PSNR 27.9, SSIM 0.911, NMI 0.859, and MAE 0.014 averaged over the three generators, exceeding the strongest baseline (LDM). The ~1.8 PSNR gap between ACADiff and ACADiff-emb is presented as evidence that language-model semantic encoding of clinical data outperforms learned embeddings — a notable claim given that both variants condition on identical metadata content.

Downstream classification (AD vs. HC):

Missing rate	Oracle	Best baseline (LDM)	ACADiff
20%	0.920 ACC / 0.943 AUC	0.885 / 0.902	0.894 / 0.910
40%	—	0.877 / 0.892	0.889 / 0.906
60%	—	0.842 / 0.871	0.878 / 0.883
80%	—	0.764 / 0.757	0.775 / 0.763

With 20% missingness, imputation recovers 97.2% of oracle accuracy. The advantage widens as missingness grows: at 80% missing data, ACADiff retains 77.5% accuracy while Drop and Mean imputation collapse to 58.2% and 55.1%, respectively. ACADiff consistently outperforms ACADiff-emb across all rates, reinforcing the semantic-guidance claim. These results imply that diffusion-based imputation preserves diagnostically relevant information even under extreme incompleteness, though the simulated-missingness design means performance on naturally occurring (non-random) missing patterns remains unverified.

Limitations and open questions

Several constraints qualify the findings. First, missingness is artificially induced by random removal; real-world missingness in ADNI correlates with disease stage and site, which could inflate or deflate reported gains. Second, evaluation is restricted to binary AD vs. HC classification; the MCI class (the largest cohort subgroup) is not evaluated downstream, leaving three-way classification performance unknown. Third, the framework requires three separately trained generators rather than a unified model, and the benefit of GPT-4o encoding versus simpler text encoders is not isolated — only learnable embeddings are compared. Fourth, computational cost of 1000-step diffusion with Monte Carlo inference on 4×A100 hardware is reported but not benchmarked against baselines' training/inference budgets. Finally, whether synthesized PET images are reliable for quantitative biomarker extraction (e.g., regional SUVR analysis), rather than classification features alone, is left unexamined.

Conclusion

ACADiff combines adaptive input-dependent fusion, GPT-4o-encoded clinical prompts, and modality-specialized latent diffusion generators for cross-modal brain image synthesis. On 1,028 ADNI subjects it outperforms five generative baselines in voxel-level fidelity and sustains 77.5% AD-vs-HC accuracy at 80% simulated missingness, approaching oracle performance at moderate missing rates. The evidence supports clinical-aware conditioning — particularly language-model-encoded cognitive scores — as a meaningful contributor to synthesis fidelity, while questions about realistic missingness patterns, multiclass diagnosis, and quantitative biomarker validity remain open.