SLaM-DiMM: Diffusion MRI Modality Synthesis

• The paper introduces SLaM-DiMM, a two-stage framework that leverages a shared latent space and latent diffusion to synthesize any missing brain MRI modality while achieving high SSIM scores.
• SLaM-DiMM overcomes limitations of prior GAN and transformer-based methods by combining efficient 2D slice-wise synthesis (MMG) with a 3D coherence enhancement network (CEn) for improved volumetric consistency.
• Experiments on BraTS-derived data demonstrate robust reconstruction quality and enhanced segmentation performance for glioma and metastasis, addressing clinical challenges of modality dropout.

Searching arXiv for the target paper and a few directly related methods mentioned in the source data. SLaM-DiMM, short for Shared Latent Modeling for Diffusion-Based Missing Modality Synthesis in MRI, is a missing-modality synthesis framework for multi-modal brain MRI that targets the common clinical setting in which one or more of the standard sequences are unavailable. In the formulation presented for brain tumor imaging, the four modalities are T1-weighted (T1w), contrast-enhanced T1-weighted (T1ce), T2-weighted (T2w), and FLAIR, and the objective is to synthesize any missing target modality from the remaining available modalities while preserving anatomical plausibility, structural faithfulness, and downstream utility for tasks such as segmentation and anomaly detection. The method combines a shared latent, modality-invariant representation with a latent diffusion bottleneck and a second-stage 3D coherence refinement network, yielding an explicitly two-stage pipeline designed to balance slice-wise fidelity with volumetric consistency (Sandbhor et al., 19 Sep 2025).

1. Clinical problem and methodological position

In multi-modal brain MRI, a subject is typically imaged with four sequences—T1w, T1ce, T2w, and FLAIR—and each modality emphasizes different tissue characteristics and tumor components, including edema, non-enhancing core, and enhancing tumor (Sandbhor et al., 19 Sep 2025). Modern deep learning systems for tumor segmentation, anomaly detection, and radiomics are explicitly designed to exploit this multi-contrast information, but their performance deteriorates when one or more sequences are missing. The reported causes include protocol variation, time constraints, contrast agent contraindications, and corrupted acquisitions.

Within this setting, the missing modality synthesis problem is defined as follows: given a subset of modalities $\{X_m: m\in \mathcal{M}_{\text{available}}\}$, synthesize the missing modality $X_{m_{\text{tgt}}}$ such that it is anatomically plausible and structurally faithful to the subject anatomy and pathology, and useful for downstream tasks such as segmentation with standard BraTS-style pipelines (Sandbhor et al., 19 Sep 2025). This framing is significant because it places missing-modality generation not merely in the domain of image translation, but in the stricter regime of clinically relevant reconstruction under cross-modal constraints.

The paper situates SLaM-DiMM against several prior methodological families. GAN-based approaches such as pGAN, cGAN, HF-GAN, and autoencoder-based collaborative attention GAN are described as strong at capturing realistic texture and local appearance, but limited by training instability, mode collapse, difficulty in modeling multi-modal conditional distributions, and a tendency toward modality-specific rather than unified any-to-any synthesis. Transformer-based or hybrid CNN/ViT methods such as ResViT and SwinUNETR are characterized as providing better long-range context but remaining deterministic regression-like mappings with limited ability to model uncertainty and variability in missing-modality synthesis. Diffusion models in pixel or wavelet space, including DDPM, WDM, and cWDM, are noted to provide high-fidelity generation but to be computationally expensive in full 3D, and frequently organized around modality-pair conditioning rather than shared latent modeling. Regression, autoencoder, and VAE-style methods are described as typically minimizing $\ell_1/\ell_2$ losses and as tending toward underfitting of high-frequency detail, blurry reconstructions, and limited stochastic modeling (Sandbhor et al., 19 Sep 2025).

Against this background, SLaM-DiMM is presented as addressing three specific gaps: a unified any-to-any cross-modal synthesis framework leveraging all four brain MRI modalities; a shared latent, modality-invariant representation refined by a latent diffusion bottleneck to capture high-level anatomy and pathology across modalities; and a two-stage design that combines slice-wise 2D intensity synthesis with a dedicated 3D coherence enhancement network to enforce volumetric consistency (Sandbhor et al., 19 Sep 2025).

2. Pipeline design and system decomposition

The framework consists of two main components: MMG (Missing Modality Generation) and CEn (Coherence Enhancement Network) (Sandbhor et al., 19 Sep 2025). MMG performs slice-wise synthesis using a shared encoder, a latent diffusion bottleneck, and modality-specific decoders, while CEn operates on the reconstructed 3D volume to reduce inter-slice artifacts and enforce 3D anatomical coherence.

The MMG component begins with a shared encoder $E$ that maps multi-modal 2D axial slices into a latent representation. A latent diffusion bottleneck $U$ refines this shared latent, and four modality-specific decoders $D_m$ reconstruct each modality from the refined latent. This first stage is explicitly intended to provide high-fidelity slice-wise synthesis of any target modality from the available modalities. The second stage, CEn, is a 3D UNETR-based model that acts on the whole reconstructed 3D volume and refines MMG outputs to reduce inter-slice artifacts and enforce 3D anatomical coherence (Sandbhor et al., 19 Sep 2025).

The resulting pipeline is therefore sequential: first synthesize missing modality slices with MMG, then enhance 3D coherence with CEn. This decomposition reflects a computational compromise. The paper directly states that diffusion models are computationally expensive in full 3D, while 2D synthesis alone has a known failure mode in volumetric consistency. SLaM-DiMM addresses this by separating intensity synthesis from volumetric refinement. A plausible implication is that the architecture is designed to preserve the efficiency of 2D modeling without accepting the coronal and sagittal artifacts that typically emerge when independently generated axial slices are stacked into a volume.

The overall design is also structured around a single shared latent space rather than modality-pair-specific generators. This is central to the paper’s claim of any-to-any synthesis: during inference, given any subset of modalities with the missing modality channel zeroed, the corresponding target modality can be synthesized via its decoder (Sandbhor et al., 19 Sep 2025).

3. Shared latent modeling and latent diffusion bottleneck

A 3D brain MRI volume in modality $m$ is defined as

$$X_m \in \mathbb{R}^{H \times W \times D}, \quad m \in \{\text{T1w}, \text{T1ce}, \text{Flair}, \text{T2w}\},$$

with height $H$, width $W$, and depth $X_{m_{\text{tgt}}}$0. For training, the method operates on 2D axial slices

$X_{m_{\text{tgt}}}$1

and constructs a 4-channel input slice

$X_{m_{\text{tgt}}}$2

where each channel corresponds to one modality (Sandbhor et al., 19 Sep 2025). To simulate missing-modality scenarios during training, one channel is randomly zero-masked in $X_{m_{\text{tgt}}}$3, forcing the network to learn a shared representation that can infer the masked modality from the remaining three.

The shared encoder maps the input slice to a latent representation

$X_{m_{\text{tgt}}}$4

Architecturally, $X_{m_{\text{tgt}}}$5 is modeled after the downsampling path of the OpenAI DDPM U-Net, with 3 encoder blocks and 5 residual sub-blocks in each block (Sandbhor et al., 19 Sep 2025). No modality-specific filters are used in the encoder; all four channels are processed jointly, producing what the paper calls a truly modality-agnostic shared latent space. Conceptually,

$X_{m_{\text{tgt}}}$6

where $X_{m_{\text{tgt}}}$7 denotes the 4-channel slice with one modality potentially zeroed out. The latent $X_{m_{\text{tgt}}}$8 is described as encoding shared anatomy and pathology information independent of modality labels.

The latent diffusion module $X_{m_{\text{tgt}}}$9 operates directly in this feature space rather than in pixel space. Given $\ell_1/\ell_2$0, it produces a refined latent

$\ell_1/\ell_2$1

where $\ell_1/\ell_2$2 is a DDPM-style U-Net working in the latent feature space (Sandbhor et al., 19 Sep 2025). The paper distinguishes this choice from conventional latent diffusion models that first compress pixel images via a VAE and then apply diffusion in that latent space; here, the encoder output itself is treated as the diffusion state. This suggests an attempt to combine stochastic refinement with tighter alignment to a representation already optimized for cross-modal anatomy and pathology.

For each modality $\ell_1/\ell_2$3, a decoder reconstructs image space from the refined latent:

$\ell_1/\ell_2$4

Each decoder follows the upsampling path of the OpenAI U-Net, but with a crucial constraint: there are no skip connections from encoder to decoder (Sandbhor et al., 19 Sep 2025). The stated purpose is to force reconstruction to rely solely on the refined latent $\ell_1/\ell_2$5, thereby avoiding leakage of modality-specific information that might bypass the shared representation. The four decoders are structurally identical but trained separately to specialize in each modality’s contrast properties.

The shared latent modeling is summarized as

$\ell_1/\ell_2$6

for $\ell_1/\ell_2$7 (Sandbhor et al., 19 Sep 2025).

4. Diffusion formulation, optimization objectives, and regularization

The forward diffusion process begins from $\ell_1/\ell_2$8 and adds Gaussian noise over timesteps $\ell_1/\ell_2$9:

$E$0

with variance schedule $E$1. The equivalent closed form is

$E$2

where

$E$3

After $E$4 steps, $E$5 (Sandbhor et al., 19 Sep 2025). A linear noise schedule is used in which $E$6 increases from $E$7 to $E$8, with $E$9 timesteps during training. During inference, DDPM sampling with a fixed intermediate timestep $U$0 is used, described as a form of shortened sampling or partial denoising.

The reverse process is defined as

$U$1

where a DDPM U-Net parameterized by $U$2 predicts the mean and covariance, or equivalently the noise $U$3, conditioned on $U$4 (Sandbhor et al., 19 Sep 2025). Although the paper does not explicitly write the DDPM training objective, it states that the method follows the standard noise-prediction formulation:

$U$5

where $U$6 and

$U$7

Beyond the diffusion objective, the MMG stage uses a spatially weighted reconstruction loss and an SSIM-based structural loss. The reconstruction term is

$U$8

where $U$9 is the BraTS segmentation mask (Sandbhor et al., 19 Sep 2025). Tumor voxels receive weight $D_m$0, background voxels weight $D_m$1, and the experiments use $D_m$2, thereby emphasizing reconstruction fidelity in tumor regions. The second term is a latent-space consistency regularizer with $D_m$3, intended to encourage diffusion to enhance semantic richness without drifting too far from the encoder representation.

The SSIM term is

$D_m$4

and the complete MMG loss is

$D_m$5

with $D_m$6 selected by hyperparameter search to maximize SSIM (Sandbhor et al., 19 Sep 2025). The loss is applied across all four modalities simultaneously even though one modality is masked in the input. The paper emphasizes the reconstruction and SSIM terms as the main driver of cross-modality fidelity.

5. Coherence Enhancement Network and volumetric consistency

The CEn stage is motivated by the fact that MMG operates slice-wise in 2D. When reconstructed slices are stacked into a 3D volume, the result can exhibit inter-slice intensity fluctuations, structural discontinuities across depth, and vertical blurry artifacts that are particularly visible in coronal and sagittal views (Sandbhor et al., 19 Sep 2025). These artifacts degrade 3D segmentation performance and are visually apparent outside the axial plane.

CEn is a 3D volumetric network based on UNETR. In the formulation provided, UNETR combines transformer-based self-attention in the encoder to model long-range spatial dependencies with a 3D U-Net-like decoder to reconstruct volumetric outputs (Sandbhor et al., 19 Sep 2025). The input to CEn is, for each modality $D_m$7, the MMG reconstructed volume

$D_m$8

obtained by stacking synthesized slices, and the output is a refined volume $D_m$9 with improved inter-slice consistency.

To manage memory on large 3D volumes, each volume is split into overlapping sub-volumes of size

$m$0

with subvolume factor $m$1, which for typical $m$2 yields 16-slice subvolumes (Sandbhor et al., 19 Sep 2025). A sliding window with stride $m$3 is used along depth, so adjacent subvolumes overlap. CEn is trained on these subvolumes and the refined outputs are reassembled after processing.

The CEn objective is

$m$4

where $m$5 is a 3D voxel-wise reconstruction loss (L2 / MSE) between refined and ground-truth volumes, $m$6 enforces 3D structural similarity, and $m$7 was selected empirically (Sandbhor et al., 19 Sep 2025). The learned mapping is written as

$m$8

This second stage is central to the framework’s claim of structural coherence across the depth of the volume. The paper describes it as a dedicated mechanism to enforce volumetric consistency, which is identified as a key failure mode of slice-wise synthesis (Sandbhor et al., 19 Sep 2025).

6. Data regime, training configuration, and empirical results

The reported experiments use the BraSyn-2025 / BraTS-Lighthouse-2025 dataset derived from multiple BraTS collections, specifically GLI, METS, and MENINGIOMA (Sandbhor et al., 19 Sep 2025). The training split contains 1,251 glioma patient volumes and 238 metastasis patient volumes; the validation split contains 219 glioma and 31 metastasis volumes. Each volume includes four MRI modalities—T1w, T2w, FLAIR, and T1ce—plus segmentation masks for WT, TC, and ET.

Preprocessing includes intensity normalization via percentile clipping, with values clipped to the $m$9 percentiles to mitigate outliers. The top and bottom 15 axial slices are removed to discard non-brain or low-content slices, and the remaining axial slices are saved individually for MMG training (Sandbhor et al., 19 Sep 2025). During MMG training, all four modalities are available but one modality channel is randomly masked to zero for each training sample. At validation and test time, one modality is randomly omitted per sample and the model must synthesize it from the remaining modalities.

The MMG component is implemented in PyTorch and trained on an NVIDIA RTX A6000 with 48 GB memory. The encoder, diffusion bottleneck, and decoders each use 3 blocks with 5 residual sub-blocks per block; the decoders have no skip connections. Optimization uses Adam with learning rate $$X_m \in \mathbb{R}^{H \times W \times D}, \quad m \in \{\text{T1w}, \text{T1ce}, \text{Flair}, \text{T2w}\},$$0, batch size 4 for 2D slices, and up to 1600 epochs, each with 500 randomly selected image slices (Sandbhor et al., 19 Sep 2025). The diffusion process uses $$X_m \in \mathbb{R}^{H \times W \times D}, \quad m \in \{\text{T1w}, \text{T1ce}, \text{Flair}, \text{T2w}\},$$1 timesteps, a linear $$X_m \in \mathbb{R}^{H \times W \times D}, \quad m \in \{\text{T1w}, \text{T1ce}, \text{Flair}, \text{T2w}\},$$2 schedule from $$X_m \in \mathbb{R}^{H \times W \times D}, \quad m \in \{\text{T1w}, \text{T1ce}, \text{Flair}, \text{T2w}\},$$3 to $$X_m \in \mathbb{R}^{H \times W \times D}, \quad m \in \{\text{T1w}, \text{T1ce}, \text{Flair}, \text{T2w}\},$$4, and inference with $$X_m \in \mathbb{R}^{H \times W \times D}, \quad m \in \{\text{T1w}, \text{T1ce}, \text{Flair}, \text{T2w}\},$$5. The loss weights are $$X_m \in \mathbb{R}^{H \times W \times D}, \quad m \in \{\text{T1w}, \text{T1ce}, \text{Flair}, \text{T2w}\},$$6, $$X_m \in \mathbb{R}^{H \times W \times D}, \quad m \in \{\text{T1w}, \text{T1ce}, \text{Flair}, \text{T2w}\},$$7, and $$X_m \in \mathbb{R}^{H \times W \times D}, \quad m \in \{\text{T1w}, \text{T1ce}, \text{Flair}, \text{T2w}\},$$8 is tuned over $$X_m \in \mathbb{R}^{H \times W \times D}, \quad m \in \{\text{T1w}, \text{T1ce}, \text{Flair}, \text{T2w}\},$$9, with best SSIM at $H$0. For CEn, the subvolume factor is $H$1, stride is 8 slices, training runs up to 250 epochs with batch size 2 for 3D volumes, each epoch samples 100 randomly selected 3D volumes, and $H$2 is tuned over $H$3 with best SSIM at $H$4.

Quantitative synthesis quality is reported using SSIM over entire 3D volumes for each missing-modality scenario and pathology type. For glioma, selected results are: missing T1w, MMG 94.96 and MMG+CEn 94.81; missing T2w, 93.82 to 93.86; missing FLAIR, 91.75 to 91.94; missing T1ce, 92.44 to 92.26 (Sandbhor et al., 19 Sep 2025). For metastasis, the corresponding values are: missing T1w, 91.61 to 91.97; missing T2w, 89.61 to 90.39; missing FLAIR, 89.41 to 90.29; missing T1ce, 89.67 to 89.98. The paper states that MMG alone achieves high SSIM above 90% across modalities, particularly in glioma, and that CEn yields small but consistent improvements for metastasis and for some modalities, while slight SSIM reductions in some glioma cases indicate a trade-off between local per-slice fidelity and global 3D coherence.

Downstream segmentation is evaluated using the BraTS orchestrator model to generate pseudo-labels on synthesized data. For glioma, Tumor Core (TC) Dice and HD95 are 0.7601 and 3.6651 for MMG versus 0.7301 and 4.0226 for MMG+CEn; Whole Tumor (WT) is 0.8993 and 2.1149 versus 0.8876 and 2.4600; Enhancing Tumor (ET) is 0.7873 and 3.3372 versus 0.7702 and 3.6991 (Sandbhor et al., 19 Sep 2025). For metastasis, TC is 0.5571 and 11.3043 versus 0.5046 and 11.9173; WT is 0.8211 and 10.7675 versus 0.8155 and 5.1284; ET is 0.7912 and 13.6090 versus 0.7836 and 7.9839. The interpretation provided is that MMG yields strong segmentation performance, especially for glioma WT and ET, while for metastasis CEn substantially reduces HD95 for WT and ET, indicating better boundary localization and volumetric consistency even when Dice changes are modest.

Qualitatively, the figures are described as showing low residuals in original-minus-MMG maps, with fine textures, tumor boundaries, and tissue contrast preserved. Sagittal and coronal views reveal vertical blurry lines in MMG outputs that are reduced after CEn refinement. The net effect is summarized as MMG excelling at per-slice accuracy and high SSIM, whereas MMG+CEn sacrifices a small amount of local SSIM in exchange for improved global 3D anatomical coherence (Sandbhor et al., 19 Sep 2025).

7. Applications, limitations, and prospective extensions

The paper identifies several potential uses in clinical and research workflows. One is sequence completion, in which a missing sequence such as FLAIR can be synthesized from the available three modalities so that standard BraTS-style segmentation models remain usable (Sandbhor et al., 19 Sep 2025). Another is robust training, where downstream models are trained assuming four modalities and SLaM-DiMM is used at deployment to fill missing modalities and mitigate performance degradation in heterogeneous clinical datasets. The paper also notes radiomics and quantitative analysis as a possible application, while explicitly stating that validation is still needed for quantitative consistency. A further use case is tumor monitoring and planning, for which coherent 3D synthesis is especially relevant because radiologists and radiotherapy workflows depend on consistent volumetric anatomy.

The framework is reported only on BraSyn-2025 brain tumor data, but the shared latent modeling design and diffusion bottleneck are described as modality-agnostic and potentially adaptable to other organs such as prostate or liver with multi-sequence MRI, and even to multi-modal imaging beyond MRI, including CT–MRI synthesis, with appropriate pretraining of the encoder and decoders (Sandbhor et al., 19 Sep 2025). This suggests that the core architectural idea is intended as a general cross-modal synthesis template rather than a brain-tumor-specific generator, although such broader applicability is presented as a prospective extension rather than an experimentally established result.

The limitations discussed or implied are structurally important. Slice-wise MMG requires CEn to correct volumetric artifacts; without CEn, non-axial views can be inconsistent. Performance is better for glioma than for metastasis, reflecting dataset imbalance and more complex metastasis anatomy. The paper also notes that synthesized images, while high-quality, may still differ subtly from true physics-based modalities, so caution is needed for tasks sensitive to absolute signal intensity (Sandbhor et al., 19 Sep 2025). These caveats are directly relevant to translational use, because they distinguish visually plausible sequence completion from full physical equivalence to acquired scans.

The main contributions are summarized as a shared latent modeling framework jointly trained across T1w, T1ce, T2w, and FLAIR; a latent diffusion bottleneck operating in a learned shared feature space rather than in pixel space; task-aware losses with tumor-region weighting and SSIM terms; a two-stage MMG plus CEn pipeline combining efficient 2D synthesis with 3D volumetric refinement; and a comprehensive evaluation on BraSyn-2025 with high SSIM and segmentation analysis on synthesized data (Sandbhor et al., 19 Sep 2025). Proposed or implied future directions include extending the framework beyond brain MRI, integrating synthesis and segmentation into a joint model, introducing explicit modality tokens or cross-attention, optimizing inference through fewer diffusion steps or distillation, leveraging diffusion for uncertainty quantification, and evaluating robustness and domain adaptation across multi-center and multi-scanner data.

Taken together, SLaM-DiMM occupies a specific position within the missing-modality literature: it is a unified any-to-any synthesis system that treats modality inference as a shared latent reconstruction problem, uses diffusion to refine modality-invariant features, and then explicitly repairs the volumetric incoherence introduced by slice-wise generation. Its empirical profile indicates that the principal tension is not between realism and blur alone, but between local slice fidelity and global 3D consistency, a trade-off the architecture makes explicit through its two-stage design (Sandbhor et al., 19 Sep 2025).