Recover Semantics First, Generate Better: Improved Latent Modeling for 3D MRI Reconstruction and Cross-Contrast Synthesis

Abstract: Multi-contrast magnetic resonance imaging (MRI) provides complementary information for clinical diagnosis. However, acquiring all MRI sequences is often time-consuming and costly. Recent generative models perform cross-contrast synthesis to address this issue by inferring absent contrasts from the available ones. Nevertheless, synthesizing 3D MRI presents significant challenges. Due to the massive volume sizes, operating directly in the pixel space is computationally prohibitive; therefore, a common approach is to first compress the 3D volumes into a latent space and subsequently train generative models in that space. We observe that existing compression architectures face several critical issues: they under-preserve long-range anatomical coherence, discard clinically meaningful semantics, and rely on optimization objectives that lead to over-smoothed reconstructions. Ultimately, these shortcomings compromise the performance of subsequent generative models. In this work, we propose a semantics-first latent modeling framework for 3D MRI reconstruction and cross-contrast synthesis. Specifically, we introduce a Latent Harmonization Encoder (LHE) to capture global anatomical dependencies, ensuring coherent volumetric representations. To mitigate semantic degradation during latent compression, we further design a Semantic Recovery Block (SRB) that injects high-level priors from a self-supervised semantic teacher, enhancing contrast-aware separability in the latent space. Additionally, we propose an Anatomy-aware Frequency Loss (AFL) to adaptively preserve diagnostically relevant high-frequency structures. Extensive experiments on two public multi-contrast MRI datasets demonstrate consistent improvements in reconstruction fidelity and cross-contrast synthesis quality. Our code is available at https://github.com/script-Yang/RSF.

• The paper presents a semantics-first approach that recovers high-frequency details and long-range anatomical coherence using a novel latent modeling framework.
• It integrates a Latent Harmonization Encoder, a Semantic Recovery Block, and an Anatomy-aware Frequency Loss to achieve effective semantic alignment and structure preservation.
• The proposed method outperforms traditional models on BraTS and IXI datasets, achieving higher PSNR/SSIM and lower LPIPS scores in MRI reconstruction and synthesis tasks.

Semantics-First Latent Modeling for 3D MRI: A Comprehensive Framework

Introduction

High-fidelity multi-contrast MRI synthesis remains a fundamental challenge due to the high-dimensional structure of volumetric data and the complexity of anatomical presentations across different contrasts. Traditional pixel-space generative models for 3D MRI are computationally infeasible due to the volume size, leading to a prevalent two-stage paradigm: volumetric compression into a latent space followed by generative modeling therein. However, existing latent compression architectures have substantial deficiencies, notably in the preservation of long-range anatomical dependencies, semantic disentanglement, and high-frequency detail fidelity. The present work proposes a semantics-first paradigm, introducing three innovative components for robust 3D MRI reconstruction and cross-modality synthesis: the Latent Harmonization Encoder (LHE), the Semantic Recovery Block (SRB), and the Anatomy-aware Frequency Loss (AFL).

Figure 1: Overview of the semantic-first latent modeling framework integrating the Latent Harmonization Encoder, Semantic Recovery Block, and Anatomy-aware Frequency Loss for structured representation and robust 3D MRI synthesis.

Semantic-Preserving Latent Compression

Latent Harmonization Encoder (LHE)

Conventional volumetric compressors rely predominantly on convolutional architectures with limited receptive fields, resulting in fragmented long-range anatomical coherence. LHE addresses this limitation by integrating local convolutional features with slice-wise ViT-derived global context features, aligned statistically prior to fusion for harmonized representation. This dual-pathway encoding ensures that the discretized latent codes maintain anatomical continuity critical for downstream generative modeling by effectively leveraging both local and non-local information flows.

Semantic Recovery Block (SRB)

Standard latent compressors prioritize local pixel fidelity at the expense of contrast-specific semantics, resulting in severe feature entanglement as visualized via t-SNE of raw FSQ latents (Figure 2(a)). The SRB counteracts this by aligning the quantized latent codes with semantic prototypes acquired by a frozen, DINO-pretrained ViT teacher. The student latent vectors, projected into the semantic space via a learned MLP, are directly optimized to match these teacher features, effectively restoring contrast-aware separability (Figure 2). This semantic alignment is critical for accurate cross-contrast mapping and robust downstream synthesis performance.

Figure 2: (a) t-SNE of raw FSQ latents showing strong entanglement; (b) contrast-wise clustering by DINO teacher features; (c) SRB-aligned latents with restored semantic separability.

Frequency-Aware Structural Preservation

Anatomy-aware Frequency Loss (AFL)

AFL supplements pixel-wise objectives, which notoriously favor low-frequency consistency and induce over-smoothing, by enforcing high-frequency structure preservation specifically within anatomically and semantically relevant regions. The mechanism constructs attention maps from volumetric gradients (capturing local edges) and semantic self-attention distributions from the pretrained teacher, jointly weighting high-frequency residuals in the loss. This yields reconstructions that better retain subtle boundaries, textures, and diagnostically critical features, as highlighted in AFL’s architecture (Figure 3).

Figure 3: Anatomy-aware Frequency Loss uses a joint anatomy-semantic attention to enforce high-frequency fidelity in significant regions.

Experimental Validation

The framework is evaluated on BraTS and IXI datasets for both volumetric reconstruction and cross-contrast synthesis (e.g., T1 $\rightarrow$ T1C; T2 $\rightarrow$ FLAIR). Quantitatively, the model surpasses VQVAE, VQGAN, and FSQ baselines, achieving peak PSNR/SSIM (up to 33.65 PSNR, 0.9377 SSIM, lowest LPIPS of 0.0450 on IXI). When adapted into CycleGAN and Latent Diffusion backbones, the semantics-first latents yield significant translation quality improvements, e.g., a +3.48 dB gain for Latent CycleGAN (T1→T1C). Reconstruction and synthesis visualizations consistently confirm improved anatomical continuity and preservation of fine details (Figure 4).

Figure 4: Visual comparison emphasizes the improved fidelity and structural preservation of the proposed semantics-first framework compared to conventional baselines.

Ablation and Analysis

Module ablation demonstrates LHE’s contribution to long-range coherence, SRB’s enhancement of latent separability, and AFL’s ability to recover fine-grained, high-frequency information. Optimal results are achieved only when all modules are combined, evidencing their complementary design. The semantic teacher ablation further indicates that DINO-pretrained ViTs provide stronger anatomy-aware supervision than ResNet-50 or MAE, resulting in maximal PSNR/SSIM and minimal perceptual discrepancy.

Implications and Future Directions

The explicit recovery and alignment of contrast-specific semantics and fine anatomical structure within the latent space set new benchmarks for both 3D MRI reconstruction and synthesis. This framework is hardware-efficient, compatible with orthogonal advances in generative models, and agnostic to subsequent generator design, thus directly porting to GANs and diffusion models for various clinical settings. The paradigm also presents a robust foundation for integrating VLM/LLM-driven supervision for further context-aware guidance and generalization beyond brain MRI to multi-organ, multi-modal clinical imaging.

Conclusion

The semantics-first latent modeling framework systematically addresses entanglement, loss of coherence, and over-smoothing prevalent in conventional volumetric compression for 3D MRI. Through synergistic application of LHE, SRB, and AFL, the model consistently enhances both quantitative and qualitative outcomes for reconstruction and cross-contrast synthesis. The modular and data-driven design presents significant potential for extension to broader medical imaging and multi-modal generative learning research.