Latent Harmonization Encoder for 3D MRI

• Latent Harmonization Encoder (LHE) is a hybrid encoder that fuses local convolutional features with slice-wise transformer outputs to yield anatomically coherent latent codes.
• It harmonizes channel statistics between the two branches before quantization, addressing long-range anatomical dependencies in 3D MRI reconstruction.
• Integrated within a semantics-first latent modeling system, LHE enhances reconstruction quality and boosts performance in downstream latent-space synthesis.

Searching arXiv for recent and foundational papers related to latent harmonization encoder, especially the explicit LHE paper and closely related latent-space harmonization work. Latent Harmonization Encoder (LHE) is an encoder-side module introduced for semantics-first latent modeling in 3D MRI reconstruction and cross-contrast synthesis. In that formulation, LHE is designed to produce a compressed volumetric representation that is not merely low-dimensional, but also globally anatomically coherent, by combining local convolutional features with global context features before finite scalar quantization. Its stated purpose is to address a specific weakness of standard latent compression architectures for 3D MRI: insufficient preservation of long-range anatomical dependencies, which can compromise both reconstruction quality and downstream latent generative modeling (Chen et al., 16 Jun 2026).

1. Definition and conceptual scope

Within the 2026 framework that explicitly names it, LHE is the front-end latent encoder of a two-stage pipeline for 3D MRI reconstruction and cross-contrast synthesis. The input is a 3D MRI volume $x$; LHE first extracts local volumetric features through a convolutional branch and global context features through a parallel slice-wise Vision Transformer branch, then harmonizes their feature statistics and fuses them prior to discretization. The resulting latent codes are used both by the reconstruction decoder and by later latent-space synthesis models such as latent CycleGAN and latent diffusion (Chen et al., 16 Jun 2026).

A central nuance is that the term harmonization in LHE does not, in the original formulation, denote direct scanner harmonization or explicit domain removal at the encoder output. Rather, it refers specifically to channel-wise feature alignment between the convolutional and transformer pathways before fusion. In other words, the harmonization is internal to latent construction: transformer features are aligned to the statistical domain of convolutional features so that the fused representation is structurally coherent before quantization. This distinguishes LHE from methods in which harmonization is performed later by a latent translator, diffusion model, or adversarial module (Chen et al., 16 Jun 2026).

The same paper frames LHE as addressing only the first of three identified weaknesses of standard 3D latent compressors: long-range anatomical incoherence. The other two weaknesses—semantic degradation and over-smoothing from voxel-wise losses—are handled by the accompanying Semantic Recovery Block (SRB) and Anatomy-aware Frequency Loss (AFL), respectively. This suggests that LHE is best understood as a specialized encoder component inside a broader semantics-first latent modeling system rather than as a complete harmonization framework by itself (Chen et al., 16 Jun 2026).

2. Architectural formulation

The LHE architecture consists of two heterogeneous encoder branches. The convolutional branch $E_c$ adopts the baseline 3D FSQ encoder structure and extracts local volumetric features,

$h_c = E_c(x).$

In parallel, a slice-wise Vision Transformer branch $E_t$ extracts global context features,

$h_t = E_t(x).$

The paper explicitly characterizes the transformer pathway as slice-wise, not as a full volumetric transformer. This means the design injects non-local context without claiming a fully 3D tokenized attention model (Chen et al., 16 Jun 2026).

Because the two branches produce features with different statistics, LHE introduces an explicit harmonization step before fusion: $$h_t^* = \frac{h_t - \mu_t}{\sigma_t + \epsilon}\cdot \sigma_c + \mu_c,$$ where $\mu_t,\sigma_t$ are the channel-wise mean and standard deviation of transformer features and $\mu_c,\sigma_c$ are the corresponding statistics of convolutional features. Fusion is then residual,

$h = h_c + h_t^*.$

The fused representation $h$ is subsequently discretized via finite scalar quantization. The paper states this operation in prose; equivalently, the stated step can be written as $E_c$0, with $E_c$1 then decoded by the baseline 3D FSQ decoder (Chen et al., 16 Jun 2026).

Several architectural properties follow directly from this formulation. First, LHE is not purely convolutional: it inserts a transformer-based context stream before latent discretization. Second, the harmonization is statistical rather than attentional; the explicit operation is channel-statistics alignment, not cross-attention between branches. Third, the fusion described in the paper is residual and pre-quantization. Fourth, although LHE is presented as a 3D MRI encoder, the global-context branch is implemented through slice-wise ViT processing, and the paper does not specify the exact slice aggregation procedure, ViT depth, patch size, channel widths, latent tensor shape, or fusion granularity (Chen et al., 16 Jun 2026).

These omissions matter for interpretation. LHE is therefore precisely defined at the level of branch roles and fusion equations, but only partially specified at the level of implementation detail. It is a hybrid conv-transformer latent encoder with channel-statistics harmonization prior to FSQ quantization, not a fully enumerated layer-by-layer architecture (Chen et al., 16 Jun 2026).

3. Mathematical role inside semantics-first latent modeling

LHE operates inside a larger objective that combines reconstruction, semantic supervision, and anatomy-aware high-frequency preservation. The full training loss of the framework is

$E_c$2

Here, $E_c$3 is the baseline FSQ reconstruction loss, while the two additional terms are designed to correct limitations that LHE alone does not fully solve (Chen et al., 16 Jun 2026).

The Semantic Recovery Block uses a frozen DINO-pretrained ViT-B/16 teacher. A random 2D slice $E_c$4 from the volume is passed through the teacher to produce a semantic embedding

$E_c$5

and the quantized latent code $E_c$6 from LHE+FSQ is mapped through a learnable projector $E_c$7 to obtain

$E_c$8

The semantic recovery loss is then

$E_c$9

This places semantic pressure on the latent space defined upstream by LHE. The paper’s interpretation is that LHE improves structural coherence, whereas SRB reshapes the resulting latent codes to be more semantically organized and more contrast-separable (Chen et al., 16 Jun 2026).

The Anatomy-aware Frequency Loss acts at reconstruction level but feeds back into encoder learning. It first defines an anatomical attention map

$h_c = E_c(x).$0

combines it with a semantic attention map $h_c = E_c(x).$1 derived from teacher self-attention to obtain

$h_c = E_c(x).$2

and defines high-frequency residuals by

$h_c = E_c(x).$3

where $h_c = E_c(x).$4 is a volumetric smoothing operator. The AFL term is

$h_c = E_c(x).$5

Its stated role is to mitigate over-smoothing while maintaining robustness. In practical terms, this forces the entire compression system, including LHE, to retain information required for attention-weighted high-frequency anatomy (Chen et al., 16 Jun 2026).

A common misconception is that LHE itself is the semantic module of the framework. The paper’s actual division of labor is stricter: LHE supplies a globally coherent pre-quantization representation; SRB injects high-level semantic priors; AFL preserves diagnostically relevant high-frequency structures. This suggests that the defining contribution of LHE is not semantic supervision in itself, but a context-aware harmonization stage before latent discretization (Chen et al., 16 Jun 2026).

4. Empirical behavior in reconstruction and cross-contrast synthesis

The clearest direct evidence for LHE comes from the ablation study on BraTS reconstruction. With baseline FSQ alone, the reported average metrics are PSNR $h_c = E_c(x).$6, SSIM $h_c = E_c(x).$7, and LPIPS $h_c = E_c(x).$8. Adding LHE only raises the averages to PSNR $h_c = E_c(x).$9, SSIM $E_t$0, and LPIPS $E_t$1. The full model, combining LHE with SRB and AFL, reaches PSNR $E_t$2, SSIM $E_t$3, and LPIPS $E_t$4. The paper interprets the LHE-only gain as evidence that the encoder captures long-range anatomical dependencies, while the full framework yields the best overall reconstruction behavior (Chen et al., 16 Jun 2026).

The same pattern appears in comparison against alternative compression baselines. On BraTS average reconstruction, VQVAE reports $E_t$5, VQGAN $E_t$6, FSQ $E_t$7, and the proposed system $E_t$8. On IXI average reconstruction, FSQ reports $E_t$9 and the proposed system $h_t = E_t(x).$0. These numbers are reported for the complete semantics-first framework rather than for LHE in isolation, but the ablation establishes that LHE contributes positively before the additional semantic and frequency losses are introduced (Chen et al., 16 Jun 2026).

The framework also evaluates whether improved latent construction benefits downstream latent synthesis. After stage-1 training, the encoder, decoder, and quantization modules are frozen, and latent-space cross-contrast synthesis is trained for T1 $h_t = E_t(x).$1 T1C and T2 $h_t = E_t(x).$2 FLAIR. Relative to baseline latent CycleGAN, the proposed latent space improves T2 $h_t = E_t(x).$3 FLAIR from PSNR $h_t = E_t(x).$4 to $h_t = E_t(x).$5, SSIM from $h_t = E_t(x).$6 to $h_t = E_t(x).$7, and LPIPS from $h_t = E_t(x).$8 to $h_t = E_t(x).$9; for T1 $$h_t^* = \frac{h_t - \mu_t}{\sigma_t + \epsilon}\cdot \sigma_c + \mu_c,$$0 T1C, it improves PSNR from $$h_t^* = \frac{h_t - \mu_t}{\sigma_t + \epsilon}\cdot \sigma_c + \mu_c,$$1 to $$h_t^* = \frac{h_t - \mu_t}{\sigma_t + \epsilon}\cdot \sigma_c + \mu_c,$$2, SSIM from $$h_t^* = \frac{h_t - \mu_t}{\sigma_t + \epsilon}\cdot \sigma_c + \mu_c,$$3 to $$h_t^* = \frac{h_t - \mu_t}{\sigma_t + \epsilon}\cdot \sigma_c + \mu_c,$$4, and LPIPS from $$h_t^* = \frac{h_t - \mu_t}{\sigma_t + \epsilon}\cdot \sigma_c + \mu_c,$$5 to $$h_t^* = \frac{h_t - \mu_t}{\sigma_t + \epsilon}\cdot \sigma_c + \mu_c,$$6. For latent diffusion, T2 $$h_t^* = \frac{h_t - \mu_t}{\sigma_t + \epsilon}\cdot \sigma_c + \mu_c,$$7 FLAIR improves from PSNR $$h_t^* = \frac{h_t - \mu_t}{\sigma_t + \epsilon}\cdot \sigma_c + \mu_c,$$8 to $$h_t^* = \frac{h_t - \mu_t}{\sigma_t + \epsilon}\cdot \sigma_c + \mu_c,$$9, and T1 $\mu_t,\sigma_t$0 T1C from $\mu_t,\sigma_t$1 to $\mu_t,\sigma_t$2. The paper attributes these gains to the improved latent representation, with LHE providing anatomical coherence, SRB improving semantic preservation and contrast-aware separability, and AFL improving high-frequency retention (Chen et al., 16 Jun 2026).

The implementation regime further situates LHE as a large-scale 3D encoder rather than a lightweight feature head. Stage 1 uses 8 RTX4090 GPUs, Adam, initial learning rate $\mu_t,\sigma_t$3, total batch size 2, and 500 epochs. Stage 2 uses Adam, learning rate $\mu_t,\sigma_t$4, batch size 32, and 200 epochs. The paper also reports that all 3D volumes are resampled to a uniform spatial resolution and that BraTS and IXI official train/val/test splits are used, but it does not provide crop size, patch size, intensity normalization method, or augmentation details (Chen et al., 16 Jun 2026).

5. Research lineage and related latent harmonization formulations

LHE emerged within a broader line of work that treats harmonization as a latent-space operation rather than a purely image-space one. Earlier MRI harmonization systems already relied on encoder-driven factorization or latent-space transport, but they did not use the term Latent Harmonization Encoder and often placed the actual harmonization operator outside the encoder proper.

Paper	Latent mechanism	Relation to LHE
CALAMITI (Zuo et al., 2021)	Separate anatomy code $\mu_t,\sigma_t$5 and contrast code $\mu_t,\sigma_t$6	Strong precursor for anatomy/style factorization
DLEST (Wu et al., 2024)	Site-agnostic latent autoencoder plus EBM latent style translation	Encoder is harmonization backbone, translation is downstream
HCLD (Wu et al., 2024)	3D autoencoder plus conditional latent diffusion	LHE-like at system level, not encoder-only
DiffusionCT (Selim et al., 2023)	Encoder-decoder plus conditional latent DDPM in bottleneck	CT standardization analogue of latent harmonization

CALAMITI is especially close to a canonical LHE interpretation because it decomposes each MR image into an anatomical latent code $\mu_t,\sigma_t$7 and a contrast latent code $\mu_t,\sigma_t$8, with harmonization performed by keeping $\mu_t,\sigma_t$9 and replacing or averaging $\mu_c,\sigma_c$0. Its anatomy encoder is spatial and low-capacity, its contrast encoder is compact and variational, and the model aligns anatomy latents across sites through a $\mu_c,\sigma_c$1-discriminator. Although the term LHE does not appear, the paper is directly encoder-centered and makes explicit the anatomy/content versus contrast/site split that later LHE discussions inherit (Zuo et al., 2021).

DLEST takes a different route. Its Site-Invariant Image Generation module uses a latent autoencoder with encoder $\mu_c,\sigma_c$2 and decoder $\mu_c,\sigma_c$3, but the encoder does not explicitly disentangle anatomy from style. Instead, a downstream energy-based model performs target-domain latent transport. This makes DLEST a strong reference for latent harmonization frameworks but only a partial match to LHE in the narrow sense, because the harmonization behavior emerges from the encoder together with the EBM and shared decoder rather than from explicit encoder-side factorization (Wu et al., 2024).

HCLD similarly supports an LHE-like reading at the system level. It uses a pre-trained 3D autoencoder to obtain a spatial latent tensor, then performs harmonization through conditional latent diffusion, guided by instance normalization, AdaIN, content loss, and style loss. The paper explicitly treats the latent map as containing both style and content/anatomical information, but the actual harmonization is iterative and diffusion-based rather than being executed directly by the encoder (Wu et al., 2024).

DiffusionCT extends the same latent-space principle to CT standardization. An encoder maps non-standard and standard CT images into a shared bottleneck, a conditional latent DDPM transforms a non-standard latent into a standard one, and the decoder reconstructs the standardized image. The paper is therefore highly relevant as a modular encoder–translator–decoder template for latent harmonization, even though it addresses CT rather than MRI and uses paired intra-scanner training in its main experiments (Selim et al., 2023).

A broader theoretical analogue appears in geometry-preserving latent generative modeling. The geometry-preserving encoder framework of 2025 is not a harmonization method in the MRI site-effects sense, but it formalizes an encoder objective that preserves pairwise geometric structure through a Gromov-style cost and shows that downstream decoder training can be substantially accelerated. This suggests a plausible direction for future LHE design: latent harmonization may benefit not only from domain disentanglement but also from explicit control of intrinsic latent geometry (Lee et al., 16 Jan 2025).

Interpretability-oriented latent harmonization also appears in PL-SE-ADA, which disentangles a domain-invariant code $\mu_c,\sigma_c$4 from a domain-specific code $\mu_c,\sigma_c$5 and reconstructs images by a pseudo-linear sum $\mu_c,\sigma_c$6. In that line, the content latent itself functions as a harmonized representation. A plausible implication is that future LHE variants may combine the global-context modeling of the 2026 LHE with the image-space interpretability of explicit content/style decoders (Abe et al., 16 Oct 2025).

6. Limitations, misconceptions, and open directions

The most immediate limitation of LHE as currently specified is incomplete architectural detail. The paper does not state the exact 3D FSQ encoder structure, number of stages, channel sizes, strides, latent dimensionality or resolution, precise ViT configuration, or how slice-wise ViT outputs are reassembled into a volume-aligned feature map. This means LHE is conceptually clear and mathematically identifiable, but not fully recoverable as a unique implementation from the paper text alone (Chen et al., 16 Jun 2026).

A second limitation is scope. LHE is explicitly introduced to solve long-range anatomical incoherence, not the full latent compression problem by itself. Semantic degradation is handled by SRB, and over-smoothing is handled by AFL. It is therefore misleading to describe LHE as a complete standalone harmonizer or as a self-sufficient semantics module. More precisely, it is the encoder-side structural component of a larger semantics-first system (Chen et al., 16 Jun 2026).

A third limitation concerns the meaning of harmonization. In the named LHE paper, harmonization refers to alignment between the feature statistics of the convolutional and transformer branches. By contrast, earlier MRI harmonization literature often uses the term for site, scanner, or contrast standardization across domains. This difference in usage can create terminological confusion. A careful reading suggests that the 2026 LHE internalizes harmonization as pre-quantization feature-statistics alignment, while latent MRI harmonization papers such as CALAMITI, DLEST, and HCLD externalize harmonization as anatomy-preserving latent translation across sites or contrasts (Zuo et al., 2021).

A fourth limitation is that downstream gains cannot be attributed to LHE alone. The reported synthesis improvements come from the latent space produced by the full stage-1 system. The authors themselves describe the effective combination as follows: LHE improves anatomical consistency, SRB improves semantic preservation and contrast-aware separability, and AFL improves high-frequency retention. This suggests that the present evidence supports LHE as a necessary but not sufficient ingredient of the best-performing latent representation (Chen et al., 16 Jun 2026).

Taken together, these points motivate a precise encyclopedia definition. LHE is best regarded as a hybrid local-global encoder for 3D MRI latent compression that performs channel-statistics harmonization between convolutional and transformer features prior to FSQ quantization, thereby improving global anatomical coherence of the latent space. It belongs to a broader family of latent harmonization methods in medical imaging, but it differs from prior site-harmonization encoders by locating the harmonization operation inside feature fusion rather than only in downstream latent translation (Chen et al., 16 Jun 2026).