Heterogenous Latents Fusion

• Heterogenous Latents Fusion is a framework that integrates non-homologous latent codes from diverse sources to enable unified prediction and robust control.
• It employs methods like decode–encode projections, optimal transport, and Gaussian process fusion to align and merge disparate latent spaces.
• Applications span video restoration, multimodal sensor fusion, and decentralized Bayesian learning, yielding measurable gains in performance and efficiency.

Heterogenous Latents Fusion refers to a broad class of techniques for integrating latent representations originating from multiple sources, modalities, or models whose latent spaces are non-identical, non-homologous, or otherwise structurally incompatible. Such methods enable cross-modality inference, improved generalization, and transfer of inductive biases in settings ranging from video diffusion, sensor and multi-source data fusion, to decentralized Bayesian learning. The core challenge addressed is the migration, alignment, or fusion of latent representations between disparate models or data sources, allowing joint or improved prediction, restoration, or control. Implementations frequently involve decode–encode projections, explicit latent-space embedding, optimal transport, or kernelized mappings.

1. Foundational Principles and Motivations

Heterogenous Latents Fusion (HLF) arises when integrating models or data sources with incompatible latent parameterizations, such as 3D video- and 2D image-based variational autoencoders (VAEs), tabular/categorical versus real-valued data, or sensor-modality-specific neural representations. The motivation is to exploit complementary strengths from independent models (e.g., video temporal priors vs. image spatial priors), align datasets for transfer, or enable Bayesian fusion in decentralized or multi-fidelity regimes. HLF is essential when:

• latent codes are non-homologous (different VAE architectures or modalities),
• input parameterizations differ (heterogeneous feature spaces),
• fusion across independently designed or trained models is required without retraining.

HLF enables decoupling model and modality definitions, supports plug-in architectures, and, when properly orchestrated with adaptive weighting or alignment, delivers significant improvements in temporal coherence, robustness, and uncertainty quantification (Cao et al., 29 Jan 2026, Yilmaz et al., 2021, Comlek et al., 2024).

2. Mathematical Formulations and Core Algorithms

Canonical HLF procedures implement one or more of the following approaches:

Decode–Encode Projection (Video–Image Diffusion)

Given two latent spaces (e.g., 3D VAE for video and 2D VAE for image), migration operates via a round-trip decode and re-encode:

• Decode $\hat z₀^\mathrm{V2}$ using $\mathcal{D}_{3D}$ to $V$ (pixel-space video)
• Encode each frame via $\mathcal{E}_{2D}$, yielding $\hat z₀^{V2→I}$
• Reconstruct the corresponding noisy latent for IR/IE (image restoration/enhancement) at time $t$,

$$z_t^{V2→I} = z_t^{V2} + \sqrt{\bar α_t^I} (\hat z₀^{V2→I} - \hat z₀^{V2})$$

• Fuse with the IR/IE latent $z_t^I$ by weighted average, with fusion weight $\lambda_t^{F2}$.

This pipeline is iterated during DDIM sampling, often with additional fusion of homologous latents (Cao et al., 29 Jan 2026).

Latent Variable Gaussian Process Fusion

For multi-source regression, each source $s$ receives a learned embedding $z_s \in \mathbb{R}^d$. Joint GP regression proceeds over tuples $w = [x; z_s]$: $$k(w_i, w_j) = \sigma^2 \exp\left(-\sum_\ell \phi_\ell(x_{i\ell}-x_{j\ell})^2 - \lVert z_{s_i} - z_{s_j} \rVert^2 \right)$$ Latent source locations $z_s$ are optimized via marginal likelihood, enabling the model to learn cross-source similarity and facilitate uncertainty-aware fusion for prediction (Ravi et al., 2024, Comlek et al., 2024, Oune et al., 2021).

Optimal Transport-Based Latent Alignment

Subject- or domain-specific autoencoders yield latent representations $\{x_i^{(s)}\}$ per domain. Fused Gromov–Wasserstein optimal transport is solved for coupling matrices $\Omega$ minimizing combined geometry and label distance: $$\min_\Omega \sum_{i,j,i',j'} \left[ (1-\alpha)M_{ij} + \alpha |D_{ii'} - D'_{jj'}|^2 \right] \Omega_{ij} \Omega_{i'j'}$$ The barycentric mapping aligns latent geometries and class structures across domains without requiring explicit correspondence (Yuan et al., 2024).

KL-Divergence Mean-Field Component Fusion

Independent variational posteriors $q_{j}(z_l|\theta^j_l)$ (possibly with differing $L_j$) are fused to a global $q(z_g|\bar \theta_g)$ via a regularized assignment and averaging scheme. Assignments $P^j_{lg}$ match local to global components; global natural parameters are averaged: $$\bar \eta_g = \frac{1}{M_g} \sum_{j=1}^D \sum_{l=1}^{L_j} P^j_{lg} \eta(\theta^j_l)$$ This alternates with convex assignment updates and regularization for nonparametric model selection (Claici et al., 2020).

3. Heterogenous Latents Fusion in Video Diffusion Restoration

HLF is exemplified in state-of-the-art video restoration pipelines. In “Zero-Shot Video Restoration and Enhancement with Assistance of Video Diffusion Models,” HLF enables injection of strong temporal priors from 3D VAE-based text-to-video (T2V) diffusion models into image restoration/enhancement (IR/IE) methods with incompatible (2D) VAE latent spaces. The method consists of:

1. Running the T2V model and extracting video latents.
2. Decoding to video, then re-encoding each frame in the IR/IE VAE.
3. Projecting these latents to reconstruct noisy latents at each timestep.
4. Linearly fusing the re-projected latents into the ongoing IR/IE DDIM sampling (Eq. 4–5 in (Cao et al., 29 Jan 2026)).

Fusion weights ($\lambda_t^{F2}$, $\lambda_t^{F}$) are adaptively selected at each step using a chain-of-thought (COT) style best-of-$M$ search over candidate weights, with metrics including CLIP-IQA and Warping Error. This strategy yields measurable improvements such as PSNR $\uparrow$ by 0.43 dB and Frame Video Distance (FVD) $\downarrow$ by 18.2, over homologous-only fusion. The method is fully training-free and generalizes to arbitrary T2V models regardless of latent design.

Contrast is drawn with homologous fusion, which is restricted to T2V and IR/IE models sharing a VAE (no decode–encode step; cheaper but less general). HLF accommodates any state-of-the-art video diffusion model, at the cost of two VAE passes per integration step.

4. Applications Beyond Diffusion: Data Fusion, Multimodal Embeddings, and Networks

HLF strategies appear in other heterogeneous learning settings:

• Generative modeling for multimodal data: By formulating a joint exponential family generative model with a shared low-dimensional latent $c_i$ linking heterogeneous attributes (e.g., real-valued and categorical), multimodal fusion is attained, with variational/Laplace–Bernstein approximations providing tractable inference in high dimensions (Yilmaz et al., 2021).
• Network analysis: In heterogeneous multilayer networks, latent fusion decomposes each embedding into shared and network-specific components, allowing for oracle-rate estimation of shared signals (error $O_p(1/T)$ in number of networks) via spectral and gradient-score approaches (Tian et al., 2024). This sharply improves interpretability and estimation over layer-by-layer analyses.
• Sensor and resource-limited fusion: Latent Sensor Fusion reuses a modality-agnostic, unified encoder (e.g., VQ-VAE) jointly with a fusion network, compressing multimodal biosignals for lightweight devices without loss in predictive quality (Ahmed et al., 13 Jul 2025).

Other approaches such as process-based spatial fusion model all data via shared latent Gaussian processes, with flexible change-of-support operators and scalable priors, applicable to spatial, lattice, and point-process data (Wang et al., 2019).

5. Adaptive Fusion Ratios, Validation Strategies, and Theoretical Guarantees

Optimal fusion weights or ratios are a critical part of HLF design:

• Adaptive ratio search (COT/best-of-$M$): Fusion ratios are selected at each sampling step via candidate generation, candidate decoding (sample short video clip), and test-time ranking with both perceptual and temporal rewards (CLIP-IQA and Warping Error) (Cao et al., 29 Jan 2026).
• Soft Information Sharing: In Meta Fusion, cross-model latent ensembles use soft mutual learning, in which only the best cohorts inform each learner, provably reducing aleatoric variance and improving generalization error (Liang et al., 27 Jul 2025).
• Approximate fusion and consistency: Heterogeneous Bayesian decentralized fusion employs Covariance Intersection or subset-based conservative approximations to support scalability and filter consistency under communication constraints (Dagan et al., 2021).

Theoretical error rates, identifiability, and convergence guarantees have been established for various formulations (Tian et al., 2024, Claici et al., 2020).

6. Quantitative Impact and Deployment Observations

Innovation in HLF yields state-of-the-art results in restoration, generalization, and uncertainty quantification:

• Video restoration: HLF achieves PSNR gains $+0.43$ dB, SSIM $+0.022$, and FVD reduction $-18.2$ on blind video SR benchmarks when combined with adaptive ratio selection and temporal-strengthening (Cao et al., 29 Jan 2026).
• Multimodal data fusion: Latent variable GP fusion outperforms both single-source and source-unaware models on RMSE metrics by up to $60\%$ (Ti6Al4V test case) (Ravi et al., 2024, Comlek et al., 2024).
• Lightweight fusion: Latent Sensor Fusion reduces parameter count by $64\%$ and inference time by $1.4\times$ over modality-specific encoders, maintaining AUC for biosignal classification (Ahmed et al., 13 Jul 2025).
• Stability and overfitting: KL-divergence HLF supports nonparametric pruning and mean-field label-alignment, ensuring tuning-free model selection (Claici et al., 2020).

Typical real-world deployments exploit HLF for on-the-fly system integration, improved sample complexity in data-scarce regimes, and resource-constrained applications.

7. Limitations, Extensions, and Prospects

While HLF methods are powerful, deployment involves computational and architectural tradeoffs:

• Cost in decode/encode steps: Video diffusion HLF incurs twice the VAE cost per fusion step versus homologous fusion (Cao et al., 29 Jan 2026).
• Optimization complexity: Nonconvexities in autoencoder-based and transport-based HLF require multi-start or careful initialization (Yuan et al., 2024, Piechocki et al., 2022).
• Alignment error: Latent mappings must accurately capture inter-source or inter-model relationships; misalignment may result in diminished transfer or overconfidence (Oune et al., 2021).
• Fairness and source-awareness: Fusing strongly dissimilar sources demands interpretable embeddings (e.g., learned latent Euclidean distances and “dissimilarity metrics” per (Ravi et al., 2024)).

Future directions include richer posterior approximations (mixture-of-experts, importance-weighted ELBOs), cross-modal private/shared latent decompositions, generalized optimal transport cost functions, and further scaling to high-dimensional sensor, video, or spatial-temporal applications. Emerging end-to-end and training-free plug-in frameworks suggest HLF will remain foundational for robust, cross-domain AI systems.