Joint Latent Space Models

• Joint latent space models are probabilistic frameworks that map diverse data sources into shared, low-dimensional embeddings for enhanced inference and optimization.
• They utilize synchronized training of encoders to ensure cross-modality correlation, improving tasks such as Bayesian optimization and generative modeling.
• Their applications range from hierarchical representation learning in vision to structured social network analysis, offering robust statistical coupling across modalities.

A joint latent space model is a probabilistic or neural framework that maps multiple heterogeneous data sources—such as structured inputs and outputs, modalities, or observed/latent layers—into one or more shared or interlinked low-dimensional representations (latent variables), with the joint geometry optimized for a downstream objective: generative modeling, inference, prediction, multi-modal alignment, or sample-efficient decision-making. The paradigm generalizes classical latent variable models by enforcing that the embeddings themselves, rather than being learned independently for each data source or via decoupled autoencoders, are constructed and optimized jointly to ensure statistical or functional coupling across sources. In recent literature, joint latent space models have been systematized in composite Bayesian optimization (Maus et al., 2023), multi-layer generative models with energy-based priors (Cui et al., 2023), multimodal representation learning, and network/social data analysis.

1. Foundations and Definition of Joint Latent Space Models

Let $x \in \mathbb{R}^d$ be a high-dimensional input, $y = h(x) \in \mathbb{R}^m$ an intermediate or auxiliary output, and $f(x) = g(h(x))$ the final target variable (potentially scalar or low-dimensional). A joint latent space model introduces parametric mappings,

• $z_x := E_x(x) \in \mathbb{R}^{d'}$ (input encoder)
• $z_y := E_y(y) \in \mathbb{R}^{m'}$ (output or intermediate encoder), with $d', m' \ll d, m$, often end-to-end neural networks, such that both $x$ and $y$ are compressed into representations that are optimized jointly for a statistical objective—e.g., marginal likelihood, cross-entropy loss, or ELBO.

The key distinction from traditional decoupled representations is the synchronous parameterization and training of $E_x, E_y$, possibly together with downstream probabilistic or neural models operating on $(z_x, z_y)$, so that information relevant for determining $y = h(x) \in \mathbb{R}^m$0 (or for reconstructing any target of interest) is actively selected and retained in the compressed spaces (Maus et al., 2023).

Joint latent spaces also arise naturally in multi-layer (hierarchical) generative models, where instead of separate priors for each layer, an energy-based model or a diffusion model is fitted over the joint set of stacked latent variables $y = h(x) \in \mathbb{R}^m$1 (Cui et al., 2023).

2. Model Architectures: Encoders, Joint Priors, and Probabilistic Graphs

Encoder Design

In practice, encoders mapping high-dimensional sources to latent coordinates—whether for inputs (e.g., molecular or robotic parameters) or intermediate outputs (e.g., images, trajectories, texts)—are implemented as neural networks:

• For tabular/vector inputs: multi-layer perceptrons with intermediate ReLU activations.
• For image inputs/outputs: convolutional networks possibly followed by dense layers.
• For sequences/structured data: causal convolutions or transformers.

In composite BO (Maus et al., 2023), two encoders are optimized: $y = h(x) \in \mathbb{R}^m$2 and $y = h(x) \in \mathbb{R}^m$3, mapping $y = h(x) \in \mathbb{R}^m$4 and $y = h(x) \in \mathbb{R}^m$5.

Latent Priors and Energy-based Models

Classical hierarchical VAEs utilize factorized Gaussians $y = h(x) \in \mathbb{R}^m$6. Joint latent space EBMs instead parameterize a prior of the form

$y = h(x) \in \mathbb{R}^m$7

where $y = h(x) \in \mathbb{R}^m$8 is a deep scalar-valued function (neural network) combining all layers' latents, modeling cross-layer and intra-layer dependencies and encouraging complex, multi-scale latent structure (Cui et al., 2023).

In sequential, multimodal, or self-supervised settings, joint Gaussian or Gaussian-mixture models are fitted to concatenated latent representations, yielding analytic or mixture-form conditional predictions (Huang, 26 Mar 2026).

Probabilistic Models in Latent Space

Typical constructions include:

• Multioutput GPs: e.g., $y = h(x) \in \mathbb{R}^m$9, $f(x) = g(h(x))$0, forming a chain of GPs over compressed coordinates for multi-stage modeling (Maus et al., 2023).
• Latent-space diffusion: score-based (denoising) diffusion models operating on joint latent codes for direct sampling and generation (Krishnan et al., 22 Jan 2025).
• Hierarchical energy terms: sums of layer-wise or contextual energy functions, as in multi-layer EBMs (Cui et al., 2023, Cui et al., 2023).

3. Joint Training Objectives and Inference

Joint latent space models are always trained by maximizing a combined objective that is sensitive to the entire chain of data sources and targets—typically a sum of marginal log-likelihoods, variational lower bounds, or score-matching losses that depend on both encoder parameters and probabilistic model parameters.

For example, in Joint Composite Latent Space Bayesian Optimization (JoCo) (Maus et al., 2023), the loss decomposes as: $f(x) = g(h(x))$1 with both $f(x) = g(h(x))$2 and $f(x) = g(h(x))$3 appearing inside the arguments to the GP marginal likelihoods. This forces encoders to be task-aware—selecting subspaces of $f(x) = g(h(x))$4 and $f(x) = g(h(x))$5 maximally informative for predicting $f(x) = g(h(x))$6.

In energy-based joint priors (Cui et al., 2023, Cui et al., 2023), the objective is a variational ELBO,

$f(x) = g(h(x))$7

where $f(x) = g(h(x))$8 is an approximate inference model (bottom-up encoder) and $f(x) = g(h(x))$9 is the joint EBM prior.

Posterior and prior expectations are estimated by MCMC or Langevin updates, and encoders/decoders are trained end-to-end. Short-run MCMC or amortized inference reduces computational overhead.

4. Applications Across Domains

Bayesian Optimization of Composite or Multi-stage Systems

JoCo (Maus et al., 2023) demonstrates that explicitly modeling both intermediate outputs and final outcomes with coupled latent spaces and probabilistic models enables sample-efficient optimization even for systems where $z_x := E_x(x) \in \mathbb{R}^{d'}$0 or $z_x := E_x(x) \in \mathbb{R}^{d'}$1 are extremely high-dimensional (e.g., generative AI outputs, molecular graphs). The use of jointly trained latent spaces provides compression and computational tractability, with acquisition optimized in latent space and candidates selected in the original space via Thompson sampling or analogous strategies.

Performance: Across simulated and real-world benchmarks (Langermann/Rosenbrock, environmental PDEs, rover planning, LLM toxicity, etc.), JoCo consistently outperforms vanilla BO and high-dimensional BO techniques such as TuRBO/SAASBO.

Hierarchical Representation Learning and Expressive Generative Models

Joint latent space energy-based priors (Cui et al., 2023, Cui et al., 2023) outperform independent Gaussian priors in capturing multi-level abstractions, enabling models to generate coherently structured samples and, crucially, to disentangle features at multiple scales—early layers capture global structure, later layers embed fine-grained details.

Empirical Results: On CIFAR-10, CelebA, and similar benchmarks, joint EBM priors lower negative log-likelihood by 0.1–0.2 bits/dim and improve FID (e.g., 16.8 vs. 19.5). In feature transfer tasks, higher semantic richness appears at early latent layers (Cui et al., 2023).

Structured, Multimodal, and Network Data

Classical joint latent space models for networks (APLSM (Wang et al., 2019), LSJM (Gollini et al., 2013), multilayer networks (Sosa et al., 2021)) embed both nodes and covariates or multiple network "views" into shared latent coordinates, enabling joint prediction, informative visualization, and better link/attribute reconstruction. Cumulative shrinkage priors (Lv et al., 23 Sep 2025) further enable posterior adaptation of latent dimension.

Empirical Validity: On social/friendship networks with node attributes, joint latent spaces explain >60% of variance (vs. 50% for one modality alone) and achieve AUC ≈0.90 for links and ≈0.85 for attribute prediction (Wang et al., 2019).

5. Model Selection, Scalability, and Theoretical Guarantees

Model selection (particularly the determination of latent dimension $z_x := E_x(x) \in \mathbb{R}^{d'}$2) is a critical aspect. Classical approaches use cross-validated ELBO, information criteria (BIC/AIC), or elbow plots (Wang et al., 2019, Sosa et al., 2021). Recent innovations apply cumulative ordered spike-and-slab (COSS) priors to automate latent dimension selection, with theoretical guarantees for posterior concentration on the true latent dimension $z_x := E_x(x) \in \mathbb{R}^{d'}$3 and minimax-optimal convergence rates in Hellinger distance (Lv et al., 23 Sep 2025).

Computational Aspects:

Inference in joint latent space models may be variational (mean-field or Gaussian), MCMC (blocked Gibbs, Langevin), or with amortized inference. For large-scale applications (networks with millions of nodes), stochastic variational inference and parallelization are critical (Wang et al., 2019, Sosa et al., 2021).

Short-run MCMC or amortized Langevin is essential for practical training of joint latent space EBMs (Cui et al., 2023, Cui et al., 2023). In practice, training with inner-loop MCMC increases cost by ≈2× but is necessary for non-Gaussian, correlated latent priors.

6. Comparative Analysis: Joint vs. Decoupled or Single-Space Methods

Joint latent space models consistently outperform approaches that fit latent spaces in isolation for each data source, or that pretrain autoencoders separately. Decoupled methods fail to:

• Focus the latent embedding on features relevant to the end task: an unsupervised encoder on $z_x := E_x(x) \in \mathbb{R}^{d'}$4 may retain irrelevant variation (Maus et al., 2023).
• Propagate supervision from target variables back to the encoding of inputs or intermediates.
• Update as more reward (or task-labeled) data arrives—fixed autoencoders cannot adapt (Maus et al., 2023).
• Avoid redundancy: decoupled models may require large, unlabeled datasets for pretraining; joint models learn adaptively as labeled data accrues.

Empirical ablations confirm the necessity of continuous, joint adaptation. Freezing encoders or decoupling model components sharply degrades performance, especially in high-dim or generative tasks (e.g., text-to-image attack detection, molecular BO).

7. Broader Impact and Extensions

Joint latent space modeling enables practical, scalable, and expressive probabilistic modeling in domains characterized by high-dimensional, structured, or multi-modal data—ranging from composite system design to generative vision/language, network science, and hierarchical self-supervised learning.

The framework unifies and interrelates:

• Multi-stage and multi-output Bayesian optimization (Maus et al., 2023)
• Hierarchical generative models with expressive (EBM or diffusion) joint priors (Cui et al., 2023, Cui et al., 2023)
• Self-supervised joint representation learning (Gaussian mixtures/diffusions over joint embeddings) (Huang, 26 Mar 2026)
• Bayesian social/network models incorporating multiple data sources and model selection (Wang et al., 2019, Sosa et al., 2021, Lv et al., 23 Sep 2025)

Limitations include increased computational cost for MCMC or large-scale diffusion, challenges in reliably estimating high-dimensional joint densities, and the need for careful architectural and prior design to prevent collapse or loss of interpretability.

---

Key Papers:

• Joint Composite Latent Space Bayesian Optimization ("Joint Composite Latent Space Bayesian Optimization") (Maus et al., 2023)
• Learning Hierarchical Features with Joint Latent Space Energy-Based Prior ("Learning Hierarchical Features with Joint Latent Space Energy-Based Prior") (Cui et al., 2023)
• Attribute and Person Latent Space Model ("Joint Latent Space Model for Social Networks with Multivariate Attributes") (Wang et al., 2019)
• Adaptive Bayesian Joint Latent Space Modeling via Cumulative Shrinkage ("Adaptive Bayesian Joint Latent Space Modeling via Cumulative Shrinkage") (Lv et al., 23 Sep 2025)
• Gaussian Joint Embeddings for Representation Learning ("Gaussian Joint Embeddings For Self-Supervised Representation Learning") (Huang, 26 Mar 2026)