Joint Latent Generation Paradigms

• Joint latent generation is a modeling paradigm that constructs a unified latent space encoding multiple interdependent data sources.
• It uses concatenative and structured decomposition techniques with diffusion methods to achieve consistent multi-modal synthesis.
• Applications span music, video-depth, 3D reconstruction, and language tasks, demonstrating state-of-the-art coherence and conditional generation.

Joint latent generation is a modeling paradigm in which a generative model learns to synthesize or manipulate multiple interdependent aspects (modalities, hierarchical representations, part-level features, etc.) of structured data by explicitly parameterizing and sampling from a unified latent space. This latent space encodes the joint statistics of the underlying factors—be they modalities (audio/video, geometry/texture), sources (instrumental stems), layers of abstraction, or compositional parts—such that sampling from it yields globally consistent and interdependent outputs without the need for cascade, post-hoc fusion, or task-specific correction. The approach is central to recent advances in multi-modal generative modeling, hierarchical diffusion models, structured latent representations for geometry and physics, and conditional generation tasks that require coherent synthesis across separate but related factors.

1. Formulation of Joint Latent Spaces

Joint latent generation frameworks construct a latent space by compressing multiple, often heterogeneous, data sources or representations into a single, structured code. In many applications, this code is the concatenation or structured amalgamation of per-factor (modality, part, or abstraction-level) latents.

• Concatenative Construction: In multi-source latent diffusion for music, each instrumental source is encoded into its own latent representation by a shared VAE, and the joint latent is formed as $$Z' = \text{concat}_k(z_k) \in \mathbb{R}^{K C \times T}$$, with $K$ instruments and $C$ the channel dimension (Xu et al., 2024). For geometry-texture synthesis in 3D terrain, heightmap and texture latents are encoded separately and then concatenated channel-wise before joint diffusion (Higo et al., 7 May 2025).
• Structured Decomposition: In factorized or structured object modeling, each part (e.g., joint in 3D humans or articulated objects) receives its own sub-latent for geometry and attributes (Ji et al., 2024, Chen et al., 24 Oct 2025).
• Unified Categorical and Continuous Spaces: In hierarchical models, a stack of latent variables corresponding to different layers of abstraction is jointly embedded, with each layer’s latent contributing to the overall generative process (Cui et al., 2023).

This construction enables the generative process to explicitly capture cross-factor dependencies by ensuring all relevant latents are visible to the generative or diffusion backbone at each sampling step.

2. Inference and Generation Algorithms

Joint latent spaces require generative and inference procedures capable of reasoning over and sampling consistent configurations across all constituent factors.

• Joint Diffusion or Flow Matching: Most current architectures employ a score-based diffusion process or stochastic flow matching in the joint latent space, operating over the full tensor or structured set of latent codes. In multi-source diffusion for music, joint inference is performed by solving

$dZ = -\sigma(t) s(Z, \sigma(t))\, dt$

where $s$ is a score network defined on the full concatenated latent (Xu et al., 2024). For multimodal video-depth or audio-video, U-Net or DiT backbones process both modalities together, often concatenated or aligned along a channel or sequence dimension (Zhai et al., 2024, Torre, 17 Mar 2026).

• Conditional and Partial Inpainting: To support manipulation, extraction, and conditional generation, many frameworks implement masked or inpainting-oriented sampling, wherein parts of the joint latent are clamped to known values while others are subject to noise and denoising (Chae et al., 29 May 2025). The diffusion backbone learns to infer the unknowns conditioned on the knowns via sampling schemes such as

$$\nabla_{Z_{\mathcal{I}^c}} \log p(Z_{\mathcal{I}^c} \mid Z_{\mathcal{I}}) \approx S^\theta([Z_{\mathcal{I}^c}, \hat{Z}_{\mathcal{I}}], \sigma)$$

for subsets $\mathcal{I}, \mathcal{I}^c$ (Xu et al., 2024).

• Cascaded Diffusion: For structured or hierarchical latent spaces, cascaded generation may apply sequential diffusion sampling over, e.g., skeleton structure and then surface geometry, each conditioned on the result of the prior cascade stage (Ji et al., 2024).

3. Enforcing and Exploiting Cross-Factor Correlations

Crucial to joint latent generation is the implicit or explicit modeling of cross-factor correlations, which ensures resulting samples are mutually consistent. Different strategies have been adopted:

• Shared Diffusion Backbone: Sharing the entire denoising U-Net or transformer across modalities or parts, with parameter-sharing and explicit modality embeddings, directly entangles the generation process and forces correlation learning (Higo et al., 7 May 2025, Zhai et al., 2024).
• Cross-Modal Attention: Spatial, temporal, or part-wise cross-attention layers are introduced to explicitly allow information flow and gradient propagation across distinct factors, e.g., kernel attention between video and depth in human-centric joint video-depth generation (Zhai et al., 2024), or bidirectional attention between visual and "physics" latents in physics-infused video modeling (Shen et al., 9 Apr 2026).
• Joint Loss Functions and Regularization: Additional training losses such as motion consistency, cross-attention map alignment, forward-kinematics losses, or mutual reconstruction penalize inconsistency between related latents and explicitly address alignment (Zhai et al., 2024, Ji et al., 2024, Ling et al., 9 Mar 2026).

4. Representative Architectures and Model Classes

A variety of architectures embody the joint latent generation paradigm, tailored for task-specific requirements:

Model / Domain	Joint Latent Structure	Core Backbone
Multi-source music (MSLDM, MGE-LDM)	Concatenated instrument latents	1D U-Net/Transformer
Terrain (TerraFusion)	Concatenated (height, texture)	Doubled-channel U-Net
Articulated objects (ArtiLatent)	Structured voxel + articulation	3D Conv VAE + Transformer
Human shape (JADE, PRISM)	Per-joint or (pose, shape) latents	Two-cascade DiT or VAE/T
Video+depth (IDOL)	Concatenated, cross-attention	Shared 3D U-Net
Hierarchical generator EBM (Cui et al., 2023)	Multi-layer latent stack	Energy-based prior + generator
Language (LDLM)	Latent encoder + diff. decoder	Perceiver-resampler + Transformer

All these architectures leverage strong per-factor encoders (often VAEs), with the joint latent concatenation or grid input format maintained for all downstream generative models.

5. Applications and Empirical Findings

Joint latent generation models have achieved state-of-the-art or competitive results across a range of generative and conditional synthesis tasks:

• Music Generation and Separation: Multi-source latent diffusion models outperform mixture-based approaches in Frechet Audio Distance (FAD) and human-rated coherence, while enabling partial generation and instrument imputation not feasible with mixture-only models (Xu et al., 2024, Chae et al., 29 May 2025).
• Multimodal Video and Geometry/Image Synthesis: Joint video-depth generation yields spatially aligned, temporally consistent outputs unachievable with individual expert models (Zhai et al., 2024). Joint geometry-appearance methods in 3D human reconstruction (JGA-LBD) yield state-of-the-art PSNR/SSIM/Chamfer and improved realism (Tang et al., 1 Jan 2026).
• Hierarchical and Structured Generation: Energy-based multi-layer priors over joint latents lead to substantial improvements in downstream generative quality, anomaly detection, and OOD performance, validating the model's ability to capture hierarchical structure and intra-layer dependencies (Cui et al., 2023).
• Text, Language, and Sketch Synthesis: Joint latent diffusion with unified, jointly trained encoders yields higher generation quality and much faster inference than discrete diffusion or partially frozen latent methods (Meshchaninov et al., 8 May 2026).

6. Extensions, Limitations, and Open Problems

• Latent Space Construction: The efficacy of joint latent generation critically depends on the latent space design—both in terms of compression quality and compatibility across factors. Joint training of latent encoders and denoisers, as emphasized in latent diffusion LLMs, is essential to avoid latent geometry degeneracies (Meshchaninov et al., 8 May 2026).
• Scalability: Parameter sharing, multimodal attention, and inpainting mechanisms scale well across number of factors, provided encoder and score model capacity are sufficient. However, increased factorization can challenge training stability, as in codebook swapping or very high-dimensional latent concatenation (Dai et al., 7 Feb 2025, Higo et al., 7 May 2025).
• Generalization and Conditioning: Flexible conditioning (e.g., masked inpainting or text-prompt extraction) is possible but may require careful design of the conditional score and attention mechanisms (Chae et al., 29 May 2025). Open challenges remain in generalizing to highly heterogeneous or weakly coupled modalities, particularly when modality-specific priors are poorly aligned.
• Interpretability and Disentanglement: Joint factorization, when combined with explicit structure (per-joint, per-part, or per-layer), promotes disentanglement and fine-grained editing, but the precise boundaries of factor independence vs. interdependence remain empirically driven (Ji et al., 2024, Donahue et al., 2017).

7. Summary Table: Key Features in Joint Latent Generation

Feature	Example Frameworks	Role in Model
Joint latent concatenation	MSLDM, StrokeFusion, TerraFusion	Cross-modal and source coherence
Hierarchical latent stack	Joint-EBM multi-layer generator	Inter-layer abstraction, OOD detection
Per-part decomposition	JADE, ArtiLatent, PRISM	Local control and semantic manipulation
Cross-modal attention	Phantom, IDOL	Physical/plausible temporal alignment
Latent inpainting	MGE-LDM, LDLM	Partial/conditional generation
Explicit structure prior	JGA-LBD, ArtiLatent	Physically valid reconstructions

Joint latent generation thus provides a theoretical and algorithmic foundation for generative models requiring globally consistent, structured, and controllable outputs across interdependent factors. Its adoption spans music, vision, graphics, language, and physical modeling, with ongoing research targeting more efficient, disentangled, and interpretable latent constructions and inference algorithms (Cui et al., 2023, Xu et al., 2024, Ji et al., 2024, Zhou et al., 31 Mar 2025, Chae et al., 29 May 2025, Shen et al., 9 Apr 2026, Tang et al., 1 Jan 2026, Meshchaninov et al., 8 May 2026).