Unified Geometry–Appearance Latents

Updated 7 June 2026

Unified geometry–appearance latents are high-dimensional representations that merge structural (geometry) and visual (appearance) data into a single, manipulable latent space.
They underpin deep generative and reconstruction models, enabling joint synthesis, editing, and efficient sampling across 2D images, 3D objects, and dynamic scenes.
Advanced encoder designs and unified training objectives ensure high-fidelity output across diverse formats, while balancing computational efficiency with detail preservation.

Unified geometry–appearance latents are high-dimensional learned representations that simultaneously encode structural (geometry) and visual (appearance) information in a single latent space or tensor. These unified representations underpin a new class of deep generative and reconstruction models that can synthesize, edit, or reconstruct images, 3D objects, dynamic scenes, or articulated assets, with consistent geometric structure and appearance. Unlike earlier systems with segregated geometry and appearance streams, unified latents are purposefully designed—typically via a VAE, flow-based, or diffusion formulation—to support joint manipulation, efficient sampling, and high-fidelity decoding to diverse output formats (images, meshes, 3D Gaussians, radiance fields, etc.).

1. Mathematical Formulations and Latent Structures

Unified latent spaces are instantiated in architectures spanning 2D images, 3D objects, 4D dynamic scenes, and articulated models. Canonical examples include:

Dense vectorized latents: As in StyleGAN-Human (Qu et al., 2024), unified latents are vectors in an extended space (e.g., $\mathbb{R}^{18\times512}$ ) where subsets of coordinates explicitly control geometry (pose, contours, blocks 1–8) and appearance (identity, clothing, texture, blocks 9–18). Latent fusion is performed by combining (e.g., slicing and concatenating along the block dimension) geometry and appearance codes.
Shared tensor/field representations: In Orchid (Krishnan et al., 22 Jan 2025), a 2D convolutional VAE encodes stacked image, depth, and normal maps as a single latent tensor $z\in\mathbb{R}^{H/8\times W/8\times 8}$ , where all modalities are fused channelwise. Similarly, SLAT (Xiang et al., 2024), SLat comprises a set of $(\mathbf{p},\mathbf{z})$ pairs—voxel spatial index plus a high-dimensional latent vector encoding both geometric and appearance cues.
Sparse structured grids: In 3D domains, Unified VAE (UniLat) (Wu et al., 29 Sep 2025), O-Voxel (Xiang et al., 16 Dec 2025), and SLat (Xiang et al., 2024) architectures operate on sparse or densely compressed voxel grids, where each activation stores a $d$ -dimensional latent vector integrating geometric primitives (e.g., occupancy, dual vertices) and surface/material appearance (e.g., color, PBR attributes).
Temporal or articulated latents: MORPHOS (Kwon et al., 1 Jun 2026) generalizes the concept to 4D, with T-SLATs comprising per-frame sequences of sparse $D$ -dimensional voxel features, supporting dynamic geometry–appearance evolution. ArtiLatent (Chen et al., 24 Oct 2025) further augments each spatial cell with explicit articulation parameters, fusing part geometry, joint state, and appearance.

Tabular summary:

Model/Framework	Latent Structure	Fusion Principle
Sketch2Human (2D human)	$W^+$ , $\mathbb{R}^{18\times512}$	Block-wise slicing, concatenation
Orchid (2D RGB/depth/norm)	$H/8\times W/8\times8$ tensor	Channelwise stacking, end-to-end
UniLat3D (3D assets)	$M^3\times d$ (dense/sparse 3D)	Voxelwise joint embeddings
O-Voxel (3D assets)	$K\times d$ sparse grid	Native geometry/material tokens
MORPHOS (4D)	$z\in\mathbb{R}^{H/8\times W/8\times 8}$ 0 sequence	Per-voxel, per-frame co-embedding
SLAT/SLat (3D)	$z\in\mathbb{R}^{H/8\times W/8\times 8}$ 1 set	Learned pooled multiview

2. Unified Latent Construction and Training

Construction of a unified geometry–appearance latent is typically achieved by designing the encoder and bottleneck to jointly process all relevant input modalities, with network architectures and loss functions tailored to maintain both geometric fidelity and appearance realism.

Encoder design: Input features (sketches, semantic maps, RGB images, depth, normals, per-view 2D features, or part parameters) are either stacked channelwise for convolutional encoders (Orchid, 2.5D Latent (Yang et al., 27 May 2025)) or mapped to tokens underpinning transformers or sparse voxel networks (SLat, O-Voxel, UniLat3D, ARTI-Latent). In the VAE bottleneck, position and appearance information are propagated together.
Latent disentanglement and fusion: Some systems (e.g., Sketch2Human) begin from architectures that only partially disentangle geometry and appearance, requiring fine-tuning with synthetic content-mixing (e.g., layer-swapping in StyleGAN-Human, blocks 1–8 from geometry, blocks 9–18 from appearance) combined with targeted losses—style, content, and segmentation—to encourage orthogonal control (Qu et al., 2024).
Unified training objectives: Modern systems optimize multi-term objectives:
- Perceptual [LPIPS], adversarial, and L1/L2 image reconstruction for realism;
- Content/style transfer (Gram, VGG) for attribute preservation;
- Semantic segmentation losses (Dice, CE) for part-aware structure;
- KL divergence and distributional alignment between modality-specific and shared latents (e.g., Orchid’s KL and distillation, Gen3R’s cross-manifold KL (Huang et al., 7 Jan 2026));
- For 3D/4D, sparse reconstruction losses (occupancy, pose, PBR attributes) and rendering-based perceptual metrics (PSNR, SSIM, multiview LPIPS).

3. Generative Modeling With Unified Latents

Unified geometry–appearance latents are leveraged in several generative modeling frameworks, providing benefits of joint coherence, flexible conditioning, and efficient sampling:

Diffusion and flow-based generative models: Latent diffusion (Orchid (Krishnan et al., 22 Jan 2025), DRF (Kim et al., 13 Aug 2025), UniLat3D (Wu et al., 29 Sep 2025)), conditional flow-matching (SLat, O-Voxel, UniLat3D), and autoregressive flow (MORPHOS (Kwon et al., 1 Jun 2026)) enable synthesis directly within the shared latent manifold. Usually, these models inject stochasticity (Gaussian noise) followed by repeated denoising, all steps operating upon a single, fused latent code.
Bidirectional interaction and cross-modal feedback: Recent systems implement recursive feedback mechanisms (DRF (Kim et al., 13 Aug 2025)) or cross-branch attention (FantasyWorld (Dai et al., 25 Sep 2025)), allowing for continual mutual alignment between geometry and appearance signals during the generative process. In DRF, recursive refinement steps jointly steer the latent towards both structure- and appearance-conforming fixed points by alternating updates on the same initial noise, effectively sculpting a coupled manifold.
Single-stage vs. multi-stage generation: UniLat3D demonstrates that a single flow-matching model operating in a unified latent space eliminates the sequential geometry→texture pipeline, enabling faster inference and mitigating geometry–appearance misalignment (Wu et al., 29 Sep 2025). In contrast, prior two-stage approaches suffer from increased computational cost and frequent texture–geometry mismatch.
Autoregressive and temporally consistent diffusion: T-SLAT of MORPHOS generalizes the latent to a 4D (space × time) sequence, supporting causal attention and handling dynamic topologies.

4. Decoding, Output Formats, and Downstream Applications

Decoding from unified latents enables high-fidelity outputs in a variety of formats, with a single latent serving as the source for divergent decoders:

Image/Video synthesis: Decoding of appearance–geometry latents in 2D/2.5D yields photorealistic full images, semantic segmentations, depth, and normal maps (Sketch2Human (Qu et al., 2024), Orchid (Krishnan et al., 22 Jan 2025), DRF (Kim et al., 13 Aug 2025)).
3D output formats: Unified latents are decoded via specialized heads to 3D Gaussians (SLat, O-Voxel, UniLat3D, JGA-LBD (Tang et al., 1 Jan 2026)), meshes (O-Voxel, SLat, UniLat3D), radiance fields (SLat), or signed distance fields (ArtiLatent (Chen et al., 24 Oct 2025)), allowing for simultaneous access to geometry and appearance data. For articulated assets, per-voxel joint parameters propagate through the decoder, ensuring pose-aware surface and interior rendering.
Dynamic and articulated scene modeling: Sequences of unified latents represent temporally evolving assets (T-SLAT/MORPHOS), while articulation-aware decoders in ArtiLatent predict previously unobserved appearances exposed by motion (e.g., the inside surfaces of a drawer).
Applications: Unified latents underpin controllable image/3D editing, pose transfer and cross-domain generation (DRF), fast image→3D asset creation (UniLat3D, SLat, O-Voxel), human reconstruction from RGB (JGA-LBD), and dynamic 4D asset synthesis with scene-level consistency (MORPHOS, FantasyWorld).

5. Quantitative Results and Empirical Advantages

Unified geometry–appearance latents demonstrate empirical superiority over decoupled representations across a range of benchmarks:

Fidelity, consistency, and speed: UniLat3D and O-Voxel (Xiang et al., 16 Dec 2025) achieve state-of-the-art FID, LPIPS, and Chamfer metrics on standard 3D datasets with significantly faster inference and tighter appearance–geometry alignment compared to two-stage pipelines.
Structural and visual congruence: DRF yields superior self-similarity (structure) and identity-cosine (appearance) scores, and in user studies is strongly preferred for text alignment and content preservation (Kim et al., 13 Aug 2025).
Dynamic and articulated assets: T-SLAT, ArtiLatent, and JGA-LBD surpass single-modality or sequential methods on dynamic scene, articulation-aware, and real-world human reconstruction metrics (FVD, 3D consistency, appearance, and geometry errors).
Ablation studies: Removing unified latent coupling reliably worsens geometry–appearance consistency, as evidenced by increased depth–normal inconsistency (Orchid), increased Chamfer Distance or PSNR drop (Gen3R (Huang et al., 7 Jan 2026)), and more severe multi-view incoherence (FantasyWorld (Dai et al., 25 Sep 2025)).

6. Challenges, Limitations, and Prospects

While unified geometry–appearance latents have enabled major advances, open challenges include:

Residual entanglement: Partial disentanglement may persist, as in face-identity drift (Sketch2Human), or failure of simple KL-alignment (Gen3R) to tightly couple distributions. Extensions to contrastive or adversarial alignment are a potential remedy (Huang et al., 7 Jan 2026).
Resolution vs. compression trade-offs: Increased latent compression (e.g., coarser $z\in\mathbb{R}^{H/8\times W/8\times 8}$ 2 grids, SLat/O-Voxel) may sacrifice high-frequency detail, while higher resolutions carry quadratic or cubic computational cost (Wu et al., 29 Sep 2025, Xiang et al., 16 Dec 2025).
Generalization to articulated and dynamic scenes: Current systems (ArtiLatent, MORPHOS) face limitations on kinematic complexity or require vast articulated training data.
Future extensions: Unified latents could benefit from spatial guidance (ControlNet-style (Qu et al., 2024)), end-to-end encoder–decoder–generator training, incorporation of richer semantic/material parameters, and integration with generalist video/3D foundation models. A plausible implication is that further advances in large unified latent architectures may support wider generalization to open-world, temporally complex, and multi-modal generative tasks.