UniLat3D: Unified 3D Asset Generation

• UniLat3D is a unified framework that combines geometry and appearance into a single latent representation, eliminating the traditional two-stage process.
• It leverages a unified variational autoencoder and a conditional flow-matching model to efficiently generate high-resolution 3D assets with precise texture-geometry alignment.
• Empirical evaluations show that UniLat3D achieves superior fidelity and reduced computational cost compared to conventional multi-stage 3D asset generation pipelines.

UniLat3D is a unified framework for high-fidelity 3D asset generation that integrates geometry and appearance into a single latent representation, enabling direct, single-stage synthesis from a single image prompt. As opposed to the standard two-stage pipelines common in recent 3D asset generators, UniLat3D eliminates the typical shape–appearance separation, which is known to cause geometry–texture misalignment and to incur redundant computational costs. Leveraging a geometry–appearance unified variational autoencoder (VAE) and a single conditional flow-matching model, UniLat3D achieves efficient, well-aligned, and high-quality 3D asset generation. The method is trained exclusively on publicly available datasets and is capable of producing diverse 3D representations, including both 3D Gaussians and meshes, at high resolution (Wu et al., 29 Sep 2025).

1. Motivation and Context

Traditional high-quality 3D generative models predominantly adopt a two-stage paradigm: first generating geometry primitives (such as voxels, meshes, or Gaussian splats), then synthesizing appearance (texture or color). While conceptually modular, this design introduces two persistent challenges: geometry–texture misalignment—manifests as seams, blurring at object boundaries, or spatial drift between structure and color—and computational inefficiency, since two separate generative models (typically diffusion or flow-based) are executed sequentially.

UniLat3D is motivated by the goal of fusing structural and visual information within a single latent, to enhance geometry–appearance consistency and to simplify inference by collapsing the generation pipeline to a single denoising process. This reflects a broader trend toward end-to-end, single-stage systems in computer vision and graphics.

2. Geometry–Appearance Unified VAE Model

At the core of UniLat3D is a unified VAE that compresses high-resolution sparse geometry–appearance features into a compact, dense latent—termed UniLat. This model operates as follows:

Encoder

• Sparse Visual Feature Extraction: Given a ground-truth 3D asset $\mathcal{O}$ (mesh or Gaussian splat), multi-view renderings are extracted and encoded to per-pixel features using a pretrained vision backbone (typically DINOv2 or DINOv3). These features are projected into 3D to yield a sparse set $\mathbf{f} = \{(f_i, p_i)\}_{i=1}^L$, where $p_i \in \mathbb{R}^3$ denotes occupied/surface voxels.
• Sparse Appearance Encoding: Sparse Transformer blocks (following TRELLIS) operate on these feature–position pairs to generate sparse appearance latents $$\mathbf{z}_{sparse} = \{(z_{\text{sparse},i}, p_i)\}_{i=1}^L$$.
• Sparse to Dense Densification: Unoccupied voxels are assigned zeros, producing a dense tensor $$\mathbf{z}_{dense} \in \mathbb{R}^{N\times N\times N\times c}$$.
• Dense Feature Compression: 3D convolutions downsample $\mathbf{z}_{dense}$ by a factor $s$, yielding $$\mathbf{z}_{uni} \in \mathbb{R}^{M\times M\times M\times d}$$, with $M = N/V$, tightly fusing geometry and appearance.

UniLat Properties

• Dimensionality: Typical $M = 16$ or $\mathbf{f} = \{(f_i, p_i)\}_{i=1}^L$0, $\mathbf{f} = \{(f_i, p_i)\}_{i=1}^L$1.
• Structure: Dense, low-resolution 3D grid, preserving locality for efficient processing in subsequent stages.

Decoder

• Upsampling and Sparsification: $\mathbf{f} = \{(f_i, p_i)\}_{i=1}^L$2 is upsampled via 3D transpose convolutions, yielding $\mathbf{f} = \{(f_i, p_i)\}_{i=1}^L$3 and occupancy logits $\mathbf{f} = \{(f_i, p_i)\}_{i=1}^L$4. Pruning to voxels where $\mathbf{f} = \{(f_i, p_i)\}_{i=1}^L$5 yields a sparse set $\mathbf{f} = \{(f_i, p_i)\}_{i=1}^L$6.
• 3D Format-Specific Decoding: For 3D Gaussians, sparse Transformers and linear heads predict primitive attributes (position, covariance, color, opacity). For meshes, an octree upsampling and sparse 3D convolution hierarchy predicts SDF, deformations, and interpolation weights, with extraction via SparseFlex and Poisson reconstruction.

VAE Objective

The loss function is:

$\mathbf{f} = \{(f_i, p_i)\}_{i=1}^L$7

The reconstruction term is further decomposed into color ($\mathbf{f} = \{(f_i, p_i)\}_{i=1}^L$8, LPIPS, SSIM), geometry (Dice), and regularization losses.

3. Single-Stage Flow-Matching Generation

UniLat3D replaces multi-stage denoising with a single rectified flow-matching model:

• Conditional Flow-Matching Model: A sparse 3D Transformer $\mathbf{f} = \{(f_i, p_i)\}_{i=1}^L$9 maps standard Gaussian noise $p_i \in \mathbb{R}^3$0 to UniLat, conditioned on an image $p_i \in \mathbb{R}^3$1 and time $p_i \in \mathbb{R}^3$2:

$p_i \in \mathbb{R}^3$3

The loss follows the conditional flow-matching formulation:

$p_i \in \mathbb{R}^3$4

where $p_i \in \mathbb{R}^3$5.

• Single Denoising Pass: As both shape and color are fused within $p_i \in \mathbb{R}^3$6, a single flow model suffices for asset generation; no additional appearance denoiser or separate refinement stage is required.

4. Training Data and Optimization Regimes

• Training Datasets: 450,000 assets from Objaverse (XL), ABO, 3D-FUTURE, and HSSD; evaluation on Toys4K and a 1000-instance “complex” set (500 Sketchfab, 500 Toys4K).
• VAE Training: Initial weights drawn from TRELLIS encoder/decoder. Training progresses in stages—first dense encoder and upsampling decoder, then full model, with further mesh decoder specialization for high-resolution output.
• Optimizer: Adam (learning rate $p_i \in \mathbb{R}^3$7, batch size $p_i \in \mathbb{R}^3$8).
• Flow Model Architecture: 36 sparse Transformer blocks, $p_i \in \mathbb{R}^3$91.3B parameters, $$\mathbf{z}_{sparse} = \{(z_{\text{sparse},i}, p_i)\}_{i=1}^L$$0 latent resolution, channels 32, DINOv2/DINOv3 for conditioning.
• Training Schedule: 500K iterations (batch 256, lr $$\mathbf{z}_{sparse} = \{(z_{\text{sparse},i}, p_i)\}_{i=1}^L$$1), fine-tune for 160K (batch 1024, lr $$\mathbf{z}_{sparse} = \{(z_{\text{sparse},i}, p_i)\}_{i=1}^L$$2).
• Data Augmentation: Depth and normal rendering for fine mesh detail; multi-scale occupancy supervision using octree subdivision.

5. Empirical Evaluation and Ablation

Quantitative Metrics

• Appearance Fidelity: CLIP score between rendered samples and condition image.
• Distributional Similarity: Fréchet Distance on DINOv2 embeddings ($$\mathbf{z}_{sparse} = \{(z_{\text{sparse},i}, p_i)\}_{i=1}^L$$3).
• Geometry Quality: ULIP and Uni3D (scaled by 100).
• Efficiency: Single A100 GPU generation time—8 seconds (3D Gaussians), 36 seconds (meshes).

Method	CLIP (3DGS)	FD$$\mathbf{z}_{sparse} = \{(z_{\text{sparse},i}, p_i)\}_{i=1}^L$$4 (3DGS)	Geometry (Mesh)
UniLat3D	90.87	47.68	Matches/surpasses prior SOTA
TRELLIS	90.70	52.54	Lower

• UniLat3D matches or slightly surpasses previous two-stage models (TRELLIS, Step1X-3D, Hunyuan3D-2.1) in mesh metrics on ULIP and Uni3D.

Qualitative and User Study Outcomes

• Generated assets span toys, furniture, and complex objects with high-fidelity alignment between texture and structure.
• Meshes at $$\mathbf{z}_{sparse} = \{(z_{\text{sparse},i}, p_i)\}_{i=1}^L$$5 resolution exhibit crisp boundaries and well-registered color.
• In a 23-prompt/19-participant study, UniLat3D received more than 35% of user preference votes, outperforming Hunyuan3D-2.1 and comparable models.

Ablation Findings

• Increasing UniLat resolution ($$\mathbf{z}_{sparse} = \{(z_{\text{sparse},i}, p_i)\}_{i=1}^L$$6) yields monotonic improvement in VAE PSNR and LPIPS.
• Replacing DINOv2 with DINOv3 as the conditioning encoder decreases FD$$\mathbf{z}_{sparse} = \{(z_{\text{sparse},i}, p_i)\}_{i=1}^L$$7 by approximately 2.7 on complex objects.

6. Strengths, Limitations, and Prospects

Advantages

• Single-stage 3D generation with a unified flow-matching model.
• Intrinsic geometry–texture alignment due to joint latent encoding.
• Comparable or superior fidelity to state-of-the-art multi-stage pipelines, with reduced computation.
• The UniLat prior is extensible to future multimodal, temporal (4D), or large-scale vision-language-3D applications.

Limitations

• High-resolution single-stage flows (beyond $$\mathbf{z}_{sparse} = \{(z_{\text{sparse},i}, p_i)\}_{i=1}^L$$8) remain computationally expensive; scaling and efficient sparse attention mechanisms are active research directions.
• The training set is restricted to public asset datasets; broader asset diversity could improve generality.
• Mesh generation is still relatively slow (∼36 s); further acceleration in pruning and sparse operations is desirable.

Future Directions

• Scaling UniLat resolution and devising lighter-weight flow architectures.
• Extending the unified latent principle to dynamic (4D) asset generation.
• Embedding UniLat as a foundational component in large vision-language-3D models for jointly learned understanding and synthesis.

UniLat3D provides empirical evidence that a geometry–appearance unified latent representation can robustly supplant traditional two-stage approaches, achieving fast, well-aligned, and high-quality 3D asset generation via a single denoising pass (Wu et al., 29 Sep 2025).