---
title: 'LaviGen: 3D Layout Generation Framework'
url: https://www.emergentmind.com/topics/lavigen
type: topic
---

# LaviGen: 3D Layout Generation Framework

LaviGen is a framework for 3D layout generation that directly leverages pretrained 3D generative models to arrange sets of objects within a scene, ensuring both semantic coherence and physical plausibility. LaviGen departs from approaches that rely on serializing layouts as language or optimizing 2D projections; instead, it formulates scene synthesis in the native 3D latent space. The method implements an autoregressive process, augmented with an adapted 3D diffusion mechanism and a novel dual-guidance distillation strategy, significantly improving efficiency and quality relative to prior work [2604.16299].

## 1. Task Definition and Motivation

3D layout generation involves sequentially arranging a collection of 3D objects $\{O_1,\dots,O_N\}$ within a defined scene such that the resulting configuration is semantically appropriate (e.g., tables and chairs sensibly grouped) and physically valid (no collisions, objects contained within scene boundaries). Early methods such as ATISS used regression to predict 3D bounding box coordinates directly, disregarding higher-order geometric relationships, which led to frequent placement errors, overlaps, and penetration artifacts.

Subsequent methods, including LayoutGPT, Holodeck, and I-Design, reformulated the task as serializing scene layouts in formats such as JSON and employing large language models (LLMs). However, the lack of an explicit 3D representation introduced recurring issues with spatial artifacts such as object floating or inversion due to insufficient geometric priors. LayoutVLM introduced 2D optimization to reduce violations, but 2D supervision cannot enforce volumetric consistency and incurs high computational cost.

LaviGen instead harnesses a pretrained 3D generative model as a spatial prior, decomposing the layout process into a sequence of incremental object placements directly in 3D latent space. This autoregressive scheme supports scene editing and completion, and its probabilistic process is factorized as:
$$
P(x_1,\dots,x_N) = \prod_{t=1}^N P(x_t \mid x_{<t}),
$$
where each $x_t$ describes geometry, position, and orientation for the $t$-th object relative to the current scene context.

## 2. Core Architecture: Autoregressive 3D Latent Modeling

At each autoregressive step $i$, the current scene state $S_{i-1}$ is encoded via a 3D VAE into voxel-based latent $s \in \mathbb{R}^{N \times d}$, with $N$ voxels and latent width $d$. The target object $O_i$ is similarly encoded as $o \in \mathbb{R}^{N \times d}$. The core model learns a conditional diffusion process to predict the updated 3D latent corresponding to scene $S_i$ after adding $O_i$, formally:
$$
p_\theta(S_i \mid S_{i-1}, O_i, c) \approx p_\theta(x_0 \mid s, o, c),
$$
where $x_0$ is the denoised latent and $c$ encodes textual instructions provided by a language model.

Post-denoising, the model decodes the latent difference $x_0 - s$, aligns it to the object $O_i$ using Iterative Closest Point (ICP) registration, and extracts the necessary transformation parameters: translation $t_i \in \mathbb{R}^3$, rotation $R_i \in SO(3)$, and scale $s_i > 0$.

## 3. Adapted 3D Diffusion Process and Conditioning

LaviGen adopts a continuous-time, flow-matching style 3D diffusion process, where latent state $x(t)$ is increasingly noised over $t \in [0, 1]$ via
$$
x(t) = (1-t) x_0 + t \epsilon,\quad \epsilon \sim \mathcal{N}(0, I).
$$
The reverse denoising follows
$$
\frac{d}{dt} x = v(x, t),
$$
with loss
$$
\mathcal{L}_{\mathrm{flow}} = \mathbb{E}_{x_0, \epsilon, t} \Vert v_\theta(x, t) - (\epsilon - x_0) \Vert_2^2.
$$

Multimodal conditioning is realized by concatenating latent tokens $[x, s, o]$ with identity-aware rotary positional embeddings (RoPE). Textual scene instructions $c$ are integrated via cross-attention, and the network predicts a score field $v_\theta(x; s, o, c, t)$. This conditioning enables the model to express scene, object, and language cues jointly.

## 4. Dual-Guidance Self-Rollout Distillation

To address exposure bias common to autoregressive training, LaviGen introduces a two-phase distillation approach:

- **Holistic guidance:** The model's distribution over the final scene $S_N$ is matched to a teacher's using distribution-matching distillation (DMD), with loss
  $$
  \mathcal{L}_{\mathrm{holistic}} = \mathrm{DMD}( p_\theta(S_N \mid \mathcal{C})\,\|\, p_{\mathcal{T}_S}(S_N \mid c) ).
  $$
- **Step-wise guidance:** Per-step distributions are similarly regularized:
  $$
  \mathcal{L}_{\mathrm{step}} = \sum_{i=1}^N \mathrm{DMD}( p_\theta(S_i \mid \mathcal{C}_i)\,\|\, p_{\mathcal{T}_P}(S_i \mid \mathcal{C}_i) ).
  $$
Both terms are combined for training:
$$
\mathcal{L}_{\mathrm{dual}} = \mathcal{L}_{\mathrm{holistic}} + \mathcal{L}_{\mathrm{step}}.
$$

Gradient computation is performed via critic networks, enabling few-step student rollouts and improving inference speed without loss of spatial accuracy.

## 5. Semantic and Physical Plausibility: Constraints and Embeddings

All object placements are governed by native 3D latent priors, which natively encode inter-object relations without the need for explicit collision loss. The identity-aware positional encoding prevents token inversion or floating, ensuring that newly inserted objects are consistently located and oriented relative to the existing scene.

Key physical correctness metrics include the collision-free (CF) probability and the in-bound (IB) probability:
$$
\mathrm{CF} = \Pr[\text{no pairwise collision}],\quad
\mathrm{IB} = \Pr[\text{all objects within scene boundary}].
$$
This approach maintains geometric fidelity implicitly via the latent space, as opposed to heuristically defined losses.

## 6. Training Protocols and Inference Pipeline

Training proceeds in three stages:
1. **Base bidirectional model:** 400,000 steps on the union of Objaverse-XL, ABO, 3D-FUTURE, and HSSD datasets;
2. **Teacher fine-tuning:** 100,000-step autoregressive finetuning phase;
3. **Self-rollout distillation:** 5,000 further steps with few-step sampling and critic guidance.

Evaluation is performed on LayoutVLM with scenes containing 8–10 objects. At inference, scene instructions are parsed using Qwen2.5-VL, yielding an ordered object list. Sequential object placement is performed using 50-step diffusion sampling with classifier-free guidance (CFG=3); transformations are decoded for mesh output.

## 7. Performance Evaluation and Component Analysis

LaviGen demonstrates a pronounced improvement in both physical validity and efficiency compared with prior models. The following table summarizes main comparison results for scenes with 8–10 objects:

| Method         | CF (↑) | IB (↑) | Pos (↑) | Rot (↑) | PSA (↑) | T(s) (↓) |
|----------------|--------|--------|---------|---------|---------|----------|
| LayoutGPT      | 83.8   | 24.2   | 80.8    | 78.0    | 16.6    | 21.3     |
| Holodeck       | 77.8   |  8.1   | 62.8    | 55.6    |  5.6    | 58.2     |
| I-Design       | 76.8   | 34.3   | 68.3    | 62.8    | 18.0    | 179.2    |
| LayoutVLM      | 81.8   | 94.9   | 77.5    | 73.2    | 58.8    | 75.5     |
| LaviGen (full) | 97.3   | 98.6   | 76.9    | 77.1    | 78.8    | 24.3     |

LaviGen achieves a 15.5 percentage-point CF gain and a 3.7 percentage-point IB improvement over LayoutVLM, corresponding to approximately 19% higher physical plausibility. Inference time is reduced by 65%, with semantic alignment (Pos/Rot) remaining competitive. User studies (430 responses) show LaviGen preferred for physical plausibility (52.1% votes) and overall quality (55.6%) [2604.16299].

Ablation analyses show that identity-aware embeddings dramatically decrease collision rates, while holistic distillation improves speed. The addition of step-wise guidance corrects per-object errors, restoring both physical and semantic metrics.

## 8. Implications, Applications, and Limitations

LaviGen demonstrates that pretrained 3D generative models, coupled with autoregressive latent-space diffusion and dual-guidance distillation, set a new standard for efficient, high-fidelity 3D scene assembly. Immediate applications include synthetic indoor scene generation, robotics, AR/VR content creation, and simulation environments demanding strong spatial and physical guarantees. A plausible implication is that this methodology generalizes to layout editing and incremental scene construction due to its autoregressive and modular composition scheme. Limitations may include dependence on the coverage and diversity of the 3D generative prior and reliance on ICP registration accuracy during decomposition.

For further details and implementation code, see the official repository: https://github.com/fenghora/LaviGen [2604.16299].

Source: https://www.emergentmind.com/topics/lavigen