LaviGen: 3D Layout Generation Framework

• LaviGen is a framework for 3D layout generation that uses autoregressive latent modeling and a 3D diffusion process to arrange objects with semantic and physical integrity.
• It introduces a novel dual-guidance distillation strategy that combines holistic and step-wise supervision to reduce errors and improve inference efficiency.
• The method outperforms previous approaches in collision-free placements and in-bound probabilities, making it ideal for applications in AR/VR, robotics, and simulation environments.

LaviGen is a framework for 3D layout generation that directly leverages pretrained 3D generative models to arrange sets of objects within a scene, ensuring both semantic coherence and physical plausibility. LaviGen departs from approaches that rely on serializing layouts as language or optimizing 2D projections; instead, it formulates scene synthesis in the native 3D latent space. The method implements an autoregressive process, augmented with an adapted 3D diffusion mechanism and a novel dual-guidance distillation strategy, significantly improving efficiency and quality relative to prior work (Feng et al., 17 Apr 2026).

1. Task Definition and Motivation

3D layout generation involves sequentially arranging a collection of 3D objects $\{O_1,\dots,O_N\}$ within a defined scene such that the resulting configuration is semantically appropriate (e.g., tables and chairs sensibly grouped) and physically valid (no collisions, objects contained within scene boundaries). Early methods such as ATISS used regression to predict 3D bounding box coordinates directly, disregarding higher-order geometric relationships, which led to frequent placement errors, overlaps, and penetration artifacts.

Subsequent methods, including LayoutGPT, Holodeck, and I-Design, reformulated the task as serializing scene layouts in formats such as JSON and employing LLMs. However, the lack of an explicit 3D representation introduced recurring issues with spatial artifacts such as object floating or inversion due to insufficient geometric priors. LayoutVLM introduced 2D optimization to reduce violations, but 2D supervision cannot enforce volumetric consistency and incurs high computational cost.

LaviGen instead harnesses a pretrained 3D generative model as a spatial prior, decomposing the layout process into a sequence of incremental object placements directly in 3D latent space. This autoregressive scheme supports scene editing and completion, and its probabilistic process is factorized as:

$$P(x_1,\dots,x_N) = \prod_{t=1}^N P(x_t \mid x_{<t}),$$

where each $x_t$ describes geometry, position, and orientation for the $t$-th object relative to the current scene context.

2. Core Architecture: Autoregressive 3D Latent Modeling

At each autoregressive step $i$, the current scene state $S_{i-1}$ is encoded via a 3D VAE into voxel-based latent $s \in \mathbb{R}^{N \times d}$, with $N$ voxels and latent width $d$. The target object $O_i$ is similarly encoded as $$P(x_1,\dots,x_N) = \prod_{t=1}^N P(x_t \mid x_{<t}),$$0. The core model learns a conditional diffusion process to predict the updated 3D latent corresponding to scene $$P(x_1,\dots,x_N) = \prod_{t=1}^N P(x_t \mid x_{<t}),$$1 after adding $$P(x_1,\dots,x_N) = \prod_{t=1}^N P(x_t \mid x_{<t}),$$2, formally:

$$P(x_1,\dots,x_N) = \prod_{t=1}^N P(x_t \mid x_{<t}),$$3

where $$P(x_1,\dots,x_N) = \prod_{t=1}^N P(x_t \mid x_{<t}),$$4 is the denoised latent and $$P(x_1,\dots,x_N) = \prod_{t=1}^N P(x_t \mid x_{<t}),$$5 encodes textual instructions provided by a LLM.

Post-denoising, the model decodes the latent difference $$P(x_1,\dots,x_N) = \prod_{t=1}^N P(x_t \mid x_{<t}),$$6, aligns it to the object $$P(x_1,\dots,x_N) = \prod_{t=1}^N P(x_t \mid x_{<t}),$$7 using Iterative Closest Point (ICP) registration, and extracts the necessary transformation parameters: translation $$P(x_1,\dots,x_N) = \prod_{t=1}^N P(x_t \mid x_{<t}),$$8, rotation $$P(x_1,\dots,x_N) = \prod_{t=1}^N P(x_t \mid x_{<t}),$$9, and scale $x_t$0.

3. Adapted 3D Diffusion Process and Conditioning

LaviGen adopts a continuous-time, flow-matching style 3D diffusion process, where latent state $x_t$1 is increasingly noised over $x_t$2 via

$x_t$3

The reverse denoising follows

$x_t$4

with loss

$x_t$5

Multimodal conditioning is realized by concatenating latent tokens $x_t$6 with identity-aware rotary positional embeddings (RoPE). Textual scene instructions $x_t$7 are integrated via cross-attention, and the network predicts a score field $x_t$8. This conditioning enables the model to express scene, object, and language cues jointly.

4. Dual-Guidance Self-Rollout Distillation

To address exposure bias common to autoregressive training, LaviGen introduces a two-phase distillation approach:

• Holistic guidance: The model's distribution over the final scene $x_t$9 is matched to a teacher's using distribution-matching distillation (DMD), with loss

$t$0

• Step-wise guidance: Per-step distributions are similarly regularized:

$t$1

Both terms are combined for training:

$t$2

Gradient computation is performed via critic networks, enabling few-step student rollouts and improving inference speed without loss of spatial accuracy.

5. Semantic and Physical Plausibility: Constraints and Embeddings

All object placements are governed by native 3D latent priors, which natively encode inter-object relations without the need for explicit collision loss. The identity-aware positional encoding prevents token inversion or floating, ensuring that newly inserted objects are consistently located and oriented relative to the existing scene.

Key physical correctness metrics include the collision-free (CF) probability and the in-bound (IB) probability:

$t$3

This approach maintains geometric fidelity implicitly via the latent space, as opposed to heuristically defined losses.

6. Training Protocols and Inference Pipeline

Training proceeds in three stages:

1. Base bidirectional model: 400,000 steps on the union of Objaverse-XL, ABO, 3D-FUTURE, and HSSD datasets;
2. Teacher fine-tuning: 100,000-step autoregressive finetuning phase;
3. Self-rollout distillation: 5,000 further steps with few-step sampling and critic guidance.

Evaluation is performed on LayoutVLM with scenes containing 8–10 objects. At inference, scene instructions are parsed using Qwen2.5-VL, yielding an ordered object list. Sequential object placement is performed using 50-step diffusion sampling with classifier-free guidance (CFG=3); transformations are decoded for mesh output.

7. Performance Evaluation and Component Analysis

LaviGen demonstrates a pronounced improvement in both physical validity and efficiency compared with prior models. The following table summarizes main comparison results for scenes with 8–10 objects:

Method	CF (↑)	IB (↑)	Pos (↑)	Rot (↑)	PSA (↑)	T(s) (↓)
LayoutGPT	83.8	24.2	80.8	78.0	16.6	21.3
Holodeck	77.8	8.1	62.8	55.6	5.6	58.2
I-Design	76.8	34.3	68.3	62.8	18.0	179.2
LayoutVLM	81.8	94.9	77.5	73.2	58.8	75.5
LaviGen (full)	97.3	98.6	76.9	77.1	78.8	24.3

LaviGen achieves a 15.5 percentage-point CF gain and a 3.7 percentage-point IB improvement over LayoutVLM, corresponding to approximately 19% higher physical plausibility. Inference time is reduced by 65%, with semantic alignment (Pos/Rot) remaining competitive. User studies (430 responses) show LaviGen preferred for physical plausibility (52.1% votes) and overall quality (55.6%) (Feng et al., 17 Apr 2026).

Ablation analyses show that identity-aware embeddings dramatically decrease collision rates, while holistic distillation improves speed. The addition of step-wise guidance corrects per-object errors, restoring both physical and semantic metrics.

8. Implications, Applications, and Limitations

LaviGen demonstrates that pretrained 3D generative models, coupled with autoregressive latent-space diffusion and dual-guidance distillation, set a new standard for efficient, high-fidelity 3D scene assembly. Immediate applications include synthetic indoor scene generation, robotics, AR/VR content creation, and simulation environments demanding strong spatial and physical guarantees. A plausible implication is that this methodology generalizes to layout editing and incremental scene construction due to its autoregressive and modular composition scheme. Limitations may include dependence on the coverage and diversity of the 3D generative prior and reliance on ICP registration accuracy during decomposition.

For further details and implementation code, see the official repository: https://github.com/fenghora/LaviGen (Feng et al., 17 Apr 2026).