---
title: 'DistScene Framework: 3D Scene Generation'
url: https://www.emergentmind.com/topics/distscene
type: topic
---

# DistScene Framework: 3D Scene Generation

DistScene is a framework for single-image compositional 3D scene generation that jointly models an explicit environment and multiple independently addressable objects. Given one RGB image, it generates a decomposable scene in which the environment, object geometry, spatial arrangement, and object–environment relationships are represented in a shared coordinate frame. Its principal components are Scene-Frame Generation, Object-Centric Refinement, and Object-to-Scene Distillation. The method uses sparse voxel representations, structured latent features, image-conditioned flow-matching transformers, and automatically composed synthetic training scenes [2610.06960].

## 1. Problem formulation and design objectives

Given an RGB image $I$, DistScene aims to infer a decomposable 3D scene

$$
S'=\{\widehat O_1,\ldots,\widehat O_N,\widehat E\},
$$

where $\widehat E$ is an environment component and each $\widehat O_i$ is an independently recoverable object. Every object is assigned a scene-space position, scale, and orientation. The generated scene is intended to preserve spatial coherence, support object-level editing and rendering, and provide sufficiently detailed geometry for downstream simulation or interaction.

Single-image 3D generation is intrinsically underdetermined. The image does not uniquely specify the number or identity of objects, their occluded geometry, metric depth and scale, object poses, object–object relationships, or support relationships with floors, walls, roads, and other environmental surfaces. DistScene treats these ambiguities as a joint generative problem rather than as a sequence of independently estimated quantities.

The framework is motivated by limitations in three classes of prior systems. Modular reconstruction methods such as SAM3D, 3D-Fixer, and CAST commonly perform segmentation or detection, depth and pose estimation, object reconstruction, and scene composition in sequence. Errors in earlier stages propagate to later stages, and independently predicted coordinate decisions may be mutually inconsistent. Object-centric feed-forward systems such as MIDI and SceneGen jointly generate multiple objects but generally do not model the surrounding environment as an explicit geometric component. Single-object generators such as TRELLIS.2, TRELLIS, Hunyuan3D, and Craftsman3D provide canonical object geometry but do not directly determine scene placement, relative scale, support, or interaction with environmental geometry.

DistScene addresses these limitations through the decomposition

$$
S'=\mathcal{R}\bigl(\mathcal{G}(I)\bigr),
$$

where $\mathcal{G}$ denotes Scene-Frame Generation and $\mathcal{R}$ denotes object-centric refinement. The intermediate scene is

$$
S=\mathcal{G}(I)=\{O_1,\ldots,O_N,E\}.
$$

The environment is therefore not treated as an image-only background. It is generated as a spatially represented 3D component that participates in the same scene-level modeling process as the objects.

## 2. Sparse voxel and structured-latent representation

DistScene uses sparse voxels as the common representation for objects and environments. A component consists of active voxel coordinates, learned geometry features, and learned appearance or material features. The active coordinates form the component’s support, which captures its coarse spatial occupancy and extent but does not contain all geometric or appearance information.

For component $i$, the sparse support is decoded from a structure latent $H_i$:

$$
S_i=\mathcal{D}_{\mathrm{SS}}(H_i),
$$

where $\mathcal{D}_{\mathrm{SS}}$ is the sparse-structure decoder. Geometry and appearance use structured latent representations:

$$
M_i=\mathcal{D}_{\mathrm{shape}}(S_i,Z_i^{\mathrm{shape}}),
$$

$$
(M_i,\mathcal{A}_i)=
\mathcal{D}_{\mathrm{tex}}
(S_i,Z_i^{\mathrm{shape}},Z_i^{\mathrm{tex}}).
$$

Here, $M_i$ denotes decoded geometry or mesh and $\mathcal{A}_i$ denotes appearance or material. The per-voxel latent features are referred to as Structured Latents, or SLATs.

During Scene-Frame Generation, all components are represented in a common scene coordinate system:

$$
\{S_i^{\mathrm{sf}},Z_i^{\mathrm{sf}}\}_{i=0}^{N},
$$

with one index corresponding to the environment and the remaining indices corresponding to objects. Scene-frame generation consequently encodes object translation, scale, orientation, spatial extent, proximity, contact, and relationships with the environment directly in the generated representation.

The underlying single-asset formulation represents an asset as

$$
\mathbf f =
\left\{
\left(f_i^{\mathrm{shape}},f_i^{\mathrm{mat}},p_i\right)
\right\}_{i=1}^{L},
$$

where $L$ is the number of active voxels, $f_i^{\mathrm{shape}}$ and $f_i^{\mathrm{mat}}$ are shape and material features, and $p_i$ is a voxel coordinate. A sparse VAE encodes this representation as $z=\mathcal{E}(\mathbf f)$, while an image-conditioned flow-matching generator produces $\widehat z$ and a sparse decoder reconstructs the asset.

DistScene extends this formulation from one asset to multiple objects and an environment. Sparse supports provide coarse spatial structure, while SLATs encode more detailed geometry and appearance. This separation permits global placement to be modeled at scene scale and individual object detail to be recovered at higher effective local resolution.

## 3. Scene-Frame Generation

Scene-Frame Generation jointly produces the environment and objects in a shared scene frame. For an input image, DistScene initializes one latent for each object and one latent for the environment:

$$
\{z_i\}_{i=1}^{N+1}.
$$

The latent sequences are flattened and concatenated:

$$
Z=[z_1;z_2;\ldots;z_{N+1}].
$$

Learnable component or type embeddings identify object slots and the environment slot. A sparse diffusion or flow transformer, implemented as a sparse DiT, performs self-attention over the concatenated sequence. Objects can attend to other objects, objects can attend to the environment, and the environment can attend to the objects. The environment latent consequently functions as a coupling component for scene-level coordination.

The supplementary formulation separates scene generation into joint sparse-structure generation and scene-frame geometry generation. Starting from noise, the structure transformer generates

$$
\{H_i\}_{i=0}^{N},
$$

which are decoded into scene-frame supports:

$$
S_i^{\mathrm{sf}}=\mathcal{D}_{\mathrm{SS}}(H_i).
$$

A second sparse flow transformer then generates scene-frame shape SLATs conditioned on the image and generated supports:

$$
\{Z_i^{\mathrm{sf}}\}_{i=0}^{N}
=
\mathcal{F}_{\mathrm{SLAT}}
\left(I,\{S_i^{\mathrm{sf}}\}_{i=0}^{N}\right).
$$

The scene-frame geometries are decoded as

$$
M_i^{\mathrm{sf}}
=
\mathcal{D}_{\mathrm{SLAT}}
\left(S_i^{\mathrm{sf}},Z_i^{\mathrm{sf}}\right).
$$

The environment and objects remain separate components, but their coordinates are already expressed in the same scene frame. This avoids requiring an external placement module after independent object generation.

DistScene supports a fixed maximum number of components. Training scenes are capped at 30 components, including the environment, and are randomly padded with empty object slots. Empty supports are discarded during inference, which allows scenes with different numbers of visible objects to use a fixed transformer structure. Inference uses 20 component slots by default.

The scene generator is adapted using LoRA while the pretrained sparse transformer backbone remains frozen. For component $i$, with noise latent $z_0^i$, target latent $z_1^i$, interpolated latent $z_t^i$, and predicted velocity $v_\theta$, the scene flow-matching objective is

$$
\mathcal{L}_{\mathrm{scene}}
=
\sum_{i=1}^{N+1}
\mathbb{E}_{t,z_0^i,z_1^i}
\left[
\left\|
v_\theta(z_t^i,t)-(z_1^i-z_0^i)
\right\|^2
\right].
$$

The objective is applied across object and environment components. Although the compact expression does not explicitly display the image condition, the architecture is image-conditioned and processes the component sequence jointly.

## 4. Object-Centric Refinement

Scene-frame generation provides global placement and relationships, but an object occupying a small part of a scene-scale sparse voxel grid may receive insufficient spatial resolution. Object-Centric Refinement addresses this problem by transforming each object into a normalized local frame, refining it with scene context, and mapping it back to its original scene position.

For object $O_i$, the procedure extracts the object from the scene-frame representation, estimates its scene-space extent, re-centers it, applies uniform scaling, and voxelizes it in a local coordinate frame. If $T_i$ denotes the scene-to-local transformation,

$$
S_i^{\mathrm{loc}}
=
\operatorname{Voxelize}
\left(T_i(M_i^{\mathrm{sf}})\right).
$$

The local transform preserves object orientation while increasing its effective spatial resolution. After refinement, the geometry is transformed back into the scene frame:

$$
\widehat M_i
=
T_i^{-1}
\left[
\mathcal{D}_{\mathrm{SLAT}}
\left(S_i^{\mathrm{loc}},\widehat Z_i^{\mathrm{loc}}\right)
\right].
$$

Refinement is not independent object super-resolution. The refinement model receives the input image, all scene-frame latents, the local support of the target object, and a learnable marker embedding $e_i$ identifying the corresponding scene-frame component:

$$
\widehat O_i
=
\mathcal{R}
\left(
O_i\mid I,\{x_j\}_{j=1}^{N+1},e_i
\right).
$$

The marker establishes correspondence between the local object being refined and its global scene-frame token. This is relevant when several objects have similar categories or appearances.

In the more detailed formulation, the refined local latent is

$$
\widehat Z_i^{\mathrm{loc}}
=
\mathcal{F}_{\mathrm{ref}}
\left(
I,\{Z_j^{\mathrm{sf}}\}_{j=0}^{N},
S_i^{\mathrm{loc}},e_i
\right).
$$

The final decomposable scene is

$$
S'=\{\widehat O_1,\ldots,\widehat O_N,E\}.
$$

The environment remains in its original scene frame, while each refined object replaces its coarse scene-frame geometry at the same location. Refinement therefore combines global scene consistency with object-level geometric detail.

Refinement training uses self-supervised coarse-to-fine pairs. A high-quality object latent $z_1$ is downsampled, passed through the sparse VAE encoder and decoder, and upsampled or voxelized to obtain a degraded latent $z_0$. The refinement model learns to recover $z_1$ from $z_0$ using

$$
\mathcal{L}_{\mathrm{refine}}
=
\mathbb{E}_{t,z_0,z_1}
\left[
\left\|
v_\phi
\left(
z_t,t\mid I,\{x_j\}_{j=1}^{N+1},e
\right)
-
(z_1-z_0)
\right\|^2
\right].
$$

This training procedure does not require manually annotated high-resolution refinement pairs.

## 5. Object-to-Scene Distillation and synthetic data

Scene-Frame Generation requires decomposed scenes containing an environment, multiple objects, scene-frame transformations, rendered conditioning images, and geometry and appearance targets for every component. Existing scene datasets are limited in scale and object diversity and may not provide all of these aligned signals. DistScene therefore constructs a synthetic corpus from a pretrained object generator through Object-to-Scene Distillation.

A language model generates structured descriptions for an environment and its objects, including approximate object sizes and composition information. A pretrained object generator, instantiated as TRELLIS.2, generates the environment and objects:

$$
E=\mathcal{G}(d_E),
$$

$$
\{O_i\}_{i=1}^{N}
=
\{\mathcal{G}(d_i)\}_{i=1}^{N}.
$$

The same generator produces the environment and objects, making the environment a geometrically detailed 3D asset rather than an empty background.

An adjustment procedure places the objects:

$$
\{T_i\}_{i=1}^{N}
=
\mathcal{A}\bigl(\{O_i\},E\bigr),
$$

subject to a physical plausibility constraint

$$
\mathcal{C}(S)=1.
$$

The composed scene is

$$
S=\{(O_i,T_i)\}_{i=1}^{N}\cup\{E\}.
$$

The physical plausibility procedure estimates valid floor regions and support surfaces, drops objects onto valid supports, rejects floating or penetrating placements, performs axis-aligned bounding-box intersection tests, inspects overlapping mesh regions, adjusts positions when possible, rejects unresolved object–object intersections, and checks collisions against non-floor environment surfaces.

For each accepted scene, multiple camera views are rendered. Views are retained when all objects are visible. The resulting training data contain rendered conditioning images, scene-frame geometry and structure targets, scene-frame object and environment latents, canonical object latents, component transforms, local object supports, and shape and texture targets.

The reported synthetic corpus contains approximately:

- $125{,}000$ synthetic training scenes;
- $70{,}000$ generated objects;
- $46{,}000$ empty environments;
- $106{,}000$ indoor scenes;
- $18{,}000$ outdoor scenes.

The generated data are rendered at resolution $512$. The model is trained at this resolution and supports $1024$-resolution inference.

Object-to-Scene Distillation transfers object-generation priors into scene-level generation without requiring manually annotated decomposable scene data. The ablation comparing distilled data with the MIDI training set indicates stronger cross-domain scene-level generalization for the distilled-data model: on Gen3DSR-test, scene CD decreases from $0.1936$ to $0.1367$ and scene F-score increases from $46.68$ to $62.04$.

## 6. Training, inference, evaluation, and limitations

### Training and inference

DistScene uses TRELLIS.2 as its pretrained object generator, LoRA adapters of rank $32$, frozen pretrained transformer backbones, AdamW optimization, learning rate $10^{-4}$, global batch size $64$, and $20{,}000$ iterations per adapted module. Structure, scene-frame geometry, refinement, and texture branches are trained with flow-matching objectives.

At inference, the system initializes $N+1$ noise latents, jointly samples sparse-structure latents, decodes scene-frame supports, generates scene-frame shape SLATs, and decodes the environment and coarse object geometries. Each object is then transformed into a local frame, refined using the image and all scene-frame latents, decoded, and transformed back. Separate texture branches generate environment and object appearance. The final output is

$$
S'=\{\widehat E,\widehat O_1,\ldots,\widehat O_N\}.
$$

### Benchmarks and metrics

Indoor evaluation uses MIDI-test and Gen3DSR-test. For MIDI-test, the authors re-rendered scenes with floor, walls, and objects because the original inputs did not contain the complete background required by DistScene. They also rendered aligned masks and depth maps for methods requiring such inputs.

Reported indoor metrics include scene-level Chamfer distance $CD_{\mathrm S}$, scene-level F-score $FS_{\mathrm S}$, object-level Chamfer distance $CD_{\mathrm O}$, object-level F-score $FS_{\mathrm O}$, and scene-coordinate bounding-box IoU. The F-score threshold is $0.1$ and is reported in percent. Outdoor evaluation uses UrbanScene3D with CD-L1 and F-score at threshold $0.05$.

On UrbanScene3D, DistScene reports CD-L1 $0.0772$ and F-score $0.722$. On MIDI-test, it reports scene CD $0.0877$, scene F-score $71.59$, object CD $0.1529$, object F-score $51.64$, and box IoU $0.3688$. On Gen3DSR-test, it reports scene CD $0.0958$ and scene F-score $79.68$.

The principal ablations support the framework’s three design choices. Adding an explicit environment improves scene-level geometry and object-level metrics. Object-centric refinement improves object detail and scene quality. Removing scene context from refinement degrades scene CD, object CD, bounding-box IoU, and scene F-score, indicating that refinement should not be treated as an isolated object super-resolution operation.

A user study with 43 participants and 20 randomly selected text-to-image-generated input images compared five anonymized methods. DistScene received preference rates of $61.95\%$ for image consistency, $67.80\%$ for geometry quality, and $66.46\%$ for appearance quality, with an average preference rate of $65.41\%$.

### Limitations

DistScene’s indoor synthetic environments may omit ceilings because the pretrained object generator has difficulty producing fully enclosed rooms with reliable ceilings. The resulting model may therefore learn an open-top room prior and omit ceilings even when the input depicts a closed room.

Large outdoor scenes are more difficult because a finite number of scene-frame voxels must cover a larger spatial extent. This reduces local geometric resolution and can produce holes, missing fine structures, and incomplete environment geometry. The outdoor training set is also smaller than the indoor set, with approximately $18{,}000$ outdoor scenes compared with $106{,}000$ indoor scenes.

Inference is computationally expensive. Scene generation at $1024$ resolution takes approximately $36.4$ seconds, and refinement takes approximately $38.5$ seconds per object, with peak memory of approximately $21.8$ GB. Since objects are refined independently, scenes with many objects can have high total inference time. The $512$-resolution scene-plus-refinement configuration provides a more favorable efficiency–quality trade-off.

The framework also depends on the quality and biases of language-model descriptions, text-to-image conditioning images, the pretrained object generator, generated empty environments, and procedural placement and collision checks. Synthetic compositions may not represent all real-world arrangements, materials, visibility patterns, or unusual geometries.

Finally, single-image generation cannot guarantee unique recovery of occluded geometry, exact depth, metric scale, or hidden object structure. DistScene generates a plausible scene consistent with the visible image and learned priors rather than claiming metric-perfect reconstruction in every case. Its central contribution is the integration of explicit environment generation, shared scene-frame modeling, context-aware local refinement, and object-to-scene distillation into a decomposable 3D scene-generation framework.

Source: https://www.emergentmind.com/topics/distscene