LLM-Driven Scene Completion

• LLM-driven scene completion is a method that uses large language models to generate, complete, and edit 3D scenes from partial inputs and multimodal cues.
• It employs diverse architectures—layout programmatic, patchwise encoding, and prompt-to-inpainting—to enable precise spatial reasoning and semantic scene synthesis.
• Robust pipelines integrate iterative blueprint refinement, diffusion-based optimization, and physics-driven editing to ensure coherent semantic and physical scene integrity.

LLM-driven scene completion refers to the use of LLMs as the central agent for inferring, generating, and editing complex 3D environments—either from partial inputs, multimodal cues, or explicit human instructions. These approaches exploit LLMs’ capabilities for spatial reasoning, symbolic scene representation, and multimodal fusion. They represent a decisive shift from tradition diffusion-based, vision-only, or template-based scene synthesis, enabling semantic completion guided by natural language and hierarchical geometric constraints.

1. Foundational Methodologies

LLM-driven scene completion systems generally follow one of three key architectural philosophies, as summarized in Table 1:

Approach	Input Representation	LLM Usage
Layout programmatic	Text (prompt, partial objects)	Blueprint synthesis
Patchwise encoding	Voxel grids, patches + text	Contextualized reasoning
Prompt-to-inpainting	Monocular image, VLM prompt	Guidance for diffusion

SceneLCM (Lin et al., 8 Jun 2025) exemplifies the layout programmatic class, wherein LLMs (e.g., GPT-4) receive a free-form textual prompt describing the desired space. The model generates a parametric blueprint as JSON, denoting floorplan vertices, furniture placements $(p,s,c,\text{name})$, and initial orientations. This is iteratively refined via feedback loops that detect and correct invalid (overlapping, out-of-bounds) configurations, with LLM dialogue handling programmatic reasoning over 3D spatial relations.

VP-LLM (Liu et al., 2024) employs patch-wise encoding, decomposing incomplete voxelized objects into independent $8\times8\times8$ patches. Each patch is VAE-encoded, projected into LLM token space, and contextualized within the prompt-augmented sequence. This allows the LLM to “speak” both text and geometry, supporting single-forward-pass volume completion.

FlashDreamer (Song et al., 2 Mar 2025) leverages LLMs (and general VLMs) for prompt generation, steering view synthesis in a latent diffusion inpainting loop that incrementally reconstructs multi-view scenes from monocular input.

2. Input Encoding and Prompt Design

Robust scene completion requires careful encoding of both geometric and semantic input. SceneLCM formulates scene blueprints in JSON, with objects described by their center, size, class, and name. Iterative message passing between LLM and automated validation ensures spatial coherence, eliminating geometric anomalies.

IL3D (Zhou et al., 14 Oct 2025) advances this domain by providing a richly annotated dataset: 27,816 layouts, 18 room types, and 29,215 assets, each annotated with instance-level natural language (e.g., “Armchair, curved backrest, dark green upholstery, 0.5m east of sofa”). Scene graphs encode positional, rotational, and categorical data in USDZ/USDA formats and JSON. Completion prompts typically specify the room type and a partial object list, requesting the LLM to output a functional, non-overlapping layout (see Fig. A4–A5 in (Zhou et al., 14 Oct 2025)).

VP-LLM: For 3D volume completion, partial occupancy grids are patchified and VAE-encoded. The text prompt (e.g., “Recover the missing airplane wing”) is concatenated with patch tokens in the LLM context, enabling linguistic steering of geometric reasoning and object completion.

FlashDreamer: A single image is mapped to a descriptive prompt via a pretrained VLM. Varying prompt length (short: “sofa, window”; long: material, orientation, object lists) affects completion fidelity—short prompts yield strong structure, long prompts increase detail but may induce hallucinations.

3. Completion and Optimization Pipelines

SceneLCM establishes a modular four-stage pipeline:

1. Layout Generation: LLM converts text into parametric 3D blueprint; iterative programmatic validation corrects overlaps and invalid placements.
2. Furniture Generation: Consistency Trajectory Sampling (CTS) loss with Latent Consistency Model (LCM) distills a diffusion prior into fast, high-quality 3D object instantiation. Key theoretical results prove that CTS loss and standard consistency loss are equivalent up to second-order local truncation errors. The distillation error is bounded by the Euler solver’s step size.
3. Environment Optimization: Multi-resolution texture fields for planar surfaces are encoded and optimized via CTS. Normal-aware cross-attention ensures texture coherence across geometrically heterogeneous surfaces: attention is restricted to reference anchors sharing surface normals.
4. Physically Editing: Physics simulation (e.g., Blender rigid-body engine) updates scene graphs for user edits (add, remove, move, rotate). The system maintains persistent physical realism post-editing.

VP-LLM implements fully parallel completion: At inference, patchified partial input is encoded and projected into the LLM; a single forward pass yields completed patch latents, which are decoded and reassembled. No cross-attention or recurrent modules are necessary.

FlashDreamer’s completion is iterative. At each step, an incomplete rendered view (via Gaussian splatting) is inpainted via diffusion conditioned on the VLM prompt. The outputs are fused incrementally via visibility masking and reconstructed with pixel-wise alignment losses. All components operate off-the-shelf without retraining.

4. Datasets, Training Protocols, and Benchmarks

The design and scale of datasets is critical. IL3D (Zhou et al., 14 Oct 2025) delivers hierarchical diversity in scene types and object assets, with high-fidelity geometric and semantic annotations:

• Multi-format exports: 3D bounding boxes, semantic point clouds, multiview images, depth/normal maps.
• Natural language annotations: Instance-level Qwen3-VL generated descriptions, explicitly specifying materials, positions, weight.

LLMs are trained via supervised fine-tuning (SFT), in which partial-to-full scene pairs (randomly masked objects) serve as input-output training examples. Loss objectives include standard autoregressive cross-entropy and optional bounding-box regression via MSE.

Benchmarking tasks:

• Asset retrieval
• Scene layout generation
• Scene completion (partial-to-full)

Quantitative metrics:

• Out-of-bound rate (OOB)
• Object overlap rate (OOR)
• CLIP-similarity (semantic alignment)
• Generation success rate (GSR)

IL3D’s Qwen3-14B model outperforms baselines (I-Design, HOLODECK) in CLIP-Sim and GSR.

VP-LLM validates on ShapeNet, using Chamfer Distance and CLIP-score over multiple degradation scenarios. VP-LLM achieves lowest CD and highest CLIP-scores versus diffusion alternatives.

5. Editing, Physical Realism, and Interactive Applications

SceneLCM uniquely supports interactive scene editing: users may select, add, delete, or transform objects, with changes propagated via a physics engine to preserve realism. The system can locally invoke its optimization pipeline to refine textures or placements in response to edits.

FlashDreamer demonstrates persistent cross-view semantic consistency by keeping the VLM prompt fixed across all steps. However, prompt length must be tuned: semantic drift can occur in highly cluttered scenes, and some objects may be hallucinated without geometric priors.

A plausible implication is that robust editing and completion pipelines require both symbolic reasoning (LLM-guided program synthesis) and continuous optimization (e.g., latent consistency models or differentiable rendering).

6. Limitations, Open Challenges, and Future Directions

Current LLM-driven scene completion methods exhibit the following limitations:

• Hallucinated geometry: Diffusion-based view inpainting (FlashDreamer) can produce artifacts such as floating objects or misaligned corners.
• Semantic drift: Long prompts in VLM-driven systems may introduce inconsistencies; highly cluttered scenes degrade descriptive fidelity.
• Geometric stability: Relying solely on autoregressive SFT may lack boundary control; best performance achieved by combining instance-level captions, asset dimension retrieval, and iterative overlap-check with scene graph feedback (IL3D).
• Scalability: VP-LLM scales to higher 3D resolutions, but scene size is ultimately constrained by LLM context length and tokenization.

Future research priorities include integrating 3D-aware diffusion priors, fine-tuning multimodal LLMs on intricate scene datasets (e.g., IL3D), ensemble prompt scheduling for more uniform coverage, and extending completion frameworks to handle dynamic or multi-agent environments.

7. Synthesis and Current Impact

LLM-driven scene completion transforms the landscape of automated 3D environment synthesis. By fusing symbolic reasoning, multimodal tokenization, and continuous geometric optimization, these systems exceed the limitations of prior template-based and vision-only paradigms. Experimental results (SceneLCM, VP-LLM, IL3D, FlashDreamer) indicate superior semantic alignment, editing flexibility, and physical coherence:

• SceneLCM achieves user-study mean score of 8.4/10 and multi-room editing capability at competitive generation speeds (Lin et al., 8 Jun 2025).
• VP-LLM delivers lowest Chamfer Distance and highest CLIP-score in held-out benchmarks against leading diffusion models (Liu et al., 2024).
• IL3D–powered SFT pushes generation success rates to 100%, with best CLIP-similarity and diverse, annotated training sets (Zhou et al., 14 Oct 2025).
• FlashDreamer demonstrates zero-shot, multi-view scene inference from monocular imagery (Song et al., 2 Mar 2025).

These results underscore the efficacy of LLMs as scene composers, completion agents, and interactive editing engines. Practical deployment should leverage large, instance-annotated datasets, programmatic spatial reasoning, and multipath optimization for robust, semantically faithful 3D scene completion.