SceneFrom3D: Geometry-Conditioned Outdoor 3D Scene Generation via View Scheduling with Object-Level Control

Abstract: Geometry-conditioned 3D scene generation enables the creation of 3D environments from user-provided geometry, offering direct control over scene structure and object layout. To generate such 3D scenes, current methods commonly adopt a three-stage design that first defines a view schedule, then synthesizes multi-view observations along the scheduled views, and finally reconstructs a 3D representation from the generated images. However, defining the view schedule becomes a major bottleneck for outdoor scenes, where large, unstructured, and unbounded geometry makes it difficult to obtain views that provide sufficient coverage while supporting stable generation. To address this bottleneck, we present SceneFrom3D, a framework that automatically schedules views from outdoor input geometries. SceneFrom3D constructs a directed generation graph whose nodes represent anchor views and whose edges represent interpolation trajectories, defining which views to synthesize, which view pairs to interpolate, and in which order generation should proceed. Beyond automatic view scheduling, SceneFrom3D further improves controllability through object-level conditioning, assigning each object an identity image for appearance guidance and a geometry-adherence parameter for region-wise control over the input geometry. Experiments demonstrate that SceneFrom3D achieves state-of-the-art geometry-conditioned outdoor 3D scene generation, producing high-quality scenes with controllable object appearance and geometry adherence.

• The paper introduces a directed generation graph that automatically schedules geometry-aware camera views, enabling outdoor scene synthesis without user-provided trajectories.
• Object-level identity images and adjustable geometry-adherence parameters control each region’s appearance and fidelity, while video interpolation and depth-aware 3DGS optimization improve reconstruction quality.
• SceneFrom3D outperforms tested baselines across five metrics, achieving CLIP Aesthetic 6.194, MUSIQ 54.474, PSNR-D 21.974, Chamfer Distance 19.255, and F-score 0.01399, but requires up to 426.7 minutes and remains limited by shadow inconsistencies and reference-count constraints.

Motivation and problem setting

Geometry-conditioned 3D scene generation decomposes naturally into a three-stage pipeline: define camera viewpoints, synthesize geometry-conditioned multi-view observations with diffusion models, and reconstruct a 3D representation such as 3D Gaussian Splatting (3DGS). While this decomposition exploits strong image and video diffusion priors without requiring paired geometry–scene training data, its first stage is a bottleneck for outdoor scenes. Indoor methods rely on bounded, structured room layouts where rule-based camera placement suffices (2607.04540), and outdoor methods such as VideoFrom3D assume user-provided camera trajectories. For arbitrary outdoor geometry—unstructured, unbounded, with no canonical camera range—no existing pipeline automatically produces the view schedule.

SceneFrom3D addresses this gap by formulating view scheduling as construction of a directed generation graph over anchor views, and additionally introduces object-level conditioning that prior geometry-conditioned methods lack: per-object identity images for appearance guidance and per-object geometry-adherence parameters controlling how strictly each region follows the input mesh.

Automatic view scheduling

The scheduler builds a directed acyclic generation graph $$\mathcal{G}_{\mathrm{gen}} = (\mathcal{V}, \mathcal{E}_{\rightarrow})$$ whose nodes are anchor views and whose directed edges are interpolation trajectories encoding generation dependencies.

Node construction begins from surface samples drawn from the input meshes (with Poisson-disk filtering and a normal-clearance filter that removes samples immediately occluded by contact configurations). A soft visibility score $V_{i,n}$ combines four criteria—field-of-view membership, preferred viewing distance $d_0$, front-facing orientation, and binary occlusion testing—and aggregation across views uses a soft union $\bar{V}_n$. Progressive initialization adds cameras along under-covered sample normals until all samples exceed a visibility threshold; because irregular geometry can spawn redundant cameras observing overlapping regions, a refinement loop removes low-contribution cameras, merges pairs with high shared visibility (cosine similarity of visibility rows), and re-densifies. Continuous pose optimization then minimizes a coverage loss plus two regularizers: a repulsion loss keeping cameras outside the input mesh by a safety margin $d_{\mathrm{safe}}$, and a tilt loss preventing cameras from collapsing toward the ground plane—a bias induced by ground meshes dominating the sample set.

Edge construction connects anchor pairs whose shared visibility $S_{i,k}=\sum_n \min(V_{i,n},V_{k,n})$ exceeds a threshold and whose connecting segment is collision-free; distance-based edges repair disjoint subgraphs and leaf nodes. Interpolation trajectories are cubic Bézier curves arcing away from a common focus region, mimicking natural in-domain camera motion, with fallback to straight segments on collision. Direction construction orients edges from lower to higher node index, guaranteeing a DAG and a topological generation order in which parent views condition their children.

The supplementary ablations establish that each visibility term is load-bearing: removing the FOV term makes optimization nearly random with respect to actual coverage, producing uninformative or near-duplicate views; removing the distance term admits cameras blocked by foreground occluders; removing the front-facing term yields oblique observations that degrade generated-view quality.

Multi-view generation with object-level conditioning

Anchor-view synthesis follows VideoFrom3D's complementary image/video strategy but replaces given trajectories with the generation graph. Each anchor view receives a conditioning tuple comprising a partial observation warped from parent views (with observation mask), a corrupted depth structure condition overlaid with HED boundary cues, identity-region pairs $(I_o, \mathbf{M}_{j,o})$, and a rule-based text prompt. The corruption strength map $\mathbf{S}_j$ is set to $1-\alpha_o$ per object, so larger adherence parameters $\alpha_o$ enforce stricter geometric fidelity while smaller values permit flexible appearance—the paper demonstrates this continuum concretely, e.g., a house whose dormer, door, and window positions are reproduced at high $V_{i,n}$0 but relaxed at low $V_{i,n}$1.

The generator is FLUX.2-klein-9B fine-tuned with LoRA (rank 128) on approximately 17K synthetically constructed identity-to-anchor pairs built from a 349-image identity bank across 22 categories. Notably, architecture components use a 3×3 grid of multi-angle identity references because single building references induce strong pose bias in the base model. During inference, a semantic overflow check (threshold 0.01, up to six candidates) filters stochastic failures where object content escapes its intended region. Edge interpolation uses Wan2.1-VACE-14B with 49 frames per edge, endpoint anchors fixed, and normalized depth renderings as intermediate control frames.

3DGS optimization

Reconstruction combines L1 RGB, DSSIM, LPIPS, and metric-depth losses against mesh-rendered depth. Two design choices matter: the LPIPS weight is applied only to interpolated edge frames with a midpoint-normalized weight $V_{i,n}$2 (zero at anchors), following CAT3D's rationale of suppressing perceptual inconsistencies inherited from generative models; and the depth loss suppresses floater artifacts that would otherwise occlude scene elements.

Experimental results

Evaluation uses nine outdoor layouts (9–16 objects each), ten generated scenes per baseline, roughly 400–700 held-out test frames per scene, and metrics covering visual quality (CLIP Aesthetic, MUSIQ) and structural fidelity (PSNR-D, Chamfer Distance, F-score). The full method achieves CLIP Aesthetic 6.194, MUSIQ 54.474, PSNR-D 21.974, Chamfer Distance 19.255, and F-score 0.01399, outperforming all baselines across all five metrics. Baselines were constructed by substituting individual stages: UrbanArchitect (SDS-based optimization) suffers blurry appearances and missing textures; YoNoSplat (feedforward reconstruction) exhibits a domain gap on generated views and cannot scale beyond ~100 input views; Zhang et al.'s aerial path planning biases viewpoints toward overhead angles, which renders coarse-geometry depth conditions ambiguous and degrades eye-level quality. SceneFrom3D's scheduling is markedly more compact than the drone-planning alternative while maintaining layout coverage.

Ablations confirm each component: removing the tilt loss tilts cameras toward the ground and leaves incomplete appearances; removing the repulsion loss lets cameras enter the mesh, invalidating depth maps and disconnecting graph nodes; removing video interpolation starves 3DGS optimization of sufficient views. One caveat deserves emphasis: the no-interpolation variant attains the best MUSIQ score (59.653), which the authors attribute to high-frequency artifacts favored by the metric rather than genuine quality—an instance where the quantitative table alone is misleading. Runtime is substantial: total pipeline latency ranges from 86.8 minutes (Small Village, 200 views) to 426.7 minutes (Tribe Town, 600 views), with multi-view generation dominating (e.g., 372.5 minutes on Tribe Town).

Limitations and open questions

The paper states two limitations directly. Anchor-view generation fails when a single view contains more than eight distinct object identities, due to the limited number of reference images the pretrained model can effectively incorporate. And absent an explicit global illumination prior, shadow direction and size vary across anchor views, producing visible shadow inconsistencies in the final 3DGS output. Beyond these, the evaluation relies on synthetic training data and manually created or free-asset layouts, so generalization to arbitrary real-world geometry distributions remains unverified, and the reliance on a fixed preferred viewing distance $V_{i,n}$3 may not suit scenes requiring close inspection or wide aerial context.

Conclusion

SceneFrom3D removes the requirement of explicit camera trajectories from geometry-conditioned outdoor scene generation by casting view scheduling as directed graph construction over anchor views, and extends controllability to the object level through identity images and per-object geometry-adherence parameters. Quantitative results show consistent superiority over SDS-based, feedforward-reconstruction, and drone-path-planning alternatives across both perceptual and structural-fidelity metrics. The remaining constraints—reference-count limits per view, illumination inconsistency, and long generation runtimes—define the concrete open problems for subsequent work on scalable, object-controllable outdoor scene synthesis.