SceneForge 3D Supervision Framework

• SceneForge is a framework that creates consistent multimodal supervision from an editable, persistent 3D world state.
• It utilizes intervention-consistent supervision by propagating semantic, geometric, and physical dependencies through controlled scene edits.
• Its instantiation with Infinigen and Blender produces multi-view datasets that enhance object and scene removal performance.

SceneForge is a world-centric data generation framework for multimodal learning that derives supervision from editable, persistent 3D world states rather than from observation-level image pairs. Its central premise is that supervision should remain consistent across edits, viewpoint changes, object-level and scene-level interventions, and scene-level side effects such as shadows, reflections, occlusion, and revealed background. In its released instantiation, SceneForge uses Infinigen and Blender to build a licensing-clean indoor supervision resource with over 2K scenes, object-linked layers, single-view and registered multi-view supervision, and large numbers of counterfactual object-removal examples; under matched training budgets, this supervision improves both object removal and scene removal performance (Li et al., 14 May 2026).

1. Conceptual basis

SceneForge is motivated by a limitation of observation-level datasets: they usually provide rendered snapshots, before/after pairs, masks, or textual instructions, but not the underlying scene state or the mechanisms by which edits propagate through geometry, visibility, support relations, and lighting (Li et al., 14 May 2026). As a result, such datasets rarely provide, for the same scene, physically consistent counterfactuals, matched multi-view renders of an edit, amodal or object-linked layers, relation-aware annotations, effect-aware channels, or aligned supervision across modalities.

The framework addresses this by defining supervision at the level of an editable 3D world rather than through image-space postprocessing. In SceneForge, a scene is treated as a structured persistent world state whose entities, attributes, relations, and environmental factors are known before rendering. An intervention—such as object removal, object relocation, pose adjustment, lighting modification, occlusion manipulation, or controlled camera variation—is applied to that world state, its consequences are propagated through scene dependencies, and all outputs are then rendered from the updated state (Li et al., 14 May 2026).

The paper refers to this as structured world supervision: supervision generated from a scene represented as a structured persistent world state rather than as a flat image. It also defines intervention-consistent supervision as supervision that remains consistent with the effects of an explicit world-level edit. This distinction is fundamental. SceneForge is not a post hoc image-editing pipeline; its alignment comes from shared state and explicit state transitions, not from later matching heuristics (Li et al., 14 May 2026).

2. Persistent world state and dependency propagation

A scene at time $t$ is represented as

$$w_t = (\mathcal{S}_t, \mathcal{D}_t, \mathcal{A}_t, \mathcal{R}_t, \mathcal{E}_t),$$

where $\mathcal{S}_t$ denotes structural elements, $\mathcal{D}_t$ dynamic objects, $\mathcal{A}_t$ attributes, $\mathcal{R}_t$ structured relations, and $\mathcal{E}_t$ environmental factors (Li et al., 14 May 2026). The distinction between structural and dynamic entities is central: floors, walls, large support surfaces, and fixed background geometry are treated as structural anchors, whereas dynamic objects are the editable entities that may be inserted, removed, moved, replaced, or otherwise modified.

SceneForge models three broad dependency classes. Semantic dependencies include category, instance identity, object type, support status, containment, and scene plausibility or coherence. Geometric dependencies include pose, extent, occupancy, relative position, visibility, occlusion, and collisions. Physical or rendering-related dependencies include support or contact interactions, lighting-related properties, material interactions, shadows, reflections, and projected effects (Li et al., 14 May 2026). This makes interventions nonlocal by construction: removing one object can alter revealed background, occlusion structure, support consistency, and appearance effects.

An intervention $\iota$ is applied at the world level as

$\tilde{w}_{t+1} = \mathcal{I}(w_t, \iota),$

followed by a dependency-aware propagation step

$w_{t+1} = \mathcal{P}(\tilde{w}_{t+1}).$

Supervision is then rendered via

$$w_t = (\mathcal{S}_t, \mathcal{D}_t, \mathcal{A}_t, \mathcal{R}_t, \mathcal{E}_t),$$0

where $$w_t = (\mathcal{S}_t, \mathcal{D}_t, \mathcal{A}_t, \mathcal{R}_t, \mathcal{E}_t),$$1 is a camera or view configuration and $$w_t = (\mathcal{S}_t, \mathcal{D}_t, \mathcal{A}_t, \mathcal{R}_t, \mathcal{E}_t),$$2 indexes output modalities (Li et al., 14 May 2026).

Not every intervention is valid. SceneForge enforces constraints such as valid edit targets, geometrically feasible placement, support and collision consistency, acceptable camera and view conditions, and semantic coherence. The paper describes a scene-querying stage that verifies edit targets, affected entities, and view conditions before outputs are emitted. In the released implementation, optional LLM-assisted querying is not required; the released path uses geometric checks (Li et al., 14 May 2026).

A notable implementation detail is that the released system does not require maintaining a separate symbolic world-state database at inference time. Instead, it operationalizes the world-state abstraction directly inside the rendering pipeline, using scene metadata and object categories to distinguish dynamic objects from structural elements at render time (Li et al., 14 May 2026). This suggests a pragmatic realization of the formalism rather than a separate symbolic runtime.

3. Aligned outputs and rendering semantics

Once a consistent pre- and post-intervention state is available, SceneForge renders aligned supervision from that shared world state. The paper lists a broad output set: RGB images, counterfactual edited images, multi-view observations, decomposition outputs, geometry-aware signals, semantic and instance labels, amodal targets, language descriptions, relation annotations, and effect-aware channels (Li et al., 14 May 2026). The key claim is alignment by construction: all outputs share object identity, geometry, and camera consistency because they are generated from the same maintained state and intervention history.

Counterfactual observations are a primary output. In the object-removal setting, the before and after images are not paired by image editing; they are rendered from a true updated world state. This yields correct revealed background, updated occlusion structure, scene-consistent appearance changes, and potentially updated lighting effects (Li et al., 14 May 2026). The same logic extends to multi-view supervision: because all views originate from the same updated persistent state, registered multi-camera observations of the same intervention remain spatially consistent.

A distinctive aspect of SceneForge is its emphasis on effect-aware signals. The framework can produce shadows, reflections, direct light contributions, and projected interactions, particularly through Blender/Cycles multi-pass decomposition (Li et al., 14 May 2026). In addition to RGB-like outputs, it emits aligned annotations including depth maps, instance segmentation, surface normals, language labels, amodal targets, object-linked layers, and relation-aware annotations.

For layer supervision, the paper states that Blender scenes provide separate Cycles passes of the same .blend, and that these layers can be recomposed front-to-back according to depth-map ordering to recover a render nearly identical to the original image (Li et al., 14 May 2026). This is important because it ties object-linked decomposition, amodal structure, and effect-aware rendering to a single maintained source scene rather than to independent pipelines.

The consistency mechanism is therefore not a learned post hoc alignment model. Visibility and occlusion are recomputed globally, shadows and reflections arise from render-time effects, and multi-view consistency follows from the shared world state (Li et al., 14 May 2026). A plausible implication is that SceneForge is best understood as a supervision engine for intervention-aware learning, not merely as a counterfactual image generator.

4. Instantiation with Infinigen and Blender

The released SceneForge instantiation is built primarily from Infinigen indoor scenes, editable .blend scenes, and Blender Cycles multi-pass rendering (Li et al., 14 May 2026). Although the framework is described as asset-source agnostic, the current release focuses on a licensing-clean indoor supervision resource. The engine starts from editable scenes, operationalizes the dynamic-versus-structure distinction using scene metadata and object categories, and then configures rendering passes to support both object-level removal and scene-level structural-only removal.

The practical pipeline is described in seven stages: editable 3D scene creation or acquisition; abstraction into persistent world state; intervention specification; intervention application; dependency-aware propagation; rendering of aligned supervision; and dataset assembly (Li et al., 14 May 2026). For multi-view data, candidate cameras are proposed around a base camera and then filtered using same-room Line-of-Sight checks, wall-clearance constraints, and 7-point target bounding-box visibility verification. The main paper also describes a tiered protocol with one group of high-overlap views $$w_t = (\mathcal{S}_t, \mathcal{D}_t, \mathcal{A}_t, \mathcal{R}_t, \mathcal{E}_t),$$3 and another group of more diverse viewpoints such as bird’s-eye and corner views.

The released resource is organized into two raw corpora and several derived datasets:

Resource	Setting	Key counts
SceneForge (S)	Single-camera raw corpus	2,369 rooms; 25,688 object-linked layers; average items per scene 10.84
SceneForge (M)	Multi-camera raw corpus	517 rooms; 2,772 registered multi-view scene instances; 34,865 object-linked image layers; 8 camera settings; average items per scene 12.58
SceneForge-Removal (S)	Single-view object removal	13,419 edited images from 2,358 rooms; split 12,734 / 685
SceneForge-Removal (M)	Multi-view object removal	3,999 edited images from 330 rooms; split 3,803 / 196
SceneForge-IndoorDynamicRemoval	Scene-level dynamic-object removal	2,435 room-level scenes/images; split 2,313 / 122

Both removal datasets discard edits below a 0.3% mask area threshold (Li et al., 14 May 2026). SceneForge-Removal (M) is further partitioned into 642 base_camera, 1,415 camera_similar, and 1,942 camera_diff examples. The supplement notes that the single-camera and multi-camera room sets are disjoint under a UUID-based room key (Li et al., 14 May 2026).

The raw corpora include aligned annotations such as depth maps, instance segmentation, surface normals, and language labels. The paper notes that the layers may include dependency-aware groupings and amodal completion when occluded objects can be recovered from the maintained world state (Li et al., 14 May 2026).

5. Experimental results and downstream effects

The empirical study evaluates SceneForge from four angles: dataset statistics, compositional layer decomposition, intervention-driven counterfactual generation, and downstream performance on removal tasks (Li et al., 14 May 2026). The main downstream experiments use Qwen-Image-Edit-2509 with LoRA fine-tuning and mask-conditioned orig/mask/target triplets. All runs use 8 × NVIDIA H200 GPUs for about 20 hours under a matched-budget protocol in which architecture, optimization, compute, and training duration are fixed.

For object-level removal, the paper compares three training settings: OpenData-30K, OpenData+SceneForge-30K, and SceneForge-16K. Despite using fewer total samples than either 30K setting, SceneForge-16K achieves the best result on every reported metric across all four test sets (Li et al., 14 May 2026). On SceneForge-Removal (S) test, the reported PSNR/SSIM/LPIPS are 33.4079/0.9602/0.0360 for OpenData-30K, 37.2342/0.9795/0.0201 for OpenData+SceneForge-30K, and 38.3180/0.9824/0.0157 for SceneForge-16K. On SceneForge-Removal (M) test, the corresponding values are 30.4853/0.9469/0.0529, 33.4978/0.9641/0.0352, and 34.7007/0.9699/0.0272 (Li et al., 14 May 2026).

The same trend appears on external benchmarks. On RORem test_300, SceneForge-16K reaches 20.3384 PSNR, 0.8803 SSIM, and 0.1328 LPIPS, compared with 19.4663/0.8508/0.1622 for OpenData-30K. On RemovalBench, SceneForge-16K reaches 28.7500/0.8584/0.0870, compared with 26.3099/0.8457/0.1242 for OpenData-30K (Li et al., 14 May 2026). The paper interprets this as evidence that cleaner, intervention-consistent supervision can outweigh larger but more heterogeneous data pools.

The evaluation is deliberately full-image rather than mask-only, so restoration of newly revealed background, shadows, and correlated visual effects influences the reported metrics (Li et al., 14 May 2026). The paper also introduces Removal-HardEffects, a qualitative benchmark of 60 open-source images with strong shadows or reflections and manual masks. On these cases, the SceneForge-trained model more reliably removes associated shadows, reflections, and other local side effects (Li et al., 14 May 2026).

For scene-level removal, the training set is SceneForge-IndoorDynamicRemoval with a 2,313 / 122 train/test split. Using the same backbone and optimization recipe, the reported full-image scores are PSNR 30.594, SSIM 0.9498, and LPIPS 0.0477 (Li et al., 14 May 2026). The paper further reports qualitative transfer to realistic real-world cases, which suggests some robustness beyond synthetic scenes, although it also states explicitly that synthetic supervision does not remove domain shift (Li et al., 14 May 2026).

6. Scope, misconceptions, and relation to adjacent work

SceneForge is a supervision framework, not a general-purpose interactive scene editor or a universal text-to-3D scene generator. Its usefulness depends on world-state fidelity, dependency propagation quality, rendering quality, and asset licensing; the empirical study mainly focuses on removal tasks (Li et al., 14 May 2026). A common misconception is to equate it with systems whose primary objective is scene authoring or real-time scene manipulation. The paper positions it instead as a flexible supervision engine whose broader applicability is promising but not exhaustively validated.

This distinction becomes clearer when SceneForge is compared with adjacent work. ASSIST is an object-centric neural scene representation and editing framework for indoor simulation, organized around independent scene nodes and online interaction over panoptic radiance fields (Zhong et al., 2023). SceneCraft, by contrast, is a layout-guided 3D scene generation pipeline that renders user-specified semantic layouts into multi-view proxy maps, synthesizes images with a diffusion model, and distills them into a NeRF representation (Yang et al., 2024). SceneFoundry targets language-guided generation of apartment-scale 3D worlds with articulated furniture and walkability constraints (Chen et al., 9 Jan 2026), while SceneCode compiles natural-language prompts into executable Blender Python programs and simulation-ready SDF assets for editable indoor scenes with articulated objects (Wang et al., 19 May 2026). In autonomous driving, HorizonForge similarly combines editable 3D assets with a temporally aware generative renderer, but for trajectory-conditioned driving scene editing rather than indoor supervision (Wang et al., 24 Feb 2026).

There is also a naming ambiguity. A separate paper titled “SCENEFORGE: Enhancing 3D-text alignment with Structured Scene Compositions” uses the same name for a batch-level compositional augmentation method for 3D-text contrastive learning, not for structured world supervision from editable 3D interventions (Sbrolli et al., 19 Sep 2025). The two works address different problems, use different representations, and operate at different points in the multimodal pipeline.

In that broader landscape, SceneForge’s distinctive claim is that aligned multimodal supervision should be generated from maintained world states and explicit state transitions. This suggests a shift from supervision as paired observations to supervision as structured world evolution. A plausible implication is that its main long-term significance lies less in any single benchmark and more in formalizing editable worlds as the source of consistent multimodal supervision (Li et al., 14 May 2026).