DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priors

Abstract: We present DeWorldSG, a novel framework that generates spatio-temporally robust 3D Semantic Scene Graphs from RGB-D sequences. Existing methods often struggle to construct reliable 3D scene graphs due to unstable 3D object representations and missing relations caused by frame-wise inference. DeWorldSG addresses these issues by estimating instance-level geometric 3D Gaussian distributions through depth-guided filtering and representing each object as a probabilistic 3D node rather than a single projected point. To mitigate relational sparsity from frame-wise inference, our framework further aggregates spatiotemporal evidence across object pairs and refines relations using contextual priors derived from a world model (V-JEPA 2). Experiments on the 3DSSG and ReplicaSSG datasets demonstrate state-of-the-art (SoTA) performance in both object and predicate prediction, while producing temporally consistent scene structures. In particular, our method improves triplet recall by 77.4% and predicate recall by 23.2% over prior SoTA approaches, making it suitable for robotic manipulation and AR applications. Our code and models are open-sourced.

• The paper introduces depth-aware 3D Gaussian node construction that enhances object stability and relational accuracy via probabilistic modeling.
• It fuses temporal visual context with world-model priors to resolve ambiguous relationships and enable real-time incremental graph merging.
• The system outperforms prior methods with significant gains in object, predicate, and relationship recall on 3DSSG and ReplicaSSG benchmarks.

DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priors

Introduction and Motivation

DeWorldSG introduces a framework for 3D Semantic Scene Graph (SSG) generation from RGB-D sequences, a critical step for enabling reliable spatial reasoning in robotic and AR systems. The primary innovation centers around constructing spatio-temporally robust 3D SSGs by jointly modeling geometrically grounded object representations and exploiting world-model priors to resolve relational ambiguity over time. Unlike previous SSG methods that largely depend on per-frame inference and explicit geometry reconstructions—both of which are susceptible to noisy depth, unstable object merging, and missing relations—DeWorldSG encodes each object instance as a probabilistic 3D Gaussian, and refines relations by aggregating visual context temporally, reinforced by predictive world-model priors.

Figure 1: DeWorldSG generates a spatio-temporally robust 3D semantic scene graph from RGB-D observations. It combines 2D scene graph generation, depth-aware 3D lifting, and world-model-based relation enhancement to produce stable object nodes and context-aware relational edges for AR and embodied robotic systems.

Methodology

DeWorldSG comprises three principal modules: depth-aware 3D Gaussian node construction, incremental graph merging, and world-model-grounded spatiotemporal relation refinement.

Depth-Aware 3D Gaussian Object Nodes

The framework initiates with frame-wise 2D scene graph generation. Each detected object is assigned a segmentation mask, and its geometry is inferred not via a single depth, but through direct estimation of a Gaussian distribution over filtered depth observations. The filtering pipeline, called Dual-Domain Depth Refinement (DR), removes spatial and depth outliers by combining local neighborhood consistency checks with adaptive 1D clustering in depth space. The resultant 3D Gaussians encode both the geometric mean and covariance, allowing robust statistical fusion over multiple observations.

Incremental Graph Merging

To construct the global 3D SSG, the method merges local object nodes from consecutive frames via distributional similarity (Hellinger distance between Gaussians) and class-consistency constraints. Object association is executed only if both geometric and semantic thresholds are satisfied. Over time, merged Gaussians are adaptively re-estimated using weighted averaging, which stabilizes global object nodes and prevents fragmentation or duplicity.

Spatiotemporal Relation Enhancement with World Models

Visual relation inference relies on aggregating predictions across time for object pairs that are spatially and semantically proximate. Temporal visual context is exploited by encoding spatiotemporal union-crop clips for each object pair with a frozen V-JEPA 2 world-model backbone, followed by an MLP probe for predicate classification. The video-based prior is fused into the relation prediction only when visual evidence is ambiguous (high entropy). This strategy leverages predictive structural regularities learned by the world model to recover missing or uncertain relationships that single-frame models frequently overlook.

Figure 2: Overview of DeWorldSG. Given RGB-D image sequences, DeWorldSG first constructs frame-wise 2D scene graphs (a), estimates depth-aware probabilistic 3D Gaussians for object instances (b), and fuses spatiotemporal predicate priors from a world model with visual predictions for incremental global scene graph construction (c).

Figure 3: For each object pair, a 16-frame union-crop is encoded using V-JEPA 2, and an MLP probe outputs the clip-level predicate distribution for relation refinement.

Experimental Results

Benchmarks and Metrics

DeWorldSG is evaluated on the 3DSSG and ReplicaSSG benchmarks, both covering real or high-fidelity synthetic indoor environments with semantic and relational annotations. Metrics include Object, Predicate, and Triplet (Relationship) Recall, as well as class-balanced mRecall.

SOTA Results and Ablations

DeWorldSG outperforms all prior approaches across every reported metric. On 3DSSG, it achieves a +77.4% improvement in Relationship Recall, +20.2% in Object Recall, and +23.2% in Predicate Recall compared to FROSS and MonoSSG, with mRecall substantially elevated—especially under class imbalance. Critically, these high scores are obtained without relying on full 3D point cloud reconstructions or ground-truth camera poses, demonstrating resilience to pose estimation noise and effective real-time incremental processing (average 108.53 ms/frame). On ReplicaSSG, DeWorldSG maintains its dominance even with predicted trajectory input, validating robust operation in typical embodied or AR environments.

Qualitative analysis reveals that depth-aware Gaussian estimation and temporal fusion suppress spurious object duplications and incorrect relation edges prevalent in prior SSG methods. Ablations show that each component—mask-conditioned depth extraction, DR filtering, and world-model predicate priors—yields additive performance gains, with the world model particularly decisive for relation inference.

Figure 4: Qualitative comparison of 3D SSG generation results on ReplicaSSG, showing DeWorldSG's superior spatial consistency and relational accuracy compared to prior methods.

Theoretical and Practical Implications

DeWorldSG validates the thesis that frame-level, geometry-first SSG pipelines are fundamentally limited by depth noise, recognition ambiguity, and time-local relational evidence. By representing instance geometry probabilistically and fusing spatiotemporal evidence—weighted by world-model priors—the proposed system dramatically reduces relational sparsity, improves node stability, and supports robust, incremental updates suitable for embodied agents.

From a practical perspective, these contributions directly impact the reliability of downstream tasks: robot manipulation, AR content anchoring, and spatially-anchored reasoning can now operate over robust, context-complete scene graphs. The method’s design is also amenable to real-time, noise-tolerant operation, promoting deployment in dynamic, unstructured environments.

Theoretically, DeWorldSG sets a new paradigm for integrating predictive world models with explicit 3D graph reasoning—pointing toward future research on cross-modal world models, semantic-temporal memory systems, and generalization across unseen environments.

Limitations and Future Work

A notable limitation is the dependence on 2D object detectors; frame-level category prediction errors can induce node duplication or fragmented relationships, especially under occlusions or significant viewpoint shifts. The authors suggest that advancing cross-frame semantic identity modeling or adopting more robust 2D detection architectures could address this gap.

Furthermore, the current system restricts relation proposal to geometrically proximal pairs, which, while suppressing hallucinated edges, may miss non-local but semantically important interactions. Extending scene graph construction to capture more global context, perhaps with higher-capacity or multi-scale world models, is a promising direction. Finally, deeper integration of spatial-language priors or unsupervised cross-modal signals could facilitate broader generalization to complex, open-domain scenes.

Conclusion

DeWorldSG establishes a new state-of-the-art for 3D semantic scene graph generation from RGB-D sequences by fusing depth-aware probabilistic object modeling with world-model-based relational priors. Its consistent gains on challenging benchmarks, real-time operation, and robust performance under noisy pose estimation highlight both immediate application value and the significance of its architectural insights for scalable 3D scene understanding in embodied AI and AR systems. Future research should explore advanced cross-frame identity mechanisms, richer world-model priors, and novel modalities for enhancing spatial reasoning capabilities in three-dimensional environments.