FROSS: Fast 3D Semantic Scene Graphs

• FROSS is a framework for instantly building 3D semantic scene graphs from RGB-D images by directly lifting 2D scene graphs into 3D without dense point-cloud reconstruction.
• The method models each object as a 3D Gaussian and uses incremental Gaussian merging with Hellinger distance based data association to maintain a global, view-invariant scene graph.
• FROSS achieves state-of-the-art performance in relationship and object recall at low latency, making it ideal for robotics, AR/VR, and embodied AI applications.

FROSS, introduced in "FROSS: Faster-than-Real-Time Online 3D Semantic Scene Graph Generation from RGB-D Images" (Hou et al., 26 Jul 2025), is a framework for building 3D semantic scene graphs from streaming RGB-D data in an online and faster-than-real-time manner on a single GPU/CPU core, without constructing or processing dense 3D point clouds. A 3D semantic scene graph represents objects as nodes and their interrelationships as directed edges in a consistent 3D coordinate frame. FROSS replaces dense reconstruction with direct lifting of 2D scene graphs into 3D and models each object as a 3D Gaussian distribution, using incremental Gaussian merging and majority-vote relationship aggregation to maintain a global graph as new frames arrive.

1. Problem formulation and motivation

A 3D semantic scene graph is a graph representation of a 3D environment in which nodes correspond to individual physical objects and edges correspond to directed relationships between object pairs. A node typically carries a semantic class, a 3D location or extent, and possibly attributes or features. Edge semantics can be spatial, such as on, under, behind, near, standing on, supported by, and attached to, or functional and semantic, such as has, with, and part of. Because nodes are grounded in 3D space, the representation supports reasoning over persistent object identities and relations across viewpoints (Hou et al., 26 Jul 2025).

The paper positions 3D SSG generation from RGB-D as particularly relevant to robotics and embodied AI, AR/VR, and 3D scene understanding and synthesis. In robotics and embodied AI, the representation supports high-level task planning, object-centric navigation, manipulation, and reasoning about support, containment, and reachability. In AR/VR, it supports semantic anchoring of virtual content and scene editing or guidance based on object relationships. In 3D scene understanding and synthesis, it supports graph-based editing, completion, and generative layout models.

The motivation for FROSS is a set of limitations in prior 3D SSG methods. First, many methods are offline or batch-oriented, requiring full scene meshes, instance-segmented point clouds, or complete RGB-D sequences. Second, real-time approaches such as SceneGraphFusion and Wu et al. (2023) depend on on-the-fly dense mapping and point-cloud segmentation, which introduces substantial computational overhead. Third, even systems described as online often only match sensor frame rate when using multi-core CPUs and GPUs. Fourth, many methods are strongly tied to precise geometry and pose, although approximate positions and relations are sufficient for many reasoning and planning tasks. FROSS addresses these constraints by targeting semantics and relations rather than full reconstruction fidelity.

2. Core architecture and probabilistic object representation

At time $t$, the input is an RGB image $I_t$, a depth image $D_t$, camera intrinsics $K$, and camera pose $(R_t, \mathbf{t}_t)$ in world coordinates. After each frame, the output is a global 3D semantic scene graph

$$\mathcal{G}_\text{global} = (\mathcal{V}_\text{global}, \mathcal{E}_\text{global}),$$

with nodes

$v_i = (c_i, \mu_i^{3D}, \Sigma_i^{3D}),$

where $c_i$ is the semantic class, $\mu_i^{3D} \in \mathbb{R}^3$ is the 3D mean, and $\Sigma_i^{3D} \in \mathbb{R}^{3\times3}$ is the covariance (Hou et al., 26 Jul 2025).

The per-frame pipeline has four stages. First, FROSS performs 2D object detection with RT-DETRv2-M and retains objects with confidence above threshold $I_t$0. Second, it performs 2D relationship extraction with EGTR, a transformer-based 2D scene graph model that uses preserved RT-DETR self-attention features, and keeps the top 10 relationships per 2D scene graph. Third, it lifts 2D objects to 3D as Gaussians using depth and camera pose. Fourth, it merges the resulting local 3D graph into the global graph.

The 2D bounding box of object $I_t$1 is modeled as a 2D uniform distribution with center $I_t$2, width $I_t$3, and height $I_t$4. Its covariance is

$I_t$5

The 3D mean is obtained by depth back-projection followed by transformation into world coordinates:

$I_t$6

For covariance lifting, FROSS uses a local affine approximation of the 3D-to-2D projection with Jacobian

$I_t$7

Using the pseudo-inverse $I_t$8, the paper approximately inverts the projection relation according to

$I_t$9

Because 2D covariance does not encode depth variance, the depth variance is heuristically set to the average of the other two, after which the covariance is rotated into world coordinates:

$D_t$0

Each detected object therefore becomes a 3D Gaussian $D_t$1, and each 2D relation becomes an edge between corresponding local nodes.

A central architectural feature is the explicit avoidance of dense point-cloud processing. FROSS does not reconstruct TSDFs, voxel grids, meshes, or dense online maps as part of the 3D SSG generation procedure. The local graph is instead a sparse set of Gaussian-parameterized nodes and predicate-labeled edges.

3. Online merging, data association, and relation consolidation

The central online problem is multi-view association and fusion of noisy object observations while preventing an explosion in node count. FROSS restricts candidate matches to nodes of the same semantic class and uses the Hellinger distance between Gaussian object representations for data association (Hou et al., 26 Jul 2025).

For two Gaussian distributions $D_t$2 and $D_t$3, FROSS first computes

$D_t$4

then the Bhattacharyya distance

$D_t$5

and finally the Hellinger distance

$D_t$6

If $D_t$7, the two nodes are considered to correspond to the same physical object and are candidates for merging. The paper reports a tuned threshold such as $D_t$8.

Once two nodes are associated, FROSS merges them by weighted Gaussian fusion, where the weights $D_t$9 and $K$0 represent detection frequency:

$K$1

$K$2

The updated weight is $K$3. This is the standard moment-matching fusion of Gaussians and explicitly accounts for both averaged covariance and dispersion between means. After merging, all edges incident on the merged nodes are rewired to the fused node.

Relationship integration is handled separately from object fusion. Once local nodes have been mapped to global nodes, predicates between a pair of global nodes are aggregated across frames, and the final relationship label is selected by majority voting. The paper presents this as a mechanism for robustness across viewpoints: a single incorrect predicate from one frame can be dominated by consistent predictions from other frames.

The algorithmic update is entirely local and analytic. For each frame, FROSS runs detection and relation extraction, lifts objects to 3D Gaussians, places local nodes into a queue, compares each queued node with same-class nodes already in the queue and in the global graph, merges when the minimum Hellinger distance falls below threshold, otherwise inserts a new node, and finally updates edges by vote accumulation. There is no optimization over the whole graph.

4. ReplicaSSG, evaluation protocol, and baselines

To support evaluation, the paper introduces ReplicaSSG by extending the Replica dataset with 3D inter-object relationship annotations (Hou et al., 26 Jul 2025). Replica is described as a set of 18 indoor scenes with photo-realistic RGB renderings, accurate instance-segmented meshes, and good geometric detail. ReplicaSSG maps objects to Visual Genome classes rather than the original Replica labels, enabling zero-shot use of 2D scene graph models trained on Visual Genome without retraining on Replica. The dataset uses 7 scenes for validation and 11 scenes for testing.

Evaluation follows Wu et al. (2023). Because FROSS does not output explicit 3D segments, predicted objects are matched to ground-truth instances by back-projecting 3D points from the central region of each predicted 2D bounding box, specifically an $K$4 patch. For each predicted point, the nearest ground-truth point must lie within $K$5 m. A predicted object matches a ground-truth object if more than 50% of its back-projected points have nearest neighbors inside that ground-truth object’s volume, and if the ratio of second-largest overlap count to largest overlap count is less than $K$6, which is used to avoid ambiguous matching. FROSS also enforces one-to-one matching: a ground-truth object can only match one prediction.

The reported metrics are object recall, predicate recall, relationship recall, object mean Recall, and predicate mean Recall. Object recall measures the fraction of ground-truth objects that are matched and have correct class labels. Predicate recall measures the fraction of predicates correctly classified between pairs of matched objects, ignoring object class correctness. Relationship recall measures the fraction of correct triplets of subject class, object class, and predicate where both objects match ground truth and the predicate is correct. Mean Recall is averaged over classes to reduce class imbalance effects. For 2D scene graph evaluation, the paper also uses standard scene-graph metrics $K$7 and $K$8 with graph constraints.

On 3DSSG, FROSS is compared to image-based and point-cloud-based baselines. The image-based set comprises IMP, VGfM, and Kim (2019), where Kim (2019) uses a local 3D SSG from RGB-D followed by incremental integration with a 3D ellipsoid representation and heuristic merging. The point-cloud-based set comprises 3DSSG, SGFN, and Wu et al. (2023). On ReplicaSSG, the paper reports ablations rather than comparisons to an established prior 3D SSG method.

5. Quantitative behavior, runtime, and ablation results

On 3DSSG, FROSS reports relationship recall $K$9, object recall $(R_t, \mathbf{t}_t)$0, predicate recall $(R_t, \mathbf{t}_t)$1, object mean Recall $(R_t, \mathbf{t}_t)$2, predicate mean Recall $(R_t, \mathbf{t}_t)$3, and latency $(R_t, \mathbf{t}_t)$4 ms per frame (Hou et al., 26 Jul 2025). The selected comparisons reported in the paper are: Wu (2023) with relationship recall $(R_t, \mathbf{t}_t)$5, object recall $(R_t, \mathbf{t}_t)$6, predicate recall $(R_t, \mathbf{t}_t)$7, object mean Recall $(R_t, \mathbf{t}_t)$8, predicate mean Recall $(R_t, \mathbf{t}_t)$9, and latency $$\mathcal{G}_\text{global} = (\mathcal{V}_\text{global}, \mathcal{E}_\text{global}),$$0 ms; SGFN with relationship recall $$\mathcal{G}_\text{global} = (\mathcal{V}_\text{global}, \mathcal{E}_\text{global}),$$1, object recall $$\mathcal{G}_\text{global} = (\mathcal{V}_\text{global}, \mathcal{E}_\text{global}),$$2, predicate recall $$\mathcal{G}_\text{global} = (\mathcal{V}_\text{global}, \mathcal{E}_\text{global}),$$3, object mean Recall $$\mathcal{G}_\text{global} = (\mathcal{V}_\text{global}, \mathcal{E}_\text{global}),$$4, predicate mean Recall $$\mathcal{G}_\text{global} = (\mathcal{V}_\text{global}, \mathcal{E}_\text{global}),$$5, and latency $$\mathcal{G}_\text{global} = (\mathcal{V}_\text{global}, \mathcal{E}_\text{global}),$$6 ms; Kim (2019) with relationship recall $$\mathcal{G}_\text{global} = (\mathcal{V}_\text{global}, \mathcal{E}_\text{global}),$$7, object recall $$\mathcal{G}_\text{global} = (\mathcal{V}_\text{global}, \mathcal{E}_\text{global}),$$8, predicate recall $$\mathcal{G}_\text{global} = (\mathcal{V}_\text{global}, \mathcal{E}_\text{global}),$$9, object mean Recall $v_i = (c_i, \mu_i^{3D}, \Sigma_i^{3D}),$0, predicate mean Recall $v_i = (c_i, \mu_i^{3D}, \Sigma_i^{3D}),$1, and latency $v_i = (c_i, \mu_i^{3D}, \Sigma_i^{3D}),$2 ms. The paper therefore attributes to FROSS the best reported relationship recall, object recall, object mean Recall, and predicate recall in that comparison, together with substantially lower latency.

The end-to-end runtime breakdown on ReplicaSSG with a single core and GPU is reported as object detection $v_i = (c_i, \mu_i^{3D}, \Sigma_i^{3D}),$3 ms, relation extraction $v_i = (c_i, \mu_i^{3D}, \Sigma_i^{3D}),$4 ms, 3D SSG merging $v_i = (c_i, \mu_i^{3D}, \Sigma_i^{3D}),$5 ms, and total approximately $v_i = (c_i, \mu_i^{3D}, \Sigma_i^{3D}),$6 ms, corresponding to approximately $v_i = (c_i, \mu_i^{3D}, \Sigma_i^{3D}),$7 FPS.

Component	Latency	Reported role
Object detection	2.31 ms	RT-DETRv2-M
Relation extraction	4.51 ms	EGTR
3D SSG merging	0.12 ms	Online Gaussian-graph update

On ReplicaSSG, using the standard FROSS configuration with a 2D scene graph model trained on Visual Genome and no tuning on ReplicaSSG, the paper reports relationship recall $v_i = (c_i, \mu_i^{3D}, \Sigma_i^{3D}),$8, object recall $v_i = (c_i, \mu_i^{3D}, \Sigma_i^{3D}),$9, predicate recall $c_i$0, object mean Recall $c_i$1, and predicate mean Recall $c_i$2. With ground-truth 2D scene graphs, performance increases substantially. On ReplicaSSG, "FROSS w/ GT" reports relationship recall $c_i$3, object recall $c_i$4, predicate recall $c_i$5, object mean Recall $c_i$6, and predicate mean Recall $c_i$7. On 3DSSG, "FROSS w/ GT" reports relationship recall $c_i$8, object recall $c_i$9, predicate recall $\mu_i^{3D} \in \mathbb{R}^3$0, object mean Recall $\mu_i^{3D} \in \mathbb{R}^3$1, and predicate mean Recall $\mu_i^{3D} \in \mathbb{R}^3$2. The paper explicitly interprets this as evidence that 2D scene graph quality is the main bottleneck rather than the 3D lifting and merging mechanism.

The ablation results further characterize this behavior. For the Hellinger distance threshold $\mu_i^{3D} \in \mathbb{R}^3$3 on 3DSSG validation, lower thresholds such as $\mu_i^{3D} \in \mathbb{R}^3$4 to $\mu_i^{3D} \in \mathbb{R}^3$5 yield higher object recall, approximately $\mu_i^{3D} \in \mathbb{R}^3$6 to $\mu_i^{3D} \in \mathbb{R}^3$7, but lower relationship recall, approximately $\mu_i^{3D} \in \mathbb{R}^3$8 to $\mu_i^{3D} \in \mathbb{R}^3$9. Higher thresholds such as $\Sigma_i^{3D} \in \mathbb{R}^{3\times3}$0 to $\Sigma_i^{3D} \in \mathbb{R}^{3\times3}$1 increase relationship recall to approximately $\Sigma_i^{3D} \in \mathbb{R}^{3\times3}$2 to $\Sigma_i^{3D} \in \mathbb{R}^{3\times3}$3 while reducing object recall to approximately $\Sigma_i^{3D} \in \mathbb{R}^{3\times3}$4. The chosen $\Sigma_i^{3D} \in \mathbb{R}^{3\times3}$5 therefore favors relational accuracy. For camera pose accuracy on ReplicaSSG, replacing ground-truth poses with ORB-SLAM3 trajectories changes relationship recall from $\Sigma_i^{3D} \in \mathbb{R}^{3\times3}$6 to $\Sigma_i^{3D} \in \mathbb{R}^{3\times3}$7, object recall from $\Sigma_i^{3D} \in \mathbb{R}^{3\times3}$8 to $\Sigma_i^{3D} \in \mathbb{R}^{3\times3}$9, and predicate recall from $I_t$00 to $I_t$01, which the paper characterizes as a minor effect. For the 2D scene graph backbone, RT-DETR+EGTR has latency of approximately $I_t$02 ms versus approximately $I_t$03 for original EGTR and is selected for the reported speed-accuracy tradeoff.

Qualitatively, the paper reports that FROSS localizes and labels objects reasonably well, merges partial observations into single coherent nodes, and correctly predicts relations such as standing on, supported by, and attached to. Compared with Wu (2023), it is reported to misclassify fewer geometrically similar objects such as curtain versus door and chair versus sofa, and to show less confusion between standing on and supported by.

6. Limitations, scope, and relation to adjacent paradigms

The paper states several limitations directly (Hou et al., 26 Jul 2025). The first is dependence on 2D scene graph quality. Performance is bounded by the quality of object detection and relationship prediction in 2D, especially under cross-domain transfer. The large gap between predicted-2D-SG and ground-truth-2D-SG settings is the clearest empirical evidence. The second is class imbalance in predicates: FROSS achieves high predicate recall overall, but predicate mean Recall is not the best on 3DSSG, and the paper attributes this to heavy predicate imbalance. The third is approximate geometry. Objects are modeled as Gaussians rather than precise shapes, which is adequate for high-level reasoning but not for tasks that require fine geometry, such as grasp synthesis. The fourth is a static-scene assumption, since evaluation is performed on static indoor scenes and moving objects are not explicitly handled. The fifth is the requirement for known camera pose per frame, even though the paper shows robustness with ORB-SLAM3. The sixth is the absence of a dedicated learned 3D relation reasoning module; graph structure is induced from 2D scene graphs rather than learned directly in 3D.

A common misconception would be to treat FROSS as a dense 3D reconstruction method. It is not. The method is explicitly designed to avoid constructing or processing dense 3D point clouds and instead maintains a sparse graph of Gaussian-distributed objects and relations. In the paper’s comparison to SLAM, TSDF, NeRF, and 3D Gaussian Splatting, those paradigms are characterized as focusing on precise geometry, surfaces, or high-fidelity rendering, whereas FROSS focuses on semantic entities and relationships. The paper therefore presents FROSS as complementary to geometric mapping rather than as a replacement for it.

Another plausible misconception is that "faster-than-real-time" implies an end-to-end embodied stack without upstream assumptions. The paper does not make that claim. Its runtime accounting excludes SLAM or mapping cost and assumes that camera poses are provided. The reported speedup arises primarily from eliminating dense 3D reconstruction and using compact Gaussian representations, closed-form Bhattacharyya and Hellinger computations, and analytic Gaussian fusion.

Within the application space described by the paper, FROSS is positioned for robotics and embodied AI, AR/VR, scene synthesis and editing, and interactive agents. This suggests a role as a semantic, persistent, view-invariant world model for task planning and relational reasoning when detailed surface reconstruction is unnecessary or too costly.