KeySG: Hierarchical 3D Scene Graph Framework

• KeySG is a framework that represents large 3D environments as hierarchical, keyframe-based scene graphs, integrating geometric, semantic, and multimodal data to support language-based reasoning.
• It employs a hierarchical retrieval-augmented generation pipeline that extracts only relevant context from floor, room, frame, and object levels, addressing scalability issues in large scenes.
• Empirical evaluations demonstrate that KeySG outperforms earlier methods in semantic segmentation and object grounding benchmarks, confirming its efficacy in human-centered applications.

Searching arXiv for the KeySG paper and closely related 3D scene graph work to ground the article. KeySG is a framework for representing large 3D environments as hierarchical, keyframe-based 3D scene graphs in which geometric structure, semantic content, and multimodal descriptions are integrated to support language-based reasoning, planning, and retrieval in human-centered environments. It is introduced as a response to two perceived limitations of prior 3D scene graph approaches: first, reliance on a predefined set of explicit relationship edges, which can make a graph optimized for one relation schema “inherently suboptimal” for other tasks; and second, poor scalability when large graphs are serialized for LLMs, where context-window overflow, “lost in the middle,” and attentional bias can degrade performance. KeySG addresses these issues by preserving task-agnostic multimodal context in selected room keyframes, organizing that information hierarchically across buildings, floors, rooms, objects, and functional elements, and retrieving only relevant context through a hierarchical retrieval-augmented generation pipeline (Werby et al., 1 Oct 2025).

1. Conceptual basis and scope

KeySG is presented as a hierarchical graph consisting of floors, rooms, objects, and functional elements, with the building as the top enclosing concept, and with nodes augmented by multimodal information extracted from selected keyframes (Werby et al., 1 Oct 2025). In standard 3D scene graph formulations, a scene graph is represented as $G = (V, E)$, where nodes represent entities such as objects, rooms, floors, or buildings, and edges encode predefined relations such as part-of, next-to, larger-than, or functional interactions. KeySG retains explicit hierarchy and containment, but shifts much of the burden of semantic representation away from predefined pairwise edges and into keyframe images and text descriptions (Werby et al., 1 Oct 2025).

The framework’s stated motivation is that a graph constructed around one family of relations may support one class of queries well while remaining weak for others. A graph dominated by spatial edges can favor spatial retrieval, whereas a graph built around functional edges can favor interaction queries. KeySG therefore proposes that scene details such as layout, semantics, object relationships, state, and affordance be stored implicitly in keyframes and their corresponding descriptions, so that relations can be inferred at query time rather than fixed in advance (Werby et al., 1 Oct 2025).

A second design premise is scalability. Prior systems that serialize large scene graphs directly into prompts can exceed LLM context windows in building-scale scenes. KeySG answers this with a hierarchical retrieval mechanism aligned with the graph topology, so that only the floor-, room-, frame-, and object-level context relevant to a query is surfaced (Werby et al., 1 Oct 2025). This suggests a reframing of the 3D scene graph from a static relation graph into a hierarchical multimodal memory structure.

2. Hierarchical scene representation

The input to KeySG is a posed RGB-D sequence

$\mathcal{I}=\{P_t, I_t, D_t\}_{t=1}^T$

where $P_t = [R_t | t_t] \in SE(3)$ is the camera pose, $I_t$ the RGB image, and $D_t$ the depth map. From this sequence, the framework reconstructs a global 3D point cloud

$\mathcal{P}_{scene} \in R^{N \times 3}.$

The point cloud is then segmented hierarchically into floors and rooms (Werby et al., 1 Oct 2025).

Floors are partitioned as $\{\mathcal{F}_i\}_{i=1}^{N_F}$, and each floor is segmented into room point clouds $\{\mathcal{R}_{ij}\}_{j=1}^{N_{R_i}}$. The implementation follows prior work cited by the authors: floors are segmented by computing a height histogram and selecting dominant peaks, while rooms are segmented using a bird’s-eye-view 2D histogram followed by Watershed segmentation (Werby et al., 1 Oct 2025). These operations establish the upper levels of the hierarchy before object- and functional-element-level processing.

The paper characterizes the full hierarchy as spanning five levels of abstraction: buildings, floors, rooms, objects, and functional elements (Werby et al., 1 Oct 2025). Operationally, floors contain floor summaries; rooms belong to floors and contain room point clouds, sparse keyframe multimodal data, and room summaries; objects belong to rooms and contain merged 3D object point clouds and visual-semantic embeddings; and functional elements are attached as subnodes to their corresponding objects. The hierarchy therefore mirrors both physical containment and semantic abstraction, with broad environment structure at the top and instance-level semantics at the bottom (Werby et al., 1 Oct 2025).

The stored representation includes floor summaries, room summaries, keyframe data, merged object geometries, CLIP embeddings, and functional-element subnodes. This is not an edge-free representation: containment remains explicit. Rather, the claim is that rich task-specific relations need not be exhaustively modeled in advance as explicit edges (Werby et al., 1 Oct 2025).

3. Keyframe sampling and multimodal augmentation

A central technical component of KeySG is room-level keyframe sampling. The authors explicitly distinguish their goal from SLAM-oriented keyframe selection: the objective is visual room coverage rather than reconstruction accuracy (Werby et al., 1 Oct 2025). For each room $\mathcal{R}_i$, frames are first assigned whose camera center lies inside the room volume: $$\mathcal{D}'_i = \{P_t \in \mathcal{I} \mid t_t \in \mathrm{Vol}(\mathcal{R}_i)\}.$$ This set is then refined to exclude frames whose camera is physically in one room but visually directed into another region. A frame is retained only if a significant fraction $\mathcal{I}=\{P_t, I_t, D_t\}_{t=1}^T$0 of its back-projected 3D points fall within the room’s 2D polygon, producing the dense room-specific frame set $\mathcal{I}=\{P_t, I_t, D_t\}_{t=1}^T$1 (Werby et al., 1 Oct 2025).

Keyframes are selected by clustering standardized 7D pose features. For each pose matrix $\mathcal{I}=\{P_t, I_t, D_t\}_{t=1}^T$2, the feature is

$\mathcal{I}=\{P_t, I_t, D_t\}_{t=1}^T$3

where $\mathcal{I}=\{P_t, I_t, D_t\}_{t=1}^T$4 is the quaternion corresponding to the rotation matrix and $\mathcal{I}=\{P_t, I_t, D_t\}_{t=1}^T$5 is a scalar rotation weight. Standardization is defined by

$\mathcal{I}=\{P_t, I_t, D_t\}_{t=1}^T$6

DBSCAN is applied to the standardized pose features, and from each cluster the medoid is selected: $\mathcal{I}=\{P_t, I_t, D_t\}_{t=1}^T$7 The resulting keyframe subset is

$\mathcal{I}=\{P_t, I_t, D_t\}_{t=1}^T$8

The paper gives a concrete example in which ScanNet scene0011_00 is reduced from 2374 frames to 23 keyframes while preserving 96.26% geometric coverage relative to the full dense point cloud (Werby et al., 1 Oct 2025).

These keyframes drive multimodal augmentation. A VLM generates object tags and functional element tags per frame, which are aggregated into room-level tag sets $\mathcal{I}=\{P_t, I_t, D_t\}_{t=1}^T$9 and $P_t = [R_t | t_t] \in SE(3)$0 (Werby et al., 1 Oct 2025). These tags guide an open-vocabulary detection and segmentation pipeline on the dense room frame set $P_t = [R_t | t_t] \in SE(3)$1, yielding object point clouds in global coordinates. Redundant segments are incrementally merged using geometric overlap, following the authors’ prior HOV-SG method as described in the KeySG paper (Werby et al., 1 Oct 2025).

For each merged object, KeySG performs best-view selection. Object masks are scored by size and distance from image boundaries, favoring large and centrally located views; the highest-scoring view is used to compute the object’s CLIP embedding (Werby et al., 1 Oct 2025). The same best view, together with room-level functional tags, is used for open-vocabulary functional-element segmentation. The paper argues that this best-view strategy materially improves open-vocabulary 3D semantic segmentation (Werby et al., 1 Oct 2025).

4. Descriptions, summaries, and implicit relations

KeySG’s distinctive claim is that many scene relations need not be encoded as explicit edges if multimodal evidence is retained and structured appropriately. To generate grounded frame descriptions, the system first determines which recovered 3D objects are visible in each keyframe. Visibility is computed geometrically: object points are transformed into camera coordinates, projected onto the image plane, points outside the image are discarded, and an occlusion test compares projected depth to the depth map. An object is considered visible if the fraction of visible points exceeds a threshold $P_t = [R_t | t_t] \in SE(3)$2 (Werby et al., 1 Oct 2025).

The keyframe image and its list of visible objects are then passed to the VLM to generate a geometrically grounded textual description. The conditioning step is intended to ensure that the description refers to entities that are actually present and visible in the reconstructed 3D scene graph (Werby et al., 1 Oct 2025). Frame-level descriptions are then aggregated into room summaries using an LLM, and room summaries are further aggregated into floor summaries (Werby et al., 1 Oct 2025).

This produces semantic context at multiple levels: local frame descriptions, room-level summaries, and floor-level summaries. The paper explicitly states that “scene details, such as layout, semantics, object relationships, state, and affordance, are implicitly stored within the keyframes and their corresponding text descriptions, alleviating the need to explicitly model these specific relations as edges in the 3D scene graph” (Werby et al., 1 Oct 2025). The intended implication is not that all graph structure disappears, but that task-specific relational structure can be recovered from retrieved multimodal evidence.

A plausible implication is that KeySG treats scene understanding as partly deferred inference: the graph stores enough multimodal context to support reasoning later, instead of committing to a narrow relation schema during graph construction.

5. Hierarchical retrieval-augmented querying

At query time, KeySG uses a hierarchical multimodal retrieval-augmented generation pipeline to interface with LLMs and VLMs while avoiding full-graph serialization (Werby et al., 1 Oct 2025). The preparation phase has three major steps: create text chunks grouped by graph level, yielding floor, room, frame, and object chunk types; compute vector embeddings for all chunks and index them by type; and build a visual vector database over scene keyframes together with a separate database of object-level CLIP embeddings (Werby et al., 1 Oct 2025).

For natural-language questions, the system may optionally parse the query into a <target object> and a set of <anchor objects> using an LLM, following the BBQ-style strategy cited in the KeySG paper (Werby et al., 1 Oct 2025). Embeddings are then computed for the target and anchors, and related text chunks, keyframes, and objects are retrieved hierarchically via cosine similarity (Werby et al., 1 Oct 2025).

The paper describes two variants for hierarchical object retrieval. In a non-RAG variant, an LLM decomposes a query into $P_t = [R_t | t_t] \in SE(3)$3, and CLIP text embeddings for each concept are matched to the corresponding graph level. In the RAG variant, the raw query embedding is compared directly to all floor summary embeddings; the highest-scoring floor is selected, then the same query embedding is compared to room summary embeddings within that floor, and finally to object embeddings within that room (Werby et al., 1 Oct 2025). Retrieval is therefore top-down: global summary first, then progressively narrower local context.

This retrieval design directly addresses the scalability problem because the entire graph never needs to be inserted into one prompt (Werby et al., 1 Oct 2025). The graph thus functions as a structured multimodal store from which only compact, relevant evidence is extracted for answer generation. KeySG is described as capable of answering direct grounding queries such as “Where is the coffee mug in the kitchen?” as well as more indirect references involving nearby objects, attributes, status, or omitted object names (Werby et al., 1 Oct 2025).

6. Empirical evaluation and reported performance

KeySG is evaluated on four benchmarks spanning segmentation and retrieval tasks, and the paper reports that it outperforms prior approaches on most metrics (Werby et al., 1 Oct 2025).

Open-vocabulary 3D semantic segmentation

On Replica (office0-office4 and room0-room2), using the ConceptGraphs evaluation protocol with prompts of the form “an image of {class name},” KeySG achieves 45.81 mAcc and 46.16 F-mIoU (Werby et al., 1 Oct 2025). The paper reports the following comparison:

Method	mAcc	F-mIoU
MaskCLIP	4.53	0.94
Mask2Former+CLIP	10.42	13.11
ConceptFusion	24.16	31.31
ConceptFusion+SAM	31.53	38.70
ConceptGraphs	40.63	35.95
ConceptGraphs-Detector	38.72	35.82
Clio	37.95	36.26
HOV-SG	38.07	40.16
KeySG	45.81	46.16

The authors interpret this result as support for the claim that best-view selection improves semantic labeling (Werby et al., 1 Oct 2025).

Functional-element 3D segmentation

On FunGraph3D, with Recall@K measured for $P_t = [R_t | t_t] \in SE(3)$4 under IoU thresholds $P_t = [R_t | t_t] \in SE(3)$5, $P_t = [R_t | t_t] \in SE(3)$6, and $P_t = [R_t | t_t] \in SE(3)$7, KeySG outperforms FunGraph and OpenFunGraph on almost all reported metrics (Werby et al., 1 Oct 2025). The paper highlights:

• At R@3, KeySG scores 46.44 / 24.23 / 13.33 versus OpenFunGraph 45.34 / 5.39 / 0.31 and FunGraph 33.56 / 22.03 / 13.04.
• At R@5, KeySG scores 53.06 / 25.19 / 13.64 versus 47.74 / 6.89 / 1.50 and 35.79 / 22.93 / 13.64.
• At R@10, KeySG scores 57.12 / 27.57 / 14.53, trailing OpenFunGraph only on IoU$P_t = [R_t | t_t] \in SE(3)$8, where OpenFunGraph gets 60.40, but exceeding it at stricter IoU thresholds (Werby et al., 1 Oct 2025).

The paper interprets this pattern as indicating that OpenFunGraph may detect more elements loosely, whereas KeySG segments them more accurately geometrically (Werby et al., 1 Oct 2025).

Hierarchical object retrieval

On Habitat Matterport 3D Semantic scenes, for room-object and floor-room-object queries, KeySG is compared with HOV-SG, with and without RAG (Werby et al., 1 Oct 2025). For simple $P_t = [R_t | t_t] \in SE(3)$9 queries at R@1, the paper reports:

• HOV-SG: 23.30 / 0.60 / 0.00
• KeySG w/o RAG: 32.62 / 26.50 / 15.50
• KeySG w/ RAG: 34.00 / 30.40 / 20.60

The paper notes that the RAG-based variant often performs best despite not explicitly parsing the query into floor/room/object first, which is presented as evidence that summary-based top-down retrieval is effective and scalable (Werby et al., 1 Oct 2025).

Nr3D object grounding

On Nr3D, with grounding accuracy at IoU $I_t$0, KeySG achieves 30.4%, compared with BBQ at 28.3% (Werby et al., 1 Oct 2025). It is reported as best on most subsets:

• w spatial language: 37.7 vs 28.1
• w/o spatial language: 37.9 vs 29.8
• w color language: 36.2 vs 25.2
• w/o color language: 38.0 vs 29.0
• w/o shape language: 38.6 vs 27.3
• w target mention: 40.1 vs 29.6

The paper also identifies failure cases: KeySG trails BBQ on shape-language queries (33.4 vs 34.3) and on queries without explicit target mention (12.2 vs 14.8) (Werby et al., 1 Oct 2025). This suggests that multimodal keyframe context broadens coverage of referring expressions, but does not eliminate all ambiguity classes.

7. Significance, limitations, and implications

The paper identifies three main contributions: KeySG is introduced as the first 3DSG framework that models environments across five hierarchical levels; it augments graph nodes with multimodal context derived from adaptively sampled keyframes and compresses that context through hierarchical summarization; and it introduces a hierarchical RAG mechanism aligned with graph topology so that LLMs can reason over large environments without full-graph serialization (Werby et al., 1 Oct 2025).

The framework is also explicitly constrained. Graph construction is described as expensive because it depends on VLMs and LLMs, so the method is currently offline and assumes a pre-reconstructed scene (Werby et al., 1 Oct 2025). The resulting graph may then be used as a persistent knowledge base and queried efficiently in real time, but construction is not yet generally online or incremental. The method also assumes a static environment and does not model dynamic objects or changing object states over time (Werby et al., 1 Oct 2025).

In robotics terms, the paper positions KeySG as a persistent world model that preserves geometric grounding, semantic identity, room/floor organization, and keyframe-derived context about layout, attributes, affordances, and state (Werby et al., 1 Oct 2025). For navigation, the floor-room-object hierarchy supports large-scale localization and target retrieval. For planning and reasoning, room and floor summaries plus frame descriptions provide compact semantic context to LLMs. For manipulation, functional-element segmentation and object-level context may support reasoning about controls and affordances (Werby et al., 1 Oct 2025).

A common misconception would be to treat KeySG as a replacement for all graph structure with free-form descriptions. The paper does not make that claim. The hierarchy and containment relations remain explicit; what changes is the role of explicit pairwise semantic edges, many of which are replaced by retrieved multimodal evidence at query time (Werby et al., 1 Oct 2025). Another possible misconception would be to view the method primarily as a compression scheme for images. The paper instead presents keyframes as a mechanism for preserving latent semantic detail from the RGB stream while still enabling scalable retrieval (Werby et al., 1 Oct 2025).

Taken together, KeySG defines a 3D scene graph not simply as a static relational data structure but as a hierarchical multimodal substrate for deferred, task-conditioned reasoning. Its keyframe-based semantic augmentation and hierarchical retrieval pipeline are the mechanisms by which it seeks to overcome semantic rigidity and prompt-scale limitations in prior 3D scene graph systems (Werby et al., 1 Oct 2025).