Hierarchical Keyframe Scene Graphs

• Hierarchical keyframe scene graph construction is a method that structures dynamic and static 3D or video environments into multi-level graphs using keyframes.
• It employs keyframe selection strategies, such as clustering and medoid approximation, to ensure high geometric coverage and optimal sparsity.
• The approach leverages multi-modal feature extraction and optimization techniques to support real-time applications in SLAM, AR/VR, and video understanding.

Hierarchical keyframe scene graph construction defines a set of methods for representing dynamic or static 3D environments by organizing visual and spatial entities across multiple semantic, geometric, and temporal levels, with node and edge structures constructed from or aligned to selected keyframes. These pipelines aggregate geometric, semantic, and multi-modal features from sparse but information-rich frames or frame clusters—termed keyframes—into a graph structure whose nodes capture entities, places, or interactivities, and whose connections encode spatial, functional, or temporal relationships. Hierarchical design enables scalable, expressive, and context-aware scene understanding critical in robotics, AR/VR, and video understanding.

1. Hierarchical Scene Graph Representations

All modern pipelines for hierarchical keyframe scene graph construction follow a multi-level abstraction, mapping the 3D or video world into a graph $G=(V,E)$ with node sets stratified by semantic or spatial containment:

• Spatial Hierarchy (3D scene domain): Building $\supset$ Floor $\supset$ Room $\supset$ Object $\supset$ Functional Element. Each level aggregates the nodes below, with “is-part-of” edges forming the backbone; explicit “adjacency” or “attribute” edges are sometimes introduced but are often supplanted by node-level multi-modal attributes (Werby et al., 1 Oct 2025).
• Spatio-temporal Hierarchy (video domain): Frames are grouped into temporal cells, first at the per-frame level (detections as nodes), then recursively merged into cells covering increasing timespans, with edges encoding similarities or interactivities. This recursion yields cells whose node feature states capture broader temporal context (e.g., Hierarchical Interlacement Graph, HIG) (Nguyen et al., 2023).
• Semantic Hierarchy: Entities are defined on open-vocabulary labels or foundation model segmentations, permitting dynamic addition, merging, or relabeling as the graph evolves in response to novel observations (Zhu et al., 17 Mar 2026).

Hierarchical indexing reduces the serialization burden and enables targeted retrieval, supporting efficient scaling to large environments where full serialization would exceed reasoning system (e.g., LLM) context capabilities (Werby et al., 1 Oct 2025).

2. Keyframe Selection Strategies

Keyframe selection is the principal step for scene abstraction: only a carefully chosen subset of frames is processed for semantic and geometric extraction, enabling sparsity without compromising coverage or information content.

• 3D Scene Keyframe Selection: For each spatial unit (typically, a room), optimal keyframes $S^*\subset D$ are chosen to maximize a weighted sum of geometric coverage $G_{geo}(S)$ (fraction of 3D points observed) and visual diversity $G_{vis}(S)$ (pose or feature spread):

$$S^* = \arg\max_{S\subset D,\ |S| \leq K} [ \alpha G_{geo}(S) + (1-\alpha) G_{vis}(S) ]$$

This NP-hard set selection is approximated by clustering (DBSCAN) in pose space, with cluster medoids as keyframes; empirical results indicate $\geq$95% geometric coverage with keyframe budget $\supset$020-30 per room (Werby et al., 1 Oct 2025). In dynamic mapping, new keyframes are inserted based on robot movement thresholds, new entity arrivals, or stale-entity timers (Giberna et al., 3 Mar 2025).

• Video/Spatio-temporal Selection: In HIG (Nguyen et al., 2023), every level of the temporal hierarchy merges shorter intervals, and the final level corresponds to the span of the whole clip. Keyframes can be selected post hoc as those with node features nearest to the final graph-level representation.

This mechanism ensures both representational compactness and high recall of structural and semantic information.

3. Multi-modal and Semantic Feature Extraction

Hierarchical keyframe scene graphs rely on extracting semantically rich features from each selected keyframe using multi-modal foundation models, combining:

• Vision-LLMs (VLMs): Keyframe images, with or without visible-entity name lists, are passed into VLMs (e.g., GPT-4V), yielding natural language descriptions, object and functional tags, and scene-level summaries. These are stored node- or room-wise as human-interpretable attributes (Werby et al., 1 Oct 2025, Zhu et al., 17 Mar 2026).
• 3D Segmentation and CLIP Embeddings: Open-vocabulary segmentation algorithms, guided by extracted tag sets, segment frames into object/element masks, which are then back-projected and merged into 3D. Each entity node (object, functional element) is assigned a best-view CLIP embedding for downstream vision-language retrieval and grounding (Werby et al., 1 Oct 2025, Zhu et al., 17 Mar 2026).
• Gaussian Scene Representations: OGScene3D (Zhu et al., 17 Mar 2026) represents objects as Gaussian primitives $\supset$1, jointly modeling appearance, geometry, and semantic class with confidence. Semantic labels and confidences are both open-vocabulary and dynamically updated, supporting progressive discovery and relabeling.

Aggregation steps (e.g., union-over-keyframe object tag sets, mask filtering by image border proximity) ensure that multi-view and multi-modal evidence is optimally captured per node.

4. Graph Construction, Optimization, and Query

Graph construction is interleaved with optimization and progressive update; query and reasoning are naturally supported by the hierarchical structure.

• Graph Construction:
  • 3D graphs are built incrementally as exploration proceeds. Initial graphs are formed after a minimum number of keyframes (e.g., 12 in OGScene3D); after that, periodic updates merge or split nodes by similarity, recalculate edge labels, and condition on confidence and spatial co-proximity (Zhu et al., 17 Mar 2026).
  • In S-Graphs for SLAM (Bavle et al., 2023, Giberna et al., 3 Mar 2025), room-local subgraphs are extracted, redundant keyframes marginalized via the Schur complement, and moving-window or global optimization is performed after loop closure. Dynamic agents and entities are represented via additional constraint/factor graph edges; global optimization jointly solves for robot and entity states subject to these constraints.
• Semantic and Spatial Edges:
  • Edges primarily encode “is-part-of” hierarchical relations but can be augmented with natural language spatial/functional relations inferred using LLMs over node-feature vectors (Zhu et al., 17 Mar 2026, Werby et al., 1 Oct 2025).
  • Spatial proximity, adjacency, and attribute edges may be stored as additional edge types or as multi-modal node attributes, depending on downstream use.
• Inference and Retrieval: KeySG (Werby et al., 1 Oct 2025) exemplifies hierarchical retrieval-augmented generation (RAG). Graph attributes at each hierarchy (floor, room, keyframe, object) are separately indexed by CLIP or language encoders (FAISS databases). At query time, retrieval is staged top-down: the most relevant floors, rooms, keyframes, and objects are selected by embedding similarity, and node summaries are concatenated—never the full graph—into LLM prompts for scalable reasoning.
• Dynamic Graph Operations: Nodes are merged/split when label confidences or semantic tags converge/diverge; edges are batched for large LLM-based relation relabeling (Zhu et al., 17 Mar 2026).

5. Computational and Algorithmic Foundations

Efficiency and scalability are achieved by architectural and algorithmic design:

• Combinatorial Optimization: Keyframe selection in KeySG is reduced to a clustering/medoid problem, sidestepping the combinatorial bottleneck of optimal set cover (Werby et al., 1 Oct 2025).
• Marginalization and Compression: Room-local marginalization via the Schur complement reduces the number of active robot poses and observation factors, substantially lowering global SLAM optimization complexity from $\supset$2 to $\supset$3 for window size $\supset$4. Empirical results show a $\supset$540% reduction in per-step computation with no observable accuracy degradation (Bavle et al., 2023).
• Incremental and Parallel Pipelines: Offline scene graph construction (3D) is parallelized at the room level, while windowed or batch local optimization amortizes the cost of full global solves. Online query in KeySG and OGScene3D is bounded-constant per query due to fixed budget retrieval per hierarchy (Werby et al., 1 Oct 2025, Zhu et al., 17 Mar 2026).
• Realtime Dynamic SLAM: Entity- and keyframe-aware constraint graphs enable real-time optimization (70–100ms per cycle on commodity hardware, 27.6% reduction in ATE over baselines), with explicit filtering of dynamic entity point clouds for loop closure robustness (Giberna et al., 3 Mar 2025).

System	Coverage/Acc.	Optim. Speedup	Keyframe Policy
KeySG (Werby et al., 1 Oct 2025)	45.8% mAcc (Replica), 30.4% (Nr3D); >95% geom. coverage	Hierarchical indexing, parallel room-wise	DBSCAN-clustering/medoid per room
S-Graphs (Bavle et al., 2023)	$\supset$62% ATE deg.; $\supset$740% time red.	Room-local marginalization	Keep first per-room, windowed optim.
OGScene3D (Zhu et al., 17 Mar 2026)	Progressive, open-vocab, high node/edge accuracy	Hierarchical/batched, local+global	per-frame, periodic batch update
HIG (Nguyen et al., 2023)	R@20 $\supset$815% (video); R@100=26.3% (PSG)	Implicit by level, temporal pyramid	Level $\supset$9 span or feature-matching

6. Benchmarks and Quantitative Evaluation

Empirical findings consistently demonstrate the advantages of hierarchical keyframe scene graph construction:

• KeySG achieves 45.8/46.2% mAcc/F-mIoU on Replica (vs. 40.6/40.2% for prior SOTA), 25.2% Recall@5@IoU≥0.10 for functionals (vs. 22.9% FunGraph), and 30.4% object grounding accuracy (Nr3D), well above competing systems (Werby et al., 1 Oct 2025).
• Constraint-based dynamic SLAM (Giberna et al., 3 Mar 2025) reduces robot trajectory error by 27.6% (mean ATE 12.45cm→9.02cm), with average entity pose errors dropping from $\supset$08cm to $\supset$15cm after joint optimization.
• S-Graphs with room-local marginalization cut optimization times by 40% on real and simulated datasets, while ATE remains essentially unchanged (Bavle et al., 2023).
• OGScene3D achieves progressive, confidence-weighted open-vocabulary labeling and supports continual scene graph update, maintaining semantic consistency despite incremental exploration (Zhu et al., 17 Mar 2026).
• HIG demonstrates improvements in multi-actor attribute recall over state-of-the-art transformers for video scene graph generation, with precise temporal segmentation and scene-change detection (Nguyen et al., 2023).

These results confirm that hierarchical keyframe scene graph formulation preserves high accuracy in semantic and geometric tasks, scales to large domains, and supports both precise localization and compositional representation.

7. Extensions, Challenges, and Research Directions

Current methods face several open challenges:

• Limited Redundancy Handling: Current marginalization in S-Graphs is typically limited to one anchor pose per room; future work may extend compression to multiple anchors or floor-level (Bavle et al., 2023).
• Dynamic and Non-standard Scenes: Handling non-rectangular structures, open-vocabulary entity discovery, and major scene changes remains less mature; advanced segmentation and relabeling strategies are active research areas (Zhu et al., 17 Mar 2026).
• Contextual Reasoning: LLM-augmented hierarchical scene graphs (e.g., KeySG, OGScene3D) reveal new opportunities for multi-step, spatially grounded reasoning but are bottlenecked by context window and embedding limitations, suggesting research on continual retrieval and memory augmentation (Werby et al., 1 Oct 2025, Zhu et al., 17 Mar 2026).
• Benchmarks: New datasets for open-vocabulary, multi-agent, and dynamic scenes—as in HIG’s ASPIRe (Nguyen et al., 2023)—drive method development towards richer representations of interactivities and functional affordances.

Hierarchical keyframe scene graph construction thus serves as an enabling abstraction for the next generation of semantic mapping, robotic interaction, and video scene understanding, combining scalable computational pipelines with the expressive power of foundation models and multimodal reasoning systems.