SCOUT: Semantic scene COverage via Uncertainty-guided Traversal

Abstract: Robots that operate over extended periods should not merely visit space; they should progressively understand it. Yet most 3D scene graph pipelines treat perception as a post-processing stage over a fixed dataset, decoupling scene representation from the decisions that determine what is observed in the first place. We present SCOUT, an online semantic exploration framework that closes this loop by coupling active traversal with probabilistic scene graph construction. Given a prior 2D occupancy map and posed RGB-D observations, SCOUT incrementally builds an uncertainty-aware 3D scene graph whose nodes maintain fused geometry and posterior beliefs over open-vocabulary object labels, while edges encode structural relations such as on, inside, belong, and next to. These beliefs are fed back to an uncertainty-guided traversal planner, which selects viewpoints by balancing expected semantic certainty gain, geometric coverage gain, and travel cost. In this way, the robot revisits ambiguous objects when additional evidence matters and expands into unseen free space when the scene remains incomplete. The resulting system treats semantic scene completeness as an operational objective rather than a passive by-product of semantic mapping, moving toward autonomous agents that can patrol, update, and reason about evolving indoor environments with minimal human intervention.

• The paper introduces SCOUT, an active framework that integrates semantic perception and 3D scene graph generation with uncertainty guidance.
• It employs a Probabilistic Scene Graph Generator and Uncertainty-Guided Traversal to fuse RGB-D data with open-vocabulary recognition and Bayesian updates.
• Empirical results show SCOUT outperforms fixed traversal strategies in complex, occluded 3D environments by optimizing semantic certainty and spatial coverage.

SCOUT: Semantic Scene Coverage via Uncertainty-Guided Traversal

Motivation and Problem Formulation

Robotic scene understanding for open-world, long-term deployment necessitates integrating perception with action, enabling not just passive scene reconstruction but iterative, evidence-guided semantic reasoning. Most prior 3D scene graph approaches decouple scene graph generation from exploration, employing fixed traversal plans followed by post-hoc synthesis of semantic maps from collected RGB-D data. This paradigm is limited, both theoretically and practically, as it does not leverage evolving scene knowledge to prioritize observations, and it cannot adapt to uncertainty or occlusion during mission time. The introduction of SCOUT addresses these limitations by formulating semantic scene completeness as an operational objective, explicitly closing the perception-action loop.

System Architecture and Algorithms

SCOUT implements an online active-perception system that fuses open-vocabulary visual recognition, geometric evidence, and an uncertainty-aware scene prior to incrementally construct and refine 3D scene graphs. The system consists of two tightly coupled modules: (1) the Probabilistic Scene Graph Generator (PSGG), responsible for multi-modal semantic fusion and uncertainty quantification, and (2) the Uncertainty-Guided Traversal (UGT) planner, which selects informative viewpoints by optimizing a weighted utility over expected semantic gain, geometric coverage, and locomotion cost.

Figure 1: System-level dataflow: PSGG synthesizes 3D scene graphs from RGB-D observations, feeding uncertainty estimates and structural relations to UGT, which then plans the next optimal viewpoint, forming a closed active loop.

Probabilistic Scene Graph Generation

PSGG processes posed RGB-D frames in real time to integrate deep open-vocabulary recognition (Grounding DINO + SAM + CLIP) with 3D geometry and history. Each candidate object instance is represented in the scene graph as a node encoding: (1) fused 3D geometry, (2) posterior probability over object classes $\mathcal{L}$, (3) an uncertainty measure (Shannon entropy), (4) history of observed views, and (5) associated structural relations (edges: on, inside, belong, next to).

Figure 2: Internal PSGG architecture, illustrating open-vocabulary detection, mask segmentation, embedding fusion, and incremental scene graph update.

Observation-integration leverages a hybrid similarity metric across geometry, vision, and language embedding dimensions for robust association and fusion. Bayesian multiplicative updates refine class posteriors, explicitly maintaining soft ambiguity. Relational edges are incrementally updated through geometric heuristics, with directional and symmetry semantics.

Uncertainty-Guided Viewpoint Planning

UGT formulates candidate viewpoints using hybrid semantic-geometric generators. Semantic candidates target regions of maximal persistent entropy in the scene graph (i.e., ambiguous objects detected so far), while geometric candidates focus on unobserved but navigable free space. Each candidate is scored via a utility function per:

$$S_{\mathrm{final}}(v) = \frac{w_{\mathrm{cert}} S_{\mathrm{cert}}(v) + w_{\mathrm{cov}} S_{\mathrm{cov}}(v)}{S_{\mathrm{travel}}(v)}$$

where certainty gain estimates expected entropy reduction via observability (3D raycasting) and view novelty (angular separation from prior viewpoints), and coverage gain measures additional FOV sweep. This formulation efficiently couples information-theoretic query selection with spatial/energy cost, driving the robot toward views that are maximally informative yet motion-efficient.

SCOUT terminates once geometric coverage and node uncertainties cross empirically chosen convergence thresholds, ensuring both spatial completeness and semantic disambiguation.

Empirical Evaluation

The quantitative assessment is performed in Gazebo simulation across two environments—the simple scenario (no occlusion, all objects visible) and the challenging scenario (rotations and strong occlusion). Baseline comparison uses a geometric-only “lawnmower” sweep—representative of state-of-the-art decoupled pipelines—where viewpoints are fixed and semantic analysis is passive. Node- and edge-level precision/recall metrics are computed against ground truth.

Figure 3: The simple scenario, illustrating relative ease of semantic completion by both methods.

Figure 4: Lawnmower trajectory, visualizing exhaustive but context-insensitive sweep.

SCOUT achieves perfect node detection (precision and recall of 1.00) in the simple scenario, outperforming the baseline (0.86/0.79). In the challenging scenario, SCOUT achieves 0.93/0.94 node-level precision/recall versus 0.72/0.74 for the baseline, indicating robust semantic disambiguation despite occlusion and ambiguities. Edge-level performance is lower—recall is notably reduced in the challenging case due to association issues and partial object visibility.

Figure 5: Mean entropy (semantic uncertainty) reduction over time, showing SCOUT’s steady, monotonic convergence versus the baseline’s stepwise and less consistent decrease.

Notably, SCOUT's entropy curves plateau when the exploration phase is dominated by structural verification rather than semantic uncertainty reduction, or when complex occlusions slow viewpoint optimization. The system occasionally fails to reach full convergence within the fixed observation budget in adverse cases, highlighting future work on more nuanced balancing of coverage and exploitation.

Implications and Future Directions

SCOUT operationalizes semantic completeness as an explicit active-perception objective, which represents a distinct shift from passive mapping and post-hoc scene synthesis approaches. This framework supports continual refinement, on-the-fly revisitation of ambiguities, and adaptive scene graph evolution as environments or observation conditions change. Practically, the architecture is particularly well-suited for applications in security, facility management, logistics, and mobile manipulation in dynamic indoor environments, where data incompleteness and limited human supervision are the norm.

Theoretically, this work points toward more deeply integrated frameworks where perception, action, and representation are co-optimized. Future research directions include extending the uncertainty model to relational edges, enabling hierarchical or object-centric graph completion, and deploying learned view-selection policies through reinforcement learning or Bayesian experimental design. Furthermore, deploying SCOUT in real-world, long-duration experiments will provide insight into robustness, real-time adaptation, and lifelong learning.

Conclusion

SCOUT presents a formal, tightly integrated framework for active 3D scene graph generation, coupling uncertainty-aware semantic fusion with information-theoretic viewpoint selection. Empirical results demonstrate superior performance over decoupled baselines, particularly in complex, ambiguous settings. The approach lays a foundation for more autonomous, adaptive, and robust robotic perception-action systems that maintain and reason about evolving semantic spatial memory with minimal human input.