SignScene: Spatial-Semantic Robot Navigation

• SignScene is a spatial-semantic navigation framework that leverages real-world signs to guide robot path planning without pre-built maps.
• It integrates multimodal perception, robust sign parsing, and VLM-based reasoning to convert abstract sign cues into actionable navigational commands.
• Empirical evaluations on a Boston Dynamics Spot demonstrate 88.6% grounding accuracy across diverse environments, significantly outperforming traditional baselines.

SignScene is a spatial-semantic framework for robot mapless navigation that leverages real-world navigational signs as the sole source of high-level guidance. It formalizes the core problem of sign grounding—mapping the diverse, abstract semantic contents of signs (arrows, icons, text) to executable navigational actions and corresponding elements of the local 3D environment. SignScene achieves this by constructing sign-centric spatial-semantic representations optimized for Vision-LLMs (VLMs), combining multimodal perception, robust sign parsing, canonicalized spatial mapping, and context-aware VLM prompt engineering. Demonstrations show end-to-end navigation on a legged robot (Boston Dynamics Spot) across diverse real-world domains, substantially outperforming existing baselines and ablations in grounding accuracy (Zimmerman et al., 13 Feb 2026).

1. Formal Problem Definition and Decomposition

SignScene operates in previously unseen, human-built environments, where a robot receives guidance solely from observed navigational signs embedding semantic location labels and high-level instructions. The workflow decomposes into two tightly coupled subproblems:

• Sign Understanding: parsing an observed sign image $I_{\rm sign}$ into a set of semantic cues,

$C = \bigl\{(\ell_j, d_j)\bigr\}_{j=1}^J,$

where each pair $(\ell_j, d_j)$ contains a location label (e.g., “Pharmacy”) and a navigational instruction (e.g., “forward-then-left”).

• Sign Grounding: given a semantic cue $(\ell_j, d_j)$ and local observations $\{I_t, u_t\}$ (RGB images and odometry), map $d_j$ to a concrete path or action in the scene that leads toward $\ell_j$. The grounding function,

$$G: \mathcal{D} \times 2^{\mathbb{O}} \to \mathcal{P},$$

takes the navigational instruction $d_j$ and a set of scene elements $\mathbb{O}$ to select a path or subgoal $C = \bigl\{(\ell_j, d_j)\bigr\}_{j=1}^J,$0 (pose in $C = \bigl\{(\ell_j, d_j)\bigr\}_{j=1}^J,$1). Correct grounding ensures that following $C = \bigl\{(\ell_j, d_j)\bigr\}_{j=1}^J,$2 enters the path signified by $C = \bigl\{(\ell_j, d_j)\bigr\}_{j=1}^J,$3 and progresses toward $C = \bigl\{(\ell_j, d_j)\bigr\}_{j=1}^J,$4 (Zimmerman et al., 13 Feb 2026).

2. Spatial–Semantic Representation and Abstract Top-view Map (AToM)

SignScene fuses multimodal perception into a compact, sign-centric 3D map, which is subsequently rendered as an Abstract Top-view Map (AToM) for VLM reasoning. The spatial-semantic representation comprises:

• 3D Map ($C = \bigl\{(\ell_j, d_j)\bigr\}_{j=1}^J,$5):
  • Navigational signs: each sign $C = \bigl\{(\ell_j, d_j)\bigr\}_{j=1}^J,$6 is encoded as $C = \bigl\{(\ell_j, d_j)\bigr\}_{j=1}^J,$7, with centroid position $C = \bigl\{(\ell_j, d_j)\bigr\}_{j=1}^J,$8, orientation $C = \bigl\{(\ell_j, d_j)\bigr\}_{j=1}^J,$9, and parsed cues $(\ell_j, d_j)$0.
  • Explicit structures: objects $(\ell_j, d_j)$1 (e.g., stairs, escalators) as oriented bounding boxes $(\ell_j, d_j)$2 plus semantic labels.
  • Implicit paths: dense traversable regions represented as point cloud $(\ell_j, d_j)$3 from segmentation.
• AToM Rendering ($(\ell_j, d_j)$4): The 3D map is recentered and rotated into the canonical (sign-centric) frame before projection,

$(\ell_j, d_j)$5

Here, $(\ell_j, d_j)$6 yields a simplified polygonal mask of traversable ground; convex protrusions produce discrete “frontier” points $(\ell_j, d_j)$7 (annotated A, B, C, ...). $(\ell_j, d_j)$8 projects explicit structures as labeled rectangles. $(\ell_j, d_j)$9 overlays textual labels at the sign center. This abstraction removes textural clutter, enabling VLMs to reason over topologically relevant scene elements and affordances.

3. System Architecture and Prompting Strategy

The SignScene pipeline consists of four modular components:

1. Sign Detection & Alignment: GroundingDINO (open-set detection) and SAM2 segmenter extract sign masks $(\ell_j, d_j)$0. Metric3Dv2 provides depth, enabling 3D cluster formation, temporally fused for robust localization. Visual servoing aligns the robot in the canonical view, within distance and angle thresholds $(\ell_j, d_j)$1.
2. Sign Understanding: GPT-5 (or Gemini-2.5-Pro) performs context-driven parsing of the observed sign, using in-context examples for symbols (arrow, stair, escalator) and textual locations, producing $(\ell_j, d_j)$2. The parser outputs simplified JSON associating instructions and labels.
3. Map Building & Exploration: RGB and odometry input GeNIE path segmentation and structure detection (GroundingDINO). Detected sign locations, objects, and ground point clouds are temporally fused. If map coverage is incomplete, AToM-driven frontier exploration targets unsampled convex protrusions.
4. VLM-Based Grounding: For a queried goal $(\ell_j, d_j)$3, fuzzy-matching is performed with all $(\ell_j, d_j)$4 in $(\ell_j, d_j)$5 (using Levenshtein distance). The relevant AToM is rendered and used as a multiple-choice prompt to GPT-5 or Gemini-2.5-Pro: “Select a letter or object name in the image closest to $(\ell_j, d_j)$6 given the parsed directions $(\ell_j, d_j)$7…” The selected candidate ($(\ell_j, d_j)$8 or $(\ell_j, d_j)$9) is back-projected to $\{I_t, u_t\}$0, yielding subgoal $\{I_t, u_t\}$1.

4. Dataset, Environments, and Evaluation Protocol

SignScene was evaluated across nine environment types: hospitals (A/B), malls (A/B/C), MRT stations, airport terminals, and campus outdoor pathways. Data collection consisted of 36 hand-held sequences with synchronized RGB, sparse depth, and visual-inertial odometry, each centered on at least one sign. The primary evaluation protocol used 114 multiple-choice queries per sign; each view presents four candidate landing points (A/B/C/D), with only one ground-truth. Metrics include:

• Sign parsing success: accurately extracting $\{I_t, u_t\}$2 for the queried label.
• Grounding accuracy: selecting the correct candidate among four.

5. Quantitative Results and Ablation Studies

Empirical results demonstrate high sign grounding efficacy and clarifies the impact of key architectural choices:

Method	Grounding Accuracy	Parsing Success
SignScene (full model)	88.6% (101/114)	82% (93/114)
No-Rotation ablation	64.9% (74/114)	n/a
ReasonNav baseline	26.3% (30/114)	n/a
Agrawal et al. (parsing)	n/a	33% (38/114)
Chandaka et al. (parsing)	n/a	47% (53/114)
Ours, w/o temporal filtering	n/a	54% (61/114)

The ablations underscore that canonical sign-centric rotation is critical for VLM reasoning, and that incorporating in-context symbol prototypes with temporal fusion markedly improves sign parsing reliability (Zimmerman et al., 13 Feb 2026).

6. Real-World Robotic Deployment and Qualitative Performance

SignScene was deployed on a Boston Dynamics Spot (Jetson Orin AGX) for live mapless navigation. The robot (1) rotates to face detected signs and parses content (e.g., 3 seconds via Gemini-2.5-Pro), (2) builds or expands $\{I_t, u_t\}$3 by exploring sign-centered frontiers, and (3) queries GPT-5 (typically 20 seconds per AToM prompt) to select the navigational path. In trials where the target (“TERRACE”) is not initially visible, the robot is able to ground intermediate signage (e.g., “stairs → TERRACE”), identify relevant affordances (such as “up-stairs”), and traverse to the correct location in the absence of a pre-built map. Empirically, SignScene demonstrates robustness to environmental variation, including metallic flooring, glass walls, and outdoor trails, enabling reliable end-to-end mapless navigation using only sign cues (Zimmerman et al., 13 Feb 2026).

7. Summary and Significance

SignScene formalizes and operationalizes the sign grounding problem as $\{I_t, u_t\}$4, leverages a sign-centric spatial-semantic representation for VLM reasoning, and utilizes in-context symbol prompting plus a canonicalized AToM for effective scene abstraction. The system achieves 88.6% grounding accuracy across nine environment domains, outperforming prior baselines and ablations substantially. By demonstrating real-world navigation on a legged robot purely from visual sign cues, SignScene advances the integration of vision, language, and spatial reasoning for autonomous robot navigation in open, mapless settings (Zimmerman et al., 13 Feb 2026).