Sketch Map-Based Visual Navigation (SkeNa)

• SkeNa is an embodied navigation paradigm that uses abstract, hand-drawn sketch maps with marked start and goal positions to guide navigation in unseen spaces.
• The approach leverages egocentric depth observations and a Ray-based Map Descriptor for aligning on-site exploration maps with the abstract sketch to infer goal locations.
• SAM-based navigation converts sketches into topological graphs with executable behaviors, enabling robust performance despite severe geometric distortions.

Searching arXiv for the specified SkeNa and Scene Action Maps papers to ground the article. Sketch map-based visual Navigation (SkeNa) is an embodied navigation paradigm in which an agent or robot must reach a specified goal in an unseen environment using an abstract 2D map rather than a metrically accurate prior map. In its formal task definition, SkeNa uses a hand-drawn sketch map that remains accessible throughout the episode and includes markers for the agent’s initial position and the goal; the agent must navigate from egocentric observations without precise floor plan, GPS, or compass information (Xu et al., 5 Aug 2025). A closely related line of work models navigation from sketches or floor-plans through Scene Action Maps (SAMs), also called “behaviour graphs,” in which an abstract, metrically inaccurate 2D map is parsed into a topological graph whose edges are executable navigational behaviours such as go-forward, turn-left, and turn-right (Loo et al., 2024). Together, these formulations define a research area centered on topology-oriented, human-like map guidance under severe geometric distortion.

1. Task definition and problem setting

SkeNa is defined as an embodied visual navigation task in which an agent must reach a specified goal in an unseen indoor environment using only its egocentric visual observations and an abstract, hand-drawn sketch map of the environment. The sketch map remains accessible throughout the episode and includes markers for the agent’s initial position and the goal. Inputs are a hand-drawn sketch map $S$, egocentric depth observations $D_t$ at each time step $t$, and an on-site exploration map $E_t$ constructed incrementally from the egocentric observations. Outputs are discrete navigation actions $$a_t \in \{\text{STOP}, \text{MOVE FORWARD}, \text{TURN LEFT}, \text{TURN RIGHT}\}$$, and an episode terminates successfully if the agent issues STOP within $T$ steps and its final position is within distance $\tau_d$ of the ground-truth goal (Xu et al., 5 Aug 2025).

The SAM formulation addresses a related but operationally distinct problem: navigating a real robot in an unseen 3D environment using only an abstract, metrically inaccurate 2D map, such as a hand-drawn sketch or floor-plan, without metric localization. The robot must plan and execute a sequence of navigational behaviours that exploit visual affordances such as corridors, junctions, rooms, and sidewalks. The assumptions are that the 2D map retains useful topological structure but is inaccurate in geometry, scale, and orientation; the robot has RGB-D sensing and odometry; navigation does not rely on precise metric localization or globally consistent geometric mapping; and the map can be parsed offline into a SAM before execution (Loo et al., 2024).

These two formulations occupy different points in the design space. SkeNavigator estimates a goal position in an evolving exploration map and learns a discrete action policy, whereas SAM-based navigation constructs an action-centric topological graph and localizes the robot on graph edges rather than in metric coordinates. This suggests that SkeNa is not a single algorithm but a family of methods unified by abstract map use, first-visit operation, and tolerance to severe geometric inaccuracy.

Aspect	SkeNavigator / SkeNa	SAM-based navigation
Map input	hand-drawn sketch map $S$ with the agent’s start and the goal marked	2D map $M$ (sketch or floor-plan); user-selected start and goal destinations
Perception	egocentric depth observations $D_t$; on-site exploration map $D_t$0	RGB-D visual stream $D_t$1; odometry
Output action	$D_t$2	continuous control commands $D_t$3
Metric prior	no precise floor plan, GPS or compass	no precise metric localization or globally consistent geometric mapping

2. Abstract maps, topology, and behavioural representations

A defining property of SkeNa is the use of sketch maps that are abstract, sparse, and can be significantly distorted. In SoR, the dataset introduced for SkeNa, two abstraction levels are used. Low-abstraction sketches approximate polygonal contours with more geometric fidelity. High-abstraction sketches emulate free-hand drawings with Bezier curves, controlled for simplicity, and may deviate more in scale and local geometry. Compared to metrically accurate CAD/BIM floor plans, these sketches are topology-oriented, human-like, and intentionally lack precise scale and exact obstacle boundaries (Xu et al., 5 Aug 2025).

SAMs provide a formal representation for exploiting such maps without metric registration. A SAM is defined as a behavioural topological graph $D_t$4 in which nodes are either destination nodes or changepoint nodes, and a changepoint is a location affording a change in behaviour, such as a junction, doorway, or bend. Directed edges are labelled by a behaviour $D_t$5 that, when executed, leads the robot from one node toward another. In the reported system, the behaviour set is $D_t$6, derived from the DECISION controller (Loo et al., 2024).

A central structural constraint in SAMs is that outgoing edges from the same node must have unique behaviour labels:

$D_t$7

This implies $D_t$8. The stated rationale is to ensure that a behaviour chosen at a node has a unique intended topological successor, thereby avoiding ambiguity in graph localization and behaviour selection. In effect, the map is reduced to a set of behaviourally meaningful transitions rather than a metrically faithful geometry (Loo et al., 2024).

The contrast between the two paradigms is instructive. SkeNavigator preserves a dense sketch image and aligns it with an incrementally built exploration map. SAM-based systems instead compress the sketch or floor-plan into destinations, changepoints, and labelled behaviour edges. A plausible implication is that the former is closer to continuous cross-map correspondence, whereas the latter is closer to topological program execution over affordances.

3. Map reading and sketch-to-environment alignment

SkeNa systems differ most sharply in how they interpret the sketch. In SkeNavigator, the key structural primitive is the Ray-based Map Descriptor (RMD), extracted at uniformly sampled points on both the sketch map and the exploration map. Let $D_t$9 be the number of keypoints arranged in a uniform lattice. For each keypoint $t$0, the method casts $t$1 rays at uniformly spaced angles $t$2, with

$t$3

If $t$4 is the set of boundary pixels or points, the ray distance is

$t$5

with $t$6 if no intersection is found within $t$7. The descriptor is

$t$8

and the full descriptor set is

$t$9

Normalization makes the descriptor partially scale-invariant; it is not strictly rotation-invariant, and it is not translation-invariant because coordinates are explicit (Xu et al., 5 Aug 2025).

Goal inference in SkeNavigator is handled by the Dual-Map Aligned Goal Predictor (DAGP). A learnable goal-associated embedding is added to the sketch descriptor nearest to the annotated sketch goal,

$E_t$0

after which self-attention is applied within sketch and exploration descriptors, cross-attention aligns exploration-map descriptors to sketch-map descriptors, and a softmax over exploration keypoints yields a predicted goal position

$E_t$1

The soft aggregation is explicitly intended to support robust goal estimates under sketch abstraction and partial exploration (Xu et al., 5 Aug 2025).

The SAM approach uses an offline parser $E_t$2 that decomposes into a node predictor and an edge predictor. The node predictor, $E_t$3, uses a MobileNetv2 encoder and a 4-layer MLP head to classify changepoint versus non-changepoint from image crops centered at sampled map locations. Training proceeds in two stages: supervised contrastive pretraining,

$E_t$4

followed by cross-entropy finetuning,

$E_t$5

Predicted changepoints are obtained by thresholding scores, clustering filtered points with GrowCompactClusters (Topomap), and selecting the highest-score point in each cluster (Loo et al., 2024).

The edge predictor, $E_t$6, also uses MobileNetv2 and processes a $E_t$7 channel input consisting of a 3-channel map crop plus $E_t$8 binary masks for neighbouring node locations. It produces a $E_t$9 behaviour cost matrix, which is normalized by Sinkhorn to a doubly-stochastic soft assignment matrix $$a_t \in \{\text{STOP}, \text{MOVE FORWARD}, \text{TURN LEFT}, \text{TURN RIGHT}\}$$0. Cross-entropy finetuning uses

$$a_t \in \{\text{STOP}, \text{MOVE FORWARD}, \text{TURN LEFT}, \text{TURN RIGHT}\}$$1

and the parser loss is

$$a_t \in \{\text{STOP}, \text{MOVE FORWARD}, \text{TURN LEFT}, \text{TURN RIGHT}\}$$2

The parser is designed to identify corridors ends, intersections, T-junctions, bends, doorways, and room entrances, and to convert local connectivity into behaviours such as go-forward, turn-right, turn-left, and room entry (Loo et al., 2024).

4. Navigation policies, localization, and control

SkeNavigator progressively aligns the agent’s egocentric experience with the abstract sketch map to infer the goal’s position in the agent’s current coordinate system, and then navigates toward it. Features from depth observation, exploration map, and sketch map are encoded as $$a_t \in \{\text{STOP}, \text{MOVE FORWARD}, \text{TURN LEFT}, \text{TURN RIGHT}\}$$3, $$a_t \in \{\text{STOP}, \text{MOVE FORWARD}, \text{TURN LEFT}, \text{TURN RIGHT}\}$$4, and $$a_t \in \{\text{STOP}, \text{MOVE FORWARD}, \text{TURN LEFT}, \text{TURN RIGHT}\}$$5. With the predicted goal position $$a_t \in \{\text{STOP}, \text{MOVE FORWARD}, \text{TURN LEFT}, \text{TURN RIGHT}\}$$6, the recurrent policy state is updated by

$$a_t \in \{\text{STOP}, \text{MOVE FORWARD}, \text{TURN LEFT}, \text{TURN RIGHT}\}$$7

and the action distribution is

$$a_t \in \{\text{STOP}, \text{MOVE FORWARD}, \text{TURN LEFT}, \text{TURN RIGHT}\}$$8

The exploration map $$a_t \in \{\text{STOP}, \text{MOVE FORWARD}, \text{TURN LEFT}, \text{TURN RIGHT}\}$$9 is incrementally constructed from depth frames using a top-down projection pipeline adapted from prior work (ESC), integrating each depth frame using the agent’s current pose and camera intrinsics and updating free and occupied cells in a top-down occupancy map (Xu et al., 5 Aug 2025).

Training combines PPO with explicit goal regression:

$T$0

with $T$1. The reward is composite,

$T$2

where $T$3, $T$4, and $T$5. The reported policy network uses a GRU with hidden size 512; PPO uses rollout length 150, 4 epochs per update, GAE with $T$6 and $T$7; DAGP uses $T$8, $T$9 keypoints, $\tau_d$0 rays, and a 2-layer attention network (Xu et al., 5 Aug 2025).

In SAM-based navigation, planning is over the behavioural graph rather than over an exploration map. Standard Dijkstra computes a shortest path over $\tau_d$1, returning an ordered sequence of edges or behaviours. Execution is mediated by a Graph Localization Network (GLN), which takes a history of RGB-D images and a local crop of the SAM centered on the last predicted pose, and outputs an edge probability vector $\tau_d$2 and an endpoint probability vector $\tau_d$3 per edge. The loss is

$\tau_d$4

The robot’s “pose” is the current edge $\tau_d$5 rather than a metric position. The selected behaviour controller then maps visual observations to continuous linear and angular velocities $\tau_d$6, and behaviour transitions are triggered when the smoothed endpoint probability for the current edge exceeds a threshold (Loo et al., 2024).

The controller library comes from the DECISION controller and includes go-forward, turn-left, and turn-right. These controllers exploit affordances such as corridor following and turning at junctions. The reported speed regime is up to $\tau_d$7 m/s, described as faster than Graphnav, which makes endpoint prediction necessary to avoid overshoot. Temporal smoothing of GLN edge predictions reduces flicker and mislocalization, while the unique-outgoing-behaviour constraint reduces branching ambiguity (Loo et al., 2024).

5. Datasets, platforms, and empirical results

The principal benchmark for SkeNa is SoR, a large-scale dataset comprising 54,774 sketch–trajectory pairs across 71 indoor scenes sourced from Matterport3D. It provides four validation subsets: val-seen and val-unseen, each with High-abstract and Low-abstract sketch styles. Each trajectory is generated by sampling start and goal positions and computing an A* shortest path on a denoised occupancy map. Sketch generation comprises environment layout construction, trajectory-related region selection, sketch style transformation, and human verification; a subset of validation sketches are genuinely human-drawn to assess generalization further (Xu et al., 5 Aug 2025).

The SAM study evaluates on real building floors with open spaces, narrow passages, clutter, and dynamic obstacles, using hand-drawn sketches of campus buildings, floor-plans of on-campus buildings, and satellite maps of industrial areas for offline parser evaluation. The robot platform is Boston Dynamics Spot with Nvidia AGX Xavier and three Intel RealSense D435i RGB-D cameras for combined $\tau_d$8 forward field of view, and the full system runs onboard (Loo et al., 2024).

SkeNavigator is reported to consistently improve over FloDiff on all metrics and subsets. Representative results are: Low-abstract val-seen, SkeNavigator SR 17.4% and SPL 16.6% versus FloDiff SR 3.6% and SPL 3.3%; Low-abstract val-unseen, SkeNavigator SR 12.7% and SPL 11.9% versus FloDiff SR 3.9% and SPL 3.3%; High-abstract val-seen, SkeNavigator SR 9.0% and SPL 7.8% versus FloDiff SR 3.5% and SPL 2.6%; High-abstract val-unseen, SkeNavigator SR 8.0% and SPL 7.6% versus FloDiff SR 3.9% and SPL 3.8%. Humans achieve SR $\tau_d$9 across all subsets, and the random agent has SR $S$0. The paper reports a 105% relative SPL improvement on the high-abstract validation set (unseen) (Xu et al., 5 Aug 2025).

In the SAM study, offline map-reading metrics quantify both node prediction and edge prediction. Node prediction precision and recall are HM: Prec 0.848, Rec 0.975; FP: Prec 0.732, Rec 0.779; SM: Prec 0.865, Rec 0.621. For edge prediction with correct node pairs only, HM Prec 0.754, Rec 0.605; FP Prec 0.820, Rec 0.643; SM Prec 0.863, Rec 0.751. For correct pair plus correct behaviour assignment, HM Prec 0.667, Rec 0.535; FP Prec 0.630, Rec 0.494; SM Prec 0.761, Rec 0.662. Online navigation uses Success Rate (SR) and Plan Completion (PC), with SR-Nav/PC-Nav including controller failures and SR-HL/PC-HL excluding controller failures (Loo et al., 2024).

Ablation and comparison in SAM-based navigation emphasize the role of endpoint prediction and graph augmentation. On ground-truth maps, SLS consistently outperforms GLN-Heu, for example on Easy HM, GLN-Heu SR-HL 45.5 and PC-HL 56.8 versus SLS SR-HL 70.0 and PC-HL 85.0; on Medium FP, GLN-Heu SR-HL 0.0 and PC-HL 18.8 versus SLS SR-HL 37.5 and PC-HL 62.5. With predicted maps, SLS-Aug often matches or exceeds performance with ground-truth SAMs, for example Hard HM: SLS (GT) 50.0/75.0, SLS (HM) 0/71.7, SLS-Aug (HM) 50.0/85.0; Hard FP: SLS (GT) 0/73.9, SLS (FP) 0/33.3, SLS-Aug (FP) 50.0/80.0. The reported takeaway is that SAMs predicted from sketches and floor-plans enable robust navigation, and that GLN endpoint prediction and graph augmentation improve robustness to map noise and defects (Loo et al., 2024).

Setting	Result	Source
SoR benchmark	54,774 sketch–trajectory pairs across 71 indoor scenes	(Xu et al., 5 Aug 2025)
High-abstract val-unseen	SkeNavigator SR 8.0%, SPL 7.6%; FloDiff SR 3.9%, SPL 3.8%	(Xu et al., 5 Aug 2025)
Ground-truth Easy HM	GLN-Heu SR-HL 45.5, PC-HL 56.8; SLS SR-HL 70.0, PC-HL 85.0	(Loo et al., 2024)
Predicted Hard FP	SLS (GT) 0/73.9; SLS (FP) 0/33.3; SLS-Aug (FP) 50.0/80.0	(Loo et al., 2024)

6. Limitations, misconceptions, and research directions

A common misconception is that sketch map-based visual navigation is simply floor-plan localization with weaker geometry. The published formulations indicate a sharper distinction. SkeNavigator assumes that sketches may be highly abstract, sparse, and significantly distorted, and addresses the problem through dual-map alignment between the sketch and an on-site exploration map. The SAM formulation explicitly avoids metric alignment and instead uses topological, behaviour-centric cues, arguing that navigation can be grounded in executable behaviours and changepoints rather than globally consistent spatial coordinates (Xu et al., 5 Aug 2025, Loo et al., 2024).

A second misconception is that stronger visual input necessarily improves performance. In SkeNavigator, RGB is optional and was studied in ablations but generally underperformed in this sketch-based setting; RGB-D tends to underperform depth-only, likely because texture and color cues do not align with sparse line drawings and act as noise for alignment. By contrast, the exploration map constructed from depth provides consistent gains, and adding DAGP yields approximately 3–5% absolute SR and approximately 3–5% SPL improvements across subsets (Xu et al., 5 Aug 2025).

The limitations reported in the two studies are complementary. SkeNavigator is sensitive to severe sketch distortions, early-stage exploration when $S$1 is too sparse for robust alignment, computational cost from attention-based dual-map alignment, and the single-floor focus that omits explicit modeling of stairs and vertical connectivity. The SAM approach reports ambiguous sketches, receptive field limits on long corridors, perceptual aliasing in visually similar places, dynamic obstacles, style/domain shift across map types, and the need for broader behaviour sets in very large graphs or outdoor spaces (Xu et al., 5 Aug 2025, Loo et al., 2024).

The research directions are correspondingly diverse. For SkeNavigator, future work includes learning invariances to rotation, scale, and sketch distortions, extending to multi-floor and outdoor navigation, human-in-the-loop adaptation to broader varieties of human-drawn sketches, and combining sketches with language and scene semantics. For SAM-based navigation, proposed directions include richer behaviour libraries such as doorway traversal, cross-open-space, approach or leave landmark, avoid dynamic obstacles, and enter or exit stairs; graph neural networks for edge prediction; uncertainty-aware planning and execution; weak metric cues such as coarse range or heading priors; multi-modal map-reading with textual directions or photos; and learning from demonstrations of human map reading and navigation traces (Xu et al., 5 Aug 2025, Loo et al., 2024).

Taken together, these works establish SkeNa as a technically heterogeneous but coherent research area. One branch treats the sketch as a dense but abstract structural signal to be aligned with an evolving exploration map; the other converts the sketch into a behavioural topological graph that supports graph localization and affordance-driven control. Both branches reject the assumption that first-visit navigation requires precise metric maps, and both make the usability of distorted human-like maps a central algorithmic question.