SkeNavigator: AR Surgery and Visual Navigation

• SkeNavigator is a label for distinct navigation systems that employ explicit spatial structure across AR surgical guidance, sketch-map alignment, and lifelong object navigation.
• In AR surgical navigation, SkeNavigator leverages preoperative CT segmentation and intraoperative point cloud registration on HoloLens to guide pedicle screw placements with quantitative error metrics.
• In embodied and lifelong navigation, the framework aligns hand-drawn sketches or semantic skeletons with exploration maps to predict goal locations and optimize long-horizon planning in unseen environments.

SkeNavigator is a name used in arXiv literature for distinct navigation systems in different technical domains. In surgical navigation, it denotes a fully standalone orthopedic navigation approach that runs on an unmodified Microsoft HoloLens and combines preoperative planning, intraoperative surface digitization for registration, holographic navigation, and quantitative evaluation for pedicle screw placement (Liebmann et al., 2020). In embodied navigation, it denotes a framework for Sketch-map-based visual Navigation (SkeNa), where an agent aligns a hand-drawn sketch map with an on-site exploration map using a Ray-based Map Descriptor (RMD) and a Dual-Map Aligned Goal Predictor (DAGP) (Xu et al., 5 Aug 2025). A related naming overlap also appears in a structured summary of SSMG-Nav, where that lifelong ObjectNav framework is glossed as “SkeNavigator” (Niu et al., 2 Mar 2026).

1. Scope and nomenclature

The supplied arXiv records do not identify a single standardized SkeNavigator platform. Instead, the name is attached to at least two different systems, with a third adjacent usage in lifelong object navigation. This matters because the underlying tasks, sensors, coordinate representations, and evaluation protocols differ substantially.

Usage of “SkeNavigator”	Domain	Core formulation
SkeNavigator (Liebmann et al., 2020)	Orthopedic AR navigation	HoloLens-based surface digitization, registration, and holographic pedicle-screw guidance
SkeNavigator (Xu et al., 5 Aug 2025)	Embodied visual navigation	Sketch-map-guided navigation in unseen indoor scenes
SSMG-Nav (“SkeNavigator”) (Niu et al., 2 Mar 2026)	Lifelong ObjectNav	Persistent semantic skeleton memory with long-horizon planning

A common source of confusion is to treat SkeNavigator as a single research line. The records instead show domain-specific systems that share a concern with explicit spatial structure: 3D/3D registration and coordinate composition in surgery, sketch-to-map alignment in embodied navigation, and topological memory graphs in lifelong ObjectNav.

2. Standalone orthopedic navigation on HoloLens

In "Registration made easy -- standalone orthopedic navigation with HoloLens" (Liebmann et al., 2020), SkeNavigator is a surgical navigation approach for pedicle screw placement that runs entirely on the Microsoft HoloLens. The system is divided into four tightly-coupled modules: preoperative planning, intraoperative surface-digitization for registration, holographic navigation, and quantitative evaluation. Its hardware stack consists of a first-generation Microsoft HoloLens with inside-out SLAM, a see-through display, four environment-tracking cameras in “Research Mode,” an IMU, and a clicker or voice-recognition interface; sterile AprilTag-style planar fiducial markers mounted on custom surgical tools; a custom pointing device for surface sampling; and a custom navigation drill-guide for screw and K-wire guidance (Liebmann et al., 2020).

The preoperative stage begins with CT segmentation in a desktop planning system. Each vertebra is reconstructed as a watertight mesh with annotated pedicle-screw entry points and trajectories. The resulting mesh and screw plans are exported as a binary package and wirelessly streamed to the HoloLens application. At startup, the application anchors the 3D model into the HoloLens world coordinate frame and waits for intraoperative registration (Liebmann et al., 2020).

The intraoperative workflow is organized around surface digitization. The surgeon grips the pointing device, whose marker is tracked by the stereo environment cameras, and presses the clicker to begin sampling. While the clicker is depressed, a continuous stream of 3D tip positions is recorded. The surgeon can issue “save” or “delete” by voice to keep or discard traced surface patches, and all retained points form the intraoperative point cloud $pc_{intra}$ (Liebmann et al., 2020).

This architecture is explicitly designed as a standalone alternative to commercial navigation systems. The paper states that state-of-the-art commercial systems improve surgical accuracy but are not gold standard in clinical practice, and it points to difficult workflow integration and unintuitive navigation feedback as possible causes. The use of augmented reality is presented as a way to address those limitations through in-situ overlays and a self-contained wearable platform (Liebmann et al., 2020).

3. Registration pipeline, coordinate systems, and holographic guidance

The registration pipeline begins with AprilTag-based tool localization from the left and right front-facing HoloLens cameras. Each frame pair yields 2D corner detections $C_i^L, C_i^R$ in 2D Camera Projection Space. After Kalman-filter smoothing, each corner is lifted into 3D Camera View Space under a unit-depth assumption and transformed into the Camera Coordinate System, after which two rays from the two camera centers are intersected via closest-point triangulation to recover four 3D points $\{\tilde p_i\}$. Applying the known marker-to-tip offset gives the pointing-device tip position in the App Coordinate System (Liebmann et al., 2020).

Coarse registration is performed through PCA extreme points. For the intraoperative cloud $\{p_i\}=pc_{intra}$ and a sampled subset of the preoperative model $\{q_j\}=pc_{pre}$, principal component analysis produces orthonormal axes $pa_1, pa_2, pa_3$. Three extreme points are then extracted along the principal axes. Because of vertebral symmetry about $pa_1$, two candidate correspondences are considered, and each is aligned by Horn’s absolute orientation to produce two candidate transforms $T^{(1)}$ and $T^{(2)}$ (Liebmann et al., 2020).

Fine registration is then obtained by iterative closest point (ICP), minimizing

$$\min_{R,t}\;\sum_{i=1}^N \Big\|\,R\,p_i + t - q_{\sigma(i)}\Big\|^2,$$

where $C_i^L, C_i^R$0 reassigns closest neighbors in $C_i^L, C_i^R$1 to transformed $C_i^L, C_i^R$2. After convergence, the RMSE of each candidate alignment is compared, and the lower-error solution is retained as $C_i^L, C_i^R$3, which maps the preoperative CT model into the HoloLens world coordinate system. The surgeon visually verifies the superimposed vertebra model on the bone surface and locks the registration through verbal confirmation (Liebmann et al., 2020).

The paper defines six coordinate frames: $C_i^L, C_i^R$4 for intrinsic CT scanner or model coordinates, $C_i^L, C_i^R$5 for the SLAM-stabilized HoloLens world frame, $C_i^L, C_i^R$6 for the HoloLens head or camera frame, $C_i^L, C_i^R$7 for the application-specific frame, $C_i^L, C_i^R$8 for the marker frame, and $C_i^L, C_i^R$9 for the tool-tip frame. The core registration output is the rigid transform

$\{\tilde p_i\}$0

and the application stores the fused anchor

$\{\tilde p_i\}$1

so that rendered holograms track head motion continuously (Liebmann et al., 2020).

Once registration is complete, SkeNavigator enters navigation mode for pedicle-screw guidance. Semi-transparent “landing pads” indicate planned entry points. After cortical piercing, the K-wire tip is rendered in green when it is within $\{\tilde p_i\}$2 of the target trajectory, yellow for $\{\tilde p_i\}$3, and red otherwise. A dynamic triangular “error wedge” connects the planned entry point $\{\tilde p_i\}$4, the current tool tip $\{\tilde p_i\}$5, and a point $\{\tilde p_i\}$6 on the planned trajectory at distance $\{\tilde p_i\}$7, while the 3D angle $\{\tilde p_i\}$8 between $\{\tilde p_i\}$9 and $\{p_i\}=pc_{intra}$0 is shown numerically in the corner of the surgeon’s view. Interaction uses the clicker, voice commands such as “start navigation,” “next level,” and “finish,” and air-tap gestures for minor plan modifications (Liebmann et al., 2020).

4. Accuracy, timing, and workflow integration in the orthopedic system

The orthopedic SkeNavigator was evaluated in a two-phantom lumbar study in which each vertebra was registered and two K-wires per level were placed under AR guidance only. Postoperative CT scans were segmented and aligned to the plan. The primary reported results were: trajectory error $\{p_i\}=pc_{intra}$1 with range $\{p_i\}=pc_{intra}$2, entry-point error $\{p_i\}=pc_{intra}$3 mm with range $\{p_i\}=pc_{intra}$4, registration RMSE $\{p_i\}=pc_{intra}$5 mm with range $\{p_i\}=pc_{intra}$6, surface digitization time $\{p_i\}=pc_{intra}$7 s with range $\{p_i\}=pc_{intra}$8, and $\{p_i\}=pc_{intra}$9 collected points with range $\{q_j\}=pc_{pre}$0 (Liebmann et al., 2020).

The evaluation uses entry-point error, angular deviation of the screw axis, and TRE, with the entry-point error defined by

$\{q_j\}=pc_{pre}$1

The paper states that the 3D/3D registration accuracy and speed are on par with state-of-the-art commercial systems, quoted as approximately $\{q_j\}=pc_{pre}$2 mm and $\{q_j\}=pc_{pre}$3 min, while avoiding radiation and large tracking rigs (Liebmann et al., 2020).

The reported end-to-end timing from application start to readiness for navigation is approximately $\{q_j\}=pc_{pre}$4 s per vertebra: $\{q_j\}=pc_{pre}$5 s for model loading and anchor placement, $\{q_j\}=pc_{pre}$6 s for surface digitization with about $\{q_j\}=pc_{pre}$7 points, and $\{q_j\}=pc_{pre}$8 s for automated registration. Screw insertion under holographic guidance adds approximately $\{q_j\}=pc_{pre}$9 s per screw. Participating surgeons reportedly considered the in-situ overlays and gesture or voice interface intuitive, suggesting that workflow integration and display semantics were central design considerations rather than peripheral interface features (Liebmann et al., 2020).

The paper also states several limitations. Hologram drift may occur under prolonged head movement, and the use of Research Mode cameras can destabilize the built-in SLAM. Future work is described as integrating high-fidelity RGB-D streams such as Azure Kinect to automate surface reconstruction and improve robustness. The results are explicitly characterized as preliminary phantom results that suggest the method may meet clinical accuracy requirements (Liebmann et al., 2020).

5. SkeNavigator in sketch-map-based visual navigation

In "SkeNa: Learning to Navigate Unseen Environments Based on Abstract Hand-Drawn Maps" (Xu et al., 5 Aug 2025), SkeNavigator is the navigation framework proposed for the SkeNa task. SkeNa is defined as sketch-map-based visual navigation in which an embodied agent must reach a goal in an unseen 3D indoor environment using only a hand-drawn sketch map as guidance. The agent receives a sketch map $pa_1, pa_2, pa_3$0 annotated with a start position $pa_1, pa_2, pa_3$1 and goal position $pa_1, pa_2, pa_3$2, and at each time step $pa_1, pa_2, pa_3$3 it observes egocentric depth $pa_1, pa_2, pa_3$4. The action space is

$pa_1, pa_2, pa_3$5

and an episode is successful if the agent issues STOP within the maximum number of steps $pa_1, pa_2, pa_3$6 and its final position $pa_1, pa_2, pa_3$7 satisfies $pa_1, pa_2, pa_3$8 (Xu et al., 5 Aug 2025).

To support this task, the paper introduces the SoR dataset, comprising $pa_1, pa_2, pa_3$9 sketch–trajectory pairs across $pa_1$0 real-world indoor scenes from Matterport3D. The dataset includes val-seen and val-unseen scene splits, as well as high-abstraction and low-abstraction sketch settings. Its sketch-generation pipeline consists of occupancy-map extraction and denoising, start and goal sampling with A* trajectory computation and trajectory-region cropping, polygonal-approximation line drawing for low abstraction, CLIPasso-style Bézier curves for high abstraction, and manual verification with a small human-drawn subset for testing (Xu et al., 5 Aug 2025).

The paper evaluates Success Rate (SR), SPL, SoftSPL, and Distance to Goal (DTG). Baselines include a random agent, human operators, and FloDiff adapted to sketches. The objective is to learn a policy $pa_1$1 that maximizes SR and SPL while generalizing across abstraction levels of sketches (Xu et al., 5 Aug 2025).

This formulation differs sharply from the orthopedic usage of SkeNavigator. Here the central problem is not surgical registration but cross-representation alignment between a sparse, abstract, hand-drawn map and a partial exploration map constructed online from depth observations.

6. RMD, DAGP, policy learning, and empirical behavior

The sketch-navigation SkeNavigator contains two core modules: the Ray-based Map Descriptor and the Dual-Map Aligned Goal Predictor. RMD is motivated by the claim that conventional CNN or ViT patch features are ill-suited for sparse sketch strokes. It represents each of $pa_1$2 map keypoints by an obstacle-distance signature along $pa_1$3 rays. With keypoints $pa_1$4, ray angles $pa_1$5, and descriptor

$pa_1$6

the method extracts

$pa_1$7

for the sketch map $pa_1$8 and the exploration map $pa_1$9 (Xu et al., 5 Aug 2025).

DAGP aligns these two descriptor sets to estimate the goal location on the exploration map. First, the keypoint nearest the annotated sketch goal is identified and augmented with a learned goal embedding $T^{(1)}$0. Second, sketch and exploration descriptors are independently contextualized with self-attention: $T^{(1)}$1 Third, cross-attention queries exploration features against sketch features: $T^{(1)}$2 Finally, an MLP plus softmax predicts weights $T^{(1)}$3, and the goal estimate is

$T^{(1)}$4

Training combines a goal-regression loss $T^{(1)}$5 with PPO through

$T^{(1)}$6

The recurrent policy state is defined as

$T^{(1)}$7

over depth, exploration-map features, sketch features, and the predicted goal (Xu et al., 5 Aug 2025).

The reported results show both the difficulty of the task and the effect of the proposed alignment mechanism. On the val-unseen, high-abstraction split, FloDiff obtains SR $T^{(1)}$8 and SPL $T^{(1)}$9, whereas SkeNavigator obtains SR $T^{(2)}$0 and SPL $T^{(2)}$1; the paper reports a relative SPL improvement of $T^{(2)}$2. On val-seen, low-abstraction scenes, FloDiff reaches SR $T^{(2)}$3 and SPL $T^{(2)}$4, while SkeNavigator reaches SR $T^{(2)}$5 and SPL $T^{(2)}$6. Human operators achieve at least approximately $T^{(2)}$7 SR in all splits, which the paper uses to confirm sketch clarity (Xu et al., 5 Aug 2025).

A plausible implication is that the principal bottleneck in SkeNa is not whether the sketches are interpretable in principle, but whether the agent can align abstract sketch structure with its incrementally constructed map under severe representation mismatch.

7. Relation to SSMG-Nav and broader navigation patterns

A third usage appears in the structured summary of "SSMG-Nav: Enhancing Lifelong Object Navigation with Semantic Skeleton Memory Graph," where SSMG-Nav is described as “SkeNavigator” (Niu et al., 2 Mar 2026). In that formulation, the agent’s memory is

$T^{(2)}$8

combining occupancy, value, and object-level maps with a skeleton graph extracted from the occupancy map by morphological thinning, node classification into endpoints, connectors, and junctions, and connector pruning. Objects are attached to their nearest skeleton node in BEV, and each node defines a subgraph that unifies entity-level semantics with space-level semantics (Niu et al., 2 Mar 2026).

For multimodal target specification, each candidate node is converted into a multimodal prompt for Qwen-VL-Plus, using textual object descriptions and a BEV snapshot, and the resulting belief scores are normalized into a probability over candidate destinations. Traversability cost is the A*-based shortest-path distance on the graph, and a 2-opt local search produces a visit sequence that minimizes expected travel distance under the destination belief distribution (Niu et al., 2 Mar 2026).

The summary reports strong gains on lifelong multimodal ObjectNav and standard ObjectNav benchmarks. On GOAT-Bench val_seen and val_unseen, SSMG-Nav achieves s-SR $T^{(2)}$9, e-SR $$\min_{R,t}\;\sum_{i=1}^N \Big\|\,R\,p_i + t - q_{\sigma(i)}\Big\|^2,$$0, and SPL $$\min_{R,t}\;\sum_{i=1}^N \Big\|\,R\,p_i + t - q_{\sigma(i)}\Big\|^2,$$1, compared with prior values of approximately $$\min_{R,t}\;\sum_{i=1}^N \Big\|\,R\,p_i + t - q_{\sigma(i)}\Big\|^2,$$2, $$\min_{R,t}\;\sum_{i=1}^N \Big\|\,R\,p_i + t - q_{\sigma(i)}\Big\|^2,$$3, and $$\min_{R,t}\;\sum_{i=1}^N \Big\|\,R\,p_i + t - q_{\sigma(i)}\Big\|^2,$$4. On HM3D and MP3D ObjectNav, it reports SR $$\min_{R,t}\;\sum_{i=1}^N \Big\|\,R\,p_i + t - q_{\sigma(i)}\Big\|^2,$$5 and SPL $$\min_{R,t}\;\sum_{i=1}^N \Big\|\,R\,p_i + t - q_{\sigma(i)}\Big\|^2,$$6, compared with best zero-shot values of approximately $$\min_{R,t}\;\sum_{i=1}^N \Big\|\,R\,p_i + t - q_{\sigma(i)}\Big\|^2,$$7 and $$\min_{R,t}\;\sum_{i=1}^N \Big\|\,R\,p_i + t - q_{\sigma(i)}\Big\|^2,$$8. The ablation study shows steady gains when adding skeleton-based memory, long-horizon planning, and revisiting (Niu et al., 2 Mar 2026).

Taken together, these records show that the label SkeNavigator has been attached to systems built around explicit spatial intermediates rather than purely end-to-end policies: point clouds and rigid transforms in orthopedic AR, ray-based map descriptors and cross-attention in sketch-map navigation, and semantic skeleton memory graphs in lifelong ObjectNav. This suggests a family resemblance at the level of representational strategy, even though the systems themselves address different tasks, operate with different sensors, and are evaluated under different notions of success.