Papers
Topics
Authors
Recent
Search
2000 character limit reached

SG2Loc: Sequential Visual Localization on 3D Scene Graphs

Published 10 Jun 2026 in cs.CV | (2606.11880v1)

Abstract: Visual localization in complex indoor environments remains a critical challenge for robotics and AR applications. Sequential localization, where pose estimates are refined over time, is important for autonomous agents. However, traditional methods often require storing extensive image databases or point clouds, leading to significant overhead. This paper introduces a novel, lightweight approach to sequential visual localization using 3D scene graphs. Our method represents the environment with a compact scene graph, where nodes represent objects (with coarse meshes) and edges encode spatial relationships. For each image in the localization phase, we extract per-patch semantic features, predicting object identities. Localization is performed within a particle filter framework. Each particle, representing a camera pose, projects the coarse object meshes from the scene graph into the image, assigning object identities to patches based on visibility. The similarity of the per-patch features, in the input image, and object features from the scene graph determines the weight of a particle. Subsequent images are incorporated sequentially, refining the pose estimate. By leveraging a compact scene graph and efficient semantic matching, our method significantly reduces storage while maintaining performance on real-world datasets. The code will be available at https://github.com/DmblnNicole/sg2loc.

Summary

  • The paper introduces a sequential localization method that leverages compact 3D scene graphs for efficient indoor pose estimation.
  • It employs a coarse-to-fine particle filter combining semantic, photometric, and depth cues to reduce storage requirements while maintaining accuracy.
  • A final PnP post-optimization and scene retrieval extension enable state-of-the-art performance on datasets like 3RScan and ScanNet.

SG2Loc: Sequential Visual Localization on 3D Scene Graphs

Introduction

SG2Loc proposes a sequential visual localization paradigm that leverages compact 3D scene graphs as the map representation, enabling efficient and accurate pose estimation in complex indoor environments. The method addresses fundamental limitations of traditional visual localization and SLAM methods—most notably, the requirement for large-scale image databases or dense point clouds—by replacing these with lightweight scene graphs composed of coarse object meshes and semantic embeddings.

Method

The core of SG2Loc is a particle filter operating in the pose space, parameterized by 3 DoF for translation and 1 DoF for yaw (gravity-aligned rotation). Localization proceeds by sequentially refining the pose hypothesis as new images and ego-motion estimates arrive. Each image is partitioned into a grid of patches; a pre-trained encoder predicts semantic descriptors for each patch. For each particle, the object-centric scene graph is raycasted into the synthetic view, assigning object embeddings to the image grid. Particle weights are computed using a combined score based on object-level semantic alignment, photometric consistency via SSIM, and depth-map congruence (when available). The likelihood model is a Gaussian centered at perfect correspondence. Figure 1

Figure 1: The observation model for a particle matches object descriptors predicted from patches in the query image to projected object labels from the scene graph mesh, weighting particles by the agreement between observation and map.

SG2Loc utilizes a coarse-to-fine, multi-round optimization: particle filtering is first performed at low image and raycasting resolution across the entire environment. Subsequent rounds downsample the search region using the Maximum Likelihood Estimate (MLE) from previous rounds, increasing the raycasting granularity and focusing on local pose refinement. Figure 2

Figure 2: Overview of the sequential localization pipeline; pose estimates are updated and refined with each incoming image and ego-motion estimate, integrating information over the entire observed sequence.

Adaptive resampling with KLD-sampling guides particle count allocation for computational efficiency and posterior approximation quality.

Pose Refinement

A final post-optimization step runs a PnP solver, seeding it with the particle filter MLE and its local neighborhood sampled along the yaw axis. 2D–3D correspondences are established by robust matching between query frame patches and synthetic renders, using up-to-date dense feature matchers. This non-parametric refinement is crucial for maximizing pose accuracy following the discrete particle filtering stage.

Scene Identification Extension

For settings involving multi-scene databases, the method is extended with a sequential scene retrieval step. The approach generalizes SceneGraphLoc’s cross-modal image-to-graph retrieval from single-frame to sequences by aggregating similarity scores over the input sequence, achieving robust and efficient place recognition in large-scale deployments.

Experimental Results

Comprehensive evaluations were conducted on 3RScan and ScanNet, including challenging cross-temporal settings to assess robustness to scene variation. SG2Loc consistently achieved the highest or competitive recall and median error rates across different sequence lengths on 3RScan and was on par with state-of-the-art baselines on ScanNet, despite a dramatically lower storage footprint. Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3: Illustration of the coarse-to-fine, multi-round particle filter; the hypothesis space contracts and pose estimation sharpens as more frames are processed with increasing detail.

In terms of resource use, SG2Loc requires nearly two orders of magnitude less storage than MeshLoc and HLoc on ScanNet and approximately an order of magnitude less on 3RScan, facilitating truly lightweight map transmission and on-device application.

Ablation studies confirmed the substantial benefit of the sequential, multi-modal scoring (semantic, depth, photometric) and coarse-to-fine optimization. Each added signal improves the fraction of high-precision localizations, while the final PnP boost yields state-of-the-art performance at a negligible extra computational cost.

Failure analysis and keyframe-based evaluations highlight that most errors stem from visually ambiguous or uninformative image sequences. Selective keyframe integration, matched to the method’s runtime, preserves accuracy while reducing redundant computation. Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4: Failure case on a highly ambiguous image sequence; the particle cloud fails to collapse onto the true pose due to repetitive structure or lack of discriminative observations.

Practical and Theoretical Implications

SG2Loc demonstrates that compact, object-centric scene graphs are competitive with or superior to legacy map representations for sequential visual localization, even under realistic map and environment variation. The integration of patch-level semantics, coarse-to-fine inference, and multi-modal cues solves the essential trade-off between map compactness and localization accuracy. The design is particularly favorable for edge-controlled or bandwidth-constrained robots and AR devices.

Theoretically, this work substantiates that full dense geometry is not always necessary for high-quality sequential relocalization; leveraging compositional scene structure and sequential integration can yield strong results with tight representational bounds.

Future Prospects

Several lines of future research are enabled by this framework. Improving the fidelity of object-level feature representations (e.g., with larger vision-LLMs or learned embeddings) could further disambiguate complex environments. On-the-fly scene graph construction or update could allow lifelong mapping and adaptivity. Integration with high-frequency event or inertial sensors offers potential for further efficiency in sequential inference. Given the method’s high degree of parallelism (on GPU), real-time large-scale deployment is plausible with further engineering.

Conclusion

SG2Loc establishes a new paradigm for efficient, sequential visual localization in indoor environments, exploiting 3D scene graphs, patch-level semantics, and particle filtering for scalable and robust operation. The approach sets a benchmark for storage-efficient localization, with accuracy on par with or exceeding methods dependent on much heavier representations. Its modularity, extensibility, and practicality recommend it as a compelling baseline for both research and deployment in robotics and AR applications (2606.11880).

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

SG2Loc: Finding Your Place Indoors Using a Smart, Lightweight Map

1) What is this paper about?

This paper is about helping robots and AR devices figure out exactly where they are inside buildings by looking at a short video (a sequence of images) and a very compact map. Instead of storing tons of photos or super-detailed 3D point clouds, the method uses a simple “scene graph” that remembers which objects are in the room (like chairs and tables), roughly where they are, and how they relate to each other. The system updates its guess of the camera’s position step by step as new images arrive.

2) What questions are the researchers trying to answer?

The paper focuses on a few easy-to-understand goals:

  • Can we localize a camera indoors using far less memory than popular methods that store huge image databases or dense 3D maps?
  • Can we improve accuracy by using a sequence of images (not just one), updating the position over time?
  • Can a simple object-based map (a 3D scene graph) be good enough for reliable localization?
  • In multi-building setups, can we also figure out which scene (which room or apartment) we’re in using a short image sequence?

3) How does the method work? (Explained with simple ideas)

Think of the map as a smart, labeled floor plan:

  • 3D scene graph: This is a compact map where “nodes” are objects (like a sofa or lamp) and “edges” say how objects relate in space. Each object has:
    • A coarse 3D shape (a simple mesh) so you can tell if it should be visible from a certain spot.
    • A learned “fingerprint” (an embedding) describing the object’s identity.

When a new image arrives, the system:

  • Splits the image into small rectangles (patches) and predicts which object each patch most likely shows. This is like labeling little chunks of the picture as “wall,” “chair,” “table,” etc.

Then comes the key idea: a particle filter.

  • Particle filter (analogy): Imagine lots of tiny “you-are-here” stickers scattered around the room map. Each sticker is a guess of where the camera might be and which way it’s facing.
  • For each guess, the system “looks” from that spot by projecting the coarse object meshes into the camera view (raycasting—like shining rays to see which object would block your view first). This predicts which objects should be visible in each patch.
  • It then compares the predicted objects (from the map) to what the image patches actually looked like (from the picture). If they match well, that guess gets a higher score; if not, it loses points.
  • The filter also uses motion estimates between frames (like how the camera moved from step to step from a SLAM tracker), so guesses move forward realistically.
  • Over time, bad guesses are thrown out, and good ones are duplicated and refined, narrowing down to the true position.

Extra cues make it stronger:

  • Color similarity: Compare the rendered view from a guess to the real image for appearance similarity.
  • Depth similarity: If depth is available (from an RGB-D camera or a depth estimator), compare how far surfaces are from the camera.
  • Coarse-to-fine: Start with a rough search over the whole space, then zoom in on the best area with finer checks—this speeds things up and improves accuracy.
  • Final polish: Generate a few synthetic views near the best guess, match them to the real image, and run a standard pose solver (PnP) to sharpen the final result.

Also, scene selection (which room am I in?):

  • If you have many scene graphs (e.g., many rooms), they extend a previous method to use an image sequence to pick the correct scene graph. Using more than one image improves reliability while still being fast and compact.

4) What did they find, and why does it matter?

The researchers tested their method on two popular indoor datasets (3RScan and ScanNet) that contain many real rooms, often with objects moved around between recordings (which makes localization harder).

Main takeaways:

  • Accuracy: On 3RScan, their method achieved the best combined success rates across sequences of 5, 10, and 25 images. On ScanNet, it was competitive with top methods and best on longer sequences.
  • Storage: Their maps are tiny compared to others. Roughly:
    • About 10 MB per scene on 3RScan and 28 MB on ScanNet.
    • Competing methods often need hundreds to tens of thousands of MB per scene.
    • In plain words: tens to hundreds of times less storage.
  • Practical speed: Building the map for their method is very fast (no heavy training). Per image, localization runs in a few seconds, and they show how to process only selected frames (“keyframes”) to match real-time constraints. The heavy lifting (checking lots of guesses) runs in parallel on a GPU.

Why this is important:

  • Small maps are easier to store on phones, AR glasses, or small robots.
  • Less data means faster transfers and updates (useful if maps change or must be sent over a network).
  • Using objects and semantics (not just pixels) makes the method more robust when lighting changes or furniture moves around.

5) What could this change in the real world?

This approach makes indoor localization more practical for everyday devices:

  • AR glasses could place virtual objects in the right spot without needing gigabytes of maps.
  • Small robots (home robots, delivery bots) could navigate buildings smoothly even with limited memory or bandwidth.
  • Because the map stores object-level information and not giant image collections, updating or sharing maps can be lighter and potentially more privacy-friendly.

In short, SG2Loc shows that you can localize accurately indoors using a clever, object-based map and a step-by-step reasoning process, all while using far less storage than traditional methods. This could help bring reliable indoor AR and robotics to more devices and more places.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

Below is a consolidated list of concrete limitations and unresolved questions that emerge from the paper. These are framed to guide actionable follow-up research.

Assumptions, scope, and observability

  • 4-DoF assumption (known gravity, yaw-only rotation): No evaluation or approach for full 6-DoF pose when gravity is unknown or unreliable; sensitivity to small pitch/roll errors is not quantified.
  • Reliance on known intrinsics: No sensitivity study to intrinsics errors or rolling-shutter/distortion effects on raycasting and photometric/depth consistency.
  • Indoor-only scope with object-rich scenes: No analysis of performance in sparse-object environments (e.g., corridors) or minimally furnished spaces where object-level cues are scarce.
  • Fixed camera height priors: Particles are initialized at a small set of heights; impact of wrong height priors (e.g., handheld vs. head-mounted vs. robot) is not analyzed.

Map/scene-graph construction and robustness

  • Dependence on pre-built, segmented, and labeled scene graphs: The cost, latency, and accuracy of building/updating scene graphs (and coarse meshes) are not accounted for in the mapping/storage/runtime comparisons.
  • Sensitivity to segmentation/labeling errors: No ablation on how mis-segmented objects, missing objects, or wrong identities in the scene graph affect localization.
  • Mesh quality vs. performance: No systematic study of robustness to mesh coarseness, geometric errors, or missing textures (for photometric supervision).
  • Temporal changes and unknown objects: While 3RScan cross-temporal changes are used, there is no targeted analysis of how object additions/removals, layout changes, or previously unseen objects impact performance and failure modes.
  • Use of graph relations: Edges/relations in the scene graph are not used during inference; the benefit of leveraging relational constraints (e.g., pairwise spatial priors) remains unexplored.

Observation model and scoring

  • Hard semantic gating: Patch similarity is only retained when the predicted object matches the raycasted identity; the impact of misclassification noise and the potential for soft probabilistic matching (e.g., top-k matches or learned likelihoods) is not explored.
  • Fixed patch grids and downsampling: No ablation of patch resolution/grid size, stride schedule, or adaptive patching schemes vs. runtime/accuracy trade-offs.
  • Photometric supervision assumptions: SSIM is used on renders from coarse textured meshes; sensitivity to illumination changes, view-dependent effects, inaccuracies in textures, and occlusion modeling is not characterized.
  • Depth supervision scaling: The depth score uses an unnormalized L1 difference; how depth range, scaling, and noise (sensor or monocular depth estimators) affect comparability and stability is not assessed (e.g., risk of negative scores, depth-range dependence).
  • Score fusion weights: The semantic, depth, and RGB terms are equally weighted; no exploration of learning or adapting weights based on uncertainty, scene content, or per-patch confidence.

Motion model and filtering

  • Ego-motion integration: The particle filter uses fixed Gaussian noise; it does not incorporate SLAM/VIO covariance or uncertainty-aware propagation. Impact of biased/underconfident motion estimates is not studied.
  • Scalability of particle count: Runtime and accuracy scaling with particle count, object count, and scene size is not analyzed; guidelines for choosing particle budgets are absent.
  • Coarse-to-fine schedule: Stride and bounding-box shrink factors are hand-tuned; no automatic schedule, stopping criteria, or analysis of trade-offs across scenes.
  • Posterior uncertainty: Only MLE poses are reported; posterior covariance or uncertainty estimates for downstream tasks are not provided.
  • Handling multi-floor/large spaces: Initialization is uniform in a bounded region; no mechanism for multi-floor disambiguation or priors from building layouts.

PnP refinement and geometric consistency

  • Synthetic-view matching: Robustness of RoMa-based matches between real and rendered views (from coarse meshes) under texture/geometry mismatch is not analyzed; alternative strategies (edge/contour-based, semantic keypoints) are not compared.
  • Integration of PnP with filtering: PnP is a final-step refinement; potential benefits of interleaving PnP hypotheses within the particle filter loop (e.g., proposal reweighting) are not investigated.
  • Backpropagation of final MLE pose: The strategy to backpropagate the last-frame MLE to earlier frames is briefly described; its impact on temporal consistency and failure cases is not evaluated.

Retrieval and multi-scene settings

  • Coupling retrieval and localization: Retrieval and PF are independent; joint inference over scene identity and pose (e.g., Rao–Blackwellized filters over scenes) is not explored, nor is recovery from wrong scene retrieval.
  • Temporal modeling in retrieval: Sequential retrieval aggregates per-frame scores by summation, ignoring temporal ordering and dynamics; sequence models (e.g., HMMs, learned temporal fusion) are not considered.
  • Retrieval at scale: Evaluations are limited to 10/50 scenes; behavior for hundreds/thousands of scenes and approximate nearest-neighbor indexing trade-offs are not studied.

Evaluation coverage and fairness

  • Storage accounting: It is unclear whether textures and all mesh assets used for photometric supervision are counted in storage; the fairness of comparing storage to methods that require image databases vs. textured meshes is not fully clarified.
  • Mapping time accounting: Scene graph/mesh generation time is excluded for the proposed method but included (at least partially) for baselines; the impact on end-to-end deployment latency is not reported.
  • Real-time deployment: Inference is ~2.9 s/frame plus ~4.2 s final optimization on a GPU; there is no evaluation on embedded/on-device hardware or exploration of further acceleration (e.g., sparse raycasting, object culling, or learned proposals).
  • Detailed failure analysis: No qualitative/quantitative analysis of failure cases (e.g., repetitive furniture layouts, heavy occlusions, low-texture surfaces) or diagnostics to guide recovery/relapse strategies.
  • Generalization and domain shift: No tests across different sensors, intrinsics, lighting, or camera models; robustness to cross-domain deployments is unassessed.

Extensions and learning opportunities

  • End-to-end learning: The pipeline relies on a pre-trained SceneGraphLoc encoder; there is no exploration of training/fine-tuning embeddings for sequential localization objectives or learning the observation likelihood.
  • Use of language and relations: Although multimodal embeddings and relations exist, inference ignores relational and textual priors; whether they improve disambiguation remains open.
  • Additional sensors: Integration of IMU, wheel odometry, WiFi/UWB signals, or barometric height for better priors and uncertainty modeling is not explored.
  • Dynamic object handling: No strategy to detect and downweight dynamic or recently moved objects in the observation model (e.g., object persistence priors, temporal consistency checks).
  • Long-term operation: Loop closure, drift correction over long trajectories, and map updates (e.g., adding/removing objects and updating embeddings) are not addressed.
View Paper Prompt  View All Prompts 

Practical Applications

Overview

The paper introduces SG2Loc, a sequential visual localization method that uses compact 3D scene graphs (object nodes with coarse meshes and semantic embeddings) and a particle filter that integrates semantic, photometric, and depth cues. It achieves strong indoor localization accuracy with orders-of-magnitude lower map storage than image/point-cloud-based methods, and adds a sequential scene-retrieval extension to select the correct scene map in multi-scene deployments. The approach is particularly suited to memory/bandwidth-constrained devices and environments with frequent changes.

Below are actionable applications organized by deployment horizon. Each item includes sectors, potential tools/workflows, and key assumptions/dependencies.

Immediate Applications

These can be piloted/deployed now with available tools (e.g., SceneGraphFusion/CLIO, existing SLAM odometry, GPU for raycasting) and modest engineering.

  • Robust, compact indoor localization for AR navigation and guidance (AR/XR, Education, Retail)
    • What: On-device localization for indoor wayfinding (malls, museums, campuses) using small scene-graph maps; sequential retrieval picks the correct venue/zone, then SG2Loc provides pose for overlays.
    • Tools/workflows: Pre-scan building; build scene graph (SceneGraphFusion/CLIO); package map + embeddings; integrate SG2Loc via SDK (Unity/Unreal plugin) and existing VIO/SLAM; keyframe at ~0.3–1 Hz to match compute.
    • Assumptions/dependencies: Indoor only; pre-built scene graphs with object-level meshes and embeddings; device intrinsics and gravity known or estimated; GPU or fast raycasting; SLAM/VIO for ego-motion.
  • Bandwidth-efficient relocalization for AMRs in facilities and warehouses (Robotics, Logistics)
    • What: Global relocalization for mobile robots (recovery from tracking failure; loop closure) using compact scene-graph maps synced over low bandwidth.
    • Tools/workflows: ROS 2 node wrapping SG2Loc; map server distributing scene-graph bundles per area; sequential retrieval to route robot to correct zone; plug-in to existing VIO/SLAM stacks (e.g., DROID-SLAM).
    • Assumptions/dependencies: Pre-mapped sites; accurate odometry for motion model; moderate GPU; periodic map maintenance for layout changes.
  • Privacy-preserving mapping and localization in public interiors (Policy, Smart Buildings)
    • What: Replace image databases with object-level graphs and coarse meshes, reducing storage of personally identifiable imagery while enabling localization.
    • Tools/workflows: Replace image/point-cloud stores with scene-graph map distribution; governance to discard raw images post-processing; audit logs for object-level updates.
    • Assumptions/dependencies: Legal/privacy review confirming coarse meshes/semantics reduce PII risks; robust object segmentation during mapping.
  • Digital twin distribution to edge devices (AECO, Facilities Management)
    • What: Use compact scene graphs as the “field-ready” version of a BIM/digital twin for on-device localization and context.
    • Tools/workflows: BIM-to-scene-graph export; periodic rescan to update object embeddings; change detection pipeline; OTA distribution to devices/robots.
    • Assumptions/dependencies: Repeatable object identification across sessions; connectors from BIM/CAD to object meshes and semantics.
  • AR-assisted maintenance and inspection workflows (MRO, Utilities)
    • What: Technicians localize and overlay step-by-step procedures anchored to recognized objects (panels, valves) even under scene changes.
    • Tools/workflows: Pre-scan plant; generate object-level meshes/labels; SG2Loc for pose; PnP refinement to tighten alignment; authoring tools to bind instructions to graph nodes.
    • Assumptions/dependencies: Sufficient object coverage/visibility; depth or good photometrics improves accuracy; safety review for critical tasks.
  • Museum/retail in-store guides and analytics (Tourism, Retail)
    • What: Device-based navigation to exhibits/products with small on-device maps; robustness to rearranged displays.
    • Tools/workflows: Per-zone scene graphs; sequential retrieval to select floor/section; SG2Loc for pose; content CMS linking to object nodes.
    • Assumptions/dependencies: Frequent restaging requires map updates; occlusion handling via sequence helps but may need staff scans.
  • GPS-denied responder navigation kits (Public Safety)
    • What: Deploy compact maps in critical buildings (stations, arenas) for firefighter/medic localization on body-worn devices with low bandwidth.
    • Tools/workflows: Pre-incident scans; scene-graph bundles on devices; keyframed sequential processing; optional depth from mobile sensors.
    • Assumptions/dependencies: Pre-scans required; thermal/smoke degrade RGB—method benefits from semantic cues but may need depth; ruggedized compute.
  • Academic benchmarking and teaching modules (Academia)
    • What: Study storage-accuracy-runtime trade-offs; teach particle-filter observation models with semantic/photometric cues; compare to HLoc/NeRF/GS.
    • Tools/workflows: Public datasets (3RScan/ScanNet); provided code; ROS bag integrations; ablation frameworks.
    • Assumptions/dependencies: GPU availability; familiarity with SLAM stacks; scene-graph generation tooling.

Long-Term Applications

These need further research, optimization, standardization, or scaling (e.g., mobile real-time rates, online scene-graph SLAM).

  • Real-time AR glasses localization at video rate (AR/XR)
    • What: Continuous 20–60 fps localization for immersive overlays.
    • Tools/workflows: GPU/accelerator kernels for raycasting and embedding; model distillation for faster per-patch semantics; adaptive particle budgets.
    • Assumptions/dependencies: Hardware acceleration on wearables; further algorithmic speedups; energy constraints.
  • Scene-graph SLAM: online mapping and relocalization with object persistence (Robotics, Software)
    • What: Build and update the 3D scene graph on the fly and localize simultaneously; maintain object identities over time.
    • Tools/workflows: Incremental object segmentation/meshing; graph optimization with relational constraints; loop closure over objects.
    • Assumptions/dependencies: Robust, real-time object tracking/segmentation; consistent IDs under dynamics; scalable graph back-end.
  • City-scale or campus-wide multi-building deployments (Smart Cities, Transportation)
    • What: Hierarchical scene graphs (building→floor→zone) with fast sequential retrieval and transfer between submaps.
    • Tools/workflows: Indexing and routing services; map streaming per zone; cross-elevator/stair transitions via graph edges.
    • Assumptions/dependencies: Scalable map ops; robust inter-scene retrieval; comprehensive coverage and maintenance.
  • Standardization: Indoor Scene Graph Map (ISGM) exchange format (Policy, Industry Consortia)
    • What: Define open standards for object-level indoor maps (geometry, embeddings, relations) for cross-vendor interoperability.
    • Tools/workflows: Schema aligned with IFC/BIM and robotics (ROS 2); privacy classifications for nodes; certification processes.
    • Assumptions/dependencies: Multi-stakeholder coordination; alignment with building owners’ data policies.
  • Lifelong autonomy in dynamic environments (Robotics, Facilities)
    • What: Continual adaptation to layout changes (furniture, shelving) with automatic change detection and versioned graph maps.
    • Tools/workflows: Delta-map updates; confidence-aware localization under inconsistencies; scheduling re-scans.
    • Assumptions/dependencies: Reliable change detection; conflict resolution between agents; minimal human supervision.
  • Cross-device object-anchored AR labels (Enterprise Collaboration)
    • What: Persistent labels anchored to object nodes (not raw geometry), shared across teams and devices.
    • Tools/workflows: Object-ID-based anchors; cloud sync; spatial permissions by object class.
    • Assumptions/dependencies: Stable object identity across updates; user-friendly authoring; access control.
  • Low-power micro-robot localization via semantic observation models (Robotics, IoT)
    • What: Use sparse cameras and low-resolution semantics for localization where compute is minimal.
    • Tools/workflows: Ultra-light encoders; FPGA/ASIC implementations for patch embeddings and ray tests.
    • Assumptions/dependencies: Extreme model compression; simplified particle filters; acceptable accuracy for tasks.
  • Hybrid scene-graph + neural rendering maps (Media, Digital Twins)
    • What: Use scene graphs for robust, compact localization and neural/GS maps for photorealistic rendering when needed.
    • Tools/workflows: Dual-map pipelines; cross-referencing object nodes to neural assets; runtime switching.
    • Assumptions/dependencies: Co-registration between representations; memory and streaming orchestration.
  • Federated map sharing across home robots and IoT devices (Consumer, Smart Home)
    • What: Standardized, secure scene-graph maps shared among vacuums, assistants, AR apps.
    • Tools/workflows: Home hub map server; permissioned node access (rooms/objects); automatic updates after furniture moves.
    • Assumptions/dependencies: Vendor cooperation; security/privacy models; simple homeowner scanning tools.
  • Reduced-bandwidth teleoperation and remote inspection (Industrial, Utilities)
    • What: Operators receive compact scene-graph context; robot localizes locally without streaming heavy video/maps.
    • Tools/workflows: Edge-local localization with periodic keyframes; shared object annotations for tasks.
    • Assumptions/dependencies: Reliable minimal telemetry; occasional ground-truth synchronization; operator UI integration.

Notes on feasibility across applications:

  • Core dependencies: pre-built 3D scene graphs with object meshes and embeddings; device intrinsics and gravity; sequential image input; ego-motion from SLAM/VIO; GPU or accelerated raycasting; optional depth improves accuracy.
  • Constraints: indoor emphasis; 4-DoF assumption (yaw rotation around gravity); current runtime (~2.9 s/frame reported) implies keyframing or offline/nearline use for now; performance depends on object segmentation quality and scene-graph freshness.
  • Advantages: orders-of-magnitude lower map storage; robustness to scene changes via semantic cues; privacy benefits over image databases; simple distribution and updates.
View Paper Prompt  View All Prompts 

Glossary

  • 2D-3D correspondences: Matches between 2D image features and 3D scene points used for pose estimation. "pose estimation from 2D-3D correspondences"
  • 3D scene graph: A graph-based map encoding objects (nodes), their attributes, and spatial relations in 3D. "We model the environment as a 3D scene graph G=(V,E)\mathcal{G} = (\mathcal{V}, \mathcal{E})"
  • absolute pose regression: Learning-based approach that directly predicts a camera’s 6-DoF pose from an image. "absolute pose regression \citep{walch2017image, kendall2015posenet}"
  • BVH-tree: Bounding Volume Hierarchy; an acceleration structure for fast ray queries (e.g., ray tracing). "BVH-tree for raytracing with particles"
  • camera intrinsics: Internal camera parameters (e.g., focal length, principal point) defining the projection model. "camera intrinsics are usually accessible from robot sensors, smartphones, and head-mounted devices"
  • Coarse-to-Fine Optimization: Strategy that begins with low-resolution/global search and progressively refines at higher resolution or in a smaller region. "Coarse-to-Fine Optimization"
  • cosine similarity: Measure of similarity between two vectors based on the cosine of the angle between them. "We compute the cosine similarity"
  • cross-modal localization: Localizing using inputs and maps from different modalities (e.g., image vs. scene graph). "cross-modal localization of a query image"
  • Depth supervision: Using depth information to guide or evaluate pose hypotheses during localization. "Depth supervision."
  • Dirac delta function: Generalized function representing a point mass used in probabilistic formulations. "where δ\delta is the Dirac delta function"
  • ego-motion: The camera/agent’s motion between frames, typically estimated by visual-inertial odometry or SLAM. "Given the current image ItI_t and its ego-motion, the pipeline updates the particle state"
  • Gaussian Splat-based methods: Rendering/mapping methods that represent scenes with 3D Gaussian primitives for fast view synthesis. "Gaussian Splat-based methods \citep{maggio2022loc, adamkiewicz2022vision, khatib2025gsvisloc, meng2025nurf}"
  • geodesic distance: Shortest-path distance on a curved space/manifold; used here to measure rotation error on SO(3). "The rotation error is calculated as the geodesic distance between the estimated rotation matrix"
  • gravity axis: The vertical axis aligned with gravity, often used to reduce pose degrees of freedom. "rotation around the known gravity axis"
  • KLD-sampling: Adaptive particle filter resampling that selects the number of particles based on KL divergence bounds. "an adaptive scheme based on KLD-sampling"
  • Kullback--Leibler divergence: A measure of dissimilarity between probability distributions. "bounds on the Kullback--Leibler divergence"
  • Maximum Likelihood Estimate (MLE): The parameter value (pose) that maximizes the likelihood under the model. "Maximum Likelihood Estimate (MLE) is shown in blue"
  • NeRF: Neural Radiance Fields; a neural scene representation enabling novel view synthesis and photometric matching. "NeRF- and Gaussian Splat-based methods"
  • particle filter: A sequential Monte Carlo method that represents a posterior over states with weighted samples (particles). "Localization is performed within a particle filter framework."
  • patch embeddings: Feature vectors computed per image patch that encode semantic/object information. "obtain a set of patch embeddings"
  • photometric consistency: Agreement between observed and rendered image appearance used as a constraint or score. "providing compact maps and constraints in addition to photometric consistency."
  • photometric error: Intensity/color discrepancy between rendered and observed images used as an optimization objective. "localize by minimizing photometric error of rendered and observed views."
  • place recognition: Identifying the scene/location depicted by an image within a database of places. "it showed promise for place recognition."
  • PnP: Perspective-n-Point; estimating camera pose from 2D image points and corresponding 3D points. "RANSAC-based PnP"
  • posterior: The probability distribution over states after incorporating observations. "To approximate posterior p(st∣I1:t)p(\mathbf{s}_t \mid I_{1:t})"
  • ray-mesh intersection: Computing the intersection points of rays with a triangle mesh, used to establish 2D–3D correspondences. "ray-mesh intersection"
  • raycasting: Projecting rays from the camera into the scene to determine visible surfaces/objects. "Raycasting."
  • RANSAC: Robust estimator that fits models by iteratively sampling minimal sets and rejecting outliers. "using RANSAC"
  • scene coordinate regression: Predicting, for each image pixel, the corresponding 3D scene coordinate for pose estimation. "ACE is a scene coordinate regression network"
  • semantic descriptor: A learned embedding that encodes object/category-level semantics for matching. "semantic descriptors, significantly reducing storage overhead"
  • Simultaneous Localization and Mapping (SLAM): Estimating the sensor trajectory while building a map of the environment. "Simultaneous Localization and Mapping (SLAM)-based techniques"
  • SO(3) manifold: The space (Lie group) of 3D rotations. "on the SO(3) manifold"
  • SSIM: Structural Similarity Index Measure; a perceptual metric for image similarity in [0,1]. "structural similarity (SSIM)"
  • stratified resampling: Particle filter resampling technique that reduces variance by stratifying the sampling space. "We then apply stratified resampling"
  • yaw: Rotation about the vertical (gravity) axis in a 3D pose parameterization. "yaw angle ϕ∈[−π,π]\phi \in [-\pi, \pi]"
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 47 likes about this paper.

Sign Up for Free