- The paper introduces LangLoc, a three-stage system that combines dual-branch GATv2 scene-graph retrieval, ray-cast visibility scoring, and Bayesian dialog to estimate an observer’s 2D position and heading from text alone.
- LangLoc achieves 76.7% Top-1 scene-retrieval recall in a 10-scene pool, outperforming Text2SGM by 8.1 percentage points and reaching 59.5% Top-1 recall on challenging LLM-generated queries.
- Interactive disambiguation reduces median position error from 1.55 to 0.80 m on 3RScan and from 1.31 to 0.069 m on ScanNet, while limitations include dependence on detailed 3D scene graphs and controlled dialog evaluation.
Problem and motivation
LangLoc addresses fine-grained indoor localization from free-form natural language: given a textual description of one's surroundings ("I'm standing in front of a bookshelf, with a blue sofa on my left"), the system estimates the observer's 2D floor position and heading within a known 3D environment. Prior language-based localization work, notably Text2Pos (collaboration et al., 2022) and Text2SGM (Wu, 2024), stops at coarse scene retrieval — identifying which room a description refers to — without resolving an intra-scene pose. LangLoc extends the task to full pose estimation, motivated by settings where camera queries are impractical or privacy-invasive: hospitals, labs, emergency dispatch, and digitally twinned environments.
The paper's central claim is that visibility geometry is sufficient to bridge the gap between coarse retrieval and metric localization: an observer must stand where the mentioned objects are simultaneously visible and nearby. This geometric prior, combined with interactive disambiguation, yields approximately 1 m median position accuracy from text alone — no image query involved.
Method
The pipeline comprises three stages.
Scene retrieval. Free-form text is parsed (via GPT-4o-mini) into a text scene graph whose nodes are objects and edges are typed spatial relations. A dual-branch GATv2 encoder processes both text graphs and dense 3D scene graphs from 3DSSG annotations. Node features concatenate the 3D centroid, mean RGB color, and a CLIP ViT-B/32 embedding of the semantic label; text-derived nodes zero-pad the geometric fields so the learned gate can rely on the relation branch alone. The two branches — a k=5 nearest-neighbor geometric branch conditioned on relative position and size, and a relation branch embedding predicates with CLIP — are fused per node via a learned sigmoid gate, then pooled with a scene-level CLIP descriptor into a 256-dim graph embedding trained with InfoNCE contrastive loss. At inference, scenes are ranked by a weighted combination of embedding similarity, global CLIP descriptor similarity, and label-overlap F1.
Fine localization. Query entities are matched to scene objects via Word2Vec cosine similarity with greedy assignment. Candidate positions are sampled on a uniform 0.25 m floor grid at 1.6 m eye height, and each candidate is scored by ray-casting against the scene mesh: the score sums visible matched-object counts plus an exponential proximity bonus. Heading is estimated by selecting the frustum orientation capturing the most visible object directions. The output is a softmax posterior over floor cells rather than a single point estimate.
Dialog-based disambiguation. When the posterior is multi-modal (e.g., two seating areas both consistent with "a sofa facing a TV"), the system discretizes poses into reference frames annotated with visible labels and relations, maps the fine-localization belief onto them via distance-weighted soft assignment, and iteratively asks yes/no questions selected by expected information gain (with a balanced-split heuristic as a lightweight alternative). Answers are modeled with soft truth and answerability probabilities plus a reliability parameter that flips yes/no responses, and the frame posterior is updated by Bayes' rule until it concentrates above a confidence threshold.
Benchmark contribution
The authors contribute a benchmark of over 13,000 pose-indexed natural-language descriptions across more than 1,300 indoor 3D scans built on 3RScan. Keyframes are curated through quality filtering (QualiCLIP), mesh-rendered visibility computation, and a two-stage determinantal point process enforcing both semantic and spatial diversity. Descriptions are generated by an LLM from per-keyframe visible-object lists and relations, supplemented by human annotation. Existing datasets such as ScanRefer and ScanQA target grounding or question answering, not viewpoint-conditioned pose estimation, so this resource fills a genuine data gap.
Scene retrieval results
On ScanScribe with a 10-scene candidate pool, LangLoc achieves 76.7% Top-1 recall, exceeding the strongest Text2SGM variant (68.6%) by 8.1 percentage points, and reaches 98.9% at Top-5. On the full 55-scene test set, it attains 83.3% Top-5 and 91.6% Top-10. Under domain shift — LLM-generated queries from scene images with reduced lexical overlap — the gap widens sharply: LangLoc obtains 59.5% Top-1 versus 34.2% for Text2SGM and 33.1% for CLIP2CLIP, a +25 pp improvement, indicating that the dual-branch encoder generalizes better to natural phrasing than methods dependent on lexical overlap with database labels.
| Method |
Top-1 (10-pool) |
Top-1 (LLM queries) |
| CLIP2CLIP |
33.09 |
33.07 |
| Text2SGM (cos-sim) |
68.27 |
34.22 |
| LangLoc |
76.70 |
59.50 |
Fine localization results
Against a midpoint baseline (floor centroid) and a zero-shot Qwen2.5-VL-2B baseline given a top-down rendering, on 100-scene subsets:
| Method |
Pos. med. (m) |
Angle med. (°) |
3D IoU |
| Midpoint (3RScan) |
1.35 |
– |
– |
| VLM (3RScan) |
1.42 |
85.5 |
0.062 |
| LangLoc w/o dialog (3RScan) |
1.55 |
37.2 |
0.172 |
| LangLoc w/ dialog (3RScan) |
0.80 |
33.4 |
0.342 |
| LangLoc w/o dialog (ScanNet) |
1.31 |
34.7 |
0.236 |
| LangLoc w/ dialog (ScanNet) |
0.069 |
5.12 |
0.593 |
Several observations bear emphasis. First, the midpoint baseline achieves a deceptively competitive raw position error (~1.4 m) purely as an artifact of small rooms; the authors correctly flag it as vacuous since it conveys neither meaningful within-room position nor heading. Second, LangLoc without dialog reports higher raw position error than midpoint because it commits to specific object clusters that may be wrong under ambiguity — but its Top-10 error (1.04 m on 3RScan) shows the correct position is almost always among high-scoring cells, confirming that residual ambiguity is cluster selection, not heading estimation. Third, dialog resolves exactly this failure mode: median position error drops 49% on 3RScan (1.55 → 0.80 m) and dramatically on ScanNet (1.31 → 0.069 m, median angular error 5.1°), aided by ScanNet frames containing more visible objects (6 vs. 4) and less spatially ambiguous layouts. On the full 1,300-scene dataset without dialog, LangLoc achieves 0.95 m median Top-10 position error and 39.8° median angular error, with 3D IoU roughly an order of magnitude above the VLM baseline (0.147 vs. 0.018).
A 10-scene human-dialog pilot with human-written descriptions and human-typed answers shows LangLoc with human dialog achieving the lowest position errors (0.66 m mean) among all systems, approaching a human localizer (0.70 m), though the human retains a clear advantage in heading estimation (20.4° vs. 45.7° mean).
An important caveat: the controlled dialog evaluations obtain answers from a human annotator (3RScan) or Qwen2.5-VL-2B using ground-truth reference-frame metadata (ScanNet), so those rows reflect controlled protocols rather than full real-user studies; only the small pilot involves genuine interaction.
Limitations and open questions
The authors concede three constraints. The pipeline requires pre-built 3D scene graphs with known object labels and relationships; extension to open-vocabulary or incrementally constructed graphs remains open. Performance degrades in large, cluttered scenes where many viewpoints share similar object configurations. Finally, despite privacy-preserving queries, the method presupposes a detailed 3D model of the environment — localizing against coarser floor plans or schematic maps is left unaddressed. Additionally, the evaluation scale for dialog is limited (100-scene subsets plus a 10-scene pilot), leaving large-scale real-user validation open, and the heading gap relative to human localizers persists even with dialog.
Conclusion
LangLoc establishes that sub-meter indoor localization from natural language alone is attainable by combining contrastively trained dual-branch GATv2 scene-graph retrieval, ray-cast visibility scoring over a dense floor grid, and Bayesian dialog disambiguation. It advances the state of the art in text-to-scene retrieval by 8 percentage points Top-1 and demonstrates that a handful of targeted yes/no questions can collapse a multi-modal pose posterior to centimeter-level accuracy under favorable conditions. Its dependence on densely annotated 3D scene graphs and controlled dialog protocols defines the boundary of what the current results support.