osmAG-LLM: Hierarchical Map Reasoning in Robotics

• osmAG-LLM is a framework combining a hierarchical, semantic, topometric area graph with LLMs to enable high-level indoor navigation reasoning.
• It supports topology and hierarchy comprehension through LLM-assisted planning, enabling effective object-goal navigation and copilot guidance over semantic maps.
• The system demonstrates improved efficiency and adaptability by leveraging lightweight map representations, dynamic replanning, and robust anomaly detection.

osmAG-LLM denotes a line of robot-navigation and map-reasoning systems that combine osmAG—a textual hierarchical semantic topometric Area Graph stored in an OpenStreetMap-like representation—with LLMs or multimodal LLMs for high-level reasoning over indoor space. In the literature, the term covers at least three closely related uses: topology-and-hierarchy comprehension over map text, LLM-assisted copilot planning over semantic maps, and zero-shot open-vocabulary object-goal navigation grounded in a lightweight semantic map rather than a dense object memory (Xie et al., 2024, Xie et al., 2024, Xie et al., 17 Jul 2025). A related development, though not titled osmAG-LLM, embeds the same representation inside a hierarchical VLN stack with an MLLM “brain” and reactive replanning, indicating that osmAG has also become a substrate for long-horizon embodied control under environmental change (Li et al., 13 Mar 2026).

1. Foundational representation and design rationale

At the center of osmAG-LLM is osmAG, a map formalism derived from the Area Graph. Its basic elements are stable across the papers: areas are represented as polygons corresponding to rooms, corridors, lobbies, sectors, floors, or buildings, and passages are line segments on area boundaries that connect neighboring areas. The representation is described as hierarchical, semantic, and topometric: topology encodes adjacency and reachability, geometry provides spatial structure, and semantics provide labels and human-meaningful annotations (Xie et al., 2024, Xie et al., 2024).

The hierarchical form extends the original Area Graph by enclosing child areas within parent areas so that containment relations such as room $\rightarrow$ wing $\rightarrow$ floor $\rightarrow$ building can be encoded. In the 2024 map-comprehension work, this hierarchy is saved as tags in OpenStreetMap XML, making the map simultaneously LLM-readable, compatible with classical robotics algorithms, compact relative to raw grids or point clouds, and human-editable or visualizable in JOSM and rviz (Xie et al., 2024).

A later formulation in HaltNav makes the dynamic aspect explicit by modeling the prior as a time-varying weighted directed graph

$\mathcal{G}_t = (\mathcal{V}, \mathcal{E}, W_t),$

where $\mathcal{V}$ are semantic nodes such as rooms or regions, $\mathcal{E} \subseteq \mathcal{V}\times\mathcal{V}$ are traversable passages, and $W_t : \mathcal{E}\to\mathbb{R}^+$ is a dynamic traversal cost. In practical terms, the map is still built from areas as closed polygons and passages as shared boundaries or doorways, but the graph can now be updated when the world no longer matches the prior (Li et al., 13 Mar 2026).

The rationale for choosing osmAG over dense metric or feature maps is consistent across the literature. Dense maps are described as computationally heavy to build and maintain, liable to modality misalignment, and vulnerable to becoming stale in dynamic environments. By contrast, osmAG is characterized as lightweight, token-efficient, naturally compatible with language reasoning, and suitable for long-horizon planning without retaining a full metric memory (Li et al., 13 Mar 2026). Early osmAG-LLM work made the additional methodological choice to drop metric information entirely during LLM reasoning because LLMs were judged to still struggle with quantitative reasoning; the focus was therefore restricted to topological and hierarchical understanding rather than metric optimization (Xie et al., 2024).

2. Topology and hierarchy comprehension

The earliest explicit osmAG-LLM formulation studies whether LLMs can comprehend the topology and hierarchy of osmAG well enough to support robot planning tasks such as route selection, branch choice at intersections, shortest-path reasoning over cycles, and containment reasoning from room to building (Xie et al., 2024). The tasks are deliberately non-metric. For topology, the model must output the sequence of room names along the path between two rooms; for hierarchy, it must recover the enclosing building by tracing parent tags through room, zone, floor, wing, and building (Xie et al., 2024).

Three prompt levels were evaluated. Level 1 provides a basic osmAG explanation and task explanation, Level 2 adds a simple example, and Level 3 adds a detailed example with an illustrated map and detailed answer. In parallel, two map simplifications were introduced to make the XML more explicit for LLMs. Variant 1 adds a tag to an area way with key "connected_area" and the connected area as value, while Variant 2 further makes the current area name and directly connected room explicit. Variant 2 performed best and was selected for later experiments; Level 3 was chosen for training because it performed slightly better overall, although examples did not always improve performance (Xie et al., 2024).

Fine-tuning was performed on LLaMA2-7B and LLaMA2-13B with LoRA, using the LLaMA Factory repository together with DeepSpeed and ZeRO memory optimization. The reported LoRA settings were rank = 8, target modules q_proj and v_proj, learning rate $5\times10^{-5}$, and a cosine learning-rate scheduler. The resulting adapters were approximately 16.8MB for 7B and 26.2MB for 13B, which the paper presents as small enough to swap across tasks (Xie et al., 2024).

Quantitatively, prompt engineering and map presentation mattered before fine-tuning. On original osmAG, ChatGPT-3.5 achieved roughly 0.42–0.54 success and ChatGPT-4 roughly 0.85–0.87; on Variant 2, ChatGPT-3.5 improved to about 0.69–0.70 and ChatGPT-4 to about 0.95–0.96. Unadapted LLaMA2-7B and LLaMA2-13B performed poorly on topology, at about 0.05–0.11 and 0.066–0.14 respectively, and reached 0.19 and 0.55 on hierarchy. After LoRA fine-tuning, performance rose sharply: Fine-tuned LLaMA2-7B reached 0.99 on Dataset 1, 0.94 on Dataset 2, and 0.89 on Dataset 3; Fine-tuned LLaMA2-13B reached 0.98, 0.95, and 0.97 on the same datasets. On hierarchy, the fine-tuned models achieved 1.0 and 0.99, and both retained 0.95 on the general-knowledge Dataset 5, indicating little catastrophic forgetting in that setup (Xie et al., 2024).

A recurring finding is prompt sensitivity. The fine-tuned 7B model dropped to 0.78 on Dataset 1 and 0.60 on Dataset 2 under unseen prompts, whereas the 13B model remained stronger at 0.91 and 0.94. This supports a narrow but important distinction: osmAG-LLM can be highly effective when prompt structure is controlled, but robustness to prompt variation is a separate capability that does not automatically follow from high in-distribution accuracy (Xie et al., 2024).

3. LLMs as navigation copilot over osmAG

A second major osmAG-LLM usage treats the LLM not as a direct planner or controller but as a copilot layered over a conventional LiDAR navigation stack. In this architecture, osmAG is the interface between high-level semantic reasoning and classical motion planning. The system comprises three reasoning and planning layers—LLM-based semantic reasoning, osmAG-based path planning, and ROS move_base execution—organized into the modules NavigationEventMonitor, PassageCostEvaluator, and osmAGPathPlanner, with Navfn as the global planner and DWA as the local planner inside move_base (Xie et al., 2024).

The graph formulation differs slightly from the map-comprehension work. Here the map is written as

$AG=(V,E),$

where $V$ are passages and $\rightarrow$0 are distances between passages inside one area. A navigation task is to find an optimal passage sequence

$\rightarrow$1

Within-area passage-to-passage distances $\rightarrow$2 are precomputed using A* and stored. The LLM contributes a context-dependent passage cost $\rightarrow$3, and the total edge cost is

$\rightarrow$4

If the NavigationEventMonitor identifies inaccessible areas, their corresponding passages are removed from the graph; start and destination nodes are then added, yielding an augmented graph $\rightarrow$5 on which A* computes the final route (Xie et al., 2024).

The LLM’s role is explicitly structured. NavigationEventMonitor inspects external sources such as emails, public notifications, web announcements, or tweets and decides whether they are relevant to navigation. If relevant, the LLM outputs structured information—stored as a JSON string—including areas to avoid, sometimes areas to try to avoid, and whether robot capability constraints such as “wheeled robot cannot use stairs” are implicated. PassageCostEvaluator receives the instruction, map information, and historical passage experience, identifies the destination area, and assigns passage costs based on passage type, robot capability, and prior success or failure. move_base then executes the selected passage sequence over an occupancy grid rendered only from the chosen osmAG areas and passages, constraining local execution to the semantically selected route (Xie et al., 2024).

The experimental setting was Gazebo simulation over a real campus osmAG map with a turtlebot3 waffle, 6 m LiDAR, an environment of over 6,200 m², and 70+ rooms per floor. Across 37 trials in 4 first-floor cases, the proposed method reduced total path length by 58% compared to move_base with obstacle-layer clearing between trials, while consistently avoiding restricted areas and reaching the destination. The average path lengths reported for the full method were 192.5, 126.5, 236.3, and 214.7 meters, compared with 347.3, 202.2, 398.5, and 267.5 meters for move_base (Xie et al., 2024).

The event-monitor results also clarify the system’s operational profile. It correctly identified all 20 notification relevance cases in the described test set. For path approval, across 95 requests over the 37 trials, 57 of 61 valid paths were approved correctly and all 34 invalid paths were rejected. However, there were 14/95 outputs in which the semantic decision was correct but the format was wrong—for example, a sector name or room number was returned instead of the expected exact area label. This provides a concrete example of a common misconception: osmAG-LLM in this form does not replace deterministic navigation software; it augments it, and structured-output reliability remains a practical integration bottleneck (Xie et al., 2024).

4. Zero-shot open-vocabulary object-goal navigation

A later system, also titled osmAG-LLM, applies the representation to zero-shot, open-vocabulary object-goal navigation. Its central claim is that for embodied object search, the map should not be treated as a perfect memory of all object instances. Instead, the map should provide environment grounding and context, while the LLM supplies semantic priors about likely object locations and online perception verifies the object at runtime (Xie et al., 17 Jul 2025).

The underlying osmAG is enriched offline from RGB-D trajectories with three forms of semantic content. Object-nodes are created from LabelMaker outputs and Mask3D instance masks; each instance centroid is inserted into osmAG with key semantic_osmAG:object_name. Viewpoint-nodes are created by prompting a VLM with “What objects are in this image?” at selected capture locations, and storing the answer with key semantic_osmAG:observed_object. Room descriptions are created by prompting a VLM with “Describe the image.”, summarizing the resulting descriptions with an LLM, and attaching them to room areas. This combination is explicitly intended to make the map open-vocabulary, because viewpoint-nodes are not limited to the label set of the object-mapping pipeline (Xie et al., 17 Jul 2025).

At query time, the LLM receives an explanation of semantic-osmAG, the retrieval objective, and a simplified map that contains only semantic nodes, hierarchy, and room relationships, while excluding coordinates and polygons. It is constrained to output candidate search targets in JSON format with maximum 3 nodes per room, maximum 3 rooms total, and nodes ordered by decreasing predicted likelihood. The search policy then navigates sequentially to the returned nodes using ROS navigation, captures images, runs an open-vocabulary detector—YOLO-World or DINO-X—and verifies candidate crops with the VLM StepFun step-1o-turbo-vision using the prompt “Is there a [object] here?”. If no detection is confirmed, the robot rotates to capture additional viewpoints and then proceeds to the next candidate node (Xie et al., 17 Jul 2025).

Navigation itself remains lightweight. The system converts osmAG to an occupancy grid in which area polygons are marked occupied and passages free, plans paths with A*, updates the rendered occupancy grid online using real-time depth, and replans when collisions are detected. The map is therefore used as permanent structure, not as a dense, ever-current object memory (Xie et al., 17 Jul 2025).

On HM3D-SEM, using 8 scenes and 5 random starting positions per scene, the main baseline was HOV-SG. For the hardest query type $\rightarrow$6, osmAG-LLM achieved R-RSR 0.66 versus 0.27, APL 12.32 m versus 20.15 m, and AMD 4.65 m versus 8.85 m. For $\rightarrow$7, R-RSR improved from 0.32 to 0.67, APL from 16.81 m to 11.54 m, and AMD from 7.87 m to 5.27 m. For $\rightarrow$8, R-RSR improved from 0.65 to 0.83, while osmAG-LLM retained far shorter paths. The reported DIR values were also higher: for $\rightarrow$9, 0.27 versus 0.05; for $\rightarrow$0, 0.25 versus 0.06; and for $\rightarrow$1, 0.31 versus 0.05. A particularly strong systems-level result is storage: across the 8 scenes, HOV-SG required 1493 MB, whereas semantic-osmAG required 3.2 MB (Xie et al., 17 Jul 2025).

The real-world experiments used a Fetch platform with base-mounted 2D LiDAR, AMCL, move_base, a head-mounted RealSense RGB-D camera, and five rooms ranging from 15 m² to 147 m². Across 30 experiments, the paper distinguishes Static objects (SO), Relocated objects (RO), and Unmapped objects (UO). For SO, R-RSR improved from 0.40 to 1.00 and APL from 47.00 m to 18.13 m. For RO, R-RSR improved from 0.40 to 1.00, O-RSR@3m from 0.50 to 0.90, AMD from 5.36 m to 1.63 m, and DIR from 0.00 to 0.70. For UO, R-RSR improved from 0.20 to 1.00, O-RSR@3m from 0.50 to 0.80, AMD from 7.42 m to 2.25 m, DIR from 0.30 to 0.80, and APL from 52.03 m to 30.68 m. Even a sparse semantic-osmAG retaining only room-level descriptions reported room-retrieval values in parentheses of 0.8–1.0, which the paper presents as evidence for scalability under token constraints (Xie et al., 17 Jul 2025).

The reported failures sharpen the interpretation of the system. There were 9 failures out of 30 trials: 6 when the robot reached a place where the object was visible but failed to retrieve it, and 3 false positives where the object was not visible. This produced 27/30 correct robot positioning, or 90%, and locates the principal residual error in detection, not navigation. Examples included confusing a ladder reflection as a ladder, confusing a hugging pillow as a neck pillow, and failing to generate a bounding box for a transparent object (Xie et al., 17 Jul 2025).

5. Hierarchical MLLM control and reactive halting

A related development extends the osmAG-LLM idea from symbolic planning and object search to long-horizon vision-language navigation in dynamic settings. HaltNav does not use the title osmAG-LLM, but it relies on the same premise that a lightweight, text-based osmAG prior can ground a high-level LLM while low-level execution is delegated to a separate navigation policy (Li et al., 13 Mar 2026).

The architecture is hierarchical. A global planner operates on an osmAG-derived passage graph by constructing a passage-level graph, computing traversal costs between passages with A* on rendered 2D occupancy submaps, and storing those costs for reuse. An MLLM-based high-level component called the Graph-Grounded Task Dispatcher (GGTD) reads a structured textual prompt of the local osmAG subgraph together with the target instruction and trajectory history, and outputs a localized macro-action

$\rightarrow$2

A low-level VLN policy $\rightarrow$3 then executes with egocentric observation $\rightarrow$4 and active macro-action $\rightarrow$5,

$\rightarrow$6

This decomposition keeps global topological reasoning and local control separate (Li et al., 13 Mar 2026).

The principal robustness mechanism is Reactive Visual Halting (RVH), defined by a halting function

$\rightarrow$7

RVH halts if either collision accumulation exceeds a threshold or the MLLM’s semantic traversability score falls below threshold. The monitored anomalies include blocked passages, closed doors, crowded corridors, unmapped obstacles, and physical hazards. When RVH fires, the system updates the passage-level graph rather than merely extending the prompt: $\rightarrow$8 Setting a traversal cost to $\rightarrow$9 effectively prunes the blocked edge and triggers replanning over the updated topology (Li et al., 13 Mar 2026).

Because off-the-shelf MLLMs were found not to be reliably grounded in robot affordances, the paper introduces a dual-engine synthesis pipeline for hard negatives. One engine injects dynamic 3D obstacles at topological bottlenecks inside Habitat; the other uses diffusion or inpainting-style generative editing to insert realistic obstacles into traversable real RGB observations. The training pairs are

$\mathcal{G}_t = (\mathcal{V}, \mathcal{E}, W_t),$0

and the MLLM is fine-tuned with LoRA under supervised negative log-likelihood

$\mathcal{G}_t = (\mathcal{V}, \mathcal{E}, W_t),$1

This places osmAG inside a broader closed-loop architecture in which language reasoning, anomaly detection, and map update are jointly operationalized (Li et al., 13 Mar 2026).

Empirically, the system degrades more gracefully than baselines under both underspecified instructions and obstacle injection. Under obstacle robustness in simulation, HaltNav retained 50.00% SR at L0-O, 43.75% SR at L1-O, and 31.25% SR at L2-O. On a Fetch robot in a university building, it retained 56.66% SR at L0-O and 46.66% SR at L2-O, while baselines often collapsed to 0% SR under obstacle injection. A plausible implication is that the osmAG-LLM paradigm scales beyond static route reasoning when the map is treated as a mutable topological prior rather than a fixed description (Li et al., 13 Mar 2026).

6. Limitations, misconceptions, and adjacent extensions

Several limitations recur across the osmAG-LLM literature. First, the paradigm is not inherently a substitute for metric optimization. The original map-comprehension work explicitly removed metric information because LLMs were judged not to handle quantitative reasoning well, which means the resulting routes are topologically correct but not guaranteed to be metric-optimal (Xie et al., 2024). Later systems reintroduced geometry through precomputed distances, occupancy-grid rendering, or passage costs, but they did so by delegating numerically sensitive components to A*, move_base, or a separate VLN executor rather than to the LLM itself (Xie et al., 2024, Li et al., 13 Mar 2026).

Second, osmAG-LLM does not generally ask the LLM to drive the robot directly. In the LiDAR copilot architecture, the LLM interprets instructions, external events, and experience while ROS move_base executes motion (Xie et al., 2024). In object navigation, the LLM ranks likely search nodes while navigation and visual verification remain separate (Xie et al., 17 Jul 2025). In HaltNav, the MLLM generates macro-actions, but the low-level policy is explicitly agnostic to the global osmAG and remains responsible for immediate control (Li et al., 13 Mar 2026). The common pattern is therefore semantic deliberation over a compact structural prior plus classical or learned execution, not end-to-end language control.

Third, the representation is deliberately biased toward permanent structure rather than exhaustive world state. This is a feature in some settings and a limitation in others. In the 2025 object-navigation work, the map is explicitly framed as grounding and context, not as a perfect object memory, precisely because objects are often relocated or were never mapped (Xie et al., 17 Jul 2025). Conversely, in dynamic navigation settings such as closed doors or crowded corridors, explicit mechanisms like RVH and topology invalidation become necessary because a static osmAG alone is brittle (Li et al., 13 Mar 2026).

Fourth, structured-output reliability and prompt dependence remain nontrivial engineering issues. The 2024 copilot system reported 14/95 formatting errors despite correct semantics, and the 2024 comprehension study showed notable degradation under unseen prompts, especially for LLaMA2-7B (Xie et al., 2024, Xie et al., 2024). These are not incidental implementation details; they delimit how directly LLM outputs can be embedded in robotic software stacks.

Adjacent non-LLM work also clarifies why osmAG persists as a common substrate. A WiFi-based global localization framework uses osmAG as a structural prior for offline AP localization and online robot localization, reporting 3.79 m mean AP localization error with 35.3\% improvement over trilateration, 3.12 m mean robot localization error in fingerprinted areas with 8.77\% improvement over KNN fingerprinting, and 3.83 m in non-fingerprinted areas with 81.05\% improvement, over a 11,025 m² multi-floor environment (Ma et al., 13 Aug 2025). This work is not itself an osmAG-LLM system, but it shows that the representation’s value is not confined to language reasoning. This suggests that osmAG’s long-term role may be as a shared intermediate representation across localization, planning, semantic reasoning, and reactive navigation, with LLMs occupying only one layer of a broader robotics stack (Ma et al., 13 Aug 2025).