--- title: 'osmAG-LLM: Hierarchical Map Reasoning in Robotics' url: https://www.emergentmind.com/topics/osmag-llm type: topic --- # osmAG-LLM: Hierarchical Map Reasoning in Robotics osmAG-LLM denotes a line of robot-navigation and map-reasoning systems that combine **osmAG**—a textual **hierarchical semantic topometric Area Graph** stored in an **OpenStreetMap-like** representation—with **large language models** or **multimodal large language models** for high-level reasoning over indoor space. In the literature, the term covers at least three closely related uses: topology-and-hierarchy comprehension over map text, LLM-assisted copilot planning over semantic maps, and zero-shot open-vocabulary object-goal navigation grounded in a lightweight semantic map rather than a dense object memory [2403.08228], [2409.08493], [2507.12753]. A related development, though not titled osmAG-LLM, embeds the same representation inside a hierarchical VLN stack with an MLLM “brain” and reactive replanning, indicating that osmAG has also become a substrate for long-horizon embodied control under environmental change [2603.12696]. ## 1. Foundational representation and design rationale At the center of osmAG-LLM is **osmAG**, a map formalism derived from the **Area Graph**. Its basic elements are stable across the papers: **areas** are represented as polygons corresponding to rooms, corridors, lobbies, sectors, floors, or buildings, and **passages** are line segments on area boundaries that connect neighboring areas. The representation is described as **hierarchical**, **semantic**, and **topometric**: topology encodes adjacency and reachability, geometry provides spatial structure, and semantics provide labels and human-meaningful annotations [2403.08228], [2409.08493]. The hierarchical form extends the original Area Graph by enclosing child areas within parent areas so that containment relations such as room \(\rightarrow\) wing \(\rightarrow\) floor \(\rightarrow\) building can be encoded. In the 2024 map-comprehension work, this hierarchy is saved as tags in **OpenStreetMap XML**, making the map simultaneously **LLM-readable**, compatible with classical robotics algorithms, compact relative to raw grids or point clouds, and human-editable or visualizable in **JOSM** and **rviz** [2403.08228]. A later formulation in HaltNav makes the dynamic aspect explicit by modeling the prior as a **time-varying weighted directed graph** \[ \mathcal{G}_t = (\mathcal{V}, \mathcal{E}, W_t), \] where \(\mathcal{V}\) are semantic nodes such as rooms or regions, \(\mathcal{E} \subseteq \mathcal{V}\times\mathcal{V}\) are traversable passages, and \(W_t : \mathcal{E}\to\mathbb{R}^+\) is a dynamic traversal cost. In practical terms, the map is still built from **areas** as closed polygons and **passages** as shared boundaries or doorways, but the graph can now be updated when the world no longer matches the prior [2603.12696]. The rationale for choosing osmAG over dense metric or feature maps is consistent across the literature. Dense maps are described as **computationally heavy** to build and maintain, liable to **modality misalignment**, and vulnerable to becoming **stale** in dynamic environments. By contrast, osmAG is characterized as **lightweight**, **token-efficient**, naturally compatible with language reasoning, and suitable for **long-horizon planning** without retaining a full metric memory [2603.12696]. Early osmAG-LLM work made the additional methodological choice to drop metric information entirely during LLM reasoning because LLMs were judged to still struggle with quantitative reasoning; the focus was therefore restricted to **topological** and **hierarchical** understanding rather than metric optimization [2403.08228]. ## 2. Topology and hierarchy comprehension The earliest explicit osmAG-LLM formulation studies whether LLMs can comprehend the topology and hierarchy of osmAG well enough to support robot planning tasks such as route selection, branch choice at intersections, shortest-path reasoning over cycles, and containment reasoning from room to building [2403.08228]. The tasks are deliberately non-metric. For topology, the model must output the **sequence of room names** along the path between two rooms; for hierarchy, it must recover the enclosing building by tracing `parent` tags through room, zone, floor, wing, and building [2403.08228]. Three prompt levels were evaluated. **Level 1** provides a basic osmAG explanation and task explanation, **Level 2** adds a simple example, and **Level 3** adds a detailed example with an illustrated map and detailed answer. In parallel, two map simplifications were introduced to make the XML more explicit for LLMs. **Variant 1** adds a tag to an `area` way with key `"connected_area"` and the connected area as value, while **Variant 2** further makes the current area name and directly connected room explicit. **Variant 2** performed best and was selected for later experiments; **Level 3** was chosen for training because it performed slightly better overall, although examples did not always improve performance [2403.08228]. Fine-tuning was performed on **LLaMA2-7B** and **LLaMA2-13B** with **LoRA**, using the **LLaMA Factory** repository together with **DeepSpeed** and **ZeRO** memory optimization. The reported LoRA settings were **rank = 8**, target modules **`q_proj`** and **`v_proj`**, learning rate **\(5\times10^{-5}\)**, and a **cosine learning-rate scheduler**. The resulting adapters were approximately **16.8MB** for 7B and **26.2MB** for 13B, which the paper presents as small enough to swap across tasks [2403.08228]. Quantitatively, prompt engineering and map presentation mattered before fine-tuning. On original osmAG, **ChatGPT-3.5** achieved roughly **0.42–0.54** success and **ChatGPT-4** roughly **0.85–0.87**; on **Variant 2**, ChatGPT-3.5 improved to about **0.69–0.70** and ChatGPT-4 to about **0.95–0.96**. Unadapted **LLaMA2-7B** and **LLaMA2-13B** performed poorly on topology, at about **0.05–0.11** and **0.066–0.14** respectively, and reached **0.19** and **0.55** on hierarchy. After LoRA fine-tuning, performance rose sharply: **Fine-tuned LLaMA2-7B** reached **0.99** on Dataset 1, **0.94** on Dataset 2, and **0.89** on Dataset 3; **Fine-tuned LLaMA2-13B** reached **0.98**, **0.95**, and **0.97** on the same datasets. On hierarchy, the fine-tuned models achieved **1.0** and **0.99**, and both retained **0.95** on the general-knowledge Dataset 5, indicating little catastrophic forgetting in that setup [2403.08228]. A recurring finding is **prompt sensitivity**. The fine-tuned **7B** model dropped to **0.78** on Dataset 1 and **0.60** on Dataset 2 under unseen prompts, whereas the **13B** model remained stronger at **0.91** and **0.94**. This supports a narrow but important distinction: osmAG-LLM can be highly effective when prompt structure is controlled, but robustness to prompt variation is a separate capability that does not automatically follow from high in-distribution accuracy [2403.08228]. ## 3. LLMs as navigation copilot over osmAG A second major osmAG-LLM usage treats the language model not as a direct planner or controller but as a **copilot** layered over a conventional LiDAR navigation stack. In this architecture, osmAG is the interface between high-level semantic reasoning and classical motion planning. The system comprises three reasoning and planning layers—**LLM-based semantic reasoning**, **osmAG-based path planning**, and **ROS `move_base` execution**—organized into the modules **NavigationEventMonitor**, **PassageCostEvaluator**, and **osmAGPathPlanner**, with **Navfn** as the global planner and **DWA** as the local planner inside `move_base` [2409.08493]. The graph formulation differs slightly from the map-comprehension work. Here the map is written as \[ AG=(V,E), \] where \(V\) are **passages** and \(E\) are **distances between passages inside one area**. A navigation task is to find an optimal passage sequence \[ \mathbf{P}=\{v_1,v_2,\ldots,v_k\}. \] Within-area passage-to-passage distances \(d_{ij}\) are precomputed using **A\*** and stored. The LLM contributes a context-dependent passage cost \(pc_i\), and the total edge cost is \[ t_{ij}=d_{ij}+pc_i+pc_j. \] If the NavigationEventMonitor identifies inaccessible areas, their corresponding passages are removed from the graph; start and destination nodes are then added, yielding an augmented graph \(AG'=(V',E'')\) on which **A\*** computes the final route [2409.08493]. The LLM’s role is explicitly structured. **NavigationEventMonitor** inspects external sources such as emails, public notifications, web announcements, or tweets and decides whether they are relevant to navigation. If relevant, the LLM outputs structured information—stored as a **JSON string**—including **areas to avoid**, sometimes **areas to try to avoid**, and whether robot capability constraints such as “wheeled robot cannot use stairs” are implicated. **PassageCostEvaluator** receives the instruction, map information, and historical passage experience, identifies the destination area, and assigns passage costs based on passage type, robot capability, and prior success or failure. `move_base` then executes the selected passage sequence over an occupancy grid rendered only from the chosen osmAG areas and passages, constraining local execution to the semantically selected route [2409.08493]. The experimental setting was **Gazebo simulation** over a real campus osmAG map with a **turtlebot3 waffle**, **6 m LiDAR**, an environment of over **6,200 m²**, and **70+ rooms per floor**. Across **37 trials** in 4 first-floor cases, the proposed method reduced total path length by **58%** compared to `move_base` with obstacle-layer clearing between trials, while consistently avoiding restricted areas and reaching the destination. The average path lengths reported for the full method were **192.5**, **126.5**, **236.3**, and **214.7** meters, compared with **347.3**, **202.2**, **398.5**, and **267.5** meters for `move_base` [2409.08493]. The event-monitor results also clarify the system’s operational profile. It correctly identified all **20** notification relevance cases in the described test set. For path approval, across **95** requests over the 37 trials, **57 of 61 valid paths** were approved correctly and **all 34 invalid paths** were rejected. However, there were **14/95 outputs** in which the semantic decision was correct but the format was wrong—for example, a sector name or room number was returned instead of the expected exact area label. This provides a concrete example of a common misconception: osmAG-LLM in this form does not replace deterministic navigation software; it augments it, and structured-output reliability remains a practical integration bottleneck [2409.08493]. ## 4. Zero-shot open-vocabulary object-goal navigation A later system, also titled **osmAG-LLM**, applies the representation to **zero-shot, open-vocabulary object-goal navigation**. Its central claim is that for embodied object search, the map should not be treated as a perfect memory of all object instances. Instead, the map should provide **environment grounding and context**, while the LLM supplies semantic priors about likely object locations and online perception verifies the object at runtime [2507.12753]. The underlying osmAG is enriched offline from **RGB-D trajectories** with three forms of semantic content. **Object-nodes** are created from **LabelMaker** outputs and **Mask3D** instance masks; each instance centroid is inserted into osmAG with key `semantic_osmAG:object_name`. **Viewpoint-nodes** are created by prompting a VLM with **“What objects are in this image?”** at selected capture locations, and storing the answer with key `semantic_osmAG:observed_object`. **Room descriptions** are created by prompting a VLM with **“Describe the image.”**, summarizing the resulting descriptions with an LLM, and attaching them to room areas. This combination is explicitly intended to make the map **open-vocabulary**, because viewpoint-nodes are not limited to the label set of the object-mapping pipeline [2507.12753]. At query time, the LLM receives an explanation of semantic-osmAG, the retrieval objective, and a simplified map that contains only **semantic nodes**, **hierarchy**, and **room relationships**, while **excluding coordinates and polygons**. It is constrained to output candidate search targets in **JSON format** with **maximum 3 nodes per room**, **maximum 3 rooms total**, and nodes ordered by **decreasing predicted likelihood**. The search policy then navigates sequentially to the returned nodes using ROS navigation, captures images, runs an open-vocabulary detector—**YOLO-World** or **DINO-X**—and verifies candidate crops with the VLM **StepFun `step-1o-turbo-vision`** using the prompt **“Is there a [object] here?”**. If no detection is confirmed, the robot rotates to capture additional viewpoints and then proceeds to the next candidate node [2507.12753]. Navigation itself remains lightweight. The system converts osmAG to an **occupancy grid** in which area polygons are marked occupied and passages free, plans paths with **A\***, updates the rendered occupancy grid online using **real-time depth**, and replans when collisions are detected. The map is therefore used as **permanent structure**, not as a dense, ever-current object memory [2507.12753]. On **HM3D-SEM**, using **8 scenes** and **5 random starting positions** per scene, the main baseline was **HOV-SG**. For the hardest query type **\((o,r,f)\)**, osmAG-LLM achieved **R-RSR 0.66** versus **0.27**, **APL 12.32 m** versus **20.15 m**, and **AMD 4.65 m** versus **8.85 m**. For **\((o,r)\)**, **R-RSR** improved from **0.32** to **0.67**, **APL** from **16.81 m** to **11.54 m**, and **AMD** from **7.87 m** to **5.27 m**. For **\((o)\)**, **R-RSR** improved from **0.65** to **0.83**, while osmAG-LLM retained far shorter paths. The reported **DIR** values were also higher: for **\((o,r,f)\)**, **0.27** versus **0.05**; for **\((o,r)\)**, **0.25** versus **0.06**; and for **\((o)\)**, **0.31** versus **0.05**. A particularly strong systems-level result is storage: across the 8 scenes, **HOV-SG** required **1493 MB**, whereas semantic-osmAG required **3.2 MB** [2507.12753]. The real-world experiments used a **Fetch** platform with base-mounted **2D LiDAR**, **AMCL**, `move_base`, a head-mounted **RealSense RGB-D camera**, and five rooms ranging from **15 m²** to **147 m²**. Across **30 experiments**, the paper distinguishes **Static objects (SO)**, **Relocated objects (RO)**, and **Unmapped objects (UO)**. For **SO**, **R-RSR** improved from **0.40** to **1.00** and **APL** from **47.00 m** to **18.13 m**. For **RO**, **R-RSR** improved from **0.40** to **1.00**, **O-RSR@3m** from **0.50** to **0.90**, **AMD** from **5.36 m** to **1.63 m**, and **DIR** from **0.00** to **0.70**. For **UO**, **R-RSR** improved from **0.20** to **1.00**, **O-RSR@3m** from **0.50** to **0.80**, **AMD** from **7.42 m** to **2.25 m**, **DIR** from **0.30** to **0.80**, and **APL** from **52.03 m** to **30.68 m**. Even a **sparse semantic-osmAG** retaining only room-level descriptions reported room-retrieval values in parentheses of **0.8–1.0**, which the paper presents as evidence for scalability under token constraints [2507.12753]. The reported failures sharpen the interpretation of the system. There were **9 failures out of 30 trials**: **6** when the robot reached a place where the object was visible but failed to retrieve it, and **3** false positives where the object was not visible. This produced **27/30** correct robot positioning, or **90%**, and locates the principal residual error in **detection**, not navigation. Examples included confusing a ladder reflection as a ladder, confusing a hugging pillow as a neck pillow, and failing to generate a bounding box for a transparent object [2507.12753]. ## 5. Hierarchical MLLM control and reactive halting A related development extends the osmAG-LLM idea from symbolic planning and object search to **long-horizon vision-language navigation** in dynamic settings. **HaltNav** does not use the title osmAG-LLM, but it relies on the same premise that a lightweight, text-based osmAG prior can ground a high-level language model while low-level execution is delegated to a separate navigation policy [2603.12696]. The architecture is hierarchical. A global planner operates on an osmAG-derived passage graph by constructing a **passage-level graph**, computing traversal costs between passages with **A\*** on rendered **2D occupancy submaps**, and storing those costs for reuse. An MLLM-based high-level component called the **Graph-Grounded Task Dispatcher (GGTD)** reads a **structured textual prompt** of the local osmAG subgraph together with the target instruction and trajectory history, and outputs a localized macro-action \[ m_i=\mathrm{GGTD}\!\left(\mathcal{P}(\mathcal{G}_t),\, I_{\mathrm{target}},\, \mathcal{H}_{i-1}\right). \] A low-level VLN policy \(\pi_{\mathrm{low}}\) then executes with egocentric observation \(o_t\) and active macro-action \(m_i\), \[ a_t \sim \pi_{\mathrm{low}}(\cdot \mid o_t, m_i; \theta), \qquad t\in[t_i,\, t_i+\Delta t_i). \] This decomposition keeps global topological reasoning and local control separate [2603.12696]. The principal robustness mechanism is **Reactive Visual Halting (RVH)**, defined by a halting function \[ \beta(o_t,m_i)=\Big[\sum_{j=0}^{k-1} c_{t-j}\ge \tau_c \;\vee\; s_{\mathrm{MLLM}(o_t,m_i)}<\tau_s\Big]. \] RVH halts if either collision accumulation exceeds a threshold or the MLLM’s semantic traversability score falls below threshold. The monitored anomalies include **blocked passages**, **closed doors**, **crowded corridors**, **unmapped obstacles**, and **physical hazards**. When RVH fires, the system updates the passage-level graph rather than merely extending the prompt: \[ C_{t+1}(p_i,p_j)= \begin{cases} \infty, & \text{if visual anomaly is detected},\\ C_t(p_i,p_j), & \text{otherwise}. \end{cases} \] Setting a traversal cost to \(\infty\) effectively prunes the blocked edge and triggers replanning over the updated topology [2603.12696]. Because off-the-shelf MLLMs were found not to be reliably grounded in robot affordances, the paper introduces a dual-engine synthesis pipeline for hard negatives. One engine injects dynamic 3D obstacles at topological bottlenecks inside **Habitat**; the other uses diffusion or inpainting-style generative editing to insert realistic obstacles into traversable real RGB observations. The training pairs are \[ \mathcal{D}=\{(X_p,Y_{\mathrm{no\text{-}halt}}),(X_a,Y_{\mathrm{halt}})\}, \] and the MLLM is fine-tuned with **LoRA** under supervised negative log-likelihood \[ \mathcal{L}_{\mathrm{SFT}}=-\sum_{(X,Y)\in\mathcal{D}}\sum_{j=1}^{|Y|}\log p_{\mathrm{MLLM}}\!\left(y_j \mid y_{