Semantic Zone-Based Map Management for Stable AI-Integrated Mobile Robots

Abstract: Recent advances in large AI models (VLMs and LLMs) and joint use of the 3D dense maps, enable mobile robots to provide more powerful and interactive services grounded in rich spatial context. However, deploying both heavy AI models and dense maps on edge robots is challenging under strict memory budgets. When the memory budget is exceeded, required keyframes may not be loaded in time, which can degrade the stability of position estimation and interfering model performance. We proposes a semantic zone-based map management approach to stabilize dense-map utilization under memory constraints. We associate keyframes with semantic indoor regions (e.g., rooms and corridors) and keyframe management at the semantic zone level prioritizes spatially relevant map content while respecting memory constraints. This reduces keyframe loading and unloading frequency and memory usage. We evaluate the proposed approach in large-scale simulated indoor environments and on an NVIDIA Jetson Orin Nano under concurrent SLAM-VLM execution. With Qwen3.5:0.8b, the proposed method improves throughput by 3.3 tokens/s and reduces latency by 21.7% relative to a geometric map-management strategy. Furthermore, while the geometric strategy suffers from out-of-memory failures and stalled execution under memory pressure, the proposed method eliminates both issues, preserving localization stability and enabling robust VLM operation. These results demonstrate that the proposed approach enables efficient dense map utilization for memory constrained, AI-integrated mobile robots. Code is available at: https://github.com/huichangs/rtabmap/tree/segment

• The paper introduces a semantic zone-based keyframe management framework that aligns memory allocation with spatial context for robust SLAM performance.
• It employs an LRU-based, zone-level eviction mechanism to prevent out-of-memory events during simultaneous SLAM and VLM operations on edge devices.
• Empirical results demonstrate reduced localization errors and improved throughput and latency, supporting continuous AI integration on mobile robots.

Semantic Zone-Based Memory Management for AI-Integrated Mobile Robot Systems

Introduction and Motivation

This paper presents a novel approach for memory management in AI-integrated mobile robotics by leveraging semantic partitioning of dense 3D maps. Advances in edge-resident Vision-LLMs (VLMs) and LLMs enable contextually rich, spatially grounded robot services, yet the confluence of heavy model workloads and detailed mapping imposes severe pressure on the limited memory typical of edge platforms. When the runtime memory budget is exceeded, timely keyframe retrieval suffers, causing failures in position estimation and feedback loops essential for reliable robot autonomy.

The core contribution is a semantic zone-based map management framework in which map keyframes are associated with human-meaningful spatial partitions (e.g., rooms, corridors). By organizing keyframe prefetch, eviction, and memory budgeting at this zone level, the method harmonizes map resource allocation with robot motion and anticipated spatial context, improving predictability and overall system stability.

Semantic Zone-Based Keyframe Management Framework

Conventional map management in SLAM systems, such as RTAB-Map, utilizes geometric or recency cues for keyframe caching under memory constraints, but this can result in high churn, delayed keyframe access, and ultimately localization instability when also co-executing large AI models. The proposed approach introduces two key architectural mechanisms:

1. Semantic Association: Each dense map keyframe is labeled with a semantic zone membership derived from a priori spatial partitioning (e.g., by room/corridor). This grouping aligns memory operations with the way robots utilize space functionally.
2. Zone-Level Memory Regulation: At runtime, a Semantic Zone Manager monitors the robot’s estimated pose to determine the currently relevant semantic zone and accordingly activates or evicts batches of zone-associated keyframes as memory permits. Before activation, the system predicts whether adding the requested zone would exceed the fixed memory threshold (MemoryThr). If necessary, it applies a batch eviction using a least-recently-used (LRU) policy at the zone level until the threshold is respected, and only then loads new zone keyframes.

This architecture, illustrated in Figure 1, enables strict compliance with a fixed memory budget and reduces the frequency of load/unload operations.

Figure 1: Schematic of semantic zone-based keyframe management with memory threshold enforcement via zone-level active/unload lists.

A key aspect is that semantic zones are spatially predictive: robot motion is usually continuous and explores connected functional regions, so the system can efficiently prefetch or retain only the semantically relevant submaps, minimizing redundant memory pressure.

Figure 2: Map of the experimental environment segmented into semantic zones, overlaid with the robot’s planned route—providing context for zone-based prediction and resource allocation.

Implementation on Edge Hardware

To evaluate practical implications, the approach is integrated in RTAB-Map and deployed on an NVIDIA Jetson Orin Nano (8GB unified memory) with concurrent execution of VLMs (Gemma3, Qwen3, Qwen3.5). The mapping environment is a large-scale indoor scenario in Nvidia Isaac Sim, where semantic zones are manually specified to match rooms and corridors. Keyframe-to-zone associations are constructed offline; online, the semantic manager governs all memory operations, while pose estimation and loop-closure processing remain as in the original SLAM pipeline.

Empirical Evaluation: Stability and AI-Inference Metrics

Extensive experiments are performed to assess system stability under concurrent SLAM and VLM operation, focusing on both mapping quality and AI inference metrics.

• Baseline (“Basic”) refers to geometric-retrieval-based keyframe management.
• Proposed (“Semantic”) denotes semantic zone-based keyframe management.

Memory Usage and Robustness

The basic strategy repeatedly reaches memory saturation, causing out-of-memory events and process stalls in the presence of a VLM workload (Figure 3). Conversely, semantic zone-based management maintains memory within limits, prevents OOM events, and ensures uninterrupted operation.

Figure 3: Memory usage trace for the geometric baseline (Basic + VLM), exhibiting out-of-memory events and unstable execution.

SLAM Trajectory and Loop Closure Quality

Trajectory estimation under the basic approach degrades severely during concurrent VLM execution: loop closure opportunities drop, processing time increases more than sixfold, and trajectory errors (ATE, RPE) increase by over 100%. Meanwhile, the semantic approach sustains near-baseline update frequency and preserves loop closure integrity (the degradation in ATE is reduced from 128% to 6% when adding VLM workload). In the geometric baseline, loop closures cease after a point due to keyframe starvation; semantic management ensures continued closures throughout the trajectory (Figure 4).

Figure 4: Comparison of estimated trajectories. Semantic map management prevents the stall in loop closure seen with baseline under VLM contention.

VLM Throughput and Latency

Token generation throughput and end-to-end processing latency are evaluated for several model backends. For the Qwen3.5:0.8b model, enabling semantic zone-based map management yields a throughput improvement of 3.3 tokens/s and a latency reduction of 21.7% relative to the geometric baseline. Furthermore, distributed VLM generation metrics (PDF/CDF) indicate that semantic management exerts more predictable performance, free from the severe latency spikes—and ultimately, the frozen execution—characteristic of geometric map management.

Figure 5: Probability distribution function for token generation speed, comparing the variability and peak throughput of VLM under different map-management policies.

Figure 6: Spatial correlation of token generation throughput with the robot’s position, evidencing stable VLM operation with semantic management.

Figure 7: Token generation latency as a function of robot position; latency spikes, common in the baseline, are eliminated with the semantic approach.

Analysis and Implications

The empirical results affirm that semantic zone-based map management robustly addresses the practical limitations encountered in edge-AI-powered mobile robots. It synchronizes map memory footprint with semantically and operationally relevant environmental context, thus cushioning the impact of unpredictable VLM memory demands on SLAM performance. This approach fundamentally shifts the focus from indiscriminately maximizing geometric map content in memory to maximizing the operational and spatial relevance of map content, promoting safe and stable robot autonomy.

Crucially, the method eliminates out-of-memory failures and execution stalls that undermine both localization and AI inference—empowering continuous, high-quality service delivery on fixed-memory hardware.

Several tunable aspects and limitations are identified:

• Semantic zone definitions are currently manual; learned or adaptive segmentation, possibly driven by end-task reward or data distribution, could further optimize relevance and minimize manual configuration.
• The LRU-based zone eviction is not task-aware; integrating anticipatory or task-relevant zone selection is a promising direction.
• No online map update is considered; future extensions to dynamic or open-world environments are warranted.

Broader Implications and Future Directions

By enforcing memory compliance at the semantic level and aligning map utilization with robot behavior, this framework opens new avenues for deploying dense-mapping and foundation-model workflows on edge robots, particularly in service, logistics, and assistive domains where edge resources are fixed and remote management is infeasible. The principles may generalize to balancing any confluence of perception-heavy SLAM and AI model workloads, supporting advances in life-long learning, semantic loop closure, and real-time collaborative robotics.

Further research may incorporate adaptive zone granularity, integrate predictive motion planning, explore hierarchical semantic associations, and enable lifelong, online zone refinement. Extensions to GPU memory management, map-compression co-design, and cross-robot map sharing could further solidify the foundation for fully autonomous, memory-efficient, AI-integrated robot systems.

Conclusion

This work demonstrates that semantic segmentation of dense maps, combined with zone-level batch management, decisively mitigates the deleterious effects of memory contention endemic in edge-AI mobile robot platforms. The proposed method delivers robust localization, stable VLM throughput and latency, and, critically, prevents catastrophic failures under concurrent workload surges. These advances significantly lower the barrier to practical field deployment of spatially aware robots integrating large-scale vision-language reasoning on resource-constrained hardware (2603.29627).