OpenGuide: Indoor Assistive Robot Retrieval

• OpenGuide is a mobile assistive robot framework for indoor multi-object retrieval, designed to aid visually impaired users.
• It integrates natural language understanding, vision-language models, and semantic mapping to detect and locate multiple objects in cluttered environments.
• Utilizing POMDP planning with value decay and frontier exploration, OpenGuide recovers from missed detections and significantly improves retrieval success.

Searching arXiv for the specified OpenGuide paper and closely related name-overlapping works to ground the article. arXiv search query: (Xu et al., 2 Sep 2025) OR OpenGuide OR "Assistive Object Retrieval" OpenGuide is a mobile assistive robot framework for indoor multi-object retrieval for people who are blind or visually impaired. It is designed for settings such as homes and offices, where users may need a robot to find several named objects distributed across multiple rooms and partially hidden in clutter. Rather than treating retrieval as simple point-to-point navigation, OpenGuide combines natural language understanding, vision-language foundation models, frontier-based exploration, semantic value maps, and Partially Observable Markov Decision Process (POMDP) planning to search efficiently under partial observability and to recover from missed detections through value decay and belief-space reasoning (Xu et al., 2 Sep 2025).

1. Problem formulation and scope

OpenGuide addresses indoor multi-object search (MOS). A user issues a natural-language request such as “find the fork, bowl, and spoon” or “locate the refrigerator and the couch,” and the robot must search an unknown environment to locate all $K$ target objects among $L$ objects in the environment (Xu et al., 2 Sep 2025). The system is explicitly motivated by assistive use cases for individuals with visual impairments, for whom repeated object retrieval can be cognitively and physically expensive in cluttered and dynamic built environments.

The problem is difficult for several reasons. Indoor spaces are partially observable; targets may be spread across different rooms; objects can be small, occluded, or view-dependent; and a strong detector may still miss an object even when the robot is nearby. The decision problem is therefore not only where to go next, but whether to exploit a semantically promising region, revisit a previously explored region after a missed detection, or move to a frontier to uncover new parts of the environment (Xu et al., 2 Sep 2025).

The paper frames OpenGuide as a response to the limits of systems that emphasize navigation, obstacle avoidance, or local exploration without goal-directed multi-object retrieval. Its central claim is that robust assistive retrieval requires a joint treatment of open-vocabulary query grounding, multi-object reasoning, uncertainty-aware planning, and recovery from perception failure (Xu et al., 2 Sep 2025).

2. System architecture and semantic perception

OpenGuide has three major components: mapping / semantic perception, belief construction and frontier extraction, and POMDP-based planning and navigation (Xu et al., 2 Sep 2025). Natural-language instructions are interpreted as open-vocabulary object queries. The paper does not describe a separate heavy language parser; instead, object names from the request are used as text prompts that are grounded visually by foundation models.

The perception stack combines several models. BLIP-2 computes cosine similarity between the current RGB image and text prompts for each target object. Grounding DINO is used for open-set object detection via language grounding, YOLOv7 for object detection, and SAM for segmentation of detected objects (Xu et al., 2 Sep 2025). The robot maintains an object map containing all detected and segmented objects over time. If a target object is directly found, the robot can navigate to it. Otherwise, search is guided by a semantic value map and candidate regions derived from it.

A distinctive element is the pixelwise value map generated from BLIP-2 similarity. The paper defines a cone-shaped confidence mask over the camera field of view,

$$c(i, j) = \cos^2\left(\frac{\theta}{\theta_{\text{FOV}/2}} \times \frac{\pi}{2}\right),$$

where $\theta$ is the angle between a pixel and the optical axis, and $\theta_{\text{FOV}}$ is the horizontal field of view (Xu et al., 2 Sep 2025). This gives more weight to pixels near the center of the view. When new observations overlap with older ones, OpenGuide fuses them by confidence-weighted averaging, biasing updates toward more confident detections.

The system also constructs an obstacle map from depth and odometry and a frontier map from the boundary between explored and unexplored space. A frontier is represented as the midpoint of an explored–unexplored boundary and serves as a candidate exploration target. The key architectural point is that OpenGuide does not commit exclusively to frontier following: frontier actions are considered jointly with semantically grounded candidate actions (Xu et al., 2 Sep 2025).

3. Belief-space planning, hybrid action selection, and value decay

OpenGuide formulates multi-object search as a POMDP,

$$\mathcal{P} = \langle \mathcal{S}, \mathcal{A}, \mathcal{O}, T, O, R, \gamma \rangle,$$

with state decomposed into robot and target-object components,

$$s = \{s_r, s_t\}, \quad s_r = (x_r, y_r), \quad s_t = (x_t, y_t),$$

a binary observation space $o \in \{0,1\}$, and motion actions of the form

$a = \text{MoveTo}(g),$

where $g$ is a candidate point or frontier point (Xu et al., 2 Sep 2025).

A major contribution is the way the semantic map is turned into a planning space. After repeated exploration without confirmation, the raw value map is decayed according to

$L$0

where $L$1 is the update count at location $L$2, and $L$3 and $L$4 control the decay rate (Xu et al., 2 Sep 2025). This mechanism prevents the robot from fixating indefinitely on stale high-value regions and enables recovery from missed detections.

After thresholding the decayed value map, OpenGuide applies DBSCAN clustering to identify semantic hotspots, yielding candidate points

$L$5

The action space is then the union of candidate actions and frontier actions, so the robot can either exploit semantic evidence or explore unknown areas (Xu et al., 2 Sep 2025). Belief over target location is represented as a discrete distribution over candidate points, and in simulation the method uses standard Bayesian belief update. For real execution, the paper states that belief is updated using the decayed value map, on the rationale that the value map already provides a strong approximation of target likelihood.

The transition model assumes that the target object is static and the robot state changes deterministically under the selected motion action. The reward is defined as a trade-off among movement cost, frontier exploration, and target proximity (Xu et al., 2 Sep 2025). This structure operationalizes the exploration–exploitation balance: short paths are preferred, frontiers receive exploration reward, and being within detection range yields target reward.

Planning is solved with POUCT (Partially Observable Upper Confidence Trees). At each step, OpenGuide refreshes the value, obstacle, frontier, and object maps; extracts candidate points and a frontier point; constructs the hybrid action set; runs POUCT on a generative model $L$6; and executes the action with the highest estimated $L$7 (Xu et al., 2 Sep 2025). The operational loop stops for an object when the robot calls stop and is within 1 m of the nearest target object; an episode ends when all target objects are found or the maximum step limit of 500 is reached.

4. Experimental protocol and quantitative performance

The simulation experiments use the Habitat Matterport3D (HM3D) validation split with 5 scenes and 120 episodes total, where each episode searches for 2 or 3 objects (Xu et al., 2 Sep 2025). Performance is reported with Success Rate (SR) and MSPL (multi-object success weighted by normalized inverse path length),

$L$8

The baselines are Random Walk, VLFM, and Finder. The reported simulation results are as follows (Xu et al., 2 Sep 2025):

Method	SR	MSPL
Random Walk	0.0%	0.0
VLFM	12.5%	0.075
Finder	28.3%	0.198
OpenGuide	55.0%	0.497

These results show that OpenGuide nearly doubles SR over Finder and more than doubles MSPL over Finder. The paper attributes the weak performance of VLFM to its one-object-at-a-time frontier-based semantic navigation, which is not robust for multi-object retrieval (Xu et al., 2 Sep 2025).

The ablation study isolates the contributions of uncertainty-aware planning and missed-detection recovery:

Method	SR	MSPL
OpenGuide w/o POMDP	27.5%	0.154
OpenGuide w/o value decay	45.0%	0.410
OpenGuide full	55.0%	0.497

The removal of POMDP causes a major drop in performance, while removing value decay also degrades both SR and MSPL (Xu et al., 2 Sep 2025). The reported interpretation is that the planner becomes overly greedy without POMDP reasoning and that, without decay, the robot can get stuck repeatedly exploring regions that retain high scores despite missed detections.

5. Real-world deployment and observed behavior

For real-world experiments, the authors built GuideBot using a Turtlebot 2 base, ROS1 Noetic, a ZED stereo camera, a laptop for processing, and a handlebar for user support (Xu et al., 2 Sep 2025). The evaluation environments are a 470 sq ft studio-like apartment and a 50 $L$9 office environment.

In the home setting, the system is evaluated on 6 multi-object search tasks with average object-search success of about 50% and average steps of 107.6 (Xu et al., 2 Sep 2025). The paper reports stable performance similar to simulation and gives a qualitative example in which the robot finds a toothbrush and sink in a bathroom after first exploring from the living room, using frontier-based replanning until VLM grounding reveals the targets.

In the office setting, the evaluation uses 10 representative instructions, 30 total object instances, and reports 17 retrieved successfully, corresponding to a 56.7% success rate, with average steps of 80.6 (Xu et al., 2 Sep 2025). Performance is described as especially good on strongly related object pairs such as spray bottle plus potted plant. The paper also reports successful recovery of an occluded umbrella after replanning when the environment was fully explored.

Across 48 real-world trials, 46 initiated successfully, 42 reached the search region, and 26 of those 42 succeeded in finding all target objects, yielding an overall end-to-end task success of

$$c(i, j) = \cos^2\left(\frac{\theta}{\theta_{\text{FOV}/2}} \times \frac{\pi}{2}\right),$$0

The failure breakdown is 2 communication failures, 4 navigation failures, and 16 search failures, mostly due to perception/detection errors (11) and stopping too far from the object (5) (Xu et al., 2 Sep 2025). This identifies perception as a major remaining bottleneck.

Qualitatively, OpenGuide is reported to be effective at revisiting previously explored regions if their value remains high, recovering from missed detections, selecting better viewpoints for occluded objects, and combining semantic priors with frontier exploration (Xu et al., 2 Sep 2025). A simulation example contrasts OpenGuide with Finder: Finder moves to a new frontier and never revisits a corner where a toilet is partially occluded, whereas OpenGuide keeps that region under consideration due to its value map and eventually detects the object.

6. Significance, limitations, and naming overlap

OpenGuide is positioned as an advance over earlier assistive and semantic search systems because it combines open-vocabulary language grounding, multi-object reasoning, uncertainty-aware planning, and recovery from perception failure in a single framework (Xu et al., 2 Sep 2025). The paper argues that these ingredients are usually treated separately in prior systems. A plausible implication is that assistive retrieval in realistic indoor settings depends as much on belief management and search policy as on raw detector quality.

The paper identifies several limitations. Perception still struggles with small, occluded, or ambiguous objects; the method depends on current VLM priors; navigation can fail in narrow or cluttered physical spaces; and stopping distance is not always aligned with detection range (Xu et al., 2 Sep 2025). Proposed future directions include trying different vision-language backbones, improving navigation with adaptive obstacle avoidance, and adding dialog-based interaction so users can correct or guide the robot in real time.

The name also overlaps with several distinct systems in the broader literature. GUIDE: Graphical User Interface Data for Execution is a dataset for multimodal LLM-based GUI automation and RPA, not an assistive robot system (Chawla et al., 2024). GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation is a training-free framework for reducing planning and grounding bias in GUI agents through instructional-video retrieval and automated annotation (Xie et al., 27 Mar 2026). LearnWeb-OER: Improving Accessibility of Open Educational Resources addresses discoverability, collaborative sharing, and reuse of educational resources rather than embodied object retrieval (Singh et al., 2015). This suggests that, despite partial overlap in terminology around “guidance” and “accessibility,” OpenGuide in the strict sense denotes the assistive indoor multi-object retrieval system introduced for blind and visually impaired users (Xu et al., 2 Sep 2025).