WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigation

Abstract: Ensuring accessible pedestrian navigation requires reasoning about both semantic and spatial aspects of complex urban scenes, a challenge that existing Large Vision-LLMs (LVLMs) struggle to meet. Although these models can describe visual content, their lack of explicit grounding leads to object hallucinations and unreliable depth reasoning, limiting their usefulness for accessibility guidance. We introduce WalkGPT, a pixel-grounded LVLM for the new task of Grounded Navigation Guide, unifying language reasoning and segmentation within a single architecture for depth-aware accessibility guidance. Given a pedestrian-view image and a navigation query, WalkGPT generates a conversational response with segmentation masks that delineate accessible and harmful features, along with relative depth estimation. The model incorporates a Multi-Scale Query Projector (MSQP) that shapes the final image tokens by aggregating them along text tokens across spatial hierarchies, and a Calibrated Text Projector (CTP), guided by a proposed Region Alignment Loss, that maps language embeddings into segmentation-aware representations. These components enable fine-grained grounding and depth inference without user-provided cues or anchor points, allowing the model to generate complete and realistic navigation guidance. We also introduce PAVE, a large-scale benchmark of 41k pedestrian-view images paired with accessibility-aware questions and depth-grounded answers. Experiments show that WalkGPT achieves strong grounded reasoning and segmentation performance. The source code and dataset are available on the \href{https://sites.google.com/view/walkgpt-26/home}{project website}.

• The paper introduces a novel model that grounds language on segmentation masks enriched with depth information for safe pedestrian mobility.
• The paper employs a unified architecture with Multi-Scale Query and Calibrated Text Projectors to align visual tokens and language cues for precise spatial reasoning.
• The paper demonstrates state-of-the-art performance on the PAVE dataset while addressing challenges like reflective surfaces and blurred imagery in real-world scenes.

Grounded Vision-Language Conversation for Pedestrian Navigation

Introduction

Pedestrian navigation in complex urban environments demands systems that can effectively reason about both semantic and spatial features. WalkGPT is introduced to address this demand by providing depth-aware accessibility guidance through grounded language reasoning and segmentation. Unlike prior LVLMs that often hallucinate absent objects or require user-provided cues, WalkGPT grounds language on segmentation masks enriched with depth information, offering holistic spatial understanding crucial for safe pedestrian mobility.

Figure 1: Overview of WalkGPT for accessibility-aware grounded navigation guide. The model grounds language on segmentation masks enriched with depth information, providing holistic spatial understanding that captures both object shapes and depth cues for interpretable accessibility analysis.

Methods

WalkGPT Architecture

WalkGPT leverages a unified architecture incorporating a Multi-Scale Query Projector (MSQP) and a Calibrated Text Projector (CTP). MSQP is tasked with aggregating visual features at various scales to form semantically aligned image tokens, facilitating fine-grained spatial reasoning. Meanwhile, CTP, guided by Region Alignment Loss, links segmentation-aware representations with language embeddings, thus providing text-to-visual space mapping without direct user inputs or guidance.

Figure 2: Overview of WalkGPT for grounded navigation guidance. (a) Overall framework. (b) The Multi-Scale Query Projector (MSQP), which aggregates multi-level visual features into spatially aligned image tokens for language reasoning. (c) The Calibrated Text Projector (CTP), guided by the proposed Region Alignment Loss, maps <SEG> tokens into the visual space. Structured tokens (<SEG>, <distance>, <assessment>, <p>) link language generation with segmentation and depth reasoning.

PAVE Dataset

A significant addition is PAVE, a large-scale VQA dataset comprising 41k pedestrian-view images with accessibility-aware questions and depth-grounded answers. Built from the SANPO dataset, it enables WalkGPT to operate successfully in real-world environments, providing robust training inputs for the model to learn depth-aware guided navigation.

Figure 3: Pipeline for generating accessibility-aware VQA pairs in the PAVE dataset. The LLM receives the system prompt, detected features, their distance values, and the accessibility of the features, and generates structured outputs containing <assessment>, <distance>, <SEG>, and <p> tokens.

Experiments

WalkGPT exhibits substantial improvements over previous models in grounded reasoning and segmentation performance. It achieves state-of-the-art results by outperforming benchmarks on accessibility-aware guidance, demonstrating the effectiveness of its architectural innovations.

Grounded Navigation Performance

WalkGPT's conversational outputs are grounded both semantically and spatially, setting a benchmark in accessibility-aware AI applications for pedestrian navigation. The model surpasses existing LVLMs in text generation, segmentation, and depth estimation tasks. It shows notable superiority in recognition of accessibility features across diverse urban scenes.

Figure 4: Qualitative results of WalkGPT on the PAVE validation set. Given a scene image, WalkGPT generates grounded conversations together with segmentation masks and depth-aware distance estimates, reflecting its understanding of accessibility and spatial context. Additional examples are provided in the Appendix.

Challenges and Failure Analysis

Despite its advancements, WalkGPT faces failures in scenarios characterized by reflections or blurred imagery. Such issues highlight a need for improved depth estimation mechanisms and possibly advanced object recognition capabilities to accurately interpret ambiguities inherent in urban navigation scenes.

Figure 5: Failure case study on PAVE. WalkGPT misinterprets strong road reflections on the building facade as physical obstacles, producing incorrect guidance even though the path itself is fully accessible.

Conclusion

WalkGPT marks a significant advancement in grounded vision-LLMs by reframing pedestrian navigation as a pixel-grounded task. By integrating language reasoning with segmentation and spatial cue integration, the model lays the groundwork for future research that may explore more nuanced spatial interpretation and improved depth reasoning across multimodal AI applications.

Future Work

Future directions could include enhancing depth estimation accuracy, exploring cross-domain generalization capabilities, and expanding to additional navigation datasets to further validate WalkGPT’s utility in diverse real-world scenarios.

Figure 6: Additional qualitative results of WalkGPT on the PAVE validation set for off-road scenes. Examples illustrate the model’s ability to handle unstructured outdoor environments with uneven terrain, dense vegetation, and limited walkable surfaces.