WalkGPT: Multimodal Pedestrian Navigation

• WalkGPT is a specialized large vision–language model that integrates visual, depth, and linguistic cues for robust pedestrian navigation and accessibility.
• It employs innovative modules like MSQP and CTP to fuse image segmentation with depth-aware reasoning, enabling interpretable natural language guidance.
• Evaluated on the PAVE benchmark, WalkGPT achieves competitive segmentation, depth estimation, and navigation performance in real-world scenarios for both robotics and BLV users.

WalkGPT is a class of large vision–LLMs (LVLMs) specialized for pedestrian navigation and accessibility, synthesizing state-of-the-art multimodal understanding, pixel-level grounding, and conversational reasoning. WalkGPT architectures unify image, depth, and linguistic information to produce navigation guidance that is interpretable, depth-aware, and robust for real-world human or robotic pedestrian navigation. WalkGPT emerged in response to the limitations of earlier vision-LLMs, which lacked explicit spatial grounding and depth reasoning, and has found application in general navigation, behavior prediction, and accessibility assistance for blind and low-vision (BLV) users (Sultan et al., 11 Mar 2026, Hanlin, 2024, Huang et al., 2023, Hwang et al., 2024).

1. WalkGPT Architecture and Multimodal Fusion

WalkGPT’s architecture is grounded in the integration of a frozen pixel-level encoder—typically based on foundation segmentation models such as SAM (Segment Anything Model)—with a LLM via specialized projector and fusion mechanisms (Sultan et al., 11 Mar 2026). The multimodal input comprises a pedestrian-view RGB image $I$ that is mapped to visual patch embeddings $Z \in \mathbb{R}^{B \times L \times C}$, where $B$ is batch size, $L$ the number of spatial tokens, and $C$ the feature dimension.

A Multi-Scale Query Projector (MSQP) projects $Z$ into a condensed set of image tokens $$V_{\text{proj}} \in \mathbb{R}^{B \times Q \times H}$$ by hierarchical cross-attention pooling at multiple spatial resolutions. These tokens, concatenated with text tokens encoding system prompts and user queries (dimension $H$), are fused via transformer cross-attention in the LLM. The LLM outputs, in an autoregressive sequence, include natural-language reasoning, segmentation prompts (via <SEG> tokens), distance estimates (<distance>…</distance>), and explicit phrase grounding (<p>…</p>).

The Calibrated Text Projector (CTP) maps each <SEG> token’s hidden state into the SAM pixel-decoder space, producing binary segmentation masks. To enforce correspondence between linguistic tokens and visual regions, WalkGPT applies a Region Alignment (InfoNCE) loss that contrastively aligns text-derived region embeddings with their attended visual regions.

2. Grounding, Segmentation, and Depth-Aware Reasoning

Central to WalkGPT is fine-grained pixel grounding of navigation cues. When prompted, the model generates segmentation tokens that explicitly delineate accessible or hazardous features in the environment. Each <SEG> token is projected into the SAM decoder space, producing a binary mask corresponding to a linguistic entity (e.g., “crosswalk,” “stair,” “parked car”).

Depth-aware reasoning is achieved through explicit generation of <distance>…</distance> fields, in which the model estimates the distance to each referenced object based on underlying dense depth maps (from stereo or ZED sensors), discretized and communicated via natural language. The grounding and depth estimation are learned jointly with navigation-oriented language objectives ($L_{CE}$), segmentation losses (binary cross-entropy plus Dice, $L_{seg}$), and the region alignment loss ($Z \in \mathbb{R}^{B \times L \times C}$0), with the total loss:

$Z \in \mathbb{R}^{B \times L \times C}$1

where implementation weights are $Z \in \mathbb{R}^{B \times L \times C}$2, $Z \in \mathbb{R}^{B \times L \times C}$3 (BCE) $Z \in \mathbb{R}^{B \times L \times C}$4 (Dice), $Z \in \mathbb{R}^{B \times L \times C}$5 (Sultan et al., 11 Mar 2026).

3. Data, Evaluation Benchmarks, and Metrics

The PAVE benchmark underpins WalkGPT quantitative evaluation (Sultan et al., 11 Mar 2026). PAVE contains 41,000 pedestrian-view images sampled from the real-image subset of SANPO, each paired with an accessibility-aware question and a depth-grounded reference answer generated by GPT-5-nano and partially verified by human annotators. Each sample includes:

• RGB frame (448×448)
• Panoptic/semantic masks (30 classes)
• Dense depth map
• Object-wise accessibility labels

Standard metrics include CIDEr and METEOR for answer generation; AP$Z \in \mathbb{R}^{B \times L \times C}$6, mean IoU, and recall for segmentation; depth accuracy (fraction of predictions within [0.5$Z \in \mathbb{R}^{B \times L \times C}$7, 2$Z \in \mathbb{R}^{B \times L \times C}$8] of ground truth) and absolute relative error (AbsRel) for distance predictions; as well as CHAIR$Z \in \mathbb{R}^{B \times L \times C}$9 (hallucination rate) and coverage (proportion of visible objects mentioned).

On PAVE, WalkGPT (13B) achieves:

• CIDEr 41.17, METEOR 43.01
• mIoU 20.16
• Depth accuracy 48.95%
• AbsRel 70.66%

Zero-shot LVLMs not tuned for pixel-level grounding or depth fail to achieve nontrivial scores on these metrics (Sultan et al., 11 Mar 2026).

4. Vision-and-Language Navigation: Sequence Modeling and Policy Learning

In navigation contexts, WalkGPT draws from the Vision-and-Language Navigation Generative Pretrained Transformer (VLN-GPT) paradigm (Hanlin, 2024), which adopts a GPT-2 decoder-only architecture for trajectory modeling. The model directly consumes the entire navigation history as concatenated sequences of return, state (fused language and vision), and action tokens, eliminating the need for explicit RNN or transformer-encoder history modules.

Language instructions are first embedded using a pre-trained Sentence-BERT, while each panoramic view is encoded by a Vision Transformer (ViT). The state embedding is computed as $B$0 (element-wise multiplication), producing one state vector per candidate viewpoint. At each step, the transformer predicts the next action, conditioned on the whole trajectory prefix.

Training proceeds in two stages:

1. Offline imitation learning with sequence action prediction (SAP) loss, minimizing $B$1 over expert demonstrations.
2. Online fine-tuning with reinforcement learning, using a reward $B$2 and proximal policy optimization (PPO).

On the Room-to-Room (R2R) benchmark, WalkGPT derivatives report:

• Validation seen: TL=11.18 m, NE=2.55 m, SR=76.48%, SPL=72.18%
• Validation unseen: TL=11.51 m, NE=3.75 m, SR=65.14%, SPL=61.09%

The decoder-only approach yields lower memory usage and faster inference relative to encoder-decoder designs (Hanlin, 2024).

5. Pedestrian Behavior Prediction and Safety-Aware Assistance

WalkGPT’s capabilities extend to dynamic prediction of pedestrian intent, crossing actions, and risk assessment. By leveraging vision–LLMs such as GPT-4V for joint visual-linguistic reasoning, WalkGPT systems encode fine-grained cues—posture, gaze, bounding-box geometry—along with context-aware inference about group behavior and intention (Huang et al., 2023).

Quantitative evaluations show that GPT-4V achieves 57.0% accuracy zero-shot for pedestrian crossing prediction on the JAAD dataset, compared to ~70% for state-of-the-art, domain-specific models (PIT-Block, Pedestrian Graph+). Precision is high (82.8%), but recall lags (54.7%), with an F1 of 65.9%—performance limited by lack of domain-specific fine-tuning, failure modes on small bounding boxes, and challenges in separating ego-motion from pedestrian motion. Proposed solutions include targeted cross-attention fine-tuning, data augmentation for scale/lighting/motion, and explicit fusion of odometry or LiDAR modalities.

In assistive settings for BLV users, WalkGPT-based pipelines (as in (Hwang et al., 2024)) perform risk assessment by fusing multiview images, YOLOv8-detected objects and masks, optical-flow cues, and structured prompts to yield discrete safety scores $B$3, grounded in observable factors (approaching vehicles, light state, pedestrian signals). The output includes both quantified risk and a chain-of-thought natural-language rationale.

6. Practical System Deployment and Accessibility Implications

Full-system WalkGPT deployments operate on robotic platforms or edge devices, employing multiview cameras for spatial context. Preprocessing modules extract object bounding boxes, optical flow, and segmentation masks, relayed as overlaid images to the LVLM for inference. Inference latencies are typically ~800 ms; preprocessing accounts for an additional ~150 ms per scene (Hwang et al., 2024).

User interfaces are designed for robustness and multimodality, supporting audio (bone-conduction headset) and haptic (vibrotactile belt) feedback, with options for brevity or detailed scene description. Edge caching and lightweight vision-only fallbacks mitigate issues with cloud latency or connectivity. User feedback is integrated via a correction loop, enabling periodic model retraining for increased reliability over time.

WalkGPT’s approach to segmentation, grounding, and depth estimation provides unprecedented interpretability and personalization in pedestrian navigation assistance, enabling safe, context-sensitive mobility for BLV users while supporting general pedestrian intelligence needs in robotics and urban autonomy (Sultan et al., 11 Mar 2026, Hwang et al., 2024, Huang et al., 2023, Hanlin, 2024).

7. Research Impact and Advancement

WalkGPT establishes a technical foundation for grounded LVLMs in pedestrian-centric tasks, demonstrating that pixel-level segmentation and depth-aware reasoning can be jointly achieved within a conversational vision-language interface. The design of MSQP and CTP modules, along with the Region Alignment Loss, is empirically validated by improved segmentation, depth accuracy, and referring expression segmentation relative to previous LVLMs. The public release of the PAVE benchmark provides a first large-scale dataset for accessibility-aware navigation, advancing the state-of-the-art in both research methodology and practical deployment for accessible urban navigation (Sultan et al., 11 Mar 2026).