Visual Prompt Based Reasoning for Offroad Mapping using Multimodal LLMs

Abstract: Traditional approaches to off-road autonomy rely on separate models for terrain classification, height estimation, and quantifying slip or slope conditions. Utilizing several models requires training each component separately, having task specific datasets, and fine-tuning. In this work, we present a zero-shot approach leveraging SAM2 for environment segmentation and a vision-LLM (VLM) to reason about drivable areas. Our approach involves passing to the VLM both the original image and the segmented image annotated with numeric labels for each mask. The VLM is then prompted to identify which regions, represented by these numeric labels, are drivable. Combined with planning and control modules, this unified framework eliminates the need for explicit terrain-specific models and relies instead on the inherent reasoning capabilities of the VLM. Our approach surpasses state-of-the-art trainable models on high resolution segmentation datasets and enables full stack navigation in our Isaac Sim offroad environment.

• The paper introduces a zero-shot, prompt-based framework for offroad mapping by leveraging SAM2 segmentation and vision-language models.
• It integrates simulation, segmentation, and hierarchical planning to achieve real-time path planning and control in complex terrains.
• Quantitative tests show competitive performance with traditional methods, though sensitivity to visual cue quality remains.

Visual Prompt-Based Reasoning for Offroad Mapping with Multimodal LLMs

Introduction

The paper "Visual Prompt Based Reasoning for Offroad Mapping using Multimodal LLMs" (2604.04564) addresses the inherent challenges of visual scene understanding in off-road autonomous navigation. Traditional pipelines rely on task-specific perception modules for terrain classification, height inference, and slip estimation; these demand significant dataset coverage and dedicated model training for each task, leading to scalability and generalization issues in heterogeneous terrains. This work presents a unified, zero-shot methodology that leverages advances in foundational image segmentation (SAM2) and vision-language reasoning (VLMs), integrating these with path planning and control systems to deliver a lightweight, end-to-end navigation solution in both simulated and real off-road domains.

Figure 1: The proposed off-road autonomy pipeline combines simulated perception, segmentation, VLM reasoning, and planning/control modules for zero-shot mapping and autonomous navigation.

System Framework and Methodology

The proposed off-road autonomy stack comprises simulation, perceptual segmentation, VLM-centric reasoning, and navigation/planning subsystems. High-fidelity simulation environments are constructed in Unreal Engine and imported into NVIDIA Isaac Sim, enabling reproducible, ROS2-compatible data capture and interactive vehicle control for data collection, system prototyping, and evaluation.

Segmentation employs SAM2, seeded with a point-prompt grid optimized for mask generation efficiency. Masks are iteratively filtered by IoU and area thresholds; retained masks are overlaid on the RGB input and indexed centrally for traceability. The memory-augmented mask tracking in SAM2 reduces the frequency of VLM queries by maintaining temporal consistency of drivable area predictions, minimizing computational and walltime overhead.

Figure 2: Simulated off-road environment in Unreal/Isaac Sim, vehicle asset integration, and annotated goal locations for reachability assessment.

Prompt-based reasoning exploits the VLM's compositional understanding of image-text relations. The VLM receives a visual input (composite original + annotated segmentation) and a natural-language instruction specifying traversable terrain types, outputting numeric indices corresponding to mask IDs judged "drivable" under the provided ontology. This approach leverages the generalization capacity and context integration of large-scale multimodal models without explicit task retraining.

Planning and control is realized via a hierarchical stack: the global planner utilizes D* Lite for incremental dynamic replanning on a voxelized occupancy grid; the local planner employs Hybrid A* for kinematically feasible, smoothly connected path segments. Low-level vehicle actuation is managed by the Stanley lateral controller and a PID for longitudinal velocity regulation, enforcing model-compliant steering and acceleration constraints.

Figure 3: Pre-processing UI for annotating and refining ground-truth mask selection used for VLM benchmarking.

Experimental Design, Datasets, and Evaluation Protocol

A synthetic dataset was constructed with over 2,900 annotated frames from the high-fidelity simulator, leveraging a ZED camera for image acquisition and interactive UI tools for precise mask-to-ground-truth mapping. Additional real-world benchmarking employed three standard off-road datasets: ORFD, RUGD, and O2DTD.

Vision-LLM candidates include ChatGPT-4o, Aquila-VL-2B, Ivy-VL (3B), and MiniCPM-V (8B), selected for their efficiency and capability to operate within the system's real-time constraints. Prompt engineering covered visual prompt format (standalone annotated vs. side-by-side collage), output cardinality control (single/multi-mask index), and linguistic specificity (specific, general, full context).

Figure 4: Qualitative comparison of SAM2 automatic mask generation versus point-prompted approach—point prompting yields lower latency and improved suitability for real-time.

A formal scoring protocol based on intersection over union, as well as mask accuracy under different prompt settings, facilitated in-depth analysis of model performance relative to ground truth. The system's full stack—including perception, reasoning, planning, and control—was validated in closed-loop navigation tasks (goal reachability, trajectory tracking) with both quantitative and qualitative metrics.

Figure 5: Comparison of drivable area predictions on sample frames by different VLMs (ChatGPT-4o, Aquila, Ivy-VL, MiniCPM) against annotated ground truth (GT).

Figure 6: Illustration of the scoring logic for mask index selection: alignment with GT yields perfect score; inclusion of non-drivable areas or selection of irrelevant masks reduces score.

Results and Analysis

Segmentation and Masking

Iterative point prompting outperformed SAM2's default mask generator in inference speed (3.5s vs. 12–13s per frame), an essential property for real-time deployment, without compromising spatial coherence.

VLM Prompt Sensitivity

Restricting VLM output to a single mask index (SNP) significantly improved accuracy, especially in resource-constrained models, contrasting with the flexibility of the multi-index setting (MNP). For instance, smaller VLMs like Aquila-VL-2B responded best to specific, highly constrained prompts, while ChatGPT-4o maximized accuracy under full context instructions with collage input.

Collage visual prompts consistently enhanced performance by providing richer contextual alignment, with annotated format offering further gains depending on the linguistic structure.

Quantitative results: ChatGPT-4o achieved the highest average qualitative score (3.47/5), followed by MiniCPM-V-2_6-int4 (2.46/5). SNPs led to a mean score increase of +1.7 for Aquila-VL-2B, underlining the value of prompt discipline for small models.

Quantitative Performance

On the synthetic simulation set, GPT-5-Mini and ChatGPT-4o led in mIoU (0.727 and 0.557 respectively), with the trade-off of considerable inference time (11.86s and 3.98s per sample). Among locally executable models, Aquila-VL-2B delivered competitive IoU (0.515) with low latency (0.59s).

Real off-road benchmarks indicate that the proposed pipeline is competitive with state-of-the-art closed-set segmentation architectures, with best results on high-resolution, visually enriched datasets:

• ORFD: 0.9141 IoU (Ours) vs. 0.7929 (PathFormer)
• RUGD: 0.8059 (Ours) vs. 0.6447 (PathFormer)
• O2DTD: 0.3516 (Ours) vs. 0.8966 (PathFormer)

Performance degradation on datasets with ambiguous, low-resolution, or highly textureless terrain images highlights the sensitivity of VLM-driven segmentation to the richness of underlying visual cues.

Implications and Future Directions

Practically, the zero-shot, VLM-driven framework eliminates the need for hand-annotated, task-specific data and model re-training, promising rapid pipeline construction and adaptation to unseen environments for off-road navigation. Nevertheless, two limitations remain salient: occasional non-determinism in generative model output (resulting in inconsistent mask selection across trials) and sensitivity to prompt engineering and dataset image characteristics.

For theoretical research, this work demonstrates that vision-language alignment in current MLLMs is sufficient for semantic scene parsing and high-level reasoning in exceptionally open-ended navigation domains. However, coverage gaps in VLM visual-text data and the absence of explicit semantic hierarchy awareness restrict transference to challenging off-road contexts with weak visual cues.

Anticipated advances include: parameter tuning of VLMs (e.g., temperature, top-p constraints) for output consistency, fine-tuning on rare terrain modalities, model distillation for real-time embedded use, and the integration of multi-scale encoding (as in PathFormer or Mask2Former-type architectures) to improve robustness against visually ambiguous or low-resolution scenes.

Conclusion

This work substantiates the viability of prompt-driven, zero-shot navigation frameworks integrating cutting-edge segmentation and VLM-based reasoning in off-road contexts. Strong empirical results and practical system integration suggest immediate utility for autonomous vehicles operating in unstructured terrain, with clear pathways for extension to dynamic, multi-modal real-world settings and improved semantic scene understanding.