Spatial-Aware Visual Prompting

• The paper presents novel spatial-aware visual prompting methodologies that incorporate explicit spatial cues to significantly enhance VLMs' spatial reasoning across diverse tasks.
• Spatial-aware visual prompting is defined as techniques that insert spatial structure into visual inputs, improving tasks like grounding, robotics, and 3D understanding.
• The approach integrates both explicit overlays and learnable tokens, achieving performance boosts such as up to +12.5% accuracy in visual grounding and robust real-world task success.

Spatial-aware visual prompting refers to a diverse set of methodologies that explicitly manipulate spatial structure and localization within visual prompts to enhance the spatial reasoning, grounding, and control capabilities of vision-LLMs (VLMs) and multimodal LLMs (MLLMs). By embedding spatial cues—whether through engineered overlays, learned spatial tokens, or structured input sequences—these approaches address long-standing limitations of purely sequential or language-based prompting in tasks requiring precise localization, 3D understanding, or action selection. Spatial-aware prompt designs push VLMs beyond standard classification or generic question answering, enabling robust performance on visual grounding, robotics, spatial VQA, spatial relation inference, and fine-grained recognition tasks across both natural and domain-specific contexts.

1. Core Principles and Motivation

Spatial-aware visual prompting encompasses techniques that insert explicit spatial structure, cues, or priors into the visual or multimodal input stream. Distinct from generic visual prompting (e.g., prepended tokens or attention hints), spatially-aware methods directly encode position, region, relation, or motion, aligning the model's processing with the underlying 2D/3D geometry or temporal evolution of the input. Major motivating problems include:

• Overcoming the lack of explicit spatial coordinates in text queries or plain images, as in MLLMs' inability to align instructions like “pick up the object to the left of the red cup” with precise pixel regions (Tang et al., 19 Mar 2025).
• Closing the semantic gap in robotic control, where language or abstract intent must map to 2D/3D positions, actions, or trajectories (Nasiriany et al., 2024, Zheng et al., 2024, Wang et al., 23 Mar 2026).
• Enabling efficient, parameter-efficient model adaptation for fine-grained spatial tasks, where generic sequential prompts are insufficient (Pei et al., 2023, Rezaei et al., 2024).
• Reducing spatial relation hallucination and enforcing logical or physical consistency among predicted spatial relationships (Wu et al., 12 Feb 2025).
• Incorporating external knowledge from segmentation, OCR, or domain ontologies in a fine-grained spatial manner, rather than relying on coarse or text-only augmentation (Lin et al., 2024, Gao et al., 2 Apr 2026).

Spatial-aware prompting is essential for domains where task success depends on the ability to localize, manipulate, or reason about precise spatial arrangements, such as robotics, remote sensing, video understanding, and medical imaging.

2. Mechanisms and Variants of Spatial-Aware Visual Prompting

Spatial-aware visual prompting designs span a spectrum from explicit, manually constructed overlays to learned, adaptive spatial embeddings. Key families and representative strategies include:

2.1. Explicit Visual Overlays

• 2D graphics overlays: Directly drawing bounding boxes, crosshairs, circles, arrows, or trajectories on input images to indicate regions, proposals, or waypoints. This enables VLMs to refer to spatial proposals in a language-interpretable way (Nasiriany et al., 2024, Wang et al., 23 Mar 2026, Zheng et al., 2024).
• Numbered and annotated proposals: Overlaying candidate points, bounding boxes, or regions with numeric or textual labels, which the VLM can reference in natural language answers or selection (Nasiriany et al., 2024, Zhang et al., 2024).
• Spatial prompt maps: Constructing pixel-wise maps where each region is filled with a class, segmentation, or OCR embedding, serving as a “fine-grained knowledge prompt” fused into the visual encoding (Lin et al., 2024).
• Motion trace overlays: Visualizing the history of keypoints or object trajectories as colored polylines on the image to encode short-term or long-term motion cues for spatial-temporal tasks (Zheng et al., 2024).

2.2. Learnable Spatial Prompts

• Spatially-aligned token maps: Learning 2D prompt maps (of dimension H×W×d or similar) aligned with an image token map, enabling per-location prompting and bilateral cross-attention in transformer backbones (Pei et al., 2023).
• Global-local position encoding: Augmenting the image input with coordinate axes (global), or region-level position-aware queries (local), e.g., via learnable axis tensors, DETR-style object queries (Tang et al., 19 Mar 2025).
• Auxiliary spatial feature fusion: Encoding fine-grained external knowledge (from segmentation/OCR models) as spatial embedding maps, which are fused into visual transformers either via addition or concatenation (Lin et al., 2024).
• Knowledge-guided spatial prompts: In domain-specific settings (e.g., medical imaging), phrase-associated knowledge from ontologies is embedded and inserted as prompt tokens, sometimes in tandem with global/local visual features and mask-derived context (Gao et al., 2 Apr 2026).

2.3. Spatiotemporal and 3D-aware Prompt Construction

• Spatial-temporal keyframe encoding: Selecting semantically rich and temporally diverse frames from a video (via object class coverage, pose diversity) and marking their spatial or trajectory context for robust spatial reasoning (Li et al., 19 Sep 2025, Taguchi et al., 8 May 2025).
• Camera pose abstraction: Including per-frame camera position and orientation parameters with visual input to enable 3D spatial analysis in general-purpose MLLMs (Taguchi et al., 8 May 2025).
• Motion reconstruction overlays: Simulating camera or object trajectories and encoding object-relative or egocentric cues directly into visual prompts (Li et al., 19 Sep 2025).

3. Training, Optimization, and Inference Procedures

Spatial-aware visual prompting can be deployed under a variety of training and optimization paradigms:

• Zero-shot and training-free regimes: Most overlay-based prompt methods, as well as some keyframe and spatial map approaches, can be applied to off-the-shelf VLMs or MLLMs with no additional network retraining (Nasiriany et al., 2024, Taguchi et al., 8 May 2025, Li et al., 19 Sep 2025, Zhang et al., 2024, Yang et al., 2023, Wu et al., 12 Feb 2025).
• Self-supervised prompt learning: For learnable visual patches or position prompts, a self-supervised objective such as attention concentration (KL divergence to a Gaussian at the prompt location) may be used, with the base model frozen (Rezaei et al., 2024).
• Instruction tuning and supervised learning: When aligning prompts with outputs or injecting prompt features into encoders/decoders, models are trained with autoregressive language modeling losses (for language answers, coordinates, or box outputs), sometimes in conjunction with classification, regression, or cross-entropy losses on spatial prediction heads (Tang et al., 19 Mar 2025, Lin et al., 2024, Gao et al., 2 Apr 2026).
• Reinforcement learning with spatial rewards: In complex video or reasoning settings, prompting can be coupled with RL objectives (e.g., Group Relative Policy Optimization, GSPO) and spatial/temporal grounding rewards to directly optimize spatio-temporal attention and output precision (Ji et al., 6 Jul 2025, Lee et al., 15 Mar 2026).
• Auxiliary grounding or localization objectives: Some methods additionally introduce explicit auxiliary tasks (e.g., predicting the coordinates or bounding box indices marked by the prompt) to force the model to align its internal representations with the spatial annotations present in the input (Wang et al., 23 Mar 2026).
• Self-distillation: Prompt-guided reasoning at training time, followed by distillation from prompt-guided outputs, allows the model to internalize spatial-groundedness and achieve efficient, prompt-free inference (Lee et al., 15 Mar 2026).

4. Empirical Performance and Benchmarking

Spatial-aware visual prompting methods consistently achieve state-of-the-art or near–state-of-the-art results across a range of benchmarks:

• Visual grounding and referring expression comprehension: Structured spatial prompts—especially fine-grained and precise (e.g., Blur Reverse Mask, axis+object queries)—yield accuracy improvements between +1 and +12.5 percentage points on RefCOCO/RefCOCO+/RefCOCOg over strong baselines (Tang et al., 19 Mar 2025, Yang et al., 2023).
• Robotics (navigation/manipulation): Iterative proposal selection, visual trace overlays, and structured spatial prompting interfaces enable zero-shot action selection, outperforming no-prompt and text-only baselines by 4–8 points, with up to 3.5x improvement in real-world task success (Nasiriany et al., 2024, Zheng et al., 2024, Wang et al., 23 Mar 2026).
• 3D VQA and trajectory inference: Keyframe-driven prompts with pose encoding achieve strong zero-shot spatial QA in real 3D datasets (ScanQA, SQA3D), eliminating the need for 3D-specific retraining (Taguchi et al., 8 May 2025).
• Emotion recognition and social reasoning: Spatially precise face/landmark overlays and body/scene cues (SoV, SoVTP frameworks) double or triple zero-shot emotion recognition accuracy on both image and video datasets (Zhang et al., 2024, Wang et al., 24 Apr 2025).
• Remote sensing and domain-adapted tasks: Multi-scale prompt encoding and cross-domain staged training in dedicated MLLMs (EarthMarker) yield new SOTA in region-level, point-level, and relationship-level understanding of RS imagery (Zhang et al., 2024).
• Spatial relation reasoning and hallucination mitigation: Constraint-aware prompting via bidirectional and transitive relation chains reduces spatial hallucination rates by 5–13pp and improves consistency—without retraining—on ARO, GQA, and MMRel (Wu et al., 12 Feb 2025).
• Medical visual grounding: Knowledge-guided spatial prompts and global-local cross-attention improve box localization (AP50, mIoU) by 2–6 points on medical grounding datasets (Gao et al., 2 Apr 2026).
• Video reasoning and spatiotemporal grounding: Training-time, input-adaptive visual cue prompting, distilled into stand-alone models, achieves large boosts in spatial/temporal alignment and accuracy on V-STAR, VideoMME, and Char