Visual-Prompted Models in Vision-Language

• Visual-prompted models are techniques that adapt frozen vision and multimodal models using explicit visual cues (e.g., pixel overlays, token prompts) for efficient task adaptation.
• They enable parameter-efficient tuning, improving performance and robustness across tasks such as image classification, VQA, and semantic segmentation.
• Recent advances integrate fixed, learnable, and generative prompt strategies to enhance spatial reasoning and transferability across diverse domains.

Visual-prompted models are a class of techniques in computer vision and vision–language modeling that adapt large frozen models for new tasks by inserting explicit visual cues—often in the form of extra image pixels, patches, spatial embeddings, or internal tokens—rather than, or in addition to, traditional text prompts. Such prompts can be fixed, learned, or generated procedurally, and may operate at the pixel, patch, or feature level. Visual prompting serves as a parameter-efficient alternative to full model fine-tuning and undergirds much recent progress in both unimodal vision models and multimodal (vision–language) large models, where grounding, compositionality, and localized reasoning remain major challenges. The field now encompasses a wide array of methods, including pixel-level warping, soft token prompts, automated retrieval of visual cues, memory-injection strategies, and dynamic spatial embeddings. Current research reveals complementary strengths between visual and text prompts, domain-specific innovation in medical and scientific imaging, and active work toward generalized, transferable prompt architectures.

1. Foundational Definitions and Taxonomies

Prompt-based adaptation (PA) in vision refers to any method that keeps a large pre-trained backbone $f_\phi$ frozen, while adapting it to new tasks by either (i) enriching the input with a visual prompt or (ii) inserting additional lightweight parameters (tokens or features) internally (Xiao et al., 15 Oct 2025). Key conceptual subdivisions include:

• Visual Prompting (VP): Prompts operate in input space, such as learnable pixel-space patterns or overlays (e.g., border frames, patches).
• Visual Prompt Tuning (VPT): Prompts act internally as extra feature vectors or tokens, injected at one or more layers (shallow or deep) of, typically, a transformer encoder (Xiao et al., 15 Oct 2025, Wu et al., 5 Sep 2024).
• Prompt Generation Axis: Prompts are either non-learnable (fixed, manually designed or rule-based), learnable (parameterized and updated via gradient descent), or generative (dynamically predicted per sample by a lightweight network).
• Injection Granularity: Prompts may be at the pixel (whole-image), patch (regions of interest), or token (feature or transformer-token) level.

This unification enables comparison and systematic evaluation across methodologies, domains, and applications.

2. Visual Prompt Architectures and Design Variants

Visual-prompted models span a rich design space:

• Pixel-space prompts: Learnable tensors $\delta\in\mathbb{R}^{H\times W\times 3}$ added or overlaid to the image. EVP (Wu et al., 2022) and AutoVP (Tsao et al., 2023) propose warping the original image inward, then padding the border with prompt pixels to avoid information loss. Input diversity and gradient normalization (from adversarial literature) are essential for prompt generalization (Wu et al., 2022).
• Token-level prompts: Learned tokens (e.g., $P\in\mathbb{R}^{p\times d}$) appended to patch embeddings at the input or inserted per layer (Xiao et al., 15 Oct 2025, Xu et al., 2023). ProVP (Xu et al., 2023) proposes progressive linkage of prompt tokens across layers to stabilize and enhance training.
• Spatial semantic prompts: Injected per-pixel embedding maps carrying fine-grained information from external models (segmentation/OCR), which are spatially fused into the vision encoder’s representations (Lin et al., 5 Jul 2024).
• Memory-space prompting: Instead of extending the input length, visual information is injected by augmenting the key–value tables of each transformer FFN in the LLM with projected vision features, greatly reducing training and inference overhead (Jie et al., 9 May 2024).
• Prompt retrieval and automation: Automated systems such as AutoV rank multiple candidate prompts (e.g., attention heatmaps, overlays) for each (image, query) pair using a lightweight reward predictor trained on model-internal loss signals (Zhang et al., 19 Jun 2025).
• Self-supervised prompt learning: Prompt patterns that maximize attention alignment with designated regions, learned without any downstream labels (Rezaei et al., 5 Jun 2024).
• Task-specific variants: Visual prompts are adapted for compositional zero-shot learning via a dynamic repository and context-dependent retrieval (Stein et al., 27 Feb 2025), and for localization or region-specific attention in medical VLMs using heuristic markers (rectangles, ellipses, scribbles) (Zhu et al., 4 Jan 2025).

The following table illustrates core types:

Category	Injection Point	Example Method
Pixel overlay (learned)	Image pixels	EVP, AutoVP
Token prompts	Patch/transformer	VPT, ProVP-Ref
Spatial semantic map	Feature map	(Lin et al., 5 Jul 2024)
Memory-space	FFN weights	MemVP
Retrieval-based	Instance-specific	AutoV

3. Applications, Empirical Results, and Benchmarks

Visual-prompted models are empirically validated on a wide range of tasks:

• Image classification: Both EVP and AutoVP show that visual-prompted adaptation matches or surpasses linear probing (LP), approaching full fine-tuning performance with only 0.04%–0.1% of parameters updated (Wu et al., 2022, Tsao et al., 2023). Best-in-class EVP achieves a mean accuracy of 82.8% across 12 datasets, outperforming LP by +2.1% (Wu et al., 2022).
• Robust adaptation: Visual prompts confer superior robustness to distribution shifts (e.g., WILDS, CIFAR-C), outperforming both LP and conventional VP (Wu et al., 2022, Zhang et al., 17 Apr 2024).
• Multimodal tasks (VQA/VLMs): MedVP achieves dramatic improvements on medical VQA tasks over zero-shot and fully fine-tuned baselines (e.g., VQA-RAD accuracy: 97.3% vs. 61.4% for LLaVA-Med) by using explicit spatial markers with instruction-tuning (Zhu et al., 4 Jan 2025). Direct pixel-wise semantic embeddings (spatial prompts) further improve fine-grained reasoning in MLLMs across nine VQA and vision reasoning benchmarks (Lin et al., 5 Jul 2024).
• Compositional zero-shot learning: VAPS dynamically adjusts prompts based on visual context and achieves new SOTA on MIT-States and C-GQA (unseen accuracy up to 74.3%) by fusing attribute/object visual prompts and text adaptation (Stein et al., 27 Feb 2025).
• Transferability: TVP demonstrates the ability to train a single visual prompt on one MLLM that can immediately improve any other black-box MLLM without fine-tuning, using feature-consistency and task-semantics-enrichment constraints (Zhang et al., 17 Apr 2024).
• Localization and spatial grounding: VRPTest reveals that visual referring prompting (visual pointers/text overlays) can change LMM accuracy by up to +7.3% or -17.5% depending on prompt strategy, with proprietary GPT-4V models outperforming open-source LMMs by +22.7 pp on average (Li et al., 2023).
• Semantics segmentation: Few-shot Prompted Semantic Segmentation shows that text and visual prompts have complementary strengths, and combining both (PromptMatcher) outperforms any single modality by 2.5–3.5 IoU points (Avogaro et al., 25 Mar 2025).
• Sensor data analysis: Visual prompting can compress long sensor streams into image plots, drastically reducing token costs (15.8× savings) while gaining +10 points accuracy over text-prompting in MLLMs (Yoon et al., 15 Jul 2024).
• Speech localization: Visually-prompted keyword localization enables detection/localization in zero-resource spoken language, outperforming BoW text query baselines by 16% F1 in localization (Nortje et al., 2022).

4. Theoretical, Practical, and Transferability Considerations

Key theoretical insights and engineering considerations include:

• Prompt efficiency: Learned visual prompts typically update only 0.04%–0.1% of the backbone's weights, suiting low-resource and hardware-constrained settings (Wu et al., 2022, Tsao et al., 2023).
• Prompt placement: Warping (shrinking) the input image with a border prompt avoids information loss and improves generalization relative to direct addition overlays (Wu et al., 2022).
• Prompt composition: Mixing multiple prompt shapes/colors during training yields robust test performance, and adaptive or retrieved prompts (AutoV) further improve accuracy on a per-instance basis (Zhang et al., 19 Jun 2025, Zhu et al., 4 Jan 2025).
• Internal vs. input prompts: Internal token prompts (VPT/ProVP) enable instance-adaptive tuning via progressive or cross-layer connections; memory-space prompting injects vision features into the FFN's key–value memories, reducing the inefficiency of long visual-token sequences during training/inference (Xu et al., 2023, Jie et al., 9 May 2024).
• Transfer and generalization: FCA and TSE objectives constrain features to avoid overfitting and encode task semantics, promoting prompt transferability across architectures and datasets (Zhang et al., 17 Apr 2024).

5. Domain Extensions, Robustness, and Limitations

Visual-prompted models have been extended beyond standard 2D vision and VQA:

• 3D and Point Clouds: Prompting strategies include geometry-aware or instance-adaptive tokens for 3D transformers, as well as pixel overlays for multi-sensor remote sensing or segmentation (Xiao et al., 15 Oct 2025).
• Medical imaging: Prompt overlays (boxes, ellipses, scribbles) are used for region-specific reasoning and improve both interpretability and performance (Zhu et al., 4 Jan 2025).
• Time series and sensors: Visual plots bridge long-sequence sensor data to standard MLLMs, outperforming raw-text inputs in both economy and performance (Yoon et al., 15 Jul 2024).

Reported limitations and failure modes:

• Prompt specificity: Static or global prompts can underperform on highly diverse datasets. Instance-adaptive or dynamic retrieval mechanisms are needed for resilient performance (Tsao et al., 2023, Zhang et al., 17 Apr 2024, Zhang et al., 19 Jun 2025).
• Cross-domain generalization: Visual prompt transfer to heterogeneous architectures or highly divergent data domains, e.g., medical or 3D images, may require prompt repository expansion or fine-tuning (Stein et al., 27 Feb 2025, Zhang et al., 19 Jun 2025).
• Security and robustness: Visual prompts introduce new attack surfaces (adversarial overlays, backdoors, visually-coded instructions). Prompt-based defenses, watermarking, and safety assessment will become important as adoption widens (Xiao et al., 15 Oct 2025).

6. Future Directions and Open Challenges

Several promising research frontiers and open questions are identified in recent surveys and primary contributions:

• Hybrid prompting: Integrating pixel-level overlays, token prompts, and spatial semantic maps for unified, cross-task adaptation (Xiao et al., 15 Oct 2025, Wu et al., 5 Sep 2024).
• Dynamic and adaptive generation: Automated and context-specific prompt retrieval (as in AutoV) is likely to be combined with prompt generation via lightweight hypernetworks or LLM planners orchestrating vision toolchains (Zhang et al., 19 Jun 2025, Wu et al., 5 Sep 2024).
• Theoretical understanding: Investigations continue into why and when visual prompt learning works, with mutual information and feature alignment analyses providing intermediate explanations (Xiao et al., 15 Oct 2025).
• Compositional and 3D reasoning: Extending spatial prompt design for 3D and temporally-evolving data such as video and sensor-capture is ongoing (Stein et al., 27 Feb 2025, Wu et al., 5 Sep 2024).
• Trustworthiness and robustness: Visual prompting for robustness, fairness, and defenses against adversarial and backdoor attacks remains underdeveloped. Additional work is needed on safe visuo-text alignment and bias reduction (Xiao et al., 15 Oct 2025, Wu et al., 5 Sep 2024).
• Benchmark development: Systematic and challenging benchmarks, such as MESS for semantic segmentation (Avogaro et al., 25 Mar 2025) and VRPTest for visual referring prompting (Li et al., 2023), lay the empirical groundwork for more nuanced model evaluation.

A plausible implication is that visual-prompted models will subsume both explicit cueing (input overlays, spatial maps) and implicit adaptation (feature and token insertions), with automated and compositional prompt strategies playing a central role in next-generation, general-purpose multimodal reasoning systems.