Set-of-Mark Visual Prompting

Updated 4 March 2026

Set-of-Mark Visual Prompting is a method using explicit spatial anchors like points, tags, and overlays to condition and guide downstream vision and language tasks.
It enables precise spatial grounding and structured region referencing, improving interpretability and alignment across segmentation, detection, and robotic control models.
SoMVP integrates human-guided and automated mark generation protocols, reducing hallucinations and enhancing zero-shot performance through robust benchmarking.

Set-of-Mark Visual Prompting (SoMVP) is a class of visual prompting paradigms that utilize explicit, spatially localized visual anchors—such as points, alphanumeric tags, colored regions, bounding boxes, or geometric shapes—overlaid directly on images to guide and condition downstream vision-language, segmentation, detection, counting, or robotic systems. SoMVP enables precise spatial grounding, structured region referencing, interpretable interaction protocols, and quantitative benchmarking across diverse domains, models, and tasks.

1. Formal Definition and Scope

In SoMVP, a prompt is constructed as a finite set $\mathcal{M}=\{m_1, m_2, \dots, m_k\}$ of marks $m_i$ , each spatially defined in the image domain $I\in\mathbb{R}^{H\times W\times 3}$ . Marks may be:

Discrete points (e.g., $(x,y)$ pixel coordinates),
Alphanumeric labels rendered at specified locations,
Geometric overlays (e.g., rectangle, ellipse, arrow, boundary),
Region masks identified by segmentation models.

Mathematically, an SoM prompt transforms $I$ via an overlay process:

$\hat{I} = f(I, \mathcal{M}) = (1-\alpha)\odot I + \alpha \odot P_{RGB}$

where $P_{RGB}$ is a composite overlay image encoding the marks, and $\alpha$ is a location-dependent transparency map determined by mark occupancy or designed opacity (Cai et al., 2023). Lexical or numeric information may be embedded in $P_{RGB}$ as glyphs, aiding subsequent OCR or region identification in the consuming model (Yang et al., 2023, Yan et al., 2024).

This formalism applies to both human-driven and fully automated prompt construction pipelines across segmentation (e.g., SAM) (Quesada et al., 2024), multimodal LLMs (MLLMs/LMMs) (Yang et al., 2023, Yan et al., 2024, Li et al., 2023), detection/counting (Jiang et al., 2023), robot policy conditioning (Liu et al., 2024), and specialized pipelines such as emotion recognition (Zhang et al., 2024).

2. Prompt Construction Protocols

2.1 Human-Guided Marking

Humans interactively place marks—e.g., inclusion/exclusion points in segmentation (Quesada et al., 2024), reference regions for counting (Jiang et al., 2023), or spatial anchors for MLLM instruction following—until a target behavior is achieved (e.g., satisfactory mask preview in SAM). Inclusion and exclusion roles are indicated through color or symbol encoding, and iterative refinement protocols converge to a set of marks reflecting the user's intent and mental object boundaries.

2.2 Automated Mark Generation

Automated procedures select and overlay marks using image-derived features and algorithmic strategies. Examples include:

Clustering pixel distributions (K-Medoids) for spatial coverage (Quesada et al., 2024),
Corner or saliency detection (Shi-Tomasi, Vision Saliency Transformer) (Quesada et al., 2024),
Entropy maximization or maximal pairwise-spread sampling (Quesada et al., 2024),
Segmentation backbone-driven region proposals (MaskDINO, SEEM, SAM) (Yang et al., 2023, Yan et al., 2024),
Automated keypoint and landmark extraction (face detection, RetinaFace; object affordances) (Zhang et al., 2024, Liu et al., 2024).

Overlay algorithms must resolve mark placement to minimize overlap, maximize distinctiveness, and ensure occlusion avoidance. Marks are rendered with high-contrast, OCR-friendly glyphs or colored overlays (Yang et al., 2023, Yan et al., 2024). Some frameworks balance mark spatial distribution using convex hull coverage, pairwise spread, and boundary proximity features (Quesada et al., 2024).

3. Integration into Model Architectures

3.1 Visual Transformers and MLLMs

SoM-prompted images $\hat{I}$ are ingested by vision encoders (CLIP-ViT, Vision Transformers) as pixel images with explicit marks. Multiple CLIP layers may be concatenated and processed to form a token stream for LLM cross-attention (Cai et al., 2023). OCR capability in vision backbones is vital for glyph-based SoM protocols (Yang et al., 2023, Yan et al., 2024). Autoregressive LLMs receive the visual features alongside (potentially interleaved) textual references to mark IDs.

3.2 Segmentation and DETR-style Object Models

For segmentation models like SAM, the prompt encoder consumes spatially localized marks to initialize segmentation or mask decoders (Quesada et al., 2024). DETR-style open-set detectors aggregate prompt mark embeddings—obtained via point sampling, RoI-Align, and MLP transformation—and fuse them into the query stack of the decoder (Jiang et al., 2023). This enables prompt-conditioned detection and counting.

3.3 Robotics and Control

In robotic context (MOKA), overlaid 2D marks (dots, grid cells) are mapped to 3D workspace coordinates via depth projections, serving as grasp or manipulation targets under VLM affordance reasoning pipelines (Liu et al., 2024). Mark-based segmentation outputs are used to design motion primitives and trajectories.

4. Evaluation Protocols and Quantitative Results

SoMVP effectiveness is evaluated through systematic, domain-spanning protocols:

4.1 Benchmark Datasets and Task Diversity

Segmentation: PointPrompt dataset—6,000 images across 16 domains, human vs. automated mark performance (Quesada et al., 2024).
Vision-Language: RefCOCOg, COCO Panoptic, Flickr30K Entities, DAVIS2017 for segmentation, phrase grounding, and video object segmentation (Yang et al., 2023).
Multimodal Reasoning: ViP-Bench, Visual7W, PointQA, and Visual Commonsense Reasoning, with both synthetic and human-drawn prompts (Cai et al., 2023).
Counting: FSC147, FSCD-LVIS, CA-44 spanning 8 domains (Jiang et al., 2023).
Emotion Recognition: Custom benchmarks with difficulty stratifications (Easy–Medium–Hard) (Zhang et al., 2024).

4.2 Metrics

Segmentation: $m_i$ 0, Dice coefficient (Quesada et al., 2024), mIoU, Recall@1 (Yang et al., 2023, Zhang et al., 2024).
Counting: MAE, NMAE (Jiang et al., 2023).
Reasoning: GPT-4-judged free-form scores (Cai et al., 2023), accuracy, BLEU-4 for structured responses (Li et al., 2023).
Object Hallucination/Alignment: POPE F1, MME, SEED-I (Yan et al., 2024).

4.3 Key Results

System	Domain/Task	SoM(+) vs Baseline
SAM (human marks)	Segmentation	mIoU 0.78 vs ≈0.55 (auto)
SAM (finetuned on auto)	Segmentation	+22–68% mIoU
GPT-4V + SoM	Zero-shot seg/ground	COCO: 75.7% [P@1]
SoM-LLaVA	MLLM reasoning	+1–3 pts F1/MME/SEED-I
T-Rex	Counting	FSC147 1-shot: MAE 10.6
MOKA	Robotics	Subtask succ. 50–100%
GPT-4V + SoV (emotion)	Group emotions	+11–15 pp over baseline

SoM-enhanced models close the gap or outperform prior task-specific models in zero-shot and finetuned settings. Notably, fine-tuning on SoM prompts can persistently improve alignment and reduce hallucinations, even when explicit marks are omitted at inference (Yan et al., 2024).

5. Analysis of Mark Design, Task Grounding, and Failure Modes

Granularity Selection: Balanced region granularity enhances disambiguation (whole-object for broad queries, fine-grained for part-level or compositional reasoning). Excessively fine granularity leads to ID crowding or ambiguity; excessive coarseness blurs distinctions (Yang et al., 2023).
Mark Type: Alphanumeric tags, color-contrasted overlays, and bounding boxes are robust, with numeric/letter glyphs providing OCR-anchored region identification (critical for GPT-4V and SoM-LLaVA) (Yang et al., 2023, Yan et al., 2024). Bounding box addition and high-contrast shapes improve referential clarity.
Placement Heuristics: Centroid or maximal-distance-from-boundary placements reduce glyph overlap; saliency-based placements are prospective improvements (Yang et al., 2023).
Failure Modes: Mark overlap in crowded/concave regions, confusion with natural scene digits, and ambiguous annotation boundaries are recurrent issues. Crowding mitigation, adaptive glyph type, and algorithmic placement (e.g., convex hull, spread control) are partial remedies (Yang et al., 2023, Quesada et al., 2024).
Inductive Bias: Training with images containing indexed SoM tags improves global object–text alignment, yielding lower hallucination even when tags are absent at test time (Yan et al., 2024).

6. Extensions, Adaptations, and Best Practices

Prompt Engineering: Prompt effectiveness can be model-dependent; practitioners should tune color, shape, font, and captioning conventions per model (Li et al., 2023).
Combined Cues: Blending visual marks with in-image instructional text (full-intervention) frequently outperforms partial overlays (Li et al., 2023).
Feedback and Interactivity: Interactive refinement (expanding/reducing mark sets, including negative samples) enables real-time system correction and policy improvement in segmentation and counting (Jiang et al., 2023, Quesada et al., 2024).
Cross-Context Robustness: SoM training boosts attention–object alignment, domain adaptation, and zero-shot generalization even on out-of-distribution (OOD) domains (Quesada et al., 2024, Liu et al., 2024).
Limitations: Preprocessing costs (e.g., segmentation, landmark extraction), susceptibility to detector failure in occluded or rare configurations, and lack of grounding where language-only models are used, delimit practical deployment (Zhang et al., 2024).

7. Future Directions and Open Research Areas

Research in SoMVP is progressing toward:

Automatic region proposal strategies optimizing both interpretability and model alignment,
Integration of mark overlays with token-level or attention-mask modulations,
Benchmarking and ablation protocols isolating the contributions of visual vs. textual anchoring,
Extensions to video (temporal SoM) and multi-instance tracking (Zhang et al., 2024),
Domain-specific SoM applications in medical, seismic, and manipulation tasks (Quesada et al., 2024, Liu et al., 2024),

The systematic benchmarking, interpretability, and cross-model portability of SoMVP constitute an active frontier in multimodal grounding, interactive vision, and transparent prompt-based control (Quesada et al., 2024, Yang et al., 2023, Yan et al., 2024, Cai et al., 2023).