---
title: Set-of-Mark Visual Prompting
url: https://www.emergentmind.com/topics/set-of-mark-visual-prompting
type: topic
---

# Set-of-Mark Visual Prompting

Set-of-Mark Visual Prompting (SoMVP) is a class of visual prompting paradigms that utilize explicit, spatially localized visual anchors—such as points, alphanumeric tags, colored regions, bounding boxes, or geometric shapes—overlaid directly on images to guide and condition downstream vision-language, segmentation, detection, counting, or robotic systems. SoMVP enables precise spatial grounding, structured region referencing, interpretable interaction protocols, and quantitative benchmarking across diverse domains, models, and tasks.

## 1. Formal Definition and Scope

In SoMVP, a prompt is constructed as a finite set $\mathcal{M}=\{m_1, m_2, \dots, m_k\}$ of marks $m_i$, each spatially defined in the image domain $I\in\mathbb{R}^{H\times W\times 3}$. Marks may be:
- Discrete points (e.g., $(x,y)$ pixel coordinates),
- Alphanumeric labels rendered at specified locations,
- Geometric overlays (e.g., rectangle, ellipse, arrow, boundary),
- Region masks identified by segmentation models.

Mathematically, an SoM prompt transforms $I$ via an overlay process:
$$
\hat{I} = f(I, \mathcal{M}) = (1-\alpha)\odot I + \alpha \odot P_{RGB}
$$
where $P_{RGB}$ is a composite overlay image encoding the marks, and $\alpha$ is a location-dependent transparency map determined by mark occupancy or designed opacity [2312.00784]. Lexical or numeric information may be embedded in $P_{RGB}$ as glyphs, aiding subsequent OCR or region identification in the consuming model [2310.11441, 2404.16375].

This formalism applies to both human-driven and fully automated prompt construction pipelines across segmentation (e.g., SAM) [2410.22048], multimodal language models (MLLMs/LMMs) [2310.11441, 2404.16375, 2312.04087], detection/counting [2311.13596], robot policy conditioning [2403.03174], and specialized pipelines such as emotion recognition [2410.02244].

## 2. Prompt Construction Protocols

### 2.1 Human-Guided Marking

Humans interactively place marks—e.g., inclusion/exclusion points in segmentation [2410.22048], reference regions for counting [2311.13596], or spatial anchors for MLLM instruction following—until a target behavior is achieved (e.g., satisfactory mask preview in SAM). Inclusion and exclusion roles are indicated through color or symbol encoding, and iterative refinement protocols converge to a set of marks reflecting the user's intent and mental object boundaries.

### 2.2 Automated Mark Generation

Automated procedures select and overlay marks using image-derived features and algorithmic strategies. Examples include:
- Clustering pixel distributions (K-Medoids) for spatial coverage [2410.22048],
- Corner or saliency detection (Shi-Tomasi, Vision Saliency Transformer) [2410.22048],
- Entropy maximization or maximal pairwise-spread sampling [2410.22048],
- Segmentation backbone-driven region proposals (MaskDINO, SEEM, SAM) [2310.11441, 2404.16375],
- Automated keypoint and landmark extraction (face detection, RetinaFace; object affordances) [2410.02244, 2403.03174].

Overlay algorithms must resolve mark placement to minimize overlap, maximize distinctiveness, and ensure occlusion avoidance. Marks are rendered with high-contrast, OCR-friendly glyphs or colored overlays [2310.11441, 2404.16375]. Some frameworks balance mark spatial distribution using convex hull coverage, pairwise spread, and boundary proximity features [2410.22048].

## 3. Integration into Model Architectures

### 3.1 Visual Transformers and MLLMs

SoM-prompted images $\hat{I}$ are ingested by vision encoders (CLIP-ViT, Vision Transformers) as pixel images with explicit marks. Multiple CLIP layers may be concatenated and processed to form a token stream for LLM cross-attention [2312.00784]. OCR capability in vision backbones is vital for glyph-based SoM protocols [2310.11441, 2404.16375]. Autoregressive LLMs receive the visual features alongside (potentially interleaved) textual references to mark IDs.

### 3.2 Segmentation and DETR-style Object Models

For segmentation models like SAM, the prompt encoder consumes spatially localized marks to initialize segmentation or mask decoders [2410.22048]. DETR-style open-set detectors aggregate prompt mark embeddings—obtained via point sampling, RoI-Align, and MLP transformation—and fuse them into the query stack of the decoder [2311.13596]. This enables prompt-conditioned detection and counting.

### 3.3 Robotics and Control

In robotic context (MOKA), overlaid 2D marks (dots, grid cells) are mapped to 3D workspace coordinates via depth projections, serving as grasp or manipulation targets under VLM affordance reasoning pipelines [2403.03174]. Mark-based segmentation outputs are used to design motion primitives and trajectories.

## 4. Evaluation Protocols and Quantitative Results

SoMVP effectiveness is evaluated through systematic, domain-spanning protocols:

### 4.1 Benchmark Datasets and Task Diversity

- **Segmentation**: PointPrompt dataset—6,000 images across 16 domains, human vs. automated mark performance [2410.22048].
- **Vision-Language**: RefCOCOg, COCO Panoptic, Flickr30K Entities, DAVIS2017 for segmentation, phrase grounding, and video object segmentation [2310.11441].
- **Multimodal Reasoning**: ViP-Bench, Visual7W, PointQA, and Visual Commonsense Reasoning, with both synthetic and human-drawn prompts [2312.00784].
- **Counting**: FSC147, FSCD-LVIS, CA-44 spanning 8 domains [2311.13596].
- **Emotion Recognition**: Custom benchmarks with difficulty stratifications (Easy–Medium–Hard) [2410.02244].

### 4.2 Metrics

- Segmentation: $\mathrm{IoU}$, Dice coefficient [2410.22048], mIoU, Recall@1 [2310.11441, 2410.02244].
- Counting: MAE, NMAE [2311.13596].
- Reasoning: GPT-4-judged free-form scores [2312.00784], accuracy, BLEU-4 for structured responses [2312.04087].
- Object Hallucination/Alignment: POPE F1, MME, SEED-I [2404.16375].

### 4.3 Key Results

| System                      | Domain/Task          | SoM(+) vs Baseline        |
|:----------------------------|:---------------------|:--------------------------|
| SAM (human marks)           | Segmentation         | mIoU 0.78 vs ≈0.55 (auto) |
| SAM (finetuned on auto)     | Segmentation         | +22–68% mIoU              |
| GPT-4V + SoM                | Zero-shot seg/ground | COCO: 75.7% [P@1]         |
| SoM-LLaVA                   | MLLM reasoning       | +1–3 pts F1/MME/SEED-I    |
| T-Rex                       | Counting             | FSC147 1-shot: MAE 10.6   |
| MOKA                        | Robotics             | Subtask succ. 50–100%     |
| GPT-4V + SoV (emotion)      | Group emotions       | +11–15 pp over baseline   |

SoM-enhanced models close the gap or outperform prior task-specific models in zero-shot and finetuned settings. Notably, fine-tuning on SoM prompts can persistently improve alignment and reduce hallucinations, even when explicit marks are omitted at inference [2404.16375].

## 5. Analysis of Mark Design, Task Grounding, and Failure Modes

- **Granularity Selection:** Balanced region granularity enhances disambiguation (whole-object for broad queries, fine-grained for part-level or compositional reasoning). Excessively fine granularity leads to ID crowding or ambiguity; excessive coarseness blurs distinctions [2310.11441].
- **Mark Type:** Alphanumeric tags, color-contrasted overlays, and bounding boxes are robust, with numeric/letter glyphs providing OCR-anchored region identification (critical for GPT-4V and SoM-LLaVA) [2310.11441, 2404.16375]. Bounding box addition and high-contrast shapes improve referential clarity.
- **Placement Heuristics:** Centroid or maximal-distance-from-boundary placements reduce glyph overlap; saliency-based placements are prospective improvements [2310.11441].
- **Failure Modes:** Mark overlap in crowded/concave regions, confusion with natural scene digits, and ambiguous annotation boundaries are recurrent issues. Crowding mitigation, adaptive glyph type, and algorithmic placement (e.g., convex hull, spread control) are partial remedies [2310.11441, 2410.22048].
- **Inductive Bias:** Training with images containing indexed SoM tags improves global object–text alignment, yielding lower hallucination even when tags are absent at test time [2404.16375].

## 6. Extensions, Adaptations, and Best Practices

- **Prompt Engineering:** Prompt effectiveness can be model-dependent; practitioners should tune color, shape, font, and captioning conventions per model [2312.04087].
- **Combined Cues:** Blending visual marks with in-image instructional text (full-intervention) frequently outperforms partial overlays [2312.04087].
- **Feedback and Interactivity:** Interactive refinement (expanding/reducing mark sets, including negative samples) enables real-time system correction and policy improvement in segmentation and counting [2311.13596, 2410.22048].
- **Cross-Context Robustness:** SoM training boosts attention–object alignment, domain adaptation, and zero-shot generalization even on out-of-distribution (OOD) domains [2410.22048, 2403.03174].
- **Limitations:** Preprocessing costs (e.g., segmentation, landmark extraction), susceptibility to detector failure in occluded or rare configurations, and lack of grounding where language-only models are used, delimit practical deployment [2410.02244].

## 7. Future Directions and Open Research Areas

Research in SoMVP is progressing toward:
- Automatic region proposal strategies optimizing both interpretability and model alignment,
- Integration of mark overlays with token-level or attention-mask modulations,
- Benchmarking and ablation protocols isolating the contributions of visual vs. textual anchoring,
- Extensions to video (temporal SoM) and multi-instance tracking [2410.02244],
- Domain-specific SoM applications in medical, seismic, and manipulation tasks [2410.22048, 2403.03174],

The systematic benchmarking, interpretability, and cross-model portability of SoMVP constitute an active frontier in multimodal grounding, interactive vision, and transparent prompt-based control [2410.22048, 2310.11441, 2404.16375, 2312.00784].

Source: https://www.emergentmind.com/topics/set-of-mark-visual-prompting