Prompt-driven Video Segmentation Models

• Prompt-driven Video Segmentation Foundation Models are systems that integrate diverse prompt types with pretrained vision backbones to unify segmentation tasks across video modalities.
• They employ a modular design with image encoders, prompt encoders, and mask decoders to enable both interactive and automated segmentation workflows.
• Incorporating temporal memory and lightweight adaptation strategies, these models maintain consistency and efficiency even in rapidly changing video scenes.

Prompt-driven Video Segmentation Foundation Models (VSFMs) are a dominant paradigm in video understanding, unifying segmentation across diverse video modalities by leveraging large, general-purpose vision backbones and flexible prompt mechanisms. These models integrate user or algorithmically generated prompts—such as points, boxes, scribbles, language, or visual memory—to enable task transfer, few-shot adaptation, and universal mask prediction across video frames. VSFMs are central to applications ranging from medical imaging to autonomous driving, supporting both interactive and automated segmentation workflows with strong domain generalization, rapid adaptation, and robust performance.

1. Core Architectures and Prompt Mechanisms

VSFMs are typically built on pretrained foundation models (e.g., ViT-based Segment Anything Model (SAM), MedSAM, or All-in-One Transformers). They modularize the segmentation task into three primary components:

• Image Encoder: Extracts dense visual features from each frame.
• Prompt Encoder: Processes user- or algorithm-specified prompts (points, boxes, masks, or text), yielding prompt embeddings.
• Mask Decoder: Fuses frame and prompt features, producing pixel-wise segmentation masks.

Prompt modalities are central:

• Points/Boxes: Embedded as sparse spatial keys.
• Masks: Passed through a convolutional encoder.
• Language: Processed via a text encoder and fused (e.g., with CLIP, GPT variants). Prompts can be provided interactively by the user, automatically derived from previous segmentation outputs, or generated via motion models and trackers, enabling both real-time interactive refinement and fully automated pipelines (Zeng et al., 30 Jan 2025, Lin et al., 8 Oct 2025, Breitenstein et al., 2024, Xu et al., 30 Jul 2025).

2. Temporal Modeling and Memory Integration

Temporal consistency is addressed via various architectural strategies:

• Streaming Memory: Models like SAM2 append a FIFO queue of encoded past frame/mask pairs, with attention modules integrating memory for temporal coherence (Xu et al., 30 Jul 2025).
• Adaptation and Upgrading: SAM-I2V introduces lightweight temporal feature integrators around 2D encoders, employing 3D convolutions and spatial-temporal fusion to adapt static image backbones for video streams at <0.2% of SAM2’s compute (Mei et al., 2 Jun 2025).
• Autoregressive State-Space: AUSM formalizes segmentation as sequence modeling, with state-space models compactly encoding all prior mask information, supporting arbitrarily long sequences via efficient Mamba layers and cross-attention decoders (Heo et al., 26 Aug 2025).
• Prompt Memory: In frameworks like UniVS, each target maintains a memory pool of prompt features—averaged or cross-attended—for use as queries in subsequent frames, forming a unified prompt-driven schedule for multi-object tracking and re-identification (Li et al., 2024).

Memory usage, frame selection, and prompt update strategies are increasingly sophisticated—pruning schemes select relevant history based on frame similarity, motion affinity, or detection confidence (Mei et al., 2 Jun 2025, Xu et al., 30 Jul 2025).

3. Adaptation, Training Paradigms, and Learning Protocols

VSFMs support various adaptation and learning protocols:

• Test-Time Prompt-Guided Training (Prompt-TTT): MedSAM can be efficiently specialized for new video domains by self-supervised consistency losses on point prompts, updating only the encoder during inference, and requiring minimal annotation (single point per frame). This procedure raises Dice coefficient from 0.847 (supervised fine-tune) to 0.868, nearly matching specialist video segmentation networks on VFSS-5k (Zeng et al., 30 Jan 2025).
• Prompt-only Few-Shot Learning: Semi-parametric Deep Forest (SDForest) achieves real-time video object segmentation by fitting shallow models (random forest and logistic regressor) on frozen deep features from a CNN backbone, trained on a single frame and applied to the rest without updating the backbone. SDForest attains competitive Jaccard and contour scores on DAVIS benchmarks in purely prompt-driven regimes (Wangni, 2024).
• Efficient Multi-Modal Adaptation: X-Prompt enables adaptation to new video modalities (e.g., Depth, Thermal, Event) by training lightweight Multi-modal Visual Prompters and Multi-modal Adaptation Experts, tuning ≤4% of parameters and preserving foundation model generality. Multi-scale prompt injection and low-rank expert layers provide cross-modal transfer and outperform full fine-tuning (Guo et al., 2024).
• Universal Training and Decoupled Pipelines: UniVS and AUSM show that by casting all segmentation tasks as prompt querying, a single model can handle instance, semantic, panoptic, referring, and open-set video segmentation without heuristics or per-task customization (Li et al., 2024, Heo et al., 26 Aug 2025).

4. Task Formulations and Applications

VSFMs are applicable to a comprehensive range of tasks, achieved through flexible prompt engineering:

• Prompted Segmentation: User- or algorithm-supplied prompts (points, boxes, masks, text) guide the tracking or mask refinement of specific targets through videos.
• Unprompted/Universal Segmentation: Detection queries or learnable prompts identify and segment all objects in videos without external cues, supporting instance, semantic, and panoptic segmentation (Heo et al., 26 Aug 2025).
• Referring Video Object Segmentation (RVOS): Language prompts are grounded to candidate tracks. Tenet demonstrates that generating and selecting high-quality temporal prompts (via motion trackers, prompt preference transformers) and deferring mask prediction to a strong image foundation segmenter provides state-of-the-art results with minimal fine-tuning (Lin et al., 8 Oct 2025).
• Amodal Video Instance Segmentation: S-AModal leverages point prompts derived from visible masks, point memory for occlusion handling, and propagation/tracking modules to enable amodal mask continuity in autonomous driving and surveillance contexts, surpassing prior methods without requiring full video-wise amodal labels (Breitenstein et al., 2024).
• Multi-Modal and Multi-Domain Applications: Prompt-based adapters and visual prompters enable robust segmentation across challenging imaging conditions, e.g., medical videos with cross-modality (ultrasound, fluoroscopy), harsh illumination, or rapid scene content changes (Guo et al., 2024, Zeng et al., 30 Jan 2025).

A summary table highlights core properties across select VSFM frameworks:

Model	Prompt Types	Temporal/Memory Method	Parameter Update Regime	Reference
MedSAM-TTT	Points, Boxes	Consistency accum. over seq.	Encoder-only, at inference	(Zeng et al., 30 Jan 2025)
SDForest	Initial mask	None; per-frame prompts	Prompt params only	(Wangni, 2024)
UniVS	Visual/Text/Queries	Prompt pool with avg.+ProCA	End-to-end	(Li et al., 2024)
SAM-I2V	Points, Boxes, Mem.	3D-conv TFI + filtered mem.	Modular plug-in	(Mei et al., 2 Jun 2025)
AUSM	Points, Boxes, Mask	State-space compress.+agg.	All (universal)	(Heo et al., 26 Aug 2025)
X-Prompt	RGB+X (Multi-modal)	Parallel tokens + MAEs	MVP/MAE params	(Guo et al., 2024)
S-AModal	Points	Point-tracking + mask shift	Adapter+decoder	(Breitenstein et al., 2024)

5. Prompt Engineering, Motion Modeling, and Selection Strategies

Prompt selection critically affects segmentation efficiency and quality:

• Temporal Propagation: Previous mask predictions or tracked points/boxes are used as prompts for subsequent frames (e.g., trajectory-guided prompts, point memory, cross-attended pools) (Li et al., 2024, Breitenstein et al., 2024, Lin et al., 8 Oct 2025).
• Motion Awareness: Kalman filter, attention-based tracking, or explicit trajectory modules predict object location for prompt generation, enhancing performance under fast motion or occlusion (Xu et al., 30 Jul 2025).
• Prompt Preference Learning: Tenet uses a Transformer-based binary comparator to automatically select the best candidate temporal prompt track per query, given language and visual representations (Lin et al., 8 Oct 2025).
• Ablation Insights: Number and type of prompts are critical: e.g., MedSAM-TTT achieves highest Dice with a single point (M=1), as more points can collapse self-supervised objectives (Zeng et al., 30 Jan 2025); S-AModal sees best AP50 for K=1 point prompt, with eroded border sampling outperforming random selection (Breitenstein et al., 2024).

Supported prompt types and temporal updating are mechanisms for both domain generalization and computational efficiency.

6. Evaluation, Empirical Performance, and Security

VSFM evaluation employs standardized segmentation and tracking metrics:

• Dice coefficient, IoU (Jaccard), Hausdorff distance (HD95), Average Surface Distance (ASD), region F-score, and Sensitivity.
• Benchmark datasets: DAVIS, YouTube-VOS, VIPSeg, VFSS-5k, VisT300, ARKitTrack, VisEvent-VOS, AmodalSynthDrive, KINS-car.
• Performance: MedSAM with Prompt-TTT attains average Dice 0.868 across 12 anatomies on VFSS-5k (Zeng et al., 30 Jan 2025). SDForest surpasses SiamMask on J/F in real-time CPU regimes (Wangni, 2024). X-Prompt outperforms specialist multi-modal fine-tuning with 2–4% parameter overhead (Guo et al., 2024). AUSM approaches state-of-the-art on both prompted and unprompted video segmentation tasks, with up to 2.5× faster training relative to iterative baselines (Heo et al., 26 Aug 2025).

Security Analysis: Prompt-driven VSFMs are exposed to novel backdoor attack surfaces, as demonstrated by BadVSFM. Conventional attacks such as BadNet are ineffective (ASR <5%), but BadVSFM’s two-stage strategy of encoder steering plus decoder backdoor enforcement achieves attack success rates above 95% across varying prompt types and triggers while preserving clean segmentation quality. This indicates a fundamental vulnerability, as standard defenses (fine-tuning, pruning, spectral signatures, STRIP) are largely ineffective (Zhang et al., 26 Dec 2025).

7. Limitations and Future Directions

Challenges persist in memory redundancy, prompt inefficiency, domain generalization, and efficient motion modeling:

• Memory Redundancy: FIFO strategies often retain unnecessary history; future hierarchical memory and quality-based pruning are active research areas (Xu et al., 30 Jul 2025).
• Prompt Inefficiency and Drift: On-the-fly prompt generation can introduce error propagation; future work may focus on supervised prompt engineering, end-to-end learning of segmentation plus prompt generation, and detection-assisted screening.
• Generalization: Cross-domain adaptation, especially to medical or atypical videos, remains limited. Incorporation of multimodal LLMs and large annotated video-language datasets are proposed for robust prompt representation.
• Efficiency: Lightweight backbones, optimized memory modules, and hardware-aware pruning are important for deployment in cost-constrained or real-time applications.
• Security: The backdoor vulnerability highlighted by BadVSFM necessitates research into robust countermeasures for encoder and decoder disentanglement under attack (Zhang et al., 26 Dec 2025).
• Expansion to New Modalities: Multi-modal cues and continuous prompt spaces (e.g., polarization, medical imaging) are promising directions for extending VSFM universality (Guo et al., 2024, Xu et al., 30 Jul 2025).

Research in VSFMs is converging toward models that are universal, efficient, robust to prompt modality and domain, and aware of emerging security risks. These directions will define the foundation for the next generation of video segmentation systems.