Det-SAM2: Detection-Driven Video Segmentation

• Det-SAM2 is a detection-driven, automated video instance segmentation framework that integrates cutting-edge object detection with promptable segmentation using SAM2.
• It features a modular pipeline combining object detection, prompt generation, memory-based segmentation propagation, and mask refinement to maintain constant compute and memory usage.
• The framework has proven effective in applications like AI refereeing in sports and zero-shot cell tracking in microscopy, achieving high accuracy and scalability across domains.

Det-SAM2 is a fully automated, detection-driven video instance segmentation framework built on top of Segment Anything Model 2 (SAM2), an image and video foundation model for promptable object segmentation. Det-SAM2 achieves high levels of automation by integrating state-of-the-art object detectors to generate prompts for SAM2, thus enabling unsupervised, scalable segmentation on streaming or long-form visual data with constant compute and memory usage. Recent research reports strong accuracy for Det-SAM2-style pipelines in both natural and biomedical domains, with application scenarios ranging from AI refereeing in sports to zero-shot cell tracking in time-lapse microscopy (Wang et al., 2024, Chen et al., 12 Sep 2025).

1. Architecture and Pipeline Composition

Det-SAM2 is structured as a modular pipeline that orchestrates object detection, prompt generation, segmentation, mask refinement, and downstream application-specific analysis. The high-level data flow can be summarized as follows:

1. Input Acquisition: Frames or video streams are buffered sequentially.
2. Object Detection Module: A detection backbone, typically YOLOv8 (CSPDarknet-based FPN/CSP structure), predicts bounding boxes and class scores per frame.
3. Prompt Generation: Detections are mapped to SAM2 prompt format, with each detection converted into both point and box prompts.
4. SAM2 Video Predictor: Prompts and buffered frames are processed by SAM2's video predictor, which maintains a memory bank of features and runs memory attention and mask decoding for refinement and propagation.
5. Post-processing and Application: Segmentation masks are dispatched to application-specific logic or further analytics.

A representative pseudocode excerpt for the core inference loop is:

sam2_state = SAM2VideoPredictor.init_state(...) for each frame f_t: append f_t to buffer if len(buffer) == K: detections = YOLOv8.detect(buffer at sampling interval) prompts = convert_boxes_to_sam2_prompts(detections) sam2_outputs, sam2_state = SAM2VideoPredictor.propagate_in_video( frames=buffer, prompts=prompts, inference_state=sam2_state, max_track=M ) SAM2VideoPredictor.release_old_frames(sam2_state, max_inference_state_frames) for t, masks in sam2_outputs: post_queue.put((t, masks)) clear(buffer)

(Wang et al., 2024)

2. Automatic Prompt Generation and Detection Backbone

Det-SAM2 delegates the automation of prompts to the object detection backbone. In the canonical implementation, YOLOv8 provides detections on RGB images. Each detection $(c_j, x_j, y_j, w_j, h_j)$ per frame is converted to a SAM2-compatible prompt:

$$\text{prompt}_j = \{ \text{point} = (x_j \pm \tfrac{w_j}{2},\; y_j \pm \tfrac{h_j}{2}), \text{box} = [x_j, y_j, w_j, h_j] \}$$

Detection backbones are typically trained on COCO-like datasets and can be fine-tuned to target domains (e.g., sports equipment or biological cells). Performance is closely coupled to detection quality; YOLOv8 recalls approximately 95% of object instances in plausible deployment settings (Wang et al., 2024, Chen et al., 12 Sep 2025).

3. Segmentation and Propagation with SAM2

Upon receiving prompts, SAM2's video predictor maintains a memory bank of image features and historical masks. Each new batch of frames and prompts triggers:

• Memory Attention: Aggregates contextual features across a sliding window in time.
• Mask Decoding: Produces updated masks using sigmoid-activated decoders conditioned on the incoming prompts and memory-augmented features.

Backward and forward propagation phases can be used for tracking or correction, as in cell tracking settings, leveraging mask, point, and box prompts for expansion, linking, or lineage construction (Chen et al., 12 Sep 2025). For formal measure, offline fine-tuning uses standard IoU-based losses:

$$\mathcal{L}_{\mathrm{mask}} = 1 - \frac{|\hat{S} \cap S|}{|\hat{S} \cup S|}$$

4. Computational and Memory Scalability

A core feature of Det-SAM2 is runtime and resource scalability for long or unbounded videos. It achieves this via:

• Chunked/Windowed Processing: Only a limited buffer (parameter $K$) and backward window ($M$) are held in memory, bounding resource usage.
• State Offloading: Features and inference state can be offloaded to CPU with minimal penalty (typically +22% latency).
• Explicit Memory Management: Non-blocking offloads, half-precision storage, dynamic frame releases, and cache management further contain VRAM/RAM within deterministic envelopes.

Representative usage metrics for Det-SAM2 (YOLOv8+SAM2, RTX 3090):

Throughput (fps)	VRAM (GB)	CPU RAM (GB)	Validation IoU	Det. Recall (%)
7	~12	~10	0.85–0.92	~95

(Wang et al., 2024)

5. Representative Applications

5.1 AI Refereeing in Billiards

In this application, YOLOv8 is fine-tuned to detect billiard balls and pockets; Det-SAM2 segments and tracks each, and post-processing extracts object trajectories for rule enforcement (goal, collision, rebound detection). Event-detection logic is implemented via simple geometric and temporal conditions on centroid sequences and velocity vectors. In field testing (60 s clip):

• Goal detection: 12/12 (100%)
• Collision: 85% recall, 90% precision
• Rebound: 88% recall, 92% precision

Domain adaptation includes unique object categories per ball, precise “near_pocket” geometry, and tuned motion thresholds (Wang et al., 2024).

5.2 Zero-Shot Cell Tracking in Microscopy

A similar paradigm is applied for biomedical imaging. For 2D or small 3D datasets, existing segmentation or rough watershed outputs generate initial masks, and SAM2 performs promptable linking and refinement across frames, with backward/forward propagation and lineage graph construction. For large-scale 3D+t, rough cell detections (Laplacian-of-Gaussian) seed prompt locations; local neighborhood patch extraction and cosine similarity over memory features enables greedy assignment and mitosis detection.

Performance on benchmarks such as the Cell Tracking Challenge (13 blind datasets):

• Linking accuracy (LNK): ≈0.984 (1st place on average)
• Biological accuracy (BIO): 0.862 (top-3 performance)
• Large-scale 3D: SEG ≈0.700 (3rd), TRA ≈0.92 (2nd)

(Chen et al., 12 Sep 2025)

6. Limitations and Open Challenges

Key limitations and open research directions for Det-SAM2 include:

• Instance ID Logic: Object detector category labels do not distinguish between identical instances; persistent unique ID assignment remains unsolved.
• Prompt Loss Propagation: Missed or incorrect detections propagate as degraded segmentation due to prompt reliance.
• Memory Span Tradeoff: Fixed window size ($M$) can miss long-range dependencies or corrections; unbounded memory would improve robustness.
• Parameter Selection: Optimal tuning of window ($K$), interval, and track lengths is currently manual.
• Large-Displacement Objects: In cell tracking, significant motion or ambiguous boundaries between frames can lead to tracking failure.

Future directions proposed include weight-based recurrent memory architectures (RWKV), joint fine-tuning of detector and SAM2, and a tracking layer atop detection backbones for persistent ID management (Wang et al., 2024, Chen et al., 12 Sep 2025).

7. Comparative Analysis and Outlook

Det-SAM2 pipelines represent a generalizable solution for prompt-driven video segmentation, leveraging foundation models and modern object detectors to automate previously manual segmentation tasks. Quantitatively, segmentation accuracy closely matches or surpasses standalone SAM2 when supplied with high-quality prompts and achieves performance competitive with state-of-the-art supervised approaches in both natural video and biomedical domains (Wang et al., 2024, Chen et al., 12 Sep 2025).

The operational design, based on self-prompting and bounded computation, provides a scalable and flexible template for downstream applications requiring unsupervised, robust instance segmentation on arbitrary-length video streams. Ongoing challenges relate to robust instance-level tracking, efficient memory management for long sequences, and adapting to detection or domain-specific peculiarities.

---

References:

• "Det-SAM2:Technical Report on the Self-Prompting Segmentation Framework Based on Segment Anything Model 2" (Wang et al., 2024)
• "Segment Anything for Cell Tracking" (Chen et al., 12 Sep 2025)