SeC: Concept-Driven VOS Framework
- SeC is a novel, concept-driven approach for semi-supervised video object segmentation that integrates memory-based matching with progressive LVLM-based concept guidance.
- It enhances segmentation performance under drastic visual variations by using pixel-level association memory across up to 22 frames and a sparse keyframe bank for robust concept construction.
- On the SeCVOS benchmark, SeC delivers an 11.8-point improvement over SAM 2.1, demonstrating its effectiveness in semantically complex, multi-shot video scenarios.
SeC, short for Segment Concept, is a concept-driven framework for semi-supervised video object segmentation (VOS) that replaces a purely appearance-matching view of segmentation with the progressive construction and utilization of high-level, object-centric representations. Built on SAM 2.1-large, SeC combines pixel-level association memory with LVLM-based concept guidance, and is introduced together with SeCVOS, a benchmark designed for semantically complex, multi-shot video scenarios. The framework is motivated by the observation that conventional VOS systems remain strong under appearance continuity but weak under identity continuity when videos exhibit drastic visual variation, heavy occlusion, distractors, viewpoint changes, object disappearance and reappearance, and abrupt scene transitions. On SeCVOS, SeC reports an 11.8-point improvement over SAM 2.1, and the benchmark itself is structured to emphasize semantic discontinuity rather than only temporal mask propagation (Zhang et al., 21 Jul 2025).
1. Problem setting and conceptual motivation
SeC is formulated in the standard semi-supervised VOS setting: the target object mask is given in the first frame, and the system must segment that same object in every following frame. The paper argues that dominant VOS pipelines remain centered on memory matching: they store features from previous frames, match current-frame features against memory, and propagate masks over time. This regime is effective when the target remains visually similar and videos are temporally smooth, but it degrades in videos with severe appearance changes, heavy occlusion, distractors with similar appearance, scene transitions or shot changes, object disappearance and reappearance, viewpoint changes, and dynamic camera motion (Zhang et al., 21 Jul 2025).
The conceptual claim behind SeC is that prior systems predominantly learn surface similarity rather than an object-level concept. Even methods that incorporate object memory or semantic cues are described as remaining close to instance-level features. SeC instead frames VOS as requiring a representation of which entity is the target despite changing appearance and context. In that sense, the framework shifts VOS from “find the object by matching current appearance to stored appearances” toward “construct a concept of the object from multiple observed frames, then use that concept to interpret new frames.” The paper repeatedly distinguishes SeC from both a purely memory-based VOS system and a pure LVLM segmentation system: it is a hybrid concept-aware VOS system rather than a text-prompt-dependent language-assisted method (Zhang et al., 21 Jul 2025).
2. System architecture and progressive concept construction
SeC is built on SAM 2.1-large and adds two major modules: a pixel-level association memory and an LVLM-based concept guidance branch. The first reuses SAM 2 memory attention and is strengthened for longer temporal range; the second uses a sparse bank of representative frames to derive a semantic representation of the target object. A lightweight scene-adaptive activation strategy decides when the current frame can be handled by memory alone and when concept reasoning is necessary (Zhang et al., 21 Jul 2025).
The pixel-level association memory extends temporal positional encoding to support a wider temporal window, up to 22 frames. Following SAM2Long, it applies an object-aware filtering strategy and keeps only frames with non-zero occlusion scores, so the memory is constructed from frames in which a visible object is present. This component addresses standard temporal correspondence and remains the efficient default mechanism in ordinary frames.
The concept branch is organized around a sparse keyframe bank. It starts from the first annotated frame and adds a new frame only when it is significantly different from existing keyframes and its segmentation is confident. The bank stores the initial frame permanently and augments it with a FIFO buffer of recent representative keyframes. This design is explicitly meant to present the LVLM with a diverse set of target manifestations rather than redundant adjacent frames.
For semantic integration, the paper uses InternVL 2.5 as the LVLM and fine-tunes it with LoRA. The LVLM input is a temporally ordered sequence of reference keyframes, the current query frame, and a special <SEG> token appended at the end. Following the paper’s adaptation of the LISA idea, the hidden state of <SEG> acts as a semantic accumulator and becomes the object concept guidance vector. The visual prompting strategy marks the target using a green contour rather than alpha-blended mask overlays, with the stated motivation that the contour highlights the target without occluding visual detail (Zhang et al., 21 Jul 2025).
A defining claim of SeC is that the concept is progressively constructed online. As more diverse target observations enter the keyframe bank, the LVLM receives a richer history and the representation becomes more complete. The paper tests this directly by contrasting no concept construction, online construction, and an offline mode that first processes the whole video and then re-segments with the final concept. On SeCVOS, the reported scores are 62.2 J&F for no concept construction, 70.0 J&F for online construction, and 71.8 J&F for offline construction. The improvement from online to offline is used to support the paper’s claim that richer concept accumulation improves segmentation (Zhang et al., 21 Jul 2025).
3. Inference pipeline, adaptive activation, and feature fusion
At inference time, SeC processes frames sequentially. Each new frame first undergoes scene change detection. If the frame is judged simple, segmentation uses only the memory-based branch; if the frame is judged complex, SeC activates concept reasoning, fuses semantic guidance with current-frame features, and then updates memory and the keyframe bank (Zhang et al., 21 Jul 2025).
The scene-change detector is intentionally lightweight and HSV-based. It computes 2D color histograms over hue and saturation for the current and previous frames, normalizes them, and evaluates Bhattacharyya distance. Concept reasoning is triggered when the distance exceeds 0.35. The paper presents this as a deliberate computational tradeoff: adjacent frames are usually easy, so invoking an LVLM on every frame would be expensive and largely redundant.
When concept reasoning is triggered, the LVLM outputs the concept vector from the <SEG> token. This vector is fused with current-frame spatial features through a lightweight cross-attention module, producing concept-enhanced spatial features. These are then pointwise added to the memory-enhanced features and passed to the SAM 2 mask decoder. The system therefore combines a semantic prior with feature correspondence rather than replacing correspondence.
The adaptive policy produces a low concept-guidance frequency. The paper reports a concept-guidance ratio of 7.4% on SeCVOS and 1.0% on SA-V. It further states that enabling concept guidance on fewer than 10% of frames already gives strong performance, with only limited gains beyond that. This is a central implementation claim: SeC seeks robustness on semantically difficult frames while avoiding continuous LVLM inference (Zhang et al., 21 Jul 2025).
The formalism in the paper remains intentionally light. It does not provide full equations for memory update, cross-attention, or custom losses. The explicit quantitative implementation parameters it does provide include the Bhattacharyya distance threshold = 0.35, temporal memory support of up to 22 frames, LVLM image resize to , and learning rates of for the memory module stage and for the LVLM fine-tuning stage. The reported evaluation metrics are the standard VOS measures , , and their average (Zhang et al., 21 Jul 2025).
4. Training protocol and implementation details
SeC is not a zero-shot system. It is trained in two stages, with the memory branch and concept branch optimized separately while most of the underlying SAM 2 components remain frozen (Zhang et al., 21 Jul 2025).
In Stage 1, SeC trains the pixel-level association memory using 2k videos from the SA-V training set. The selected videos are those with the highest number of scene transitions detected by SceneDetect. Each training sample uses 24 shuffled frames randomly sampled per video. Only the memory attention module is trainable; all other components are frozen. The schedule is 40 epochs, batch size 64, and learning rate .
In Stage 2, the framework fine-tunes the LVLM concept guidance module. The base LVLM is InternVL 2.5, fine-tuned with LoRA. The data consist of about 190k object instances from the SA-V training set, restricted to instances with at least three visible masks. Each training sample includes 1 to 7 reference frames, 0 to 2 distractor frames with incorrect annotations, and one non-overlapping query frame. All images are resized to 448 × 448, the target is marked by a green contour, and the SAM 2 parameters remain frozen. The schedule is 3 epochs, batch size 64, and learning rate . Training is reported on 8 NVIDIA A800 GPUs.
The paper states that “the loss function remains consistent with that of SAM 2.” It does not introduce a new optimization objective. This is consistent with the framing of SeC as an architectural extension rather than a new loss-driven formulation.
5. SeCVOS benchmark
SeC is introduced together with SeCVOS, the Semantic Complex Scenarios Video Object Segmentation benchmark, which is designed to evaluate VOS under conditions requiring high-level conceptual reasoning and robust semantic understanding. The benchmark contains 160 manually annotated multi-shot videos, with average duration 29.36 s, average number of scenes 4.26, and disappearance rate 30.2%. The videos are selected from Shot2Story and supplemented with additional videos crawled from YouTube (Zhang et al., 21 Jul 2025).
The benchmark construction pipeline filters candidate videos using criteria that include a minimum duration of 20 seconds and semantically meaningful content following the Shot2Story filtering strategy. GPT-4o is then used to analyze videos and identify target objects that appear frequently and unambiguously across scenes. Initial object masks are generated using SAM 2, followed by multiple rounds of manual correction to produce the final annotations. The supplementary description notes diversity across indoor, outdoor, and animated settings, with targets including humans, vehicles, and animals.
SeCVOS is defined by three scenario characteristics: highly discontinuous frame sequences, frequent reappearance of objects across disparate scenes, and abrupt shot transitions and dynamic camera motion. The benchmark’s comparative statistics are used to argue that its difficulty is not merely long duration or disappearance. The paper emphasizes that the striking statistic is 4.26 scenes/video, far above DAVIS, YTVOS, MOSE, SA-V, and LVOS, whose reported average scene counts range from 1.03 to 1.47. This is the empirical basis for the claim that SeCVOS stresses semantic discontinuity more directly than earlier VOS benchmarks (Zhang et al., 21 Jul 2025).
6. Empirical performance, ablations, positioning, and limitations
The headline result is SeC’s performance on SeCVOS. The paper reports the following overall J&F scores:
| Method | Overall J&F |
|---|---|
| XMem | 48.4 |
| DEVA | 49.7 |
| Cutie-base | 52.7 |
| SAM 2.1 | 58.2 |
| SAMURAI | 62.2 |
| SAM2.1Long | 62.3 |
| SeC | 70.0 |
This yields the paper’s stated +11.8 gain over SAM 2.1. The breakdown by scene complexity is equally central. On videos with no scene change, SeC scores 84.2 versus 79.4 for SAM 2.1, a +4.8 gain. On single scene change, it scores 69.6 versus 58.5, a +11.1 gain. On multi scene change, it scores 67.5 versus 52.4, a +15.1 gain. The monotonic increase in the margin is used to support the paper’s claim that concept reasoning matters most as semantic discontinuity increases (Zhang et al., 21 Jul 2025).
The method also remains competitive on standard VOS benchmarks. Reported scores for SeC are 82.7 on SA-V val, 81.7 on SA-V test, 86.5 on LVOS v2 val, 75.3 on MOSE val, 91.3 on DAVIS 2017 val, and 88.6 on YTVOS 2019 val. Relative to SAM 2.1, the gains are +4.1 on SA-V val, +2.1 on SA-V test, +2.4 on LVOS v2 val, +0.8 on MOSE val, and +0.7 on DAVIS val, while YTVOS remains roughly on par.
The ablation studies separate the contribution of memory and concept guidance. Without either module, the base configuration gives 78.6 J on SA-V and 58.2 J on SeCVOS. Adding pixel-level association alone raises these to 82.4 J and 62.2 J. Adding concept guidance on top yields 82.7 J and 70.0 J. The paper therefore interprets concept guidance as marginal on ordinary videos but decisive on semantically difficult videos. A separate LVLM-scale ablation reports J&F values of 68.4, 69.5, 70.0, and 70.3 for 1B, 2B, 4B, and 8B models respectively, and concludes that gains saturate beyond 4B.
Efficiency is reported on one NVIDIA A800 GPU. On SeCVOS, SeC reaches 14.8 s throughput versus 22.0 s for SAM 2, with a concept-guidance ratio of 7.4%. On SA-V, SeC reaches 18.1 s versus 22.0 s0 for SAM 2, with a concept-guidance ratio of 1.0%. The paper presents this as a moderate computational cost for a large gain on difficult videos.
The method is also positioned against prior work. Compared with traditional memory-based VOS, SeC adds concept-level semantic priors rather than relying only on feature propagation. Compared with object-level memory approaches such as Cutie and ISVOS, it is described as moving beyond instance features or detector-derived semantics toward high-level concept construction. Compared with language-assisted or text-driven methods such as LISA, VISA, UFO, VideoLISA, Sa2VA, and GLUS-A, SeC does not require textual inputs or outputs during segmentation and uses the LVLM implicitly as a visual concept extractor rather than as a text interface (Zhang et al., 21 Jul 2025).
The limitations are stated directly. The scene change detector is simple, relying only on HSV histogram distance and a fixed threshold. SeCVOS is semantically hard but not ultra-long, being shorter on average than LVOS. Concept coverage is still viewpoint-limited: the supplementary highlights a failure case in which the interior view of a sailboat differs drastically from previous external views, causing mismatch with the constructed concept. Finally, LVLM reasoning adds computation relative to plain SAM 2. These limitations are consistent with the paper’s broader claim: SeC improves robustness by introducing progressive semantic identity construction, but the concept remains bounded by the diversity and reliability of the evidence accumulated online (Zhang et al., 21 Jul 2025).
