VideoSEG-O3: Multi-turn Video Segmentation

• VideoSEG-O3 is a reasoning video object segmentation system that employs a multi-turn reinforcement learning framework with a multimodal LLM and SAM2 decoder to iteratively refine segmentation.
• It formulates segmentation as an episodic Markov Decision Process, enabling dynamic temporal exploration and precise keyframe selection for improved localization.
• The framework introduces SEG-aware logit calibration and a decoupled reasoning trace to optimize both pixel-level mask quality and temporal accuracy.

Searching arXiv for papers on VideoSEG-O3 and closely related reasoning/segmentation frameworks. VideoSEG-O3 is a Reasoning Video Object Segmentation system built on a multimodal LLM + SAM2 backbone and trained with a multi-turn reinforcement learning scheme. It is designed to “think before it segments”: rather than processing a fixed set of frames once and directly outputting a mask, it iteratively explores a video temporally and spatially, uses a multi-turn temporal–spatial chain-of-thought to refine critical intervals and keyframes, and terminates with a [SEG] token whose hidden state is decoded by SAM2 into a mask. The framework is presented as the first multi-turn reinforcement learning framework for RVOS, with explicit emphasis on coarse-to-fine evidence acquisition, SEG-aware logit calibration, and a specialized cold-start dataset, VTS-CoT (Dai et al., 5 Jun 2026).

1. Problem formulation and task definition

VideoSEG-O3 addresses Reasoning Video Object Segmentation, a setting in which the target object is specified by language that may involve temporal relations, causal or logical constraints, and motion-specific properties. In the formulation used by the framework, this differs from standard video object segmentation, which either assumes an initial mask or an implicit object definition, and from earlier referring video object segmentation settings that usually treat language as a relatively shallow referring expression. The intended difficulty arises from the need to jointly resolve when in the video the relevant event occurs, where the target is within selected frames, and how linguistic constraints align with visual evidence (Dai et al., 5 Jun 2026).

The framework is motivated by the claim that existing methods are limited to reasoning over fixed initial inputs and lack the capacity to actively acquire further visual evidence. VideoSEG-O3 therefore reformulates the problem as an episodic Markov Decision Process in which the model may either continue exploring by selecting a new temporal interval and keyframe or terminate and segment. This makes temporal exploration part of the policy rather than a fixed preprocessing step.

The state at turn $t$ is defined as

$s_t = (\mathcal{V}_t, Q, \mathcal{H}_{t-1}),$

where $\mathcal{V}_t$ is the current visual observation, $Q$ is the language query, and $\mathcal{H}_{t-1}$ is the reasoning history. The action space contains an exploration action realized through <select> and a termination action realized through <answer> followed by [SEG]. This formalization makes the coarse-to-fine search over long or intricate videos part of the learned inference procedure rather than a hand-crafted heuristic.

2. Architecture and multi-turn inference pipeline

The overall architecture combines Qwen3-VL as the multimodal LLM with SAM2 as the segmentation decoder. A special [SEG] token is appended to the multimodal sequence, and its hidden state is decoded by SAM2 into a mask. The system therefore retains the implicit segmentation interface of One-Token-Seg-All-style designs, but inserts it into a multi-turn loop in which the model repeatedly refines temporal and spatial evidence before committing to segmentation (Dai et al., 5 Jun 2026).

At the initial turn, the model receives globally sampled frames that provide low-resolution temporal context together with the language query. From this initial observation it produces a structured reasoning trace and an action of the form <select> with a temporal interval $[S_t,E_t]$ and a keyframe index $k_t$. A transition function then samples low-resolution frames within $[S_t,E_t]$ for motion context and a high-resolution keyframe at $k_t$ for spatial precision, yielding the next visual observation. The updated state is fed back to the LLM together with the accumulated reasoning history. This loop can continue for multiple turns until the model emits <answer> [SEG].

The coarse-to-fine character of the design is embodied in the sampling schedule. Global frames provide broad temporal coverage, intermediate interval frames narrow down the relevant segment, and the final keyframe provides the high-resolution view required for precise segmentation. During inference the framework uses at most 3 turns and at most 20 sampled frames, increases the number of spatial frames to 8, raises spatial resolution to $256 \times 28 \times 28$, and keeps keyframes at $s_t = (\mathcal{V}_t, Q, \mathcal{H}_{t-1}),$0. The best-performing post-processing strategy uses keyframe initialization plus bidirectional SAM2 propagation and [SEG]-driven propagation.

The inference logic can be summarized procedurally: global observation yields an initial interval proposal; subsequent interval proposals become shorter and more specific; the final [SEG] embedding is decoded on the current keyframe and then propagated through time. This suggests that VideoSEG-O3 treats temporal localization and pixel segmentation as coupled but non-identical problems: the former is solved by iterative evidence selection, the latter by a prompt-conditioned SAM2 decoder.

3. Reinforcement learning objective and SEG-aware logit calibration

A central claim of VideoSEG-O3 is that standard RL over text tokens is poorly aligned with implicit segmentation. In conventional GRPO-style optimization, the policy gradient is driven by the log-probability of generating the [SEG] token, but a high token probability does not imply that the token’s hidden state is useful for SAM2. The framework therefore introduces SEG-aware logit calibration, which augments the token-level policy with a mask-likelihood term so that segmentation quality enters the importance ratio directly (Dai et al., 5 Jun 2026).

The calibrated policy is defined as

$s_t = (\mathcal{V}_t, Q, \mathcal{H}_{t-1}),$1

where $s_t = (\mathcal{V}_t, Q, \mathcal{H}_{t-1}),$2 is the hidden state of [SEG]. The mask probability is defined by spatially averaging the pixel-wise likelihood assigned by the mask decoder: $s_t = (\mathcal{V}_t, Q, \mathcal{H}_{t-1}),$3 This term is then incorporated into the GRPO importance ratio: $s_t = (\mathcal{V}_t, Q, \mathcal{H}_{t-1}),$4

The reward is composite: $s_t = (\mathcal{V}_t, Q, \mathcal{H}_{t-1}),$5 $s_t = (\mathcal{V}_t, Q, \mathcal{H}_{t-1}),$6 checks format correctness of the chain-of-thought and action syntax. $s_t = (\mathcal{V}_t, Q, \mathcal{H}_{t-1}),$7 rewards temporal localization through keyframe correctness and temporal precision. $s_t = (\mathcal{V}_t, Q, \mathcal{H}_{t-1}),$8 rewards segmentation quality, including average IoU on sampled spatial frames and the superiority of the selected keyframe over the average frame. $s_t = (\mathcal{V}_t, Q, \mathcal{H}_{t-1}),$9 is progressive, rewarding improvement over previous turns.

The total training loss combines calibrated GRPO with auxiliary segmentation supervision: $\mathcal{V}_t$0 with $\mathcal{V}_t$1. Ablations show that vanilla GRPO degrades performance relative to the CoT cold-start baseline, whereas adding SEG-aware calibration reverses this degradation and further gains are obtained by adding both spatial and keyframe segmentation losses. This directly supports the paper’s argument that RL must optimize the implicit segmentation representation rather than the [SEG] token probability alone.

4. Decoupled thinking trace and the VTS-CoT dataset

VideoSEG-O3 does not use an undifferentiated chain-of-thought. Instead, it introduces a decoupled thinking trace with three explicit components: Temporal Understanding, Spatial Description, and Expression Parsing. In the intended format, the model first reasons about when the relevant event occurs, then about where the target object is within a selected frame or segment, and finally about how the language expression aligns with these observations before emitting either a new <select> action or <answer> [SEG] (Dai et al., 5 Jun 2026).

This decomposition is operationalized in VTS-CoT, a 6K-sample Video Temporal–Spatial CoT dataset built from ReVOS, MeViS, and Long-RVOS. Its construction uses a multi-stage synthesis pipeline. Qwen3-VL-235B performs temporal labeling and mask quality assessment, identifying a best interval, a best keyframe, and a mask-quality flag. ERNIE-45-VL then produces three non-overlapping candidate intervals subject to constraints such as

$\mathcal{V}_t$2

Finally, Qwen3-VL-Thinking synthesizes a structured two-phase reasoning trace in which the model first performs global analysis and then verifies or refines the selected segment.

Each VTS-CoT item contains explicit reasoning text and associated interval and keyframe annotations. The generated traces are designed to enforce “information isolation,” meaning that each step only reasons over the evidence that has already been requested. The dataset is used in the CoT cold-start stage to teach valid interval selection, valid keyframe indices, structured action emission, and the use of different resolutions for global temporal reasoning and local spatial discrimination.

The decoupled trace serves two functions. It makes the intermediate reasoning process more interpretable, and it regularizes exploration by preventing temporal, spatial, and linguistic evidence from collapsing into a single vague rationale. A plausible implication is that this decomposition reduces ambiguity in long or reasoning-heavy videos by explicitly separating the questions of when, where, and why.

5. Training pipeline, implementation, and empirical results

Training proceeds in three stages: supervised alignment, CoT cold-start, and reinforcement learning. Stage I uses LoRA tuning of the LLM together with a trainable SAM2 decoder on video QA, temporal grounding, image referring segmentation, and video referring segmentation datasets. Stage II uses VTS-CoT to teach structured multi-turn reasoning and valid <select>/<answer> behavior. Stage III performs full-parameter fine-tuning of the LLM and SAM2 decoder, with the visual encoder frozen, using GRPO with SEG-aware calibration on ReVOS and MeViS. RL uses $\mathcal{V}_t$3 NVIDIA H20 GPUs, takes about 24 hours for the 4B model and about 16 hours for the 2B model, uses a learning rate of $\mathcal{V}_t$4, a KL coefficient $\mathcal{V}_t$5, $\mathcal{V}_t$6 generations per prompt, 1000 steps, and a maximum of 3 turns (Dai et al., 5 Jun 2026).

Stage	Main role	Key settings
Stage I – SFT	Align [SEG] and temporal indexing	LoRA, max length 8192, LR $\mathcal{V}_t$7
Stage II – CoT cold-start	Teach multi-turn temporal–spatial reasoning	VTS-CoT, decoupled multi-resolution sampling
Stage III – RL	Optimize exploration and segmentation quality	GRPO, SEG-aware calibration, $\mathcal{V}_t$8

Quantitatively, the framework reports state-of-the-art results on 8 RVOS benchmarks. On referring video object segmentation, VideoSEG-O3-4B reaches $\mathcal{V}_t$9 J&F on MeViS, $Q$0 on Ref-YouTube-VOS, $Q$1 on Ref-DAVIS17, $Q$2 on Ref-SAV, and $Q$3 on Long-RVOS. On reasoning video object segmentation, it reaches $Q$4 J&F on ReVOS, $Q$5 on zero-shot ReasonVOS, and $Q$6 on GroundMoRe. On image-level tasks it remains competitive, including $Q$7 on RefCOCO val and $Q$8 on ReasonSeg val.

The staged ablation is especially informative. Stage I alone yields an average J&F of $Q$9; adding CoT cold-start raises this to $\mathcal{H}_{t-1}$0; adding RL raises it to $\mathcal{H}_{t-1}$1, while reducing average rounds from $\mathcal{H}_{t-1}$2 to $\mathcal{H}_{t-1}$3, a $\mathcal{H}_{t-1}$4 reduction. Additional ablations show that temporal reward is crucial, that episodic reward outperforms step-wise reward, and that keyframe initialization plus [SEG]-injected propagation improves over Sa2VA-style First-N post-processing. The paper also reports that RL improves keyframe mIoU and reduces the keyframe empty ratio, indicating that the policy learns to choose frames in which the target is present and informative.

6. Position within related research, strengths, and limitations

VideoSEG-O3 belongs to a broader line of work that combines multimodal reasoning, active evidence acquisition, and segmentation. Relative to VideoSeg-R1, which formulates video reasoning segmentation as hierarchical frame sampling followed by reasoning and SAM2/XMem propagation, VideoSEG-O3 moves to a multi-turn RVOS setting in which temporal interval selection and keyframe selection are recurrent decisions inside the policy rather than a single decoupled preprocessing stage (Xu et al., 20 Nov 2025). Relative to Video-o3, which studies native interleaved clue seeking for long-video multi-hop reasoning through iterative tool invocation, VideoSEG-O3 transfers the same coarse-to-fine exploration principle into dense pixel-level localization and couples it to a segmentation-aware RL objective (Zeng et al., 30 Jan 2026).

Several strengths follow from this design. The framework is particularly effective on long and reasoning-heavy videos, as reflected by its gains on Long-RVOS, ReVOS, and GroundMoRe. It also preserves fine-grained segmentation quality because the final decision is made through SAM2 conditioned on an implicit [SEG] representation rather than sparse geometric prompts. The RL stage further learns adaptive reasoning depth: after reinforcement learning, the model uses fewer turns on average while improving accuracy.

The limitations are explicit. Multi-turn interaction and multi-resolution sampling are more expensive than single-pass methods. Performance depends materially on prompt design and on the quality of the GPT-generated VTS-CoT traces. The paper also notes language sensitivity and remaining difficulty on certain categories, especially the counterfactual category on GroundMoRe. These constraints indicate that the framework’s current gains come from a carefully engineered interaction between CoT supervision, active evidence selection, and segmentation-aware RL rather than from a single universally robust mechanism.

Taken as a whole, VideoSEG-O3 demonstrates that dense vision–language grounding can benefit from the same agentic and reasoning-oriented shift already visible in multimodal RL systems. Its central contribution is not merely that it segments with a multimodal LLM, but that it treats segmentation as the endpoint of an explicitly structured decision process whose temporal search, linguistic interpretation, and pixel-level output are jointly optimized (Dai et al., 5 Jun 2026).