RCoT-Seg: Reinforced Chain-of-Thought Video Segmentation

• The paper introduces a two-stage framework that decouples temporal reasoning from spatial segmentation to enhance video analysis.
• It employs an agentic keyframe selection module alongside a high-resolution SAM2 model to achieve precise and consistent mask propagation.
• Reinforcement learning combined with supervised fine-tuning yields significant performance gains over traditional MLLM-based approaches.

Searching arXiv for the specified RCoT-Seg paper and closely related video reasoning segmentation work. arXiv search query: "(Wen et al., 8 May 2026) RCoT-Seg Reinforced Chain-of-Thought for Video Reasoning and Segmentation" RCoT-Seg, introduced in "RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation," is a framework for Video Reasoning Segmentation (VRS) that segments target objects in videos from implicit instructions conveying human intent and temporal logic. It formulates VRS as a two-stage "video-of-thought" pipeline that explicitly separates temporal reasoning from spatial perception: Temporal Video Reasoning (TVR) selects and verifies a keyframe through an agentic chain-of-thought procedure, and Keyframe Target Perception (KTP) performs high-resolution segmentation on that keyframe and propagates masks across the sequence with SAM2-based methods. The method is positioned against MLLM-based approaches that predict masks with a [SEG] token after frame selection by simple sampling or an auxiliary MLLM, arguing that such designs suffer from limited supervision, narrow-scope keyframe choices, brittle localization, and weakened holistic temporal understanding in complex multi-object scenes (Wen et al., 8 May 2026).

1. Problem formulation and scope

RCoT-Seg addresses VRS with input

$$V=\{I_t\}_{t=1}^T,\qquad I_t\in\mathbb{R}^{3\times H\times W},$$

and a natural-language query $Q$. The output is a binary mask sequence $\{M_t\}_{t=1}^T$, where each $M_t\in\{0,1\}^{H\times W}$ identifies the queried target object at pixel level. The stated objective is to learn a multimodal policy that both performs temporal reasoning to locate a keyframe $z_{\mathrm{key}}$ and produces pixel-accurate masks in all frames by segmenting and propagating from that keyframe (Wen et al., 8 May 2026).

The notation used in the framework makes the decomposition explicit. The video frames are $V=\{I_1,\dots,I_T\}$; the text query is $Q$; the model-generated compact video description is $x_{\mathrm{vid}}$; the candidate keyframe index is $z_{\mathrm{sel}}\in\{1,\dots,T\}$; and the final verified or re-selected keyframe index is $z_{\mathrm{key}}$. On the selected keyframe, the system predicts object cues $Q$0, where each $Q$1 is a bounding box and each $Q$2 is a point. The propagated mask sequence is written as

$Q$3

This factorization is central to the method’s scope. Rather than treating temporal localization and segmentation as a single monolithic decoding problem, RCoT-Seg separates them into temporally grounded decision making and spatially precise mask generation. This suggests a deliberate architectural response to the mismatch between language-driven temporal logic and frame-level segmentation fidelity.

2. Two-stage architecture

RCoT-Seg consists of two coupled stages. In Stage 1, Temporal Video Reasoning generates a global video description $Q$4 and selects a keyframe through an agentic keyframe selector. In Stage 2, Keyframe Target Perception conditions on the verified keyframe, the query, and the video description to predict coarse object cues, then feeds those cues to a frozen SAM2 model for segmentation and propagation (Wen et al., 8 May 2026).

The pipeline is summarized by four equations:

$Q$5

$Q$6

$Q$7

and

$Q$8

Within this design, TVR supplies a temporally grounded abstraction of the full video, while KTP converts that abstraction into spatial prompts for segmentation. The architecture therefore does not rely on heuristic frame sampling or an external selector. Instead, keyframe selection is internalized as part of the reasoning policy, and segmentation is delegated to a high-resolution promptable model. A plausible implication is that the framework treats temporal ambiguity and spatial ambiguity as related but non-identical subproblems, assigning each to a different optimization regime.

3. Temporal Video Reasoning and agentic keyframe selection

The TVR stage centers on an Agentic Keyframe Selection (AKS) module. A single MLLM receives a low-resolution version of all frames together with prompts, generates a candidate keyframe index $Q$9, and emits a chain-of-thought trace. A self-evaluation step then outputs $\{M_t\}_{t=1}^T$0, where “A” means keep the frame and “B” means re-select. If the answer is “B,” the model is re-invoked with updated prompts, up to a maximum iteration count $\{M_t\}_{t=1}^T$1. At inference, the method sets $\{M_t\}_{t=1}^T$2 as a trade-off between performance and latency (Wen et al., 8 May 2026).

The AKS module is initialized by a CoT-start procedure. The paper states that a 28K-sample CoT corpus is curated by prompting Qwen2.5-VL-7B, with 20K samples for Keyframe-Target Grounding and 8K for AKS. Supervised fine-tuning is then performed on Qwen2.5-VL-3B using cross-entropy on the styled > ...<answer>...</answer> format.

Reinforcement learning further refines the selector using GRPO. The policy is $\{M_t\}_{t=1}^T$3 over outputs including the CoT trace, answer, and optional re-selection; the state is

$\{M_t\}_{t=1}^T$4

and the action is the complete reasoning trace together with $\{M_t\}_{t=1}^T$5. The reward is defined as

$\{M_t\}_{t=1}^T$6

where $\{M_t\}_{t=1}^T$7 is a format reward and $\{M_t\}_{t=1}^T$8 is an answer-accuracy reward. The total maximum reward for AKS is

$\{M_t\}_{t=1}^T$9

GRPO uses normalized advantages

$M_t\in\{0,1\}^{H\times W}$0

and a clipped surrogate objective with KL regularization to a reference policy:

$M_t\in\{0,1\}^{H\times W}$1

where

$M_t\in\{0,1\}^{H\times W}$2

A common simplification is to view keyframe selection as a one-shot retrieval problem. RCoT-Seg rejects that assumption by making the selector agentic, self-evaluative, and iterative. The method’s self-evaluation loop is therefore not an auxiliary detail but a structural claim about how temporal reasoning should be performed in VRS.

4. Keyframe Target Perception and SAM2 propagation

The KTP stage operates on the verified high-resolution keyframe $M_t\in\{0,1\}^{H\times W}$3, resized to $M_t\in\{0,1\}^{H\times W}$4, together with $M_t\in\{0,1\}^{H\times W}$5 and $M_t\in\{0,1\}^{H\times W}$6. Its output is a reasoning trace $M_t\in\{0,1\}^{H\times W}$7 and a set of object cues

$M_t\in\{0,1\}^{H\times W}$8

where each cue consists of a bounding box and an interior point (Wen et al., 8 May 2026).

These cues are passed to a frozen SAM2 model:

$M_t\in\{0,1\}^{H\times W}$9

SAM2 then propagates the mask across the full sequence. The description in the paper states that SAM2 internally warps or inpaints masks into neighboring frames, implicitly enforcing temporal consistency, and that this yields smooth mask tracks without explicit optical-flow constraints.

The integration of KTP with TVR is expressed as

$z_{\mathrm{key}}$0

The video description $z_{\mathrm{key}}$1 provides global context for multi-object disambiguation, while the keyframe cues provide object-local prompts for segmentation. This makes the KTP stage a perception module in a narrow sense: it does not perform the temporal reasoning itself, but it consumes the temporally grounded context produced upstream.

This design also clarifies a frequent source of confusion. RCoT-Seg is not a direct video analogue of image-only reasoning segmentation methods that refine a single-frame mask from attention priors. By contrast, CR-Seg is an image reasoning segmentation framework with a coarse-to-refined design built around attention extraction and GLCoT for images, whereas RCoT-Seg is specifically formulated for videos, keyframe verification, and sequence-level SAM2 propagation (Cao et al., 2 Jun 2026).

5. Training curriculum and implementation

The training strategy is staged. Stage 1 is supervised CoT-SFT, intended to warm up structured reasoning. Stage 2 runs two separate GRPO procedures, one for AKS and one for KTG, and then merges them into the unified RCoT-Seg model (Wen et al., 8 May 2026).

The dataset composition is specified as follows. The CoT-SFT stage uses RCoT-Seg-SFT-28k, built from Ref-YouTube-VOS, MeViS, and ReVOS, and split into 20K KTG samples and 8K AKS samples. The GRPO stage uses AKS-GRPO with 4K samples and KTG-GRPO with 6K samples, both distinct from the SFT set.

The hyperparameters are also given explicitly. For CoT-SFT with LoRA, the learning rate is $z_{\mathrm{key}}$2, the number of epochs is $z_{\mathrm{key}}$3, the LoRA rank is $z_{\mathrm{key}}$4, and a cosine scheduler is used. For AKS-GRPO, the learning rate is $z_{\mathrm{key}}$5, batch size is $z_{\mathrm{key}}$6, the number of sampled responses per prompt is $z_{\mathrm{key}}$7, KL coefficient is $z_{\mathrm{key}}$8, and temperature is $z_{\mathrm{key}}$9, with reward weights $V=\{I_1,\dots,I_T\}$0 and $V=\{I_1,\dots,I_T\}$1. For KTG-GRPO, the learning rate is again $V=\{I_1,\dots,I_T\}$2, batch size is $V=\{I_1,\dots,I_T\}$3, $V=\{I_1,\dots,I_T\}$4, KL coefficient is $V=\{I_1,\dots,I_T\}$5, and temperature is $V=\{I_1,\dots,I_T\}$6, with reward weights $V=\{I_1,\dots,I_T\}$7 and $V=\{I_1,\dots,I_T\}$8.

This curriculum indicates that RCoT-Seg treats structured output formatting, temporal reasoning, and grounding as partially separable competencies. A plausible implication is that the method uses supervised learning to stabilize reasoning syntax and reinforcement learning to adapt selection and grounding behavior to task-aligned rewards.

6. Benchmarks, ablations, and empirical profile

RCoT-Seg is evaluated on both reasoning segmentation and referring video object segmentation benchmarks. On ReVOS, it reports $V=\{I_1,\dots,I_T\}$9 J&F, $Q$0 J, and $Q$1 F. On Reason VOS, it reports $Q$2 J&F. On referring benchmarks, it reports $Q$3 J&F on DAVIS17, $Q$4 J&F on Ref-YouTube-VOS, and $Q$5 J&F on MeViS. The reported gains over the compared baselines on the referring benchmarks are $Q$6 on DAVIS17, $Q$7 on Ref-YouTube-VOS, and $Q$8 on MeViS (Wen et al., 8 May 2026).

Dataset	Metric	RCoT-Seg-3B
ReVOS	J&F	66.9
ReVOS	J	61.6
ReVOS	F	66.9
Reason VOS	J&F	61.6
DAVIS17	J&F	80.0
Ref-YouTube-VOS	J&F	74.6
MeViS	J&F	56.7

The ablation results identify several contributions. Adding Video Description Gen (VDG), Keyframe Gen (KFG), and Agentic Keyframe Selection (AKS) yields up to $Q$9 J&F on Reason VOS. The best maximum iteration setting for AKS is $x_{\mathrm{vid}}$0. Jointly applying CoT-SFT and GRPO on both tasks outperforms partial schedules by $x_{\mathrm{vid}}$1 to $x_{\mathrm{vid}}$2 J&F. A unified RCoT-Seg architecture is reported as $x_{\mathrm{vid}}$3 and $x_{\mathrm{vid}}$4 J&F better on referring and reasoning subsets, respectively, than separated architectures. For propagation, replacing SAM2 with simple mask-flow reduces J&F by $x_{\mathrm{vid}}$5 to $x_{\mathrm{vid}}$6 points.

Qualitative examples in the paper are described as showing resistance to occlusion and multi-object distractions via AKS, accurate handling of small targets, and coherent mask tracks across complex scenes. In aggregate, these observations frame the empirical profile of RCoT-Seg as one in which temporal reasoning quality, especially keyframe verification and reselection, is tightly coupled to downstream spatial accuracy and inter-frame consistency.