Veason-R1: RL-Enhanced Video Reasoning Segmentation

• Veason-R1 is a large vision-language model that combines reinforcement learning with Chain-of-Thought initialization to generate explicit reasoning trajectories and keyframe cues.
• It decouples temporal reasoning from mask propagation by first predicting a keyframe timestamp and bounding boxes before invoking an external segmentation model.
• Empirical benchmarks show that Veason-R1 improves temporal localization, spatial accuracy, and robustness against hallucinations compared to prior methods.

Veason-R1 is a reinforcement-learning–enhanced large vision-LLM for Video Reasoning Segmentation (VRS), a setting in which a video sequence $$\mathcal{V}_I \in \mathbb{R}^{T \times 3 \times H \times W}$$ and a free-form reasoning instruction $Q_{\text{txt}}$ must be converted into a sequence of segmentation masks $\mathcal{M} \in \mathbb{R}^{T \times H \times W}$ for the referred object across time. It is designed to “think before it segments”: rather than mapping video and language directly to masks through opaque segmentation tokens, it generates an explicit reasoning trajectory, predicts a keyframe timestamp and keyframe bounding boxes, and then invokes SAM2 to propagate these spatial cues into full-video masks. The model is built on Qwen2.5-VL, initialized through Chain-of-Thought (CoT) supervision as Veason-SFT, and then refined with Group Relative Policy Optimization (GRPO) using a holistic reward that jointly scores format compliance, temporal localization, spatial alignment, and spatiotemporal consistency (Gong et al., 15 Aug 2025).

1. Task formulation and conceptual shift

Video Reasoning Segmentation differs from classic referring video object segmentation in the nature of its language input. The query is not merely an explicit description such as a color or category label; it may encode human intent, commonsense, and temporal logic, including order of events, appearance and disappearance, speed, interactions, and “critical moments” in the video. Veason-R1 therefore treats the problem as one of reasoning-heavy grounding rather than direct per-frame mask generation (Gong et al., 15 Aug 2025).

The model formalizes the task as a decomposition. Given $\mathcal{V}_I$ and $Q_{\text{txt}}$, it first predicts a keyframe timestamp

$T_k \in \{1,2,\dots,T\},$

intended to identify the frame where the referred object is most visually salient. It then predicts a set of keyframe boxes

$\mathcal{B}_{T_k} = \{b_i\}_{i=1}^{N_k},$

where each $b_i = (x_1,y_1,x_2,y_2)$. These structured outputs are later converted into a mask sequence by SAM2. This decomposition is central to the method’s claim of interpretability: the intermediate variables are visible and inspectable, rather than being compressed into a latent <SEG> token (Gong et al., 15 Aug 2025).

A common misconception is that Veason-R1 is a dense segmentation model in the narrow sense. It is not. The LVLM itself predicts textual reasoning and structured spatial-temporal outputs; dense masks are produced by an external segmenter. Another misconception is that CoT alone is the method’s defining contribution. The paper instead presents a two-stage system in which CoT initialization establishes structured reasoning trajectories, and GRPO fine-tuning reshapes those trajectories with task-level rewards (Gong et al., 15 Aug 2025).

2. Architecture and inference pipeline

Veason-R1 is built on Qwen2.5-VL. Frames are sampled from the video, resized to $560 \times 560$, and encoded as visual tokens. Temporal information is introduced through symbolic timestamp tokens such as \<0s> and \<3s>, so the LLM receives a sequence of timestamped frame representations rather than an unstructured image set. The text query is tokenized and processed jointly with these visual tokens inside a unified transformer (Gong et al., 15 Aug 2025).

The output is explicitly structured. The model generates a reasoning trajectory inside > ... </think>, followed by an <answer> ... </answer> block containing a JSON-like dictionary with keyframe_timestamp and bbox_2d_list. The segmentation stage is external: a frozen SAM2 consumes the matched keyframe boxes and propagates them over the sampled video frames to obtain

$$\mathcal{M} = \text{SAM2}(\mathcal{B}'_{T_k}, \mathcal{V}_s, T_k),$$

where $Q_{\text{txt}}$0 denotes the matched boxes after Hungarian matching (Gong et al., 15 Aug 2025).

This architecture differs from prior LVLM-based VRS systems that encode object semantics into one or a small number of specialized segmentation tokens. In those systems, reasoning and mask prediction are entangled, and the inference process exposes little about why a frame or region was selected. Veason-R1 instead decouples reasoning from segmentation. The paper characterizes this as a “single-system” LVLM that performs temporal reasoning and spatial grounding itself, while delegating mask propagation to SAM2 (Gong et al., 15 Aug 2025).

The temporal modeling strategy is language-centric rather than based on an explicit video transformer or 3D convolution. Temporal order is represented through timestamped frame tokens, and the reasoning trajectory is expected to verbalize temporal evolution. This suggests that the model’s temporal competence is strongly mediated by instruction-following and cross-modal sequence modeling rather than by a dedicated motion backbone.

3. CoT initialization and the Veason-SFT stage

The supervised initialization stage produces Veason-SFT. Its training set contains 5.8k curated CoT samples. The dataset is built by sampling one pseudo keyframe per video from the top-5 frames with the largest target area, then prompting Seed1.5-VL to generate multi-step reasoning that analyzes the sampled frames, justifies the selected frame, and localizes the referred objects with bounding boxes. Different CoT templates are used to encourage diversity and avoid overly rigid reasoning patterns (Gong et al., 15 Aug 2025).

The model is trained autoregressively on the full output sequence, including both reasoning and answer:

$Q_{\text{txt}}$1

where $Q_{\text{txt}}$2 is the prompt containing sampled frames, timestamps, and query, and $Q_{\text{txt}}$3 is the generated token sequence. The implementation uses LLaMA-Factory and LoRA, with LoRA rank 8, learning rate $Q_{\text{txt}}$4, gradient accumulation 8, cosine annealing, and 1 epoch of fine-tuning (Gong et al., 15 Aug 2025).

The role of this stage is not merely to teach formatting. It is described as bridging video-level semantics and frame-level grounding: the model learns to explain what happens across sampled frames, identify which frame best satisfies the query, and emit keyframe boxes in a structured answer. Veason-SFT thereby acquires the ability to produce coherent CoT reasoning, basic keyframe selection, and coarse object localization (Gong et al., 15 Aug 2025).

Model	Training regime	Reported capability
Veason-SFT	Qwen2.5-VL fine-tuned with LoRA on 5.8k CoT samples	Structured CoT, basic keyframe selection, coarse bounding boxes
Veason-R1	Veason-SFT further fine-tuned by GRPO on 10k ReVOS samples	Refined temporal localization, spatial accuracy, and spatiotemporal consistency

A plausible implication is that Veason-SFT supplies the exploration prior for later RL. The paper’s formulation supports this interpretation because GRPO treats the full reasoning-and-answer sequence as the policy output, and the initial supervised distribution determines what kinds of trajectories are available for reward-driven refinement.

4. GRPO fine-tuning and the holistic reward

Veason-R1 is obtained by GRPO-based reinforcement learning on top of Veason-SFT. For each prompt $Q_{\text{txt}}$5, the model samples $Q_{\text{txt}}$6 candidate responses, computes a scalar reward for each, and normalizes rewards within the group:

$Q_{\text{txt}}$7

The policy is thus the LVLM mapping from timestamped frames and query to the full textual trajectory; the “actions” are the generated tokens, especially those that determine the reasoning chain, the keyframe timestamp, and the box list (Gong et al., 15 Aug 2025).

The total reward is

$Q_{\text{txt}}$8

with $Q_{\text{txt}}$9. Each term corresponds to a distinct failure mode in VRS (Gong et al., 15 Aug 2025).

The format compliance reward $\mathcal{M} \in \mathbb{R}^{T \times H \times W}$0 enforces the presence of <think> ... and <answer> ... </answer>, and requires the answer to contain keyframe_timestamp and bbox_2d_list. The temporal localization reward $\mathcal{M} \in \mathbb{R}^{T \times H \times W}$1 scores whether the predicted keyframe is one in which the target is maximally visible:

$\mathcal{M} \in \mathbb{R}^{T \times H \times W}$2

where $\mathcal{M} \in \mathbb{R}^{T \times H \times W}$3 is the foreground area of the referred object on sampled frame $\mathcal{M} \in \mathbb{R}^{T \times H \times W}$4 (Gong et al., 15 Aug 2025).

Spatial grounding is handled through a Hungarian-matched IoU reward. Given predicted boxes $\mathcal{M} \in \mathbb{R}^{T \times H \times W}$5 and ground-truth boxes $\mathcal{M} \in \mathbb{R}^{T \times H \times W}$6, the cost matrix is

$\mathcal{M} \in \mathbb{R}^{T \times H \times W}$7

and after Hungarian matching the reward is

$\mathcal{M} \in \mathbb{R}^{T \times H \times W}$8

This normalizes by the larger of the predicted and ground-truth counts, so both missing objects and spurious objects reduce reward (Gong et al., 15 Aug 2025).

The most distinctive component is the unified consistency reward $\mathcal{M} \in \mathbb{R}^{T \times H \times W}$9. After retaining matched keyframe boxes, SAM2 propagates them over the sampled frames, and the resulting masks are scored against ground truth:

$\mathcal{V}_I$0

This term couples temporal selection and spatial alignment into a video-level criterion. In effect, a keyframe-box prediction is good only if it remains coherent under propagation (Gong et al., 15 Aug 2025).

Training uses 10,000 ReVOS samples, the VERL framework, global batch size 16, learning rate $\mathcal{V}_I$1, KL regularization coefficient $\mathcal{V}_I$2, max response length 1024 tokens, gradient clipping with max norm 1.0, and 1 epoch of RL fine-tuning. The paper argues that this reward design also acts as an anti-hallucination mechanism, because wrong objects, wrong regions, and malformed outputs are all penalized either directly or through SAM2-propagated mask IoU (Gong et al., 15 Aug 2025).

5. Benchmarks and reported performance

Veason-R1 is evaluated on ReVOS, ReasonVOS, and MeViS. ReVOS contains both a referring subset with more explicit queries and a reasoning subset with more implicit, reasoning-heavy queries. ReasonVOS contains 91 videos with an average length of 105 frames and 253 long-form queries involving causal reasoning, hypothetical or conditional scenarios, and abstract descriptions. MeViS is a motion-centric, multi-object dataset, and Veason-R1 is tested there in zero-shot mode (Gong et al., 15 Aug 2025).

Evaluation follows standard VRS metrics: region similarity $\mathcal{V}_I$3, contour accuracy $\mathcal{V}_I$4, and their average $\mathcal{V}_I$5. The paper also reports a robustness score $\mathcal{V}_I$6 for hallucination resistance (Gong et al., 15 Aug 2025).

Benchmark	Veason-R1-7B	Reported comparison
ReVOS overall	$\mathcal{V}_I$7, $\mathcal{V}_I$8, $\mathcal{V}_I$9, $Q_{\text{txt}}$0	+1.3 $Q_{\text{txt}}$1 over VRS-HQ-13B overall
ReVOS reasoning subset	$Q_{\text{txt}}$2, $Q_{\text{txt}}$3, $Q_{\text{txt}}$4	+2.2 $Q_{\text{txt}}$5 over prior SOTA on reasoning subset
ReasonVOS	$Q_{\text{txt}}$6, $Q_{\text{txt}}$7, $Q_{\text{txt}}$8	+10.0 $Q_{\text{txt}}$9 over GLUS-7B
MeViS zero-shot	$T_k \in \{1,2,\dots,T\},$0, $T_k \in \{1,2,\dots,T\},$1, $T_k \in \{1,2,\dots,T\},$2	+0.9 $T_k \in \{1,2,\dots,T\},$3 over prior art

On ReVOS, Veason-R1-7B reports $T_k \in \{1,2,\dots,T\},$4, $T_k \in \{1,2,\dots,T\},$5, and $T_k \in \{1,2,\dots,T\},$6 on the referring subset, and $T_k \in \{1,2,\dots,T\},$7, $T_k \in \{1,2,\dots,T\},$8, and $T_k \in \{1,2,\dots,T\},$9 on the reasoning subset. The overall score is $\mathcal{B}_{T_k} = \{b_i\}_{i=1}^{N_k},$0, $\mathcal{B}_{T_k} = \{b_i\}_{i=1}^{N_k},$1, $\mathcal{B}_{T_k} = \{b_i\}_{i=1}^{N_k},$2, with robustness $\mathcal{B}_{T_k} = \{b_i\}_{i=1}^{N_k},$3. The 3B variant reaches overall $\mathcal{B}_{T_k} = \{b_i\}_{i=1}^{N_k},$4, which the paper describes as competitive with 13B token-based methods, and reports $\mathcal{B}_{T_k} = \{b_i\}_{i=1}^{N_k},$5, compared with earlier values such as 14.5 for VISA-13B, 17.9 for GLUS-7B, and 18.9 for VRS-HQ-13B (Gong et al., 15 Aug 2025).

On ReasonVOS, GLUS-7B reports $\mathcal{B}_{T_k} = \{b_i\}_{i=1}^{N_k},$6, $\mathcal{B}_{T_k} = \{b_i\}_{i=1}^{N_k},$7, and $\mathcal{B}_{T_k} = \{b_i\}_{i=1}^{N_k},$8, whereas Veason-R1-7B reports $\mathcal{B}_{T_k} = \{b_i\}_{i=1}^{N_k},$9, $b_i = (x_1,y_1,x_2,y_2)$0, and $b_i = (x_1,y_1,x_2,y_2)$1. The gain is therefore +8.5 in $b_i = (x_1,y_1,x_2,y_2)$2, +11.4 in $b_i = (x_1,y_1,x_2,y_2)$3, and +10.0 in $b_i = (x_1,y_1,x_2,y_2)$4 (Gong et al., 15 Aug 2025).

On MeViS in zero-shot evaluation, Veason-R1-7B reaches $b_i = (x_1,y_1,x_2,y_2)$5, $b_i = (x_1,y_1,x_2,y_2)$6, and $b_i = (x_1,y_1,x_2,y_2)$7, exceeding GLUS-7B at 51.3 and VRS-HQ-13B at 50.9. The abstract also states that the model exhibits robustness to hallucinations of “+8.8 R,” which is presented as part of its empirical summary (Gong et al., 15 Aug 2025).

6. Interpretability, failure modes, and place in the R1 ecosystem

Veason-R1’s most visible contribution is interpretability. During inference, the model exposes a step-by-step <think> trace explaining what appears in the sampled frames, how the query is interpreted, why a particular frame is chosen as the keyframe, and where the referred objects are localized. Qualitative cases include vehicles partially occluded by elephants, slippers touching a puppy’s feet, a warthog appearing only near the end of a video, a girl situated by the window at the end, and a “dancing bride” disambiguated from a groom in ReasonVOS. In these examples, the paper attributes errors of prior methods such as VISA to failure in fine-grained spatial relation reasoning, late-frame localization, or abstract query interpretation (Gong et al., 15 Aug 2025).

The paper also identifies explicit limitations. In one failure case, the reasoning correctly describes “the tiger closest to the camera” but the masks are misaligned, indicating a mismatch between textual reasoning and visual grounding. In another, the query “the fastest ship” leads the model to select a slower ship, suggesting limited sensitivity to motion-related concepts such as relative speed and acceleration. The appendix also notes that failures persist in very long, cluttered scenes with many similar objects and complex occlusions (Gong et al., 15 Aug 2025).

These limitations matter because they qualify a possible misunderstanding of explicit CoT. Exposed reasoning does not guarantee that the grounding stage is correct, nor that the reasoning process fully captures motion saliency. The method makes errors more auditable, but it does not eliminate cross-modal inconsistency. This suggests that interpretability and grounding fidelity, while related, remain distinct optimization targets.

Within the broader R1-style landscape, Veason-R1 belongs to a family of post-training methods that use CoT initialization and GRPO- or RLVR-style objectives to specialize large multimodal models. Vision-R1 applies criterion-driven, vision-guided RL to localization with deterministic visual feedback; VLN-R1 adapts GRPO-based reinforcement fine-tuning to continuous vision-language navigation from egocentric RGB video; Table-R1 studies inference-time scaling for table reasoning through distillation and RLVR; IAD-R1 uses PA-SFT and SC-GRPO for industrial anomaly detection; and Step-Audio-R1 introduces Modality-Grounded Reasoning Distillation for audio reasoning (Zhan et al., 23 Mar 2025, Qi et al., 20 Jun 2025, Yang et al., 29 May 2025, Li et al., 7 Aug 2025, Tian et al., 19 Nov 2025). Veason-R1 is distinctive within this group because it couples structured reasoning not merely to classification or localization outputs, but to a segmentation pipeline in which keyframe selection and box grounding are optimized jointly with video-level propagation consistency (Gong et al., 15 Aug 2025).

In that sense, Veason-R1 is not only a segmentation model but a concrete formulation of reasoning-supervised video grounding. Its central proposition is that segmentation quality in reasoning-heavy video tasks improves when the model first commits to an interpretable temporal-spatial explanation and only then segments. The reported gains on ReVOS, ReasonVOS, and zero-shot MeViS indicate that this decomposition is effective under the benchmarks studied, while the documented failure cases indicate that motion reasoning, grounding consistency, and long-video complexity remain open research problems (Gong et al., 15 Aug 2025).