Video Reasoning Segmentation (VRS)

• Video Reasoning Segmentation (VRS) is a video understanding task that fuses natural language queries with video inputs to produce temporally coherent binary mask sequences.
• The task involves complex challenges such as temporal evidence selection, spatial grounding, and maintaining mask continuity amid occlusion and dynamic scenes.
• Advanced VRS systems leverage end-to-end multimodal fusion, chain-of-thought reasoning, and reinforcement learning to enhance object segmentation accuracy and robustness.

Video Reasoning Segmentation (VRS) is a video understanding task in which a model receives a video and a free-form natural-language query, then produces a temporally coherent sequence of binary masks for the referred object or objects. In representative formulations, the input is a video $V=\{I_t\}_{t=1}^T$ and a query $x$, and the output is a mask sequence $M=\{m_t\}_{t=1}^T$ rather than a single-frame localization. What distinguishes VRS from conventional referring video segmentation is that the query may invoke dynamics, causality, temporal interactions, implicit attributes, or commonsense conditions such as “Which runner is most likely to win?” or “What caused the accident?” (Li et al., 2 Dec 2025). Recent research treats VRS as a joint problem of language-grounded temporal evidence selection, spatial grounding, and temporally stable propagation, with solutions ranging from end-to-end MLLM-based architectures to explicit reasoning chains, training-free agentic systems, and reinforcement-learning policies over multi-step decisions (Luo et al., 23 Mar 2026, Dai et al., 5 Jun 2026).

1. Problem formulation and task boundaries

A common formalization maps $(V,x)\rightarrow M$, where $V=\{I_t\}_{t=1}^T$ is the input video, $x$ is a free-form query, and $M=\{m_t\}_{t=1}^T$ is the output mask sequence (Li et al., 2 Dec 2025). In another equivalent statement, given a video sequence of $T$ frames $I=\{I_1,\dots,I_T\}$ with each $I_i\in\mathbb{R}^{H\times W\times 3}$ and a textual query $x$0 describing implicit or complex properties of a target object, the goal is to produce masks

$x$1

such that each $x$2 highlights the pixels belonging to the query-specified object in frame $x$3, with temporal consistency across the video (Zhu et al., 2 Mar 2026).

The task is consistently described as more demanding than standard RVOS. RVOS is framed as segmenting objects in videos based on textual queries, whereas VRS emphasizes implicit instructions, temporal logic, causality, and abstract reasoning. ReVSeg characterizes the task as requiring a model not merely to “paint a mask” on every frame but to think through an abstract, often causal or commonsense query; TrajSeg defines it as extending reasoning segmentation from static images to videos; and Veason-R1 stresses implicit instructions such as “the wolf that first emerges and then hides behind the tree” (Li et al., 2 Dec 2025, Luo et al., 23 Mar 2026, Gong et al., 15 Aug 2025).

Three recurring technical difficulties appear across the literature. The first is temporal ambiguity: the decisive evidence may occur only in a small subset of frames, so naive uniform or similarity-based sampling can miss the relevant moment (Wen et al., 8 May 2026). The second is spatial precision under dynamic appearance change, occlusion, and crowded multi-object scenes, even after a plausible keyframe has been found (Hong et al., 28 Mar 2026). The third is trajectory perception and identity maintenance: human instructions frequently refer to behavior over time rather than static appearance, and masks must remain coherent rather than flickering or drifting across frames (Luo et al., 23 Mar 2026).

Evaluation is usually reported with region similarity $x$4, contour accuracy $x$5, and their mean. One standard definition is

$x$6

where $x$7 denotes ground truth and $x$8 is a boundary-based $x$9 score (Li et al., 2 Dec 2025). This metric choice reflects the dual demand of object-level overlap and boundary fidelity.

2. Benchmarks, datasets, and task variants

A central benchmarking milestone was ViLLa’s VideoReasonSeg benchmark, constructed to promote research in what that work described as an unexplored area. VideoReasonSeg contains 1,934 video–instruction–mask triplets, split into train/val/test = 1000/400/534, with implicit text queries and evaluation through VIS-style segmentation AP/AR and multiple-choice QA accuracy (Zheng et al., 2024). This benchmark made explicit the link between language reasoning and video segmentation, and it anchored later comparisons with Ref-YouTube-VOS, Ref-DAVIS17, MeViS, and ReVOS.

Later work stabilized a broader benchmark suite. ReVSeg evaluates on ReasonVOS and ReVOS as reasoning-centric datasets, and on Ref-DAVIS17, Ref-YouTube-VOS, and MeViS as referring VOS datasets (Li et al., 2 Dec 2025). SDAM likewise reports results on Ref-YouTubeVOS, Ref-DAVIS17, MeViS, ReasonVOS, and ReVOS, reflecting a now-standard practice of testing both reasoning-heavy and conventional RVOS settings within a single protocol (Zhu et al., 2 Mar 2026). Refer-Agent adds GroundMoRe to this landscape, explicitly targeting five challenging benchmarks (Jiang et al., 3 Feb 2026).

A significant extension is predictive or foresight-oriented VRS. FeVOS introduces Foresight Expression Video Object Segmentation, in which the query refers to a future event and the required visual answer is the mask of the object in the observed frames. The dataset contains 968 video clips, 14,525 foresight expressions, 2,904 chain-of-thought annotations, 37,412 per-frame annotations, and 1,419 distinct target objects (Lan et al., 24 Jun 2026). This shifts the problem from “reason over observed evidence” to “reason about future object involvement from observed evidence,” which broadens VRS toward anticipation.

Another source of terminology variation is task naming. Some papers use “ReasonVOS” or “reasoning VOS” for benchmark-specific settings, while others use “VRS” as the umbrella term. The acronym is also overloaded outside this topic: RobotSeg uses “VRS” to denote a video robot segmentation dataset with 2,812 videos and 138,707 frames, which is distinct from Video Reasoning Segmentation as defined in the VLM/MLLM literature (Mei et al., 28 Nov 2025). This suggests that current terminology is not yet fully standardized across subfields.

3. Architectural paradigms

A first major design family is end-to-end fusion inside a single multimodal model. ViLLa augments a frozen LLM and a video foundation encoder with a Context Aggregation Module and a Video-Frame Decoder, training with joint text and mask losses on fixed-length clips of $M=\{m_t\}_{t=1}^T$0 frames (Zheng et al., 2024). VRS-HQ replaces a monolithic segmentation token with frame-level <SEG> tokens and a temporal-level <TAK> token, then applies Temporal Dynamic Aggregation and Token-driven Keyframe Selection before SAM2-based mask decoding and propagation (Gong et al., 15 Jan 2025). TrajSeg moves further toward explicit trajectory modeling through bidirectional text–trajectory alignment, a Frame-Level Content Integration module, and a unified mask generator that handles key and non-key frames in one end-to-end trainable structure (Luo et al., 23 Mar 2026). VIRST unifies global video reasoning and pixel-level mask prediction through Spatio-Temporal Fusion and a Temporal Dynamic Anchor Updater, conditioning a SAM-2-based decoder on fused tokens and anchor-memory cues in a single forward pass (Hong et al., 28 Mar 2026).

A second design family decomposes the task into explicit stages. ThinkVideo uses an MLLM keyframe selector with zero-shot chain-of-thought prompting, an off-the-shelf reasoning image segmenter, and a SAM2 video processor (Kao et al., 24 May 2025). VideoSeg-R1 uses a hierarchical text-guided frame sampler, a reasoning model that outputs explicit spatial cues inside > and <answer> blocks, and a segmentation–propagation stage with SAM2 and XMem (Xu et al., 20 Nov 2025). ReVSeg instantiates VRS as a two-turn dialogue with one VLM, unfolding semantics interpretation, temporal evidence selection, and spatial grounding before handing the result to a tracker such as SAM2 (Li et al., 2 Dec 2025). RCoT-Seg formalizes a similar split as Temporal Video Reasoning and Keyframe Target Perception, with the first stage selecting and reselecting keyframes and the second performing high-resolution grounding plus SAM2 propagation (Wen et al., 8 May 2026).

A third design family is training-free or nearly training-free reasoning over off-the-shelf components. PARSE-VOS is described as a hierarchical, coarse-to-fine, training-free framework powered by LLMs: it parses natural-language queries into structured semantic commands, generates candidate trajectories through a spatio-temporal grounding module, and identifies the correct target via coarse motion reasoning followed, when needed, by fine pose verification (Zhao et al., 6 Sep 2025). DecAF casts VRS as video QA, extracts MLLM attention rollout maps, refines them via contrastive object-background fusion and complementary video-frame fusion, and uses attention-guided SAM2 prompting (Han et al., 22 Oct 2025). SDAM introduces Adaptive Object Memory and spatio-temporal decoupling, using only off-the-shelf MLLM, SAM, and Cutie modules without fine-tuning (Zhu et al., 2 Mar 2026). AgentRVOS shifts the temporal reasoning burden away from early keyframe choice by letting SAM3 first generate full-video candidate mask tracks, then having an MLLM iteratively prune them as Accepted, Rejected, or Uncertain (Jin et al., 24 Mar 2026).

These families imply a central methodological axis. One end emphasizes unified token-level fusion and end-to-end optimization; the other emphasizes decomposition into semantically meaningful subproblems such as keyframe selection, grounding, tracking, reflection, or pruning. A plausible implication is that VRS system design is increasingly organized around where reasoning should occur: inside latent token spaces, inside explicit language traces, or over external object-level evidence.

4. Explicit reasoning traces and reinforcement learning

Reinforcement learning has become a prominent mechanism for making intermediate reasoning decisions explicit and optimizable. VideoSeg-R1 describes itself as the first framework to introduce reinforcement learning into video reasoning segmentation, formulating the reasoning stage as a Markov Decision Process in which a multimodal LLM generates chain-of-thought text and JSON spatial cues, with rewards for format, segmentation accuracy, diversity, negative-point distance, and a soft length penalty controlled by an instance-level difficulty score (Xu et al., 20 Nov 2025). Veason-R1 then develops a “think before you segment” paradigm by curating 5.8k chain-of-thought samples for supervised initialization and optimizing four rewards: format compliance, temporal localization, spatial alignment, and unified consistency after SAM2 propagation (Gong et al., 15 Aug 2025).

ReVSeg pushes this explicitness further by decomposing VRS into semantics interpretation, temporal evidence selection, and spatial grounding, then training the two-turn VLM policy with a GRPO variant of PPO. Its reward is

$M=\{m_t\}_{t=1}^T$1

where $M=\{m_t\}_{t=1}^T$2 measures JSON correctness, $M=\{m_t\}_{t=1}^T$3 rewards informative keyframe choice, and $M=\{m_t\}_{t=1}^T$4 rewards spatial grounding via bounding-box IoU (Li et al., 2 Dec 2025). The same line of work argues that decomposition alone provides a large boost and that RL on top of the decomposed chain yields further gains.

RCoT-Seg combines a CoT-SFT warm start with GRPO in an agentic keyframe selector that can answer “incorrect,” forbid the current frame, and re-query up to $M=\{m_t\}_{t=1}^T$5 rounds (Wen et al., 8 May 2026). VideoSEG-O3 generalizes the RL formulation to multi-turn interaction: the policy emits exploration actions of the form <select> [s,e] [k] and a termination action <answer> [SEG], and it introduces SEG-aware logit calibration so that the log-probability of [SEG] is augmented with pixel-wise mask confidence (Dai et al., 5 Jun 2026). FeVOS-R1 adapts a two-stage SFT-plus-RL pipeline to foresight expressions, freezing SAM2 and fine-tuning only the MLLM via LoRA under an IoU-based reward (Lan et al., 24 Jun 2026).

Across these systems, explicit reasoning is no longer merely auxiliary explanation. It becomes a structured intermediate variable: a keyframe timestamp, a temporal interval, a box-point prompt set, a JSON object, or a multi-turn trace. This suggests a shift from latent “reasoning tokens” toward auditable decision sequences that can be rewarded directly for correctness, brevity, and segmentation utility.

5. Training-free, zero-shot, and agentic approaches

Training-free VRS has advanced from simple prompt-based baselines to sophisticated pipelines that use MLLMs primarily for reasoning and external segmentation models for precise masks. ThinkVideo is entirely training-free and reports $M=\{m_t\}_{t=1}^T$6 on ReasonVOS and $M=\{m_t\}_{t=1}^T$7 on Refer-DAVIS-17 with a GPT-4o selector, using CoT keyframe selection, Seg-Zero for image segmentation, and SAM2 for propagation (Kao et al., 24 May 2025). DecAF remains fully training-free as well, reporting $M=\{m_t\}_{t=1}^T$8 on Ref-DAVIS with Qwen2.5VL-7B and $M=\{m_t\}_{t=1}^T$9 on ReasonVOS by refining attention maps before attention-guided SAM2 prompting (Han et al., 22 Oct 2025).

Refer-Agent recasts RVOS/VRS as collaboration among four specialized agents—frame selection, intent analysis, object grounding, and mask generation—plus a two-stage Chain-of-Reflection. In zero-shot mode with Ovis2.5-9B, it reports JF̅ of $(V,x)\rightarrow M$0 on Ref-YouTube-VOS, $(V,x)\rightarrow M$1 on MeViS, $(V,x)\rightarrow M$2 on ReasonVOS, $(V,x)\rightarrow M$3 on GroundMoRe, and $(V,x)\rightarrow M$4 on ReVOS Overall (Jiang et al., 3 Feb 2026). SDAM follows a different training-free path, separating spatial localization from temporal propagation and storing key-object memory in an Adaptive Object Memory module; it reports $(V,x)\rightarrow M$5 on Ref-YouTubeVOS, $(V,x)\rightarrow M$6 on Ref-DAVIS17, $(V,x)\rightarrow M$7 on MeViS, $(V,x)\rightarrow M$8 on ReVOS, and $(V,x)\rightarrow M$9 on ReasonVOS (Zhu et al., 2 Mar 2026).

SteerSeg identifies attention misalignment as the main bottleneck in frozen-LVLM grounding and uses learnable soft prompts with CoT-derived attributes to steer attention before SAM2-based tracklet construction. With a Qwen2.5VL-7B backbone, it reports Ref-YouTube-VOS $V=\{I_t\}_{t=1}^T$0, Ref-DAVIS $V=\{I_t\}_{t=1}^T$1, ReasonVOS $V=\{I_t\}_{t=1}^T$2, ReVOS(Reasoning) $V=\{I_t\}_{t=1}^T$3, and MeViS $V=\{I_t\}_{t=1}^T$4 (Cheraghian et al., 14 May 2026). AgentRVOS instead starts from exhaustive perception: SAM3 generates candidate mask tracks over the full spatio-temporal extent, and the MLLM reasons over those tracks with iterative spatio-temporal pruning. With Qwen3-VL-8B-Thinking, it reports $V=\{I_t\}_{t=1}^T$5 on MeViS, $V=\{I_t\}_{t=1}^T$6 on ReVOS Overall, and $V=\{I_t\}_{t=1}^T$7 on ReasonVOS (Jin et al., 24 Mar 2026).

These systems collectively dispute a common misconception that high-performing VRS must require large-scale supervised fine-tuning. Multiple papers explicitly report competitive or state-of-the-art training-free results. At the same time, the literature also documents the fragility of this regime: SDAM notes sensitivity to prompt design and single-keyframe failure, Refer-Agent notes latency growth from reflection rounds, and AgentRVOS notes the cost of SAM3 plus repeated MLLM calls (Zhu et al., 2 Mar 2026, Jiang et al., 3 Feb 2026, Jin et al., 24 Mar 2026).

6. Performance trends, limitations, and emerging frontiers

The upper end of current performance is split across several model families. VideoSeg-R1 reports $V=\{I_t\}_{t=1}^T$8 J&F on Ref-YouTube-VOS, $V=\{I_t\}_{t=1}^T$9 on Ref-DAVIS17, $x$0 on MeViS, $x$1 on ReVOS, and $x$2 gIoU on ReasonSeg (Xu et al., 20 Nov 2025). RCoT-Seg reports $x$3 J&F on ReVOS referring, $x$4 on ReVOS reasoning, $x$5 on Reason-VOS, $x$6 on DAVIS17, $x$7 on Ref-YouTube-VOS, and $x$8 on MeViS (Wen et al., 8 May 2026). VIRST reports ReVOS overall $x$9, MeViS $M=\{m_t\}_{t=1}^T$0, Ref-YT-VOS $M=\{m_t\}_{t=1}^T$1, and Ref-DAVIS17 $M=\{m_t\}_{t=1}^T$2, while emphasizing large-motion, occlusion, and reasoning-oriented scenarios (Hong et al., 28 Mar 2026). TrajSeg reports J&F of $M=\{m_t\}_{t=1}^T$3, $M=\{m_t\}_{t=1}^T$4, and $M=\{m_t\}_{t=1}^T$5 on Ref-YouTube-VOS, Ref-DAVIS, and MeViS respectively, and $M=\{m_t\}_{t=1}^T$6 / $M=\{m_t\}_{t=1}^T$7 on ReVOS referring / reasoning (Luo et al., 23 Mar 2026).

Despite these gains, limitations recur with notable regularity. Long videos remain difficult: TrajSeg currently processes at most 10 frames, VIRST is capped to $M=\{m_t\}_{t=1}^T$8, and Refer-Agent explicitly notes that very long videos could strain CLIP and MLLM prompts (Luo et al., 23 Mar 2026, Hong et al., 28 Mar 2026, Jiang et al., 3 Feb 2026). Occlusion and reappearance are still open problems, especially when propagation modules rather than end-to-end temporal decoders carry the identity signal (Zhu et al., 2 Mar 2026, Hong et al., 28 Mar 2026). Multi-instance ambiguity and rare compositional queries also remain failure modes: ThinkVideo notes grouping errors for many nearly identical instances, AgentRVOS notes difficulty with highly abstract queries or rare nouns, and VIRST reports failure cases under extremely cluttered scenes and multi-step semantic constraints (Kao et al., 24 May 2025, Jin et al., 24 Mar 2026, Hong et al., 28 Mar 2026).

Efficiency has emerged as a parallel frontier. FastReasonSeg departs from direct visual-token reasoning by constructing a digital twin representation with SAM-2 masks, DepthAnything2 depth statistics, and OWLv2 semantic labels, then distilling teacher reasoning chains into smaller students via supervised and reinforcement fine-tuning. On JiTBench and RVTBench it reports state-of-the-art reasoning segmentation performance, and its 0.6B student reaches 7.79 FPS with 2.1 GB memory consumption, while the 1.7B model reaches 4.07 FPS with 4.2 GB (Shen et al., 15 Nov 2025). This suggests that compact, structured scene abstractions may be an important route for deploying VRS outside datacenter settings.

The task frontier is also expanding conceptually. FeVOS extends VRS from present-time reasoning to pre-decisive spatio-temporal reasoning about future events, and FeVOS-R1 reports $M=\{m_t\}_{t=1}^T$9 on FeVOS versus $T$0 for Sa2VA* and $T$1 for GLUS*, with a reported $T$2 point gain from RL refinement over its SFT-only counterpart (Lan et al., 24 Jun 2026). A plausible implication is that VRS is evolving from “segment the object described by what has happened” toward a broader family of language-conditioned spatio-temporal decision problems, including anticipation, active evidence acquisition, and multi-turn reasoning over object tracks.

In that broader view, VRS is no longer a narrow variant of RVOS. It is becoming a unifying testbed for explicit multimodal reasoning, temporal evidence selection, and dense video prediction, with distinct subtraditions in end-to-end token fusion, reasoning-grounding decomposition, agentic training-free inference, and RL-optimized decision chains.