MOSS-Video: Process-Supervised Video Benchmark

• MOSS-Video is a benchmark for dynamic state prediction in videos, providing annotated object states and detailed reasoning traces for process supervision.
• It employs dual GPT-4o–based annotation pipelines to generate both coarse- and fine-grained state triplets, enabling effective reinforcement-learning fine-tuning.
• The benchmark addresses process inconsistencies by aligning generated and reference reasoning using Subsequence Dynamic Time Warping, leading to significant gains in reasoning accuracy.

MOSS-Video is a benchmark and training/evaluation substrate for process-supervised video temporal reasoning introduced with MOSS-ChatV. It is defined as “a video state prediction dataset derived from ShareGPT4Video”; each sample is annotated with object states and corresponding reasoning traces, enabling models to predict future states conditioned on visual context. In that formulation, MOSS-Video is used to study dynamic state prediction, temporally grounded explanation, and process consistency in video multimodal LLMs (MLLMs), with a training split used for reinforcement-learning fine-tuning and a held-out split reserved for evaluation (Tao et al., 25 Sep 2025).

1. Definition and conceptual scope

MOSS-Video is organized around video state prediction with explanation. Given a video $V$ and a query $q$ specifying a target object, the model must identify the object, infer its current or imminent state, and provide a temporally grounded explanation. This makes the benchmark explicitly object-centric and temporally predictive rather than purely descriptive. The paper positions dynamic state prediction as a proxy for video reasoning ability: if a model can accurately predict future states of objects from past visual evidence, it must model temporal dynamics and causal regularities rather than static appearance alone (Tao et al., 25 Sep 2025).

The benchmark is motivated by a specific failure mode in existing video MLLMs: process inconsistency. In the paper’s terminology, this is the situation in which intermediate reasoning drifts from video dynamics even when the final answer is correct. MOSS-Video addresses that problem by attaching reference reasoning traces to prediction questions, so evaluation and training can act not only on answer correctness but also on whether the reasoning trajectory follows the temporal structure of the video.

A compact comparison given in the paper places MOSS-Video against other video benchmarks along three axes:

Dataset	Understanding	Reasoning	Prediction
ViTiB	✓	✓	×
NeXT-QA	✓	×	×
Video-R1-CoT-165K	×	✓	×
MOSS-Video (train)	✓	✓	✓
MOSS-Video (test)	✓	✓	✓

This positioning is central to the benchmark’s identity. A plausible implication is that MOSS-Video is designed less as a general-purpose video QA corpus than as a controlled substrate for training and measuring temporally coherent reasoning.

2. Dataset construction and annotation schema

MOSS-Video is derived from ShareGPT4Video. The construction pipeline uses two parallel GPT-4o–based annotation streams. The first is a coarse-grained state pipeline that produces triplets of the form $$\langle \text{Object}, \text{State}, \text{Timestamp} \rangle$$. The second is a fine-grained change pipeline that produces more detailed state or transition triplets, including interval descriptions such as $$\langle \text{Object}, \text{change description}, \text{start time}, \text{end time} \rangle$$. The two streams are then integrated into a unified temporal model of object dynamics, from which question–answer pairs probing predicted future states are automatically generated (Tao et al., 25 Sep 2025).

Each sample includes a video clip, a prediction question, a ground-truth answer $a_{\text{gt}}$, and a reference reasoning trace $T_{\text{ref}}$. The reference trace is not an auxiliary commentary layer; it is part of the benchmark’s formal supervision. The paper describes these traces as temporally grounded references, because the underlying state schema is tied to timestamps or time ranges extracted from caption-derived annotations. This grounding is what later enables rule-based process supervision.

The dataset is partitioned into a training split of 11,654 samples from 1,218 unique videos and a test split of 2,836 samples from 479 unique videos. Average video length is 27.73 seconds in training and 28.21 seconds in test. These statistics matter because the benchmark is long enough to contain multiple state transitions while remaining practical for reinforcement-learning-based fine-tuning.

The reasoning traces themselves are serialized into step sequences by an NLP tool $\mathcal{N}$:

$$Seq_{\text{gen}} = \{ g_1, \dots, g_m \} = \mathcal{N}(T_{\text{gen}})$$

$$Seq_{\text{ref}} = \{ r_1, \dots, r_n \} = \mathcal{N}(T_{\text{ref}})$$

Each step corresponds to a reasoning unit, typically describing a state or transition associated with a segment of the video. This stepwise structure is the basis for process alignment during training.

3. Task formulation and role in MOSS-ChatV

Within MOSS-ChatV, MOSS-Video is used as both a benchmark and a reinforcement-learning substrate. The model receives a video $V$, a question $q$0, the ground-truth answer $q$1, and the reference reasoning trace $q$2. During GRPO fine-tuning, the policy model $q$3 samples a group of candidate outputs,

$q$4

and each output is assigned a reward composed of answer correctness, format adherence, and process alignment (Tao et al., 25 Sep 2025).

The answer-level task is straightforward: predict the future or imminent state of the queried object. The distinctive element is that the output is expected to include reasoning text in a specified format, so the benchmark can supervise not only what the model predicts but how it arrives there. This is why the paper treats MOSS-Video as a process-supervised benchmark rather than a standard QA set.

The train/test split is used asymmetrically. The training split serves reinforcement-learning fine-tuning, while the held-out test split is reserved for evaluation of both answer accuracy and reasoning quality. This design is important because it separates the use of reasoning traces for optimization from the use of similar traces for held-out assessment.

A recurring point in the paper is that MOSS-Video supports efficient process supervision without an auxiliary learned reward model. This is possible because the dataset contains explicit reference reasoning traces; the reward can therefore be computed algorithmically rather than by training a separate judge model.

4. Process Reasoning Reward and temporal alignment

The central training mechanism built on MOSS-Video is the Process Reasoning Reward (PRR). After segmenting both generated and reference traces into step sequences, the framework computes a ROUGE-based similarity for each pair of steps:

$q$5

and defines the step distance

$q$6

These distances populate a matrix $q$7 whose entries measure divergence between generated and reference reasoning steps. The alignment itself is performed with Subsequence Dynamic Time Warping (SDTW), which aligns the entire reference sequence to some subsequence of the generated trace rather than forcing full-sequence matching. The final alignment cost is

$q$8

The process reward is then defined as an exponential transform of that distance:

$q$9

The total reward for a sampled response is

$$\langle \text{Object}, \text{State}, \text{Timestamp} \rangle$$0

Here the accuracy reward is binary, the format reward is binary, and the process reward is continuous. The paper’s interpretation is that lower alignment distance implies better reasoning-process agreement with the temporally grounded reference and therefore higher reward.

A key methodological point is that SDTW treats the annotated trace as a minimal gold standard. This allows the model to include extra exploratory reasoning before or after the aligned subsequence without being heavily penalized. The paper contrasts this with naive DTW, which can lead to reward hacking by encouraging overly short reasoning traces. This suggests that MOSS-Video is not merely a dataset with rationales; it is constructed so that the rationales can function as stable reinforcement-learning signals.

5. Evaluation results and empirical significance

On the held-out MOSS-Video test split, the paper reports MOSS-ChatV at 86.6–87.2% accuracy: Table 4 lists 86.6%, Table 5 lists 86.62%, and the abstract states 87.2% (Tao et al., 25 Sep 2025). In the same comparison, Qwen2.5-VL is reported at 67.0%, LLaVA-OneVision at 48.1%, TinyLLaVA-3B at 65.9%, TinyLLaVA-3B + PRR at 82.5%, Video-UTR at 58.9%, VideoChat-R1 at 70.8%, VideoChat-R1-thinking at 70.1%, and Video-R1 at 73.3. These figures place MOSS-Video as the primary benchmark on which the gains from PRR are demonstrated.

Ablation results sharpen that interpretation. Starting from Qwen2.5-VL-7B at 67.00 on MOSS-Video test, supervised fine-tuning on MOSS-Video reaches 71.44, GRPO in a Video-R1-style setup reaches 73.26, GRPO without PRR reaches 84.17, and GRPO with PRR reaches 86.62. Factually, this shows that reinforcement learning itself provides a large gain, while the process reward adds a further improvement on top of RL.

The paper also evaluates reasoning quality with GPT-4o-as-judge on MOSS-Video samples. MOSS-ChatV is reported at 0.79 for Reasoning–Answer Consistency, 7.23 for Repetitiveness, 7.59 for Logical Coherence and Knowledge Accuracy, and 7.35 for Relevance to Video Content. In the authors’ interpretation, these scores indicate more consistent and stable reasoning traces than those produced by comparison systems.

Another empirical observation concerns response-length dynamics. With SDTW-based PRR on MOSS-Video, response lengths initially fluctuate and then converge to a stable range; without process supervision, lengths remain unstable; with naive DTW, models shorten reasoning drastically. This result is used to argue that MOSS-Video supports not only higher answer accuracy but also better-controlled reasoning-process optimization.

6. Distinctions from related benchmarks and similarly named usages

MOSS-Video should be distinguished from MOSEv2, which is a dataset for video object segmentation in complex scenes rather than a benchmark for process-supervised temporal reasoning. MOSEv2 consists of 5,024 videos and over 701,976 high-quality masks for 10,074 objects across 200 categories, and it emphasizes severe occlusions, crowding, smaller objects, adverse weather, low-light scenes, multi-shot sequences, camouflaged objects, non-physical targets, and knowledge-dependent scenarios (Ding et al., 7 Aug 2025).

This distinction matters because some later technical reports connect their methods to “MOSS-Video” in a broader sense of complex video understanding. SeC, for example, studies long-term grounding memory and concept-aware memory for MOSEv2’s semi-supervised VOS setting, while STSeg combines finetuned SAM2, finetuned TMO, and an Adaptive Pseudo-labels Guided Model Refinement Pipeline for the MOSE challenge track (Gao et al., 23 Sep 2025, Song et al., 11 Apr 2025). Those systems address multi-object segmentation under occlusion, distractors, and long-term reappearance, not temporally grounded reasoning-trace supervision.

A separate source of ambiguity is the informal solar-physics use of “moss video” for Hi-C movies of solar active-region moss. In that literature, Hi-C’s 193 Å movies provide “a resolved, time-resolved view of moss as a forest of thin, moving threads,” and rapid brightenings in such movies have been interpreted as signatures of heating events associated with reconnection in overlying hot coronal loops (Morton et al., 2014, Testa et al., 2013). That usage is unrelated to the MOSS-Video benchmark in video reasoning, despite the lexical overlap.

In present machine-learning usage, the most precise meaning of MOSS-Video is therefore the object-centric video state prediction benchmark with annotated reasoning traces introduced for MOSS-ChatV. Its defining feature is not merely that it contains videos and answers, but that it supplies temporally grounded reference processes that can be used directly for reinforcement-learning-based supervision of reasoning trajectories.