SLVMEval: Benchmark for Long-Video Evaluators

• The paper introduces SLVMEval, a benchmark that assesses evaluation systems using synthetically degraded long-video pairs to mimic real quality differences.
• It constructs test pairs from the Vript corpus by selectively degrading video segments and verifying the perceptible quality gap through crowdsourced human judgments.
• Empirical results reveal that while humans maintain high accuracy, most automatic evaluators struggle with temporal coherence and semantic fidelity over extended video durations.

Synthetic Long-Video Meta-Evaluation (SLVMEval) denotes a pairwise-comparison benchmark for meta-evaluating text-to-video evaluation systems in the long-video regime. Instead of assessing generators directly, it asks whether an evaluation system can reliably identify the better video in pairs of long human-made videos where one member has been synthetically degraded along a controlled quality aspect. In its benchmark instantiation, the framework targets videos up to 10,486 seconds, operates over ten degradation aspects, and retains only pairs whose quality difference is clearly perceptible to humans, thereby turning “easy-for-humans” long-video judgments into a testbed for the reliability of automatic evaluators (Matsuda et al., 31 Mar 2026).

1. Conceptual definition and problem setting

SLVMEval is explicitly a benchmark for evaluation systems, not for text-to-video models themselves. The underlying motivation is that modern text-to-video metrics and VLM-as-a-judge systems were largely designed and validated on short clips, whereas long-video generation is moving toward durations measured in tens of minutes and hours. The central question is therefore whether an evaluator that appears competent on short clips remains reliable when prompts are long, video durations are extreme, and failures involve long-range temporal structure rather than only frame-level defects (Matsuda et al., 31 Mar 2026).

The formal setup distinguishes three mappings. A text-to-video model is written as $g : \mathcal{P} \rightarrow \mathcal{V}$. For a prompt $p$, two models may produce videos $\{u_p, v_p\} = \{g(p), g'(p)\}$. A text-to-video evaluation system $e$ receives the prompt and the two videos and outputs which one is better,

$e(p,\{u_p,v_p\}) = z,\quad z \in \{u_p,v_p\}.$

SLVMEval replaces actual model outputs with synthetic high-versus-low quality pairs,

$$\mathcal{D} = \{(p_i,\{v_{p_i}^{+}, v_{p_i}^{-}\})\}_{i=1}^N,$$

and measures evaluator accuracy by

$$\mathrm{acc}(e,\mathcal{D}) = \frac{1}{|\mathcal{D}|} \sum_{(p,\{v_p^{+},v_p^{-}\}) \in \mathcal{D}} \mathbf{1}\!\left[e(p,\{v_p^{+},v_p^{-}\}) = v_p^{+}\right].$$

This makes pairwise preference accuracy the primary meta-metric (Matsuda et al., 31 Mar 2026).

A broader implication, also formalized elsewhere in long-video evaluation, is that long-context properties should not be reduced to aggregates of short-clip quality. Long-CODE explicitly treats long-context as an orthogonal dimension to short-video assessment and shows that metrics based only on per-shot quality can miss structural corruption such as shuffling and narrative disruption (Tang et al., 19 Apr 2026). SLVMEval is consistent with that view: it tests whether evaluators remain reliable when degradations affect temporal flow, comprehensiveness, and dynamics across long horizons rather than only isolated frames.

2. Benchmark construction from long human videos

The benchmark is built on Vript, a dense video-captioning corpus containing long, human-made videos. A video $v$ with $T$ sampled frames is denoted by frames $f_{v,t}$, and if it is split into $p$0 semantically coherent clips, the $p$1-th clip is

$p$2

Each clip $p$3 has caption $p$4, and the prompt for the whole long video is formed by concatenation,

$p$5

Videos are sampled at 1 fps and frames are resized to maximum side 512 pixels (Matsuda et al., 31 Mar 2026).

From a curated prompt–video corpus

$p$6

SLVMEval constructs aspect-specific degraded counterparts with a degradation function $p$7:

$p$8

where

$p$9

Algorithmically, $\{u_p, v_p\} = \{g(p), g'(p)\}$0 randomly samples five clips from the video, degrades only those clips according to aspect $\{u_p, v_p\} = \{g(p), g'(p)\}$1, and concatenates the degraded and untouched clips back into a long video. In pseudocode form, the paper describes:

• sampling five indices $\{u_p, v_p\} = \{g(p), g'(p)\}$2 from the $\{u_p, v_p\} = \{g(p), g'(p)\}$3 clips,
• applying DegradeClip only when $\{u_p, v_p\} = \{g(p), g'(p)\}$4,
• and concatenating the resulting clips into $\{u_p, v_p\} = \{g(p), g'(p)\}$5 (Matsuda et al., 31 Mar 2026).

This design is synthetic in a specific sense: the underlying videos are real, but the quality difference is controlled synthetically. That yields pairs with known superiority labels while preserving the visual and narrative richness of real long-form video.

3. Controlled degradation aspects

The benchmark defines ten aspects, divided into Video Quality and Video–Text Consistency. These aspects are operationalized by concrete, aspect-specific degradations rather than by abstract labels alone (Matsuda et al., 31 Mar 2026).

Category	Aspect	Synthetic degradation
Video Quality	Aesthetics	Decrease contrast with FFmpeg eq filter (contrast = -0.8)
Video Quality	Technical Quality	Downscale from long side 512 px to 256 px with LANCZOS, then upscale
Video Quality	Appearance Style	Apply OpenCV style transfer such as cartoon, oil painting, watercolor, or pencil sketch
Video Quality	Background Consistency	Extract background mentions with Qwen3-8B, remove backgrounds with rembg (U²-Net), replace with random nature images
Video–Text Consistency	Temporal Flow	Move five consecutive clips to random positions
Video–Text Consistency	Comprehensiveness	Randomly remove five clips
Video–Text Consistency	Object Integrity	Extract objects with Qwen3-8B, detect with Grounding DINO, erase with Stable Diffusion inpainting
Video–Text Consistency	Spatial Relationship	Find left/right relations with Qwen3-8B and horizontally flip those clips
Video–Text Consistency	Dynamics Degree	Replace every frame in selected motion clips with the middle frame
Video–Text Consistency	Color	Identify object-color mentions and change object color with Qwen-Image-Edit using a fixed palette

The four video-quality aspects mostly target frame-level or local perceptual properties, whereas the six video–text consistency aspects include failures that are specifically long-video in character, especially Temporal Flow and Comprehensiveness. Temporal Flow preserves clip content but breaks event order by moving five consecutive clips to random positions. Comprehensiveness removes five clips so that portions of the long prompt are no longer depicted. Object Integrity, Spatial Relationship, Dynamics Degree, and Color modify prompt-specified semantic content while keeping the rest of the video intact (Matsuda et al., 31 Mar 2026).

This controlled taxonomy makes the benchmark diagnostically useful. It separates relatively local failures such as aesthetics or technical softness from long-range semantic failures such as missing events and reordered narratives. A plausible implication is that evaluator errors can be localized by failure type rather than interpreted as a single undifferentiated score.

4. Human filtering, perceptual ground truth, and scale

Synthetic degradation alone does not guarantee that the degraded video is obviously worse to human observers. SLVMEval therefore introduces a crowdsourced verification stage. For each candidate pair and aspect, five workers are shown the prompt, both videos, an explanation of the target aspect, and the intended gold label, and they assign one of three labels: A if the degradation clearly succeeds in all selected clips, B if it succeeds in at least one but not all selected clips or is weak, and C if it fails in all selected clips (Matsuda et al., 31 Mar 2026).

A pair is retained only if

$\{u_p, v_p\} = \{g(p), g'(p)\}$6

The first condition excludes any pair for which a worker judged the degradation to have completely failed; the second requires the majority tendency to favor clear success rather than weak or partial success. The paper reports 736 annotators and 3,793 annotation tasks on Yahoo! Crowdsourcing, together with worker-level filtering for excessively short completion times, systematic disagreement on easy cases, and very low accuracy; 227 workers were excluded from further tasks after manual review (Matsuda et al., 31 Mar 2026).

The resulting benchmark contains long videos with average length 1,141 seconds and maximum length 10,486 seconds, spanning 15 content categories and supporting pairwise judgments that remain easy for humans despite the extreme duration (Matsuda et al., 31 Mar 2026). Human pairwise accuracy ranges from 84.7% to 96.8% across the ten aspects, which is central to the benchmark’s philosophy: the task is deliberately designed so that the correct choice should be straightforward for people, turning evaluator failures into evidence of unreliable long-video assessment rather than ambiguous labeling (Matsuda et al., 31 Mar 2026).

5. Evaluation systems, protocols, and empirical findings

SLVMEval evaluates several classes of automatic systems. The first class is video-based VLM-as-a-judge, instantiated with GPT-5, GPT-5-mini, and Qwen3-VL-235B. The second is text-based roundtrip evaluation, where a VLM first captions each video and a LLM then judges which caption better matches the prompt. The third is CLIPScore, adapted to long video by extracting center frames from FFmpeg-detected clips and averaging Jina-CLIP v2 text–image similarity over those frames. The fourth is VideoScore-v1.1, used through the closest mapping between its internal criteria and SLVMEval’s ten aspects (Matsuda et al., 31 Mar 2026).

Because long videos exceed the frame budgets of most VLMs, the protocol uses clip-aware sampling. Clip boundaries are detected with FFmpeg, the center frame of each clip is extracted, and systems with tight frame limits receive a random subset of these center frames. The aim is to preserve coverage across the whole video rather than overrepresenting its beginning (Matsuda et al., 31 Mar 2026).

The empirical result is that in nine of the ten aspects, automatic systems underperform humans, with gaps ranging from 6.3 to 43.2 percentage points. The sole exception is Background Consistency, where GPT-5 video-based evaluation reaches 98.9%, above human 95.0%. For more semantically demanding aspects, the gap is large: Dynamics Degree reaches 95.9% for humans but only 52.7% for the best automatic system, Temporal Flow reaches 86.6% for humans but 55.6% for the best automatic system, and Comprehensiveness reaches 84.7% for humans but 57.4% for the best automatic system (Matsuda et al., 31 Mar 2026).

Different evaluator families exhibit distinct strengths and weaknesses. CLIPScore is relatively strong on Object Integrity and Comprehensiveness, which is consistent with its frame-level text–image alignment bias, but it is weak on Temporal Flow and Dynamics Degree. Text-based roundtrip evaluation sometimes improves over direct video judging, particularly when explicit description generation helps surface what the model actually sees. The benchmark also reports a duration effect: for most systems and aspects, Spearman correlation between video duration and accuracy is negative, whereas human accuracy remains robust. Examples include strong negative correlations for Background Consistency and Color, and multiple systems show roughly $\{u_p, v_p\} = \{g(p), g'(p)\}$7 correlations for Temporal Flow (Matsuda et al., 31 Mar 2026).

The overall interpretation is not merely that long videos are difficult, but that current automatic evaluators are especially unreliable on semantic and temporal dimensions that require tracking what happens, in what order, and with what coverage over long durations.

6. Position within the broader long-video evaluation literature

SLVMEval belongs to a wider family of synthetic and long-context evaluation efforts, but its target is distinctive: it meta-evaluates evaluation systems rather than understanding models or generators. Several neighboring works provide complementary design principles.

Long-CODE isolates long-context as an orthogonal dimension of video evaluation, introduces corruption-based tests such as Shuffle, Replace, Edition, and Synthesis, and proposes Dynamic Structure Alignment and an MLLM-based metric for narrative, causality, and cross-shot consistency (Tang et al., 19 Apr 2026). DirectorBench moves in a different direction by evaluating long-form generation workflows through 80 structured metadata entries, 7 user profiles, and 40 checkpoint criteria across script, visual, audio, cross-modal, and stability dimensions, revealing a “between-unit bottleneck” in transition quality (Chen et al., 28 May 2026). NarrLV treats narrative richness through Temporal Narrative Atoms, evaluates element fidelity, unit coverage, and unit coherence, and reports that current video generation models effectively express only about two TNAs reliably (Feng et al., 15 Jul 2025). LoCoT2V-Bench likewise emphasizes event-level alignment, fine-grained temporal consistency, content clarity, and HERD for 30–60 second long-form prompts with high semantic, structural, and control complexity (Zheng et al., 30 Oct 2025).

On the understanding side, VideoNIAH shows how synthetic “needles” inserted into arbitrary videos can support scalable retrieval, ordering, and counting benchmarks without entangling labels with original video semantics (Zhao et al., 2024). ScaleLong demonstrates the value of within-content multi-timescale questioning over clip, shot, event, and story horizons (Ma et al., 29 May 2025). MLVU and LVBench show that current multimodal models struggle with long video understanding across holistic, single-detail, and multi-detail tasks, especially as context length grows (Zhou et al., 2024, Wang et al., 2024). X-LeBench extends this to egocentric life logs up to 16.4 hours and shows poor performance on temporal localization, counting, and ordering under a single-pass memory protocol (Zhou et al., 12 Jan 2025). In synthetic video construction and long-form anomaly evaluation, SVLTA, Pistachio, and related benchmarks show how controlled generation, temporal balance, and diagnostic splits can be built directly into the data pipeline (Du et al., 8 Apr 2025, Li et al., 22 Nov 2025).

Taken together, these works suggest a broader definition of synthetic long-video meta-evaluation: controlled perturbation or generation of long-form video conditions, paired with evaluation protocols that isolate temporal structure, narrative coherence, coverage, and robustness. Within that landscape, SLVMEval is the benchmark that most directly asks whether automatic evaluators can make the same obvious long-video quality judgments that humans can make on controlled, aspect-specific pairs (Matsuda et al., 31 Mar 2026).