CollabVR: Step-Level Video Reasoning

• The paper introduces CollabVR, a closed-loop framework where a Vision-Language Model acts as both planner and verifier at step-level granularity to guide video generation.
• It employs a progressive plan–simulate–verify–repair loop to address long-horizon drift and mid-clip errors by decomposing tasks into immediate sub-actions with targeted corrections.
• Empirical results on benchmarks like Gen-ViRe and VBVR-Bench show that CollabVR improves accuracy and efficiency compared to single-inference and prior iterative methods.

VLM-VGM Collaborative Video Reasoning (CollabVR) is a closed-loop framework for goal-directed video reasoning in which a Vision-LLM (VLM) and a Video Generation Model (VGM) collaborate at step-level granularity. Introduced by Kim et al. in "CollabVR: Collaborative Video Reasoning with Vision-Language and Video Generation Models" (Kim et al., 9 May 2026), the method addresses a central weakness of recent "Thinking with Video" approaches: although VGMs can produce temporally coherent Chain-of-Frames, they exhibit long-horizon drift on multi-step tasks and mid-clip simulation errors that compound. CollabVR assigns the VLM two recurring roles—planner and verifier—so that the model plans only the immediate next action, inspects the generated clip, and folds corrective diagnosis directly into the next prompt. On Gen-ViRe and VBVR-Bench, the framework improves both open-source and closed-source VGMs over single-inference, Pass@$k$, and prior test-time scaling baselines at matched compute (Kim et al., 9 May 2026).

1. Problem Setting and Design Rationale

Recent video-reasoning systems increasingly use a VGM as a simulator that externalizes reasoning into generated video clips rather than purely textual intermediate states. In this setting, the VGM contributes perceptual quality, short-horizon visual simulation, and frame-to-frame coherence, but it does not by itself provide explicit multi-step reasoning. CollabVR is motivated by two failure modes that recur on goal-directed tasks: long-horizon drift, described as overloaded-prompt failure under a single long-horizon instruction, and mid-clip execution errors, such as crossing walls, identity loss, or incomplete sub-actions, that contaminate downstream frames (Kim et al., 9 May 2026).

The framework is organized around a placement question: where, exactly, should the VLM intervene? The work argues that an upfront planner commits before any frame is generated, whereas a post-hoc critique over a whole video intervenes too late. CollabVR therefore couples the VLM with the VGM at step-level granularity. The VLM plans only the next sub-action, the VGM simulates that sub-action as a short clip, and the VLM then verifies whether the intended sub-action occurred and whether corrective action is needed (Kim et al., 9 May 2026).

This design yields a progressive, corrective loop summarized in the paper as plan–simulate–verify–repair. A plausible implication is that CollabVR treats reasoning as an online coordination problem between symbolic decomposition and short-horizon visual rollout, rather than as a one-shot prompt-engineering problem.

2. Closed-Loop Collaborative Procedure

Algorithm 1 in the paper takes an initial image $I_0$ and task prompt $q$ and processes them over at most $N_{\max}$ planning steps. At each step $t$, the planner $\pi_{\mathrm{plan}}$ receives $(I_0, q, H)$, where $H$ is the history of accepted clips, and emits the next action prompt $a_t$. The VGM generator $g$ then produces a short clip $I_0$0 from the last accepted frame $I_0$1. The verifier $I_0$2 evaluates $I_0$3 and returns a verdict $I_0$4 and a diagnosis $I_0$5 (Kim et al., 9 May 2026).

If the verdict is accept, the clip is appended to the history, the final frame of the accepted clip becomes the new conditioning frame, and the loop either terminates if the task is complete or proceeds to the next planning step. If the verdict is reject, the action prompt is evolved by injecting the verifier’s suggested fix, and generation is retried. The retry budget is bounded by $I_0$6 attempts per step. If all $I_0$7 attempts reject, the system collapses to a fallback such as best-of-$I_0$8 or single-shot generation and proceeds (Kim et al., 9 May 2026).

The planner emits a structured JSON object whose fields include "instruction", defined as a 4–8 s actionable video-generation prompt for the next sub-action, "target_state", "estimated_steps_remaining", and "task_complete". The verifier likewise emits JSON. On acceptance it returns "verdict": "accept", "reason", and "confidence"; on rejection it returns "verdict": "reject", "reason", "suggestion", and "good_fraction". The verifier is instructed to focus on whether the intended sub-action occurred while allowing partial progress, to flag fundamental errors, and to propose a concrete repair suggestion (Kim et al., 9 May 2026).

Prompt repair is intentionally lightweight. When a clip is rejected, the next prompt is updated by simply concatenating or injecting the verifier’s suggestion into $I_0$9; no extra VLM call is required for rewriting. This keeps the corrective loop within a bounded inference budget (Kim et al., 9 May 2026).

3. Formalization and State Update

The paper gives a compact mathematical description of the step-level interaction. At step $q$0,

$q$1

If the clip is accepted, the history and current frame are updated as

$q$2

If the clip is rejected, the prompt is evolved using the diagnosis and the generation step is retried:

$q$3

The paper also writes a successful accepted update symbolically as

$q$4

These equations formalize CollabVR as an explicit recurrent interaction between planning, simulation, verification, and prompt evolution rather than as end-to-end parameter adaptation (Kim et al., 9 May 2026).

Two modules are especially important in this decomposition. The first, labeled M1 in the ablation, is progressive planning: the VLM decides only the next sub-action instead of a complete long-horizon plan. The second, labeled M2, is verify-and-repair: the VLM checks the generated clip and revises the next prompt only when failure is detected. The empirical results show that these modules are complementary and that their relative value depends on task structure (Kim et al., 9 May 2026).

4. Instantiations and Implementation Parameters

The default planner and verifier are instantiated with Gemini 2.5 Pro (Google DeepMind) via API at temperature $q$5. The work also reports ablated open-source alternatives, Qwen3.5–27B and Qwen3.5–9B, to test whether the framework depends on a single proprietary supervisor (Kim et al., 9 May 2026).

Several VGM backbones are used. VBVR-Wan 2.2 is a 14 B open-source image-to-video diffusion model that is already reasoning-fine-tuned; its reported settings are resolution $q$6, 16 fps, 20 steps, and CFG $q$7. On Gen-ViRe, the first step generates 6 s and subsequent steps generate 3 s clips; on VBVR-Bench it uses matched ground-truth durations. Veo 3.1 is used as a closed-source API at 24 fps, with an 8 s first step and 4 s thereafter on Gen-ViRe only. Cosmos-Predict 2.5 is a 14 B open model with 16 fps and 5 s per step (Kim et al., 9 May 2026).

The default control parameters are $q$8 maximum planning steps with adaptive early exit and per-step attempt budget $q$9, giving up to two re-generations. The failure router can optionally choose "regen" from a good prefix, "split" into finer sub-steps, or "fallback" to single-shot generation. The prompt templates used by the system are called step_planner and step_verifier; the former instructs the VLM to emit one sub-action JSON, and the latter instructs it to judge accept or reject, estimate good_fraction, and suggest a fix (Kim et al., 9 May 2026).

These implementation choices emphasize that CollabVR is not tied to a single backbone pair. The framework is designed to wrap around off-the-shelf VGMs and VLMs, including open-source and closed-source systems, provided that the VLM can produce and verify structured step-level descriptions (Kim et al., 9 May 2026).

5. Benchmark Results and Empirical Behavior

The paper evaluates CollabVR on Gen-ViRe, which contains 72 samples across 6 categories and uses a rubric-based VLM judge via Gemini, and on VBVR-Bench, which contains 500 samples across 5 categories with in-domain and out-of-domain splits and ground-truth video references with rule-based scoring. The matched-compute baselines are Single-inference (Pass@1), Pass@$N_{\max}$0 with VLM re-ranking for $N_{\max}$1, and VideoTPO, an iterative full-video prompt-rewrite method via Gemini (Kim et al., 9 May 2026).

Setting	Matched-compute baselines	CollabVR
Gen-ViRe, VBVR-Wan2.2	Pass@1: 0.391 (6.0 s); Pass@4: 0.438 (24.0 s); VideoTPO: 0.488 (30.0 s)	0.531 (17.8 s)
Gen-ViRe, Veo 3.1	Pass@1: 0.481 (8.0 s); Pass@4: 0.509 (32.0 s)	0.550 (21.4 s)
VBVR-Bench, VBVR-Wan2.2	Pass@1: 0.671 (3.70 s); Pass@4: 0.707 (14.80 s); VideoTPO: 0.650 (11.10 s)	0.757 (10.91 s)
VBVR-Bench, Cosmos-Predict2.5	Pass@1: 0.308	0.403 at same 10.91 s

On Gen-ViRe with VBVR-Wan2.2, CollabVR reaches 0.531 overall accuracy at 17.8 s VGM cost, which is +8.3 pts vs. VideoTPO at lower cost. On VBVR-Bench with VBVR-Wan2.2, it improves from 0.671 under Pass@1 and 0.707 under Pass@4 to 0.757 at 10.91 s, while VideoTPO is reported at 0.650 and 11.10 s (Kim et al., 9 May 2026).

The per-category gains on Gen-ViRe are largest on Algorithmic (+0.191) and Planning (+0.267) tasks, consistent with the intended role of progressive decomposition. Qualitative examples show decomposition of long navigation tasks into turn-by-turn moves, sequential tool-use sub-actions, and correction of local mid-clip errors. A blind user study with $N_{\max}$2 reports human preference for CollabVR outputs over Pass@4 in 78.9 % of decisive trials (Kim et al., 9 May 2026).

A further result bears on training-time versus test-time reasoning. VBVR-Wan2.2 is already fine-tuned on VBVR reasoning data, yet CollabVR still adds +0.086 overall on VBVR-Bench and +0.140 on Gen-ViRe. This directly supports the paper’s claim that step-level test-time supervision is orthogonal to, and complementary with, reasoning-oriented VGM fine-tuning (Kim et al., 9 May 2026).

6. Ablation Findings, Misconceptions, and Relation to Adjacent Paradigms

The module ablations clarify the division of labor between planning and verification. With the VBVR-Wan2.2 backbone, M1 only (progressive planning) yields Gen-ViRe +0.120 and VBVR-Bench +0.035; M2 only (verify+repair, $N_{\max}$3) yields Gen-ViRe +0.045 and VBVR-Bench +0.063; and M1+M2 (full) yields Gen-ViRe +0.140 and VBVR-Bench +0.086. The paper concludes that M1 dominates on multi-step benchmarks, M2 on single-step, and that the two modules are complementary and adapt per instance (Kim et al., 9 May 2026).

One common misconception is that adding a VLM to a VGM necessarily means either producing a full textual plan up front or scoring complete videos afterward. CollabVR is explicitly positioned against both choices: upfront plans commit before any frame is generated, and post-hoc whole-video critiques intervene too late. Another misconception is that more decomposition is always better. The step-count sweep on Gen-ViRe shows that performance rises to $N_{\max}$4 and then plateaus or declines because of over-splitting artifacts, which justifies adaptive step termination rather than unbounded refinement (Kim et al., 9 May 2026).

The paper also measures the reliability of the VLM supervisor. On 250 clips and 72 tasks, Gemini 2.5 Pro achieves 68 % exact match for plan depth with MAE 0.37 steps, verifier overall agreement $N_{\max}$5 with reject recall 65.6 %, and evolution quality mean 2.61 on a 1–3 scale, with 67.5 % rated “well-suited.” These figures indicate that a single off-the-shelf VLM can supervise both planning depth and clip verification reasonably well, but not perfectly (Kim et al., 9 May 2026).

CollabVR occupies one point in a broader design space of VLM-VGM collaboration. In "VLMs are Good Teachers for Video Reasoning via Adaptive Test-Time Optimization" (Cheng et al., 1 Jun 2026), the VLM is not a step-level planner-verifier but a teacher that extracts task-specific rules, formulates differentiable rewards, and guides a VGM Reasoner via test-time online optimization of a lightweight LoRA module; that method reports a 16.7-point average performance gain over VLM-as-Solver and Best-of-N scaling at comparable test-time cost. In "VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Prior" (Yang et al., 30 Mar 2025), the VLM functions as a coarse-grained motion planner whose physics-aware chain-of-thought produces rough trajectories that guide a video diffusion model. In "EmboAlign: Aligning Video Generation with Compositional Constraints for Zero-Shot Manipulation" (Zhang et al., 5 Mar 2026), VLM-generated compositional constraints are applied first for constraint-guided rollout selection and then for constraint-based trajectory optimization in real-robot manipulation. This suggests that CollabVR’s distinctive contribution is not merely combining a VLM with a VGM, but locating the collaboration at the step-level closed loop where planning, simulation, and repair remain tightly interleaved.