VideoChat-R1.5: Iterative Visual Reasoning

• VideoChat-R1.5 is a multimodal model that uses iterative test-time scaling to re-perceive visual inputs and enhance spatio-temporal reasoning.
• The system employs Visual Test-Time Scaling and Iterative Perception, leveraging reinforcement fine-tuning on VTTS-80K to optimize clue prediction and answer accuracy.
• Empirical results demonstrate significant gains in video QA, temporal grounding, and tracking, though increased inference time and perceptual challenges remain.

VideoChat-R1.5 is a multimodal LLM and inference framework introduced in “Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception” (Yan et al., 25 Sep 2025). It is designed to improve video and spatio-temporal reasoning by changing how visual information is processed at test time: rather than encoding a video once and reasoning over a fixed representation, it iteratively re-perceives the input, guided by its own evolving textual predictions. The paper formalizes this approach as Visual Test-Time Scaling (VTTS), implements it through an Iterative Perception (ITP) mechanism, and trains the resulting model with reinforcement fine-tuning on VTTS-80K, a dataset containing question, reasoning, and spatio-temporal clue annotations (Yan et al., 25 Sep 2025).

1. Historical position and model identity

VideoChat-R1.5 is presented as a member of the VideoChat family built on Qwen2.5-VL, with released 3B and 7B variants (Yan et al., 25 Sep 2025). Its stated purpose is to improve “video conversation, video reasoning, and spatio-temporal perception” by scaling perceptual computation at inference time rather than only scaling language-side reasoning (Yan et al., 25 Sep 2025). In the broader VideoChat lineage, this places it after the original “VideoChat: Chat-Centric Video Understanding,” which framed video understanding as multi-round dialogue over video representations (Li et al., 2023), and alongside later VideoChat-family extensions for long video, online video, and task alignment, such as VideoChat-T in TimeSuite (Zeng et al., 2024), VideoChat-Online (Huang et al., 2024), and VideoChat-TPO (Yan et al., 2024).

The original VideoChat paper described a chat-centric video understanding system that integrates “video foundation models and LLMs via a learnable neural interface,” emphasizing spatiotemporal reasoning, event localization, and causal relationship inference (Li et al., 2023). VideoChat-R1.5 retains the chat-centric MLLM framing, but the 2025 work argues that most MLLMs still use a static perception stage: they encode the video once, then devote additional compute only to language generation or chain-of-thought (Yan et al., 25 Sep 2025). VideoChat-R1.5 is therefore positioned as a response to a specific limitation of prior multimodal reasoning systems: “Reasoning cannot fix bad or incomplete visual evidence. If the initial visual parsing misses crucial frames or regions, no amount of language-level test-time scaling will recover them” (Yan et al., 25 Sep 2025).

This positioning is consistent with later external evaluations. In “Perception First: A Frontier Native-Video Model with Self-Consistency for Implicit Video Question Answering,” VideoChat-R1.5 is grouped with “the RL-tuned video reasoners Video-R1 and VideoChat-R1.5,” and is treated as one of several “open-source baseline” systems in a training-free study of video reasoning models (Alavi, 31 May 2026). That paper does not use VideoChat-R1.5 in its final winning system, but explicitly places it in the category of “reinforcement learning–tuned native-video LMMs” associated with “test-time scaling and iterative perception” (Alavi, 31 May 2026). A plausible implication is that VideoChat-R1.5 became a reference point not only as a standalone model, but also as an exemplar of the broader strategy of learned, iterative visual refinement at inference time.

2. Visual Test-Time Scaling and Iterative Perception

The conceptual core of VideoChat-R1.5 is VTTS, defined as test-time scaling on the visual side rather than only on the language side (Yan et al., 25 Sep 2025). In the standard formulation discussed in the paper, an MLLM is trained to maximize the conditional likelihood of answer tokens given visual input and question:

$$\mathcal{L} = - \log \sum_{i=1}^{T} P_{\theta}(\mathbf{W}^a_{i} \mid \mathbf{V}, \mathbf{W}^a_{<i}, \mathbf{W}^q),$$

where $\mathbf{V}$ is the visual input, $\mathbf{W}^q$ the question tokens, and $\mathbf{W}^a$ the answer tokens (Yan et al., 25 Sep 2025).

VTTS extends this to iterative visual dependency modeling:

$$\mathcal{L}_{v} = - \log \sum_{k=1}^{K} \sum_{i=1}^{T} P_{\theta}(\mathbf{W}^a_{i,k}, \mathbf{V}_{k+1} \mid \mathbf{V}_k, \mathbf{W}^a_{<i,k}, \mathbf{W}^q), \quad \text{where} \quad \mathbf{V}_{k+1} = \delta(\mathbf{V}_k \mid \mathbf{W}^a_k).$$

Here, the visual input at iteration $k+1$ is a function of the previous visual input and the model’s own textual prediction at step $k$ (Yan et al., 25 Sep 2025). The paper interprets this as a causal dependency from language to vision: textual hypotheses predict regions of interest, and those predictions govern the next visual resampling step (Yan et al., 25 Sep 2025).

The operational mechanism is Iterative Perception. The first pass is a global, coarse view, using uniform frame sampling for video or the full image for image tasks (Yan et al., 25 Sep 2025). The model then outputs an initial answer, chain-of-thought, and “spatio-temporal clues,” expressed as temporal segments for video or bounding boxes for images (Yan et al., 25 Sep 2025). Those clues drive the next iteration: the system densifies frame sampling inside the selected temporal interval, while keeping sparse context frames outside it, or adds crops around predicted boxes while retaining the full image (Yan et al., 25 Sep 2025). This process is repeated for a fixed number of iterations, with the default ITP setting being $K=3$ (Yan et al., 25 Sep 2025).

The paper presents VTTS as an analogue of language-side test-time scaling. In language-only systems, additional test-time compute is allocated to longer chains, multiple samples, beam search, or MCTS; in VideoChat-R1.5, additional compute is allocated to “more frames per iteration,” “higher resolution for selected ROIs,” and “additional passes of the vision encoder,” with the crucial difference that this compute is concentrated on model-predicted relevant regions rather than uniformly over the entire input (Yan et al., 25 Sep 2025). The authors explicitly connect this to “humans’ hierarchical attention,” describing the procedure as a coarse-to-fine, revisiting-based perception process (Yan et al., 25 Sep 2025).

3. Reinforcement fine-tuning and VTTS-80K

The paper argues that supervised next-token learning is poorly suited to precise continuous spatio-temporal localization, because temporal intervals and bounding boxes are structured numerical targets and because many standard video QA datasets do not provide explicit annotations of “what part of the video should be looked at” (Yan et al., 25 Sep 2025). VideoChat-R1.5 therefore uses reinforcement fine-tuning based on GRPO, with rewards defined jointly over clue correctness, answer correctness, and output format (Yan et al., 25 Sep 2025).

The combined reward is:

$$\mathcal{R}(\theta) = \lambda_0 \cdot r_{\text{clue}}(\mathbf{W}^a_{\text{clue}}, \hat{\mathbf{W}}^a_{\text{clue}}) + \lambda_1 \cdot r_{\text{ans}}(\mathbf{W}^a, \hat{\mathbf{W}}^a) + \lambda_2 \cdot r_{\text{fmt}}(\mathbf{W}^a, \hat{\mathbf{W}}^a).$$

The clue reward uses IoU between predicted and ground-truth temporal segments or spatial boxes, the answer reward measures task correctness, and the format reward enforces parsable output structure (Yan et al., 25 Sep 2025). The paper reports that standard supervised fine-tuning “hardly benefits finding regions of interest and improving subsequent reasoning,” whereas RL produces substantial gains across temporal grounding, grounded video QA, OCR, and video reasoning tasks (Yan et al., 25 Sep 2025).

This RL procedure is enabled by VTTS-80K, a dataset constructed to support iterative perception. Each example may include up to five fields: Question, Options, Think, Clue, and Answer (Yan et al., 25 Sep 2025). The clue field contains either temporal intervals $[t_{\text{start}}, t_{\text{end}}]$ or spatial boxes $\mathbf{V}$0 (Yan et al., 25 Sep 2025). The dataset includes VideoQA, Temporal Clue, Image Reasoning, and Spatial Clue instances, and is built by re-annotating or filtering resources including LVBench, LongVideoBench, MVBench, PerceptionTest, LLavaVideo, STAR, Charades-STA, NextGQA, ReXTime, QVHighlight, ActivityNet, RefCOCO, RefCOCO+, RefCOCOg, GOT-10k, VisualCoT, Geo, and ScienceQA (Yan et al., 25 Sep 2025).

The paper reports approximately 15K temporal clues, 30K spatial clues, 80K “Think” chains, and 50K QA pairs in VTTS-80K (Yan et al., 25 Sep 2025). Its construction pipeline uses DeepSeek for QA verification and reasoning generation, followed by VLM-based ranking and filtering of QA–CoT pairs (Yan et al., 25 Sep 2025). A plausible implication is that VTTS-80K was designed less as a conventional QA corpus than as an explicit supervision source for evidence selection policies: the model is trained not only to answer, but also to externalize what temporal span or spatial region justifies the answer.

4. Architecture and inference pipeline

VideoChat-R1.5 is built on Qwen2.5-VL, with both 3B and 7B scales (Yan et al., 25 Sep 2025). The paper describes this as a unified visual backbone for image and video perception, coupled to the Qwen2.5-VL LLM core and standard cross-modal fusion via visual tokens (Yan et al., 25 Sep 2025). No entirely new multimodal backbone is introduced; the architectural novelty lies in the iterative construction of visual inputs and the reinforcement-trained policy that predicts where to look next (Yan et al., 25 Sep 2025).

The inference pipeline begins with uniform frame sampling for videos, using defaults such as FPS = 2.0, MIN_FRAMES = 64, and MAX_FRAMES up to 2048 (Yan et al., 25 Sep 2025). The first pass yields an initial answer, chain-of-thought, and clue. The clue is then parsed and used to build the next visual batch. For videos, later iterations allocate a “Key Ratio” of 0.5 so that half the frames are sampled from the predicted temporal interval and the other half from outside it, preserving both local detail and global context (Yan et al., 25 Sep 2025). For images, the full image is always retained while additional crops corresponding to predicted boxes are encoded (Yan et al., 25 Sep 2025).

Training uses AdamW with learning rate $\mathbf{V}$1, batch size 16, and a linear learning-rate schedule without warmup (Yan et al., 25 Sep 2025). Video training settings include 4–768 frames at FPS = 2, with video max pixels $\mathbf{V}$2 and video min pixels $\mathbf{V}$3 (Yan et al., 25 Sep 2025). Test-time VTTS settings extend to 4–2048 video frames at FPS = 2 (Yan et al., 25 Sep 2025). The paper compares “single-perception” variants, which perform one pass, to “multi-perception” variants using the default three iterations (Yan et al., 25 Sep 2025).

The authors describe an empirical “perception scaling law”: as the number of ITP iterations increases, performance rises across different task families (Yan et al., 25 Sep 2025). In the reported curves, VideoMME improves from 65.2 to 67.9, LongVideoBench from 61.4 to 62.9, and OCRBench from 856 to 866 as the number of iterations increases (Yan et al., 25 Sep 2025). This suggests that the model’s performance is monotonic in perceptual compute, at least over the tested range.

5. Empirical results across video, grounding, and tracking

The central quantitative claim of the VideoChat-R1.5 paper is that the model achieves “an average increase of over 5\%” relative to robust Qwen2.5-VL baselines “across more than 15 benchmarks” (Yan et al., 25 Sep 2025). For video benchmarks specifically, the paper states average improvements of approximately 5.4% over Qwen2.5-VL-7B and approximately 6.3% over Qwen2.5-VL-3B (Yan et al., 25 Sep 2025).

On general video understanding tasks, the 7B variant improves over Qwen2.5-VL-7B on several benchmarks (Yan et al., 25 Sep 2025). Reported numbers include MVBench 68.4 to 70.6, VideoMME 64.4 to 67.1, LongVideoBench 56.0 to 62.6, LVBench 70.2 to 70.9, MLVU 64.4 to 67.1, and VideoMMMU 45.3 to 48.4 (Yan et al., 25 Sep 2025). The paper emphasizes especially large gains on long-form video benchmarks such as LongVideoBench (Yan et al., 25 Sep 2025).

On grounded video QA, the reported improvements are larger. For NextGQA, Qwen2.5-VL-7B reaches Acc@[email protected] = 72.7, Acc@GQA = 42.3, mIoP = 54.0, and [email protected] = 54.5, while VideoChat-R1.5-7B reaches 79.9, 61.9, 74.9, and 77.6 respectively (Yan et al., 25 Sep 2025). On ReXTime, Qwen2.5-VL-7B reports Acc = 70.4, Acc@[email protected] = 20.9, and mIoU = 29.6, whereas VideoChat-R1.5-7B reports 74.8, 38.1, and 45.8 (Yan et al., 25 Sep 2025).

Temporal grounding results are similarly strong. On fine-tuned Charades-STA, Qwen2.5-VL-7B obtains mIoU = 43.6 and [email protected] = 26.2, while VideoChat-R1.5-7B obtains mIoU = 60.6 and [email protected] = 48.3 (Yan et al., 25 Sep 2025). In zero-shot temporal grounding, the same paper reports QVHighlight mIoU 30.6 to 52.7 and ActivityNet mIoU 19.1 to 35.5 from Qwen2.5-VL-7B to VideoChat-R1.5-7B (Yan et al., 25 Sep 2025). The reported GOT-10k tracking result is also notable: Qwen2.5-VL-7B achieves AO = 12.6, while VideoChat-R1.5-7B reaches 52.2 (Yan et al., 25 Sep 2025).

The RL training ablation is central to interpreting these numbers. The paper reports that SFT often degrades the baseline, while RL substantially improves it (Yan et al., 25 Sep 2025). Examples include Charades-STA mIoU 54.9 under SFT versus 60.6 under RFT, NextGQA mIoP 31.6 versus 61.9, OCRBench 815 versus 865, and VideoMME 63.9 versus 67.1 (Yan et al., 25 Sep 2025). This is one of the strongest empirical arguments in the paper: the gains are attributed not only to iterative inference, but also to the specific choice of reinforcement fine-tuning over clue-bearing trajectories.

6. Interpretation, external evaluations, and limitations

Subsequent papers place VideoChat-R1.5 in a broader and more contested landscape. In the ImplicitQA / VRR-QA study, it is evaluated as one of several “RL-tuned video reasoning models” alongside Video-R1, but is not the strongest open-source model on that benchmark (Alavi, 31 May 2026). The paper explicitly states that the “open-source ceiling” is 58.5% test accuracy, achieved by Qwen3-VL-32B-AWQ with self-consistency, and that VideoChat-R1.5 “does not beat this ceiling” (Alavi, 31 May 2026). That study’s conclusion is that the benchmark is “perception-bound rather than reasoning-bound,” and it applies this claim directly to VideoChat-R1.5: “Its generated reasoning traces will often be logically consistent … But when it misperceives depth ordering, viewpoint, object counts, or visibility, the reasoning built on those faulty premises produces wrong answers” (Alavi, 31 May 2026).

This later interpretation is important because it distinguishes two senses of “iterative perception.” In the VideoChat-R1.5 paper, iterative perception is the mechanism that drives improvement through targeted re-sampling and RL-trained clue prediction (Yan et al., 25 Sep 2025). In the ImplicitQA study, however, reasoning-side scaffolds and iterative procedures are often “neutral-to-harmful” on a benchmark dominated by low-level perceptual bottlenecks such as relative depth, viewpoint, and inferred counting (Alavi, 31 May 2026). The authors summarize that setting with the phrase “The reasoning is sound; the grounding is not” (Alavi, 31 May 2026). A plausible implication is that the effectiveness of VideoChat-R1.5 depends strongly on whether the task benefits from coarse-to-fine evidence acquisition, as opposed to requiring fundamentally better raw perceptual encoding.

The limitations section of the VideoChat-R1.5 paper is consistent with this more cautious interpretation. The authors note that multi-iteration ITP “substantially increases inference time,” especially on long videos, with multi-pass inference taking approximately 30.095s on VideoMME compared with 11.265s for direct-output Qwen2.5-VL-7B and 14.905s for its CoT variant (Yan et al., 25 Sep 2025). They also note that VTTS has been validated only in visual domains, that visual token caching across iterations is not yet implemented, and that performance depends on the quality of clue annotations in VTTS-80K (Yan et al., 25 Sep 2025). Qualitative failure cases show persistent errors when the model repeatedly focuses on incorrect spatio-temporal clues, as well as delayed correction when early ROI predictions are wrong (Yan et al., 25 Sep 2025).

Within the broader literature, VideoChat-R1.5 sits between several trajectories in VideoChat-family research. Long-video extensions such as MA-LMM (He et al., 2024), AdaCM$\mathbf{V}$4 (Man et al., 2024), TimeSuite / VideoChat-T (Zeng et al., 2024), and XComp (Zhang et al., 15 Apr 2026) address long context by memory banks, adaptive cross-modality memory reduction, temporal adapters, or extreme token/frame compression. VideoChat-Online addresses continuous streams by a Pyramid Memory Bank and offline-to-online training (Huang et al., 2024). TPO augments VideoChat with learnable task tokens and task heads for precise grounding, tracking, and segmentation (Yan et al., 2024). VideoChat-R1.5 is orthogonal to many of these efforts: it does not primarily alter the long-context memory substrate or add task-specific experts, but instead makes the perceptual budget itself iterative and answer-conditioned at inference time (Yan et al., 25 Sep 2025). This suggests a broader taxonomy in which VideoChat-R1.5 represents a “perceptual test-time scaling” branch of multimodal model design.

In sum, VideoChat-R1.5 is best understood as a VideoChat-family MLLM that replaces static one-pass visual parsing with iterative, RL-trained evidence acquisition (Yan et al., 25 Sep 2025). Its strongest reported gains occur in grounded video QA, temporal grounding, and tracking, where explicit spatio-temporal clues can guide useful re-perception (Yan et al., 25 Sep 2025). External studies indicate that this strategy does not uniformly dominate other open-source video models on every benchmark, especially when failures arise from low-level perceptual limitations rather than from insufficient procedural search (Alavi, 31 May 2026). The model therefore occupies a specific and influential position in the evolution of video MLLMs: it demonstrates that test-time scaling can be shifted from “think longer” to “look again,” while also clarifying that iterative looking remains constrained by the quality of the underlying perceptual substrate (Yan et al., 25 Sep 2025).