---
title: 'VideoChat-R1.5: Iterative Visual Reasoning'
url: https://www.emergentmind.com/topics/videochat-r1-5
type: topic
---

# VideoChat-R1.5: Iterative Visual Reasoning

VideoChat-R1.5 is a multimodal large language model and inference framework introduced in “Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception” [2509.21100]. It is designed to improve video and spatio-temporal reasoning by changing how visual information is processed at test time: rather than encoding a video once and reasoning over a fixed representation, it iteratively re-perceives the input, guided by its own evolving textual predictions. The paper formalizes this approach as Visual Test-Time Scaling (VTTS), implements it through an Iterative Perception (ITP) mechanism, and trains the resulting model with reinforcement fine-tuning on VTTS-80K, a dataset containing question, reasoning, and spatio-temporal clue annotations [2509.21100].

## 1. Historical position and model identity

VideoChat-R1.5 is presented as a member of the VideoChat family built on Qwen2.5-VL, with released 3B and 7B variants [2509.21100]. Its stated purpose is to improve “video conversation, video reasoning, and spatio-temporal perception” by scaling perceptual computation at inference time rather than only scaling language-side reasoning [2509.21100]. In the broader VideoChat lineage, this places it after the original “VideoChat: Chat-Centric Video Understanding,” which framed video understanding as multi-round dialogue over video representations [2305.06355], and alongside later VideoChat-family extensions for long video, online video, and task alignment, such as VideoChat-T in TimeSuite [2410.19702], VideoChat-Online [2501.00584], and VideoChat-TPO [2412.19326].

The original VideoChat paper described a chat-centric video understanding system that integrates “video foundation models and large language models via a learnable neural interface,” emphasizing spatiotemporal reasoning, event localization, and causal relationship inference [2305.06355]. VideoChat-R1.5 retains the chat-centric MLLM framing, but the 2025 work argues that most MLLMs still use a static perception stage: they encode the video once, then devote additional compute only to language generation or chain-of-thought [2509.21100]. VideoChat-R1.5 is therefore positioned as a response to a specific limitation of prior multimodal reasoning systems: “Reasoning cannot fix bad or incomplete visual evidence. If the initial visual parsing misses crucial frames or regions, no amount of language-level test-time scaling will recover them” [2509.21100].

This positioning is consistent with later external evaluations. In “Perception First: A Frontier Native-Video Model with Self-Consistency for Implicit Video Question Answering,” VideoChat-R1.5 is grouped with “the RL-tuned video reasoners Video-R1 and VideoChat-R1.5,” and is treated as one of several “open-source baseline” systems in a training-free study of video reasoning models [2606.01485]. That paper does not use VideoChat-R1.5 in its final winning system, but explicitly places it in the category of “reinforcement learning–tuned native-video LMMs” associated with “test-time scaling and iterative perception” [2606.01485]. A plausible implication is that VideoChat-R1.5 became a reference point not only as a standalone model, but also as an exemplar of the broader strategy of learned, iterative visual refinement at inference time.

## 2. Visual Test-Time Scaling and Iterative Perception

The conceptual core of VideoChat-R1.5 is VTTS, defined as test-time scaling on the visual side rather than only on the language side [2509.21100]. In the standard formulation discussed in the paper, an MLLM is trained to maximize the conditional likelihood of answer tokens given visual input and question:
$$
\mathcal{L} = - \log \sum_{i=1}^{T} P_{\theta}(\mathbf{W}^a_{i} \mid \mathbf{V}, \mathbf{W}^a_{<i}, \mathbf{W}^q),
$$
where $\mathbf{V}$ is the visual input, $\mathbf{W}^q$ the question tokens, and $\mathbf{W}^a$ the answer tokens [2509.21100].

VTTS extends this to iterative visual dependency modeling:
$$
\mathcal{L}_{v} = - \log \sum_{k=1}^{K} \sum_{i=1}^{T} P_{\theta}(\mathbf{W}^a_{i,k}, \mathbf{V}_{k+1} \mid \mathbf{V}_k, \mathbf{W}^a_{<i,k}, \mathbf{W}^q), \quad \text{where} \quad \mathbf{V}_{k+1} = \delta(\mathbf{V}_k \mid \mathbf{W}^a_k).
$$
Here, the visual input at iteration $k+1$ is a function of the previous visual input and the model’s own textual prediction at step $k$ [2509.21100]. The paper interprets this as a causal dependency from language to vision: textual hypotheses predict regions of interest, and those predictions govern the next visual resampling step [2509.21100].

The operational mechanism is Iterative Perception. The first pass is a global, coarse view, using uniform frame sampling for video or the full image for image tasks [2509.21100]. The model then outputs an initial answer, chain-of-thought, and “spatio-temporal clues,” expressed as temporal segments for video or bounding boxes for images [2509.21100]. Those clues drive the next iteration: the system densifies frame sampling inside the selected temporal interval, while keeping sparse context frames outside it, or adds crops around predicted boxes while retaining the full image [2509.21100]. This process is repeated for a fixed number of iterations, with the default ITP setting being $K=3$ [2509.21100].

The paper presents VTTS as an analogue of language-side test-time scaling. In language-only systems, additional test-time compute is allocated to longer chains, multiple samples, beam search, or MCTS; in VideoChat-R1.5, additional compute is allocated to “more frames per iteration,” “higher resolution for selected ROIs,” and “additional passes of the vision encoder,” with the crucial difference that this compute is concentrated on model-predicted relevant regions rather than uniformly over the entire input [2509.21100]. The authors explicitly connect this to “humans’ hierarchical attention,” describing the procedure as a coarse-to-fine, revisiting-based perception process [2509.21100].

## 3. Reinforcement fine-tuning and VTTS-80K

The paper argues that supervised next-token learning is poorly suited to precise continuous spatio-temporal localization, because temporal intervals and bounding boxes are structured numerical targets and because many standard video QA datasets do not provide explicit annotations of “what part of the video should be looked at” [2509.21100]. VideoChat-R1.5 therefore uses reinforcement fine-tuning based on GRPO, with rewards defined jointly over clue correctness, answer correctness, and output format [2509.21100].

The combined reward is:
$$
\mathcal{R}(\theta) = \lambda_0 \cdot r_{\text{clue}}(\mathbf{W}^a_{\text{clue}}, \hat{\mathbf{W}}^a_{\text{clue}}) + \lambda_1 \cdot r_{\text{ans}}(\mathbf{W}^a, \hat{\mathbf{W}}^a) + \lambda_2 \cdot r_{\text{fmt}}(\mathbf{W}^a, \hat{\mathbf{W}}^a).
$$
The clue reward uses IoU between predicted and ground-truth temporal segments or spatial boxes, the answer reward measures task correctness, and the format reward enforces parsable output structure [2509.21100]. The paper reports that standard supervised fine-tuning “hardly benefits finding regions of interest and improving subsequent reasoning,” whereas RL produces substantial gains across temporal grounding, grounded video QA, OCR, and video reasoning tasks [2509.21100].

This RL procedure is enabled by VTTS-80K, a dataset constructed to support iterative perception. Each example may include up to five fields: Question, Options, Think, Clue, and Answer [2509.21100]. The clue field contains either temporal intervals $[t_{\text{start}}, t_{\text{end}}]$ or spatial boxes $[x_1, y_1, x_2, y_2]$ [2509.21100]. The dataset includes VideoQA, Temporal Clue, Image Reasoning, and Spatial Clue instances, and is built by re-annotating or filtering resources including LVBench, LongVideoBench, MVBench, PerceptionTest, LLavaVideo, STAR, Charades-STA, NextGQA, ReXTime, QVHighlight, ActivityNet, RefCOCO, RefCOCO+, RefCOCOg, GOT-10k, VisualCoT, Geo, and ScienceQA [2509.21100].

The paper reports approximately 15K temporal clues, 30K spatial clues, 80K “Think” chains, and 50K QA pairs in VTTS-80K [2509.21100]. Its construction pipeline uses DeepSeek for QA verification and reasoning generation, followed by VLM-based ranking and filtering of QA–CoT pairs [2509.21100]. A plausible implication is that VTTS-80K was designed less as a conventional QA corpus than as an explicit supervision source for evidence selection policies: the model is trained not only to answer, but also to externalize what temporal span or spatial region justifies the answer.

## 4. Architecture and inference pipeline

VideoChat-R1.5 is built on Qwen2.5-VL, with both 3B and 7B scales [2509.21100]. The paper describes this as a unified visual backbone for image and video perception, coupled to the Qwen2.5-VL LLM core and standard cross-modal fusion via visual tokens [2509.21100]. No entirely new multimodal backbone is introduced; the architectural novelty lies in the iterative construction of visual inputs and the reinforcement-trained policy that predicts where to look next [2509.21100].

The inference pipeline begins with uniform frame sampling for videos, using defaults such as FPS = 2.0, MIN_FRAMES = 64, and MAX_FRAMES up to 2048 [2509.21100]. The first pass yields an initial answer, chain-of-thought, and clue. The clue is then parsed and used to build the next visual batch. For videos, later iterations allocate a “Key Ratio” of 0.5 so that half the frames are sampled from the predicted temporal interval and the other half from outside it, preserving both local detail and global context [2509.21100]. For images, the full image is always retained while additional crops corresponding to predicted boxes are encoded [2509.21100].

Training uses AdamW with learning rate $2 \times 10^{-6}$, batch size 16, and a linear learning-rate schedule without warmup [2509.21100]. Video training settings include 4–768 frames at FPS = 2, with video max pixels $768 \times 28 \times 28$ and video min pixels $128 \times 28 \times 28$ [2509.21100]. Test-time VTTS settings extend to 4–2048 video frames at FPS = 2 [2509.21100]. The paper compares “single-perception” variants, which perform one pass, to “multi-perception” variants using the default three iterations [2509.21100].

The authors describe an empirical “perception scaling law”: as the number of ITP iterations increases, performance rises across different task families [2509.21100]. In the reported curves, VideoMME improves from 65.2 to 67.9, LongVideoBench from 61.4 to 62.9, and OCRBench from 856 to 866 as the number of iterations increases [2509.21100]. This suggests that the model’s performance is monotonic in perceptual compute, at least over the tested range.

## 5. Empirical results across video, grounding, and tracking

The central quantitative claim of the VideoChat-R1.5 paper is that the model achieves “an average increase of over 5\%” relative to robust Qwen2.5-VL baselines “across more than 15 benchmarks” [2509.21100]. For video benchmarks specifically, the paper states average improvements of approximately 5.4% over Qwen2.5-VL-7B and approximately 6.3% over Qwen2.5-VL-3B [2509.21100].

On general video understanding tasks, the 7B variant improves over Qwen2.5-VL-7B on several benchmarks [2509.21100]. Reported numbers include MVBench 68.4 to 70.6, VideoMME 64.4 to 67.1, LongVideoBench 56.0 to 62.6, LVBench 70.2 to 70.9, MLVU 64.4 to 67.1, and VideoMMMU 45.3 to 48.4 [2509.21100]. The paper emphasizes especially large gains on long-form video benchmarks such as LongVideoBench [2509.21100].

On grounded video QA, the reported improvements are larger. For NextGQA, Qwen2.5-VL-7B reaches Acc@IoP@0.5 = 72.7, Acc@GQA = 42.3, mIoP = 54.0, and IoP@0.5 = 54.5, while VideoChat-R1.5-7B reaches 79.9, 61.9, 74.9, and 77.6 respectively [2509.21100]. On ReXTime, Qwen2.5-VL-7B reports Acc = 70.4, Acc@IoU@0.5 = 20.9, and mIoU = 29.6, whereas VideoChat-R1.5-7B reports 74.8, 38.1, and 45.8 [2509.21100].

Temporal grounding results are similarly strong. On fine-tuned Charades-STA, Qwen2.5-VL-7B obtains mIoU = 43.6 and R@0.7 = 26.2, while VideoChat-R1.5-7B obtains mIoU = 60.6 and R@0.7 = 48.3 [2509.21100]. In zero-shot temporal grounding, the same paper reports QVHighlight mIoU 30.6 to 52.7 and ActivityNet mIoU 19.1 to 35.5 from Qwen2.5-VL-7B to VideoChat-R1.5-7B [2509.21100]. The reported GOT-10k tracking result is also notable: Qwen2.5-VL-7B achieves AO = 12.6, while VideoChat-R1.5-7B reaches 52.2 [2509.21100].

The RL training ablation is central to interpreting these numbers. The paper reports that SFT often degrades the baseline, while RL substantially improves it [2509.21100]. Examples include Charades-STA mIoU 54.9 under SFT versus 60.6 under RFT, NextGQA mIoP 31.6 versus 61.9, OCRBench 815 versus 865, and VideoMME 63.9 versus 67.1 [2509.21100]. This is one of the strongest empirical arguments in the paper: the gains are attributed not only to iterative inference, but also to the specific choice of reinforcement fine-tuning over clue-bearing trajectories.

## 6. Interpretation, external evaluations, and limitations

Subsequent papers place VideoChat-R1.5 in a broader and more contested landscape. In the ImplicitQA / VRR-QA study, it is evaluated as one of several “RL-tuned video reasoning models” alongside Video-R1, but is not the strongest open-source model on that benchmark [2606.01485]. The paper explicitly states that the “open-source ceiling” is 58.5% test accuracy, achieved by Qwen3-VL-32B-AWQ with self-consistency, and that VideoChat-R1.5 “does not beat this ceiling” [2606.01485]. That study’s conclusion is that the benchmark is “perception-bound rather than reasoning-bound,” and it applies this claim directly to VideoChat-R1.5: “Its generated reasoning traces will often be logically consistent … But when it misperceives depth ordering, viewpoint, object counts, or visibility, the reasoning built on those faulty premises produces wrong answers” [2606.01485].

This later interpretation is important because it distinguishes two senses of “iterative perception.” In the VideoChat-R1.5 paper, iterative perception is the mechanism that drives improvement through targeted re-sampling and RL-trained clue prediction [2509.21100]. In the ImplicitQA study, however, reasoning-side scaffolds and iterative procedures are often “neutral-to-harmful” on a benchmark dominated by low-level perceptual bottlenecks such as relative depth, viewpoint, and inferred counting [2606.01485]. The authors summarize that setting with the phrase “The reasoning is sound; the grounding is not” [2606.01485]. A plausible implication is that the effectiveness of VideoChat-R1.5 depends strongly on whether the task benefits from coarse-to-fine evidence acquisition, as opposed to requiring fundamentally better raw perceptual encoding.

The limitations section of the VideoChat-R1.5 paper is consistent with this more cautious interpretation. The authors note that multi-iteration ITP “substantially increases inference time,” especially on long videos, with multi-pass inference taking approximately 30.095s on VideoMME compared with 11.265s for direct-output Qwen2.5-VL-7B and 14.905s for its CoT variant [2509.21100]. They also note that VTTS has been validated only in visual domains, that visual token caching across iterations is not yet implemented, and that performance depends on the quality of clue annotations in VTTS-80K [2509.21100]. Qualitative failure cases show persistent errors when the model repeatedly focuses on incorrect spatio-temporal clues, as well as delayed correction when early ROI predictions are wrong [2509.21100].

Within the broader literature, VideoChat-R1.5 sits between several trajectories in VideoChat-family research. Long-video extensions such as MA-LMM [2404.05726], AdaCM$^2$ [2411.12593], TimeSuite / VideoChat-T [2410.19702], and XComp [2604.14149] address long context by memory banks, adaptive cross-modality memory reduction, temporal adapters, or extreme token/frame compression. VideoChat-Online addresses continuous streams by a Pyramid Memory Bank and offline-to-online training [2501.00584]. TPO augments VideoChat with learnable task tokens and task heads for precise grounding, tracking, and segmentation [2412.19326]. VideoChat-R1.5 is orthogonal to many of these efforts: it does not primarily alter the long-context memory substrate or add task-specific experts, but instead makes the perceptual budget itself iterative and answer-conditioned at inference time [2509.21100]. This suggests a broader taxonomy in which VideoChat-R1.5 represents a “perceptual test-time scaling” branch of multimodal model design.

In sum, VideoChat-R1.5 is best understood as a VideoChat-family MLLM that replaces static one-pass visual parsing with iterative, RL-trained evidence acquisition [2509.21100]. Its strongest reported gains occur in grounded video QA, temporal grounding, and tracking, where explicit spatio-temporal clues can guide useful re-perception [2509.21100]. External studies indicate that this strategy does not uniformly dominate other open-source video models on every benchmark, especially when failures arise from low-level perceptual limitations rather than from insufficient procedural search [2606.01485]. The model therefore occupies a specific and influential position in the evolution of video MLLMs: it demonstrates that test-time scaling can be shifted from “think longer” to “look again,” while also clarifying that iterative looking remains constrained by the quality of the underlying perceptual substrate [2509.21100].

Source: https://www.emergentmind.com/topics/videochat-r1-5