PyVision-Video: Agentic Video Reasoning

• PyVision-Video is a video understanding model that constructs its visual context on demand by interleaving natural language reasoning with executable Python code.
• It employs an agentic scaffold built on Qwen2.5‑VL‑7B and reinforcement learning to sustain multi-turn, tool-based interactions, preventing interaction collapse.
• The model achieves competitive spatial reasoning accuracy on VSI-Bench using a fraction of visual tokens compared to conventional video MLLMs.

PyVision-Video is a video understanding model developed within PyVision-RL, a unified reinforcement-learning framework for agentic multimodal models based on open-weight MLLMs. Built on Qwen2.5‑VL‑7B and trained to interleave natural language reasoning with executable Python code, it is defined by an architectural and interactional constraint that distinguishes it from conventional video MLLMs: videos are only loaded into the Python runtime, and the model must explicitly sample and plot frames via Python to “see” them. The result is a video reasoning agent that does not pre-ingest a large set of frames, but instead constructs visual context on demand during reasoning, with reinforcement learning used to sustain multi-turn tool use and to prevent interaction collapse (Zhao et al., 24 Feb 2026).

1. Framework position and defining characteristics

Within PyVision-RL, PyVision-Video is paired with PyVision-Image as one of two concrete models trained under a shared policy backbone and RL algorithm. Both use Python as a primitive tool, interact with an external environment in the form of a Python sandbox, synthesize tools dynamically via Python code rather than selecting from a fixed tool menu, and perform multi-turn reasoning and tool use before issuing a final answer. In this sense, PyVision-Video is not merely a video-LLM but an instance of what the authors call an agentic multimodal model (Zhao et al., 24 Feb 2026).

The decisive distinction between PyVision-Image and PyVision-Video lies in scaffolding and context construction rather than neural architecture. PyVision-Image injects images into both the model context and the Python runtime. PyVision-Video, by contrast, initially injects only the system prompt into the MLLM context; the video exists only as a Python object, specifically a VideoReader instance such as video_clue_j. The model therefore cannot access raw video evidence unless it writes Python code that samples frames from video_clue_j, plots them using matplotlib, and calls plt.show() (Zhao et al., 24 Feb 2026).

This design is tied to a central problem identified by the paper: interaction collapse. During RL, the average number of tool calls can decrease, and the model may converge to short, low-interaction behaviors with fewer Python code blocks, fewer multi-turn steps, and sometimes no tool usage at all. For video reasoning, this is especially harmful because tasks such as spatial reasoning, object counting, and route planning often require sampling multiple frames from different times, comparing views across time, and zooming into specific regions. PyVision-Video is therefore constructed to force tool-based access to video and to make sustained interaction a first-class training objective (Zhao et al., 24 Feb 2026).

2. Architectural scaffold and video representation

PyVision-Video inherits the underlying multimodal architecture of Qwen2.5‑VL‑7B. The paper does not redefine the vision encoder type, temporal embeddings, or other low-level backbone components, and it does not introduce new temporal embeddings or special video heads beyond what Qwen2.5‑VL‑7B already has. The key innovation is instead agentic scaffolding, RL training, and on-demand context construction (Zhao et al., 24 Feb 2026).

Video representation is correspondingly procedural rather than front-loaded. The raw video is not converted into a pre-selected set of frames for the model. Instead, the video is read into Python as a VideoReader object, and the MLLM can only see frames if it generates code to sample them, often via decord, and then renders them as images. Frame sampling and temporal encoding details are thus delegated to Python code rather than fixed at the architecture level. The multimodal interface continues to see images in the ordinary sense, with standard visual tokens and positional encodings inherited from Qwen2.5‑VL‑7B (Zhao et al., 24 Feb 2026).

A recurrent misconception is to treat PyVision-Video as a conventional video-LLM with an unusual prompt. That characterization is incomplete. Its defining access pattern is operational: no frames are placed in context upfront, and the model is explicitly instructed that it cannot see the video directly. It must fetch evidence through code. Another misconception is to regard it as a new temporal architecture. The paper instead positions it as a new agentic scaffold around an existing open-weight multimodal LLM, coupled to RL mechanisms that stabilize tool-centric behavior (Zhao et al., 24 Feb 2026).

3. On-demand context construction and interaction loop

On-demand context construction is the central mechanism of PyVision-Video. The interaction begins with a system prompt and a user query; the video remains outside the language-model context and only inside the Python environment. The model then generates text interleaved with <code> ... </code> blocks. The Python sandbox executes those blocks, which may sample frames from video_clue_j, plot them, and return images or textual outputs. Those execution results are wrapped as <interpreter> ... </interpreter> and appended to the MLLM context, after which the model can continue reasoning and decide whether further tool use is needed. The process terminates only when the model emits a final answer in the specified <answer> format with \boxed{...} (Zhao et al., 24 Feb 2026).

Frame selection is policy-driven rather than heuristic-driven. The system prompt explicitly states: “Remember you CAN NOT see the video directly. Thus, if you need to reason based on the video, you MUST sample frames and use plt.show() to display these frames, helping your reasoning.” The model is free to choose which frame indices to sample, how many frames to sample, and whether to zoom or crop them further. The framework does not impose hard-coded sampling heuristics; RL shapes the policy so that useful frame acquisition patterns emerge implicitly (Zhao et al., 24 Feb 2026).

The paper’s case studies show a characteristic coarse-to-fine interaction pattern. In one example on absolute distance estimation, the model first samples approximately 15 frames uniformly to identify clear views, then issues a second, more focused sampling call over relevant indices, and finally uses Python for approximate spatial reasoning. In another example on object counting, it initially samples 15 frames, identifies ambiguity between two candidate tables, then samples more densely between frames 600 and 1200 to disambiguate whether the views correspond to one table or two. These trajectories instantiate a broader pattern: coarse temporal scanning, targeted resampling, and only then answer production (Zhao et al., 24 Feb 2026).

The token-efficiency consequences are substantial. On VSI-Bench, PyVision-Video is reported at 44.0% average accuracy with approximately 5K visual tokens per sample. In the efficiency comparison, Qwen2.5‑VL‑7B at 1 FPS consumes approximately 45K visual tokens per sample to reach 38.0% accuracy, while SpaceR uses approximately 25K tokens for 45.6% accuracy. The authors therefore characterize PyVision-Video as offering a favorable accuracy–efficiency trade-off, including higher accuracy than the base Qwen2.5‑VL‑7B with approximately 9× fewer visual tokens (Zhao et al., 24 Feb 2026).

4. Reinforcement-learning methodology

PyVision-Video shares the PyVision-RL training stack with PyVision-Image, but it is optimized on video spatial reasoning data. The RL algorithm is GRPO-style RL with several modifications intended to stabilize training and maintain interaction: an oversampling–filtering–ranking rollout strategy, an accumulative tool reward, and intra-group advantage estimation without standard deviation normalization (Zhao et al., 24 Feb 2026).

The oversampling–filtering–ranking pipeline operates as follows. For each RL step, the framework samples $\alpha B$ prompts, generates $G$ rollouts per prompt, executes each rollout in the Python environment, and computes rollout-level rewards. Broken trajectories—those involving timeout, runtime error, or invalid multimodal output—are marked accordingly. Groups are then filtered if all rollouts are broken or if group reward variance is zero. The remaining groups are ranked by group standard deviation in descending order, and the top $B \cdot G$ rollouts are selected for training. The paper interprets higher reward variance as a signal of more informative, moderately difficult groups, and argues that this selection reduces instability and emphasizes useful learning signal (Zhao et al., 24 Feb 2026).

The reward function couples answer accuracy to sustained tool use. For a trajectory $\tau$, let $R_{\text{acc}} \in \{0,1\}$ denote answer accuracy reward and let $n_{\text{tc}}$ denote the total number of tool calls. The final reward is

$$R = R_{\text{acc}} + 0.1 \cdot n_{\text{tc}} \cdot \mathbf{1}_{\{R_{\text{acc}}=1\}}.$$

The tool reward is added only when the answer is correct, so incorrect trajectories receive no extra reward for tool usage. This prevents the policy from exploiting the objective by merely spamming tools. In PyVision-Video, tool calls include frame sampling and plotting operations, such as the fetch_frames_and_plot category, as well as auxiliary numerical analysis (Zhao et al., 24 Feb 2026).

Advantage estimation is also modified. Rather than using standard GRPO normalization by group standard deviation, the framework computes

$$\widehat{A}_{i,t} = R(x, y_i) - \mathrm{mean}\left(\{R(x, y_i)\}_{i=1}^G\right).$$

The removal of standard deviation normalization is reported to reduce variance and improve RL stability. Together with the accumulative tool reward, a sufficiently large max turn budget, and standard deviation sorting, this modification is presented as part of the mechanism that prevents interaction collapse (Zhao et al., 24 Feb 2026).

PyVision-Video is warm-started from supervised fine-tuning. The paper reports 44K SFT samples with multi-turn tool usage for video spatial reasoning and long video understanding, sourced from SpaceR and LongVILA, and then applies RL on 15K SpaceR samples. Training dynamics in Appendix D.1 show stable entropy and gradient norm, steadily increasing validation score on a VSI-Bench subset, and steadily increasing mean tool call count and response length, which the paper interprets as confirmation that the RL signal encourages sustained multi-turn behavior (Zhao et al., 24 Feb 2026).

5. Datasets, benchmarks, and observed behavior

Training and evaluation are concentrated on tasks in which frame selection and spatial reasoning are crucial. For supervised fine-tuning, PyVision-Video uses 44K synthetic or curated samples from SpaceR and LongVILA, filtered for correct answers and at least two tool-use turns. RL then uses 15K SpaceR samples. Evaluation centers on VSI-Bench, a video spatial reasoning benchmark with categories including Obj. Count, Abs. Dist., Obj. Size, Room Size, Rel. Dist., Rel. Dir., Route Plan, and Appr. Order (Zhao et al., 24 Feb 2026).

The benchmark results reported for VSI-Bench place PyVision-Video at 44.0 average accuracy, with category scores of 53.8 on object counting, 25.8 on absolute distance, 50.8 on object size, 38.2 on room size, 44.8 on relative distance, 46.3 on relative direction, 26.3 on route planning, and 58.6 on approximate order. In the same table, Qwen2.5‑VL‑7B is reported at 36.7 average, Video-R1 at 37.1, and VITAL at 41.8; the paper states that PyVision-Video improves over Qwen2.5‑VL‑7B by 7.3 points on average and outperforms VITAL by 2.2 points on average (Zhao et al., 24 Feb 2026).

Model	Avg. on VSI-Bench	Visual tokens per sample
PyVision-Video	44.0%	~5K
Qwen2.5‑VL‑7B at 1 FPS	38.0%	~45K
SpaceR	45.6%	~25K

Evaluation uses 32K context length for both PyVision-Image and PyVision-Video, with a max turn budget of 30 for PyVision-Video and temperature 0.01 during evaluation. These settings are aligned with the model’s intended long-horizon reasoning behavior (Zhao et al., 24 Feb 2026).

Tool-use analysis further clarifies the learned policy. On VSI-Bench, 87.4% of tool calls fall into the fetch_frames_and_plot category, indicating that on-demand frame acquisition is the dominant behavioral primitive rather than a peripheral capability. The tool call-number distribution shows that PyVision-Video typically uses 3 tool calls per sample, while some complex samples reach 9 turns. The paper interprets this as evidence of long-horizon multi-turn tool-using ability for video reasoning (Zhao et al., 24 Feb 2026).

6. Research context, scope, and limitations

PyVision-Video belongs to a broader line of work that attempts to strengthen video reasoning without redesigning the entire multimodal backbone. Video-ToC, for example, also builds on Qwen2.5‑VL‑7B but improves reasoning through tree-guided visual cue localization, a reasoning-demand reward, and training on uniformly sampled frames—16 during training and 16, 32, or 64 during evaluation—using <locate> and <answer> prompting rather than mandatory Python-mediated frame access (Tan et al., 22 Apr 2026). PyVision-Video differs in that it makes visual access itself agentic: the model must explicitly decide which frames to fetch and when (Zhao et al., 24 Feb 2026).

A second relevant context comes from work questioning whether video benchmarks genuinely require temporal reasoning. The ATP framework shows that understanding of event temporality is often not necessary to achieve strong performance on several video-language tasks, because a strong image-LLM with learned frame selection can already solve many ostensibly video-level problems (Buch et al., 2022). PyVision-Video’s emphasis on SpaceR training and VSI-Bench evaluation suggests a different target regime: problems in which active frame selection and spatial reasoning are intended to be indispensable rather than incidental (Zhao et al., 24 Feb 2026).

The system also has clear boundaries. It is mainly evaluated on spatial reasoning and is not comprehensively tested on long-horizon narrative understanding, fine-grained motion or fast action sequences, or open-domain video QA and captioning. Because the underlying MLLM still treats frames mostly as independent images, a plausible implication is that tasks requiring precise temporal dynamics beyond what can be inferred from a few sampled frames may remain challenging. Another plausible implication is that the absence of an explicit token- or time-based penalty in the reward may allow oversampling in some cases, even though the reported behavior is efficient in practice. The paper also notes a deployment concern: because Python can access the host filesystem, the sandbox must be carefully controlled (Zhao et al., 24 Feb 2026).

In this configuration, PyVision-Video is best understood not as a new video backbone, but as a reinforcement-learned agentic interface to video evidence. Its contribution is to relocate the central video reasoning problem from up-front frame packing into policy-guided evidence acquisition: the model becomes an agentic Python programmer that constructs its own visual context incrementally, and performance emerges from sustained interaction rather than from a fixed ingest of uniformly sampled frames (Zhao et al., 24 Feb 2026).