FrameMind: Adaptive Video Reasoning

• FrameMind is a video understanding framework that interleaves textual reasoning with targeted visual tool calls to acquire dynamic, question-dependent evidence.
• It employs the Frame-Interleaved Chain-of-Thought (FiCOT) protocol alongside Dynamic Resolution Frame Sampling (DRFS) and DRFS-GRPO for adaptive temporal-spatial evidence acquisition.
• Empirical evaluations on benchmarks like MVBench and VideoMME demonstrate improved accuracy over baseline models with enhanced sample efficiency and strategic frame selection.

FrameMind is a video understanding framework that treats visual evidence acquisition as part of reasoning rather than a fixed preprocessing step. It introduces Frame-Interleaved Chain-of-Thought (FiCOT), in which the model alternates textual reasoning with targeted visual tool use, and it is trained with Dynamic Resolution Frame Sampling (DRFS) and DRFS-GRPO to learn question-dependent temporal-spatial evidence policies (Ge et al., 28 Sep 2025). In the reported formulation, FrameMind is built on Qwen2.5-VL-7B, uses FrameAt and VideoClip as visual tools, and is evaluated on MVBench, MLVU, and VideoMME, where it improves over both the base model and the RL baseline Video-R1 (Ge et al., 28 Sep 2025).

1. Conceptual definition

FrameMind departs from the standard video-LLM pipeline

$A = M(F, Q),$

where $Q$ is the question, $F=\{f_1,\dots,f_n\}$ is a fixed set of sampled frames, and $A$ is the answer (Ge et al., 28 Sep 2025). Its premise is that fixed frame sampling is structurally mismatched to video reasoning because different questions require different temporal-spatial evidence budgets: some require broad temporal coverage, others require high-resolution inspection of a narrow instant (Ge et al., 28 Sep 2025).

The framework is organized around a bounded multi-turn loop rather than one-shot inference. In the paper’s formulation, the reasoning trajectory is

$$\tau = \{T_1, C_1, E_1, T_2, C_2, E_2, \dots, T_n, A\},$$

where $T_k$ is textual thought, $C_k$ is a tool call, $E_k$ is returned visual evidence, and $A$ is the final answer (Ge et al., 28 Sep 2025). This makes FrameMind a tool-augmented video reasoning agent rather than a fixed-context video encoder.

Component	Function	Key details
FiCOT	Interleave reasoning and perception	Multi-turn text-think, tool-call, evidence loop
FrameAt	Fine spatial inspection	Single frame at timestamp $t$, resized to $Q$0
VideoClip	Coarse temporal localization	Interval retrieval with 8–20 frames, resized to $Q$1
DRFS	Temporal-spatial training ladder	Interpolates from many-frames/low-res to few-frames/high-res
DRFS-GRPO	RL optimization	Group-relative comparison across DRFS views

This design places FrameMind within the class of frame-adaptive systems, but not within the narrower class of fixed-budget keyframe selectors. A common simplification is to treat it as merely an adaptive sampler. That is incomplete: the paper defines it as an end-to-end framework in which sampling decisions are conditioned on intermediate reasoning states, explicit tool outputs, and RL rewards over full trajectories (Ge et al., 28 Sep 2025).

2. Frame-Interleaved Chain-of-Thought

FiCOT is the core execution protocol. At turn $Q$2, the model conditions on the question, the previous textual history

$Q$3

and the previous visual evidence $Q$4, then generates the next thought and tool call as

$Q$5

(Ge et al., 28 Sep 2025). The alternation is explicit: reason in text, call a visual tool, append the returned observation, and continue.

The tool interface contains two actions. FrameAt($Q$6) retrieves a single high-resolution frame at timestamp $Q$7. VideoClip($Q$8) retrieves a clip over a time interval, sampled uniformly into 8–20 frames (Ge et al., 28 Sep 2025). The paper motivates a two-stage usage pattern: VideoClip for broad localization and FrameAt for detail verification. Returned evidence is represented as timestamped frames

$Q$9

(Ge et al., 28 Sep 2025).

The prompt protocol is structurally constrained. The model emits > ..., optional <tool_call>...</tool_call>, and then either <turn_sum>...</turn_sum> to continue or <answer>...</answer> to terminate (Ge et al., 28 Sep 2025). Tool calls are regex-parsed; malformed calls or invalid timestamps yield explicit error messages that are appended to context, enabling self-correction on later turns (Ge et al., 28 Sep 2025). Inference is bounded by a 3-turn cap. The controller continues if a turn ends with a closed <turn_sum> and $F=\{f_1,\dots,f_n\}$0; it stops immediately if <answer> is emitted (Ge et al., 28 Sep 2025).

This protocol makes FiCOT a serialized planner-executor loop. It is not an external retrieval heuristic bolted onto a static LVLM. The same policy emits both thought tokens and action tokens, so evidence acquisition is endogenous to the reasoning process (Ge et al., 28 Sep 2025).

3. Training with DRFS and DRFS-GRPO

FrameMind is trained with reinforcement learning rather than a large SFT-heavy pipeline. The central training mechanism is Dynamic Resolution Frame Sampling (DRFS), which exposes the policy to a family of temporal-spatial input regimes (Ge et al., 28 Sep 2025). For group member $F=\{f_1,\dots,f_n\}$1,

$F=\{f_1,\dots,f_n\}$2

$F=\{f_1,\dots,f_n\}$3

$F=\{f_1,\dots,f_n\}$4

interpolating from a many-frames/low-resolution regime to a few-frames/high-resolution regime (Ge et al., 28 Sep 2025). In the reported setup, DRFS spans 32–64 frames and $F=\{f_1,\dots,f_n\}$5 to $F=\{f_1,\dots,f_n\}$6, with $F=\{f_1,\dots,f_n\}$7 DRFS views in training (Ge et al., 28 Sep 2025).

For each training example, the policy rolls out a trajectory for each DRFS view and receives a scalar reward. The group-average reward is

$F=\{f_1,\dots,f_n\}$8

and the group-relative advantage is

$F=\{f_1,\dots,f_n\}$9

(Ge et al., 28 Sep 2025). This makes the supervision comparative: the model is reinforced not only for success, but for using the temporal-spatial view that works best relative to sibling views for the same video-question pair.

The trajectory reward is

$A$0

(Ge et al., 28 Sep 2025). Accuracy reward is binary,

$A$1

format reward is

$A$2

tool reward is

$A$3

and turn reward is

$A$4

(Ge et al., 28 Sep 2025). The tool score is 1.0 for one unique tool type and 1.2 if both tool types are used (Ge et al., 28 Sep 2025). The paper’s ablation attributes successful tool learning partly to the unconditional 20% exploration component in $A$5 (Ge et al., 28 Sep 2025).

The reported RL fine-tuning uses AdamW, learning rate $A$6, weight decay $A$7, global batch size 64, KL coefficient $A$8, maximum context length 32768, and 8 × NVIDIA A100 80GB GPUs (Ge et al., 28 Sep 2025). Training data totals about 7.6K video-QA instances, drawn from datasets including PerceptionTest, LLaVA-Video-178K, STAR, CLEVRER, NeXT-QA, LongVideo-Reason, and VideoEspresso (Ge et al., 28 Sep 2025).

4. Empirical profile

On the main benchmarks reported in the paper, FrameMind improves over both the base Qwen2.5-VL-7B and Video-R1 (Ge et al., 28 Sep 2025).

Model	MVBench	MLVU	VideoMME Overall	VideoMME Long
Qwen2.5-VL-7B	62.6	41.6	53.6	44.7
Video-R1	63.9	45.4	59.3	50.2
FrameMind	64.2	48.6	60.9	57.5

The largest relative gains are on long-form regimes. On VideoMME long, FrameMind improves from 44.7 for the base model and 50.2 for Video-R1 to 57.5 (Ge et al., 28 Sep 2025). The paper also reports that FrameMind exceeds GPT-4V on VideoMME overall in its table, though GPT-4o and Gemini 1.5 Pro remain higher (Ge et al., 28 Sep 2025).

The DRFS ablation is central. At 32-frame evaluation, GRPO-32 reaches 54.0 overall on VideoMME, whereas DRFS-32 reaches 60.9; on long videos the corresponding numbers are 49.5 and 57.5 (Ge et al., 28 Sep 2025). This indicates that the improvement is not reducible to “more frames at inference”; the training regime itself alters the policy’s robustness across temporal-spatial budgets.

The paper also reports markedly smaller fine-tuning data than major baselines: 7.6K for FrameMind versus 260K for Video-R1, 257K for Video-XL, 1.3M for LongVA, 2M for VideoChat2, and 4.4M for Video-CCAM (Ge et al., 28 Sep 2025). This suggests strong sample efficiency, although the method still incurs RL-specific training cost.

5. Position within frame-aware video research

FrameMind sits in a broader family of work that rethinks video understanding around frames, evidence, and temporal structure, but its mechanism is distinct. FrameThinker also uses multi-turn frame-interrogation, with actions such as “choose frames” and “get frame number,” plus GRPO and rule-based Cognitive Consistency Verification; it reports 76.1% on LongVideo-Reason using 20.6 frames on average (He et al., 29 Sep 2025). FrameMind differs in centering tool-mediated retrieval of frames and clips, DRFS-based temporal-spatial training, and a reward design tied to active perception (Ge et al., 28 Sep 2025).

Other systems focus on one-shot selection rather than interleaved reasoning. FrameOracle predicts both frame relevance and how many frames are needed, reducing 16-frame inputs to an average of 10.4 with no loss in accuracy, and reducing 64-frame candidates to 13.9 while improving accuracy by 1.4% (Li et al., 4 Oct 2025). ReFoCUS shifts RL from textual response optimization to frame selection policy optimization, using a reward model derived from a reference LMM and an autoregressive selector (Lee et al., 2 Jun 2025). Event-Anchored Frame Selection constructs event proxies with DINOv2 and query-aware anchors, while WFS-SB uses wavelet-based semantic-boundary detection to segment videos before budget allocation and MMR-based intra-segment selection (Chen et al., 1 Mar 2026, Chen et al., 28 Feb 2026). Adaptive Greedy Frame Selection formalizes pre-answer selection as a monotone submodular objective combining relevance and facility-location coverage (Huang et al., 20 Mar 2026).

A different axis of research targets reuse rather than selection. VLMaxxing studies training-free anti-recomputation for video VLMs, especially same-video follow-up reuse, and reports 14.90–35.92× follow-up latency reduction on Qwen2.5-VL-7B-Instruct-4bit with no observed paired drift across 93 VideoMME breadth queries (Bastien et al., 5 May 2026). This is not a sampling framework, but it addresses the same systems problem of avoiding redundant visual processing.

FrameMind should also be distinguished from frame-centric generative work such as Semantic Frame Interpolation, which defines a task $A$9 for generating intermediate frames between endpoints under text and variable frame-count control (Hong et al., 7 Jul 2025). That line addresses frame-conditioned video synthesis rather than frame-adaptive reasoning.

6. Limitations and open directions

The paper identifies several constraints. FrameMind is capped at 3 turns, which bounds latency but also limits multi-stage evidence gathering on very long or structurally complex videos (Ge et al., 28 Sep 2025). It has no frame-level supervision; all learning is trajectory-level and reward-shaped, so the selector can only be indirectly supervised (Ge et al., 28 Sep 2025). Tool learning is sensitive to reward design: removing the unconditional exploration component in tool reward leads to stagnation in the ablation (Ge et al., 28 Sep 2025). The framework also introduces additional inference stages—reasoning turns, tool execution, and context accumulation—yet the paper does not report detailed wall-clock latency curves or average tool-call counts (Ge et al., 28 Sep 2025).

A second common misconception is that more frames necessarily help. The broader literature directly contradicts that. FrameOracle shows that reducing frames can preserve or improve answer quality (Li et al., 4 Oct 2025), and FrameMind itself is motivated by the failure of static large-frame inputs to balance temporal coverage and spatial detail (Ge et al., 28 Sep 2025). This suggests that frame budget should be treated as a learned control variable rather than a constant.

Several plausible extensions are already visible in adjacent work. FrameOracle makes adaptive count prediction explicit (Li et al., 4 Oct 2025); FrameThinker adds rule-based trajectory verification (He et al., 29 Sep 2025); EFS and WFS-SB impose event-aware structure before selection (Chen et al., 1 Mar 2026, Chen et al., 28 Feb 2026); VLMaxxing addresses same-video state reuse after ingest (Bastien et al., 5 May 2026). This suggests a broader research direction in which FrameMind-like systems combine active retrieval, adaptive budget prediction, event structure, and cache-aware reuse. That implication goes beyond a single paper, but it is consistent with the emerging pattern across frame-aware video reasoning research.