- The paper introduces a two-stage omni-modal agent that first scans a low-cost global preview, then learns whether and where to retrieve high-fidelity audio-video evidence using a temporal zoom tool.
- TimeAnchor binds interleaved audio-video tokens to absolute wall-clock time, improving Charades-STA mIoU from 37.9 to 41.3 and raising answer accuracy on OmniVideoBench and LVOmniBench.
- OmniReasoner combines supervised tool-use training with GRPO and improves Qwen2.5-Omni-7B across six benchmarks, with gains reaching 9.9 points on 10–30 minute videos and 15.6 points on VideoHolmes.
OmniReasoner addresses a specific bottleneck in omnimodal LLM reasoning over long audio-video streams: the decisive evidence for a query is typically sparse in time, distributed across modalities, and destroyed by uniform downsampling. Rather than improving post-hoc reasoning over a fixed observation budget, the paper trains omni-modal models to decide whether and where to acquire additional high-fidelity evidence before answering, via a learned zoom-in tool call. The framework is instantiated on Qwen2.5-Omni-7B and yields consistent gains over its base model across six benchmarks, with gains that grow monotonically with input duration (2607.19339).
Framework overview
OmniReasoner operates in two stages. The model first consumes a low-cost global observation that preserves the full temporal structure of the input at low perceptual fidelity. Conditioned on this observation and the question, the policy takes one of two actions: answer directly, or emit a zoom call specifying a temporal interval [s,e] on the original timeline. The media environment then returns a densely sampled local audio-video clip for that interval, and the final answer is generated from the question, the global context, and the retrieved evidence. The zoom-in decision is thus a learned tool-use behavior rather than a fixed preprocessing heuristic, and the post-training objective covers not only answering but also when to invoke the tool and where to point it.
TimeAnchor: temporal grounding across sampling granularities
The central technical contribution is TimeAnchor, which solves a grounding problem specific to cross-granularity tool use. Qwen-Omni interleaves audio and video tokens in two-second chunks, with time carried implicitly by chunk index relative to the sampled frame grid. When the model moves from a sparse global preview to a densely sampled zoomed clip, the same physical moment falls under a different grid, so a frame- or chunk-index argument does not round-trip to the same moment. TimeAnchor prepends a plain-text absolute-time marker (e.g., \<32 seconds>) to every two-second chunk, binding interleaved video and audio tokens to wall-clock time. Zoom arguments are therefore emitted in absolute seconds and resolved against the source timeline, independent of the sampling rate at either stage.
Notably, the mechanism adds no special tokens, no architectural change, and no separate localization loss. Its ablation shows substantial effects: on Charades-STA, TimeAnchor raises [email protected] from 58.8 to 64.9, [email protected] from 37.5 to 41.1, and mIoU from 37.9 to 41.3 relative to the same model without anchors, and it also improves answer accuracy on OmniVideoBench (32.3 → 34.8) and LVOmniBench (32.8 → 35.4). This indicates that a stable cross-granularity timeline benefits not only interval prediction but also downstream answering.
Temporal Augmented Data Engine
Supervision for tool-use behavior requires both answers and evidence intervals, which are scarce at scale. The paper avoids manual span annotation through temporal editing. Multi-segment composition stitches semantically independent clips into longer streams, so each question's source-segment offset is the ground-truth evidence interval by construction; the model must locate the relevant segment among temporal distractors. Anomaly insertion replaces the audio, visual, or both modalities within a known window of a long stream, yielding typed anomaly detection with interval labels for free. These constructions produce 13,222 and 5,319 SFT examples respectively, complemented by FineVideo open-ended trajectories, AVQA-R1 image-audio data, and CG-Bench long-video QA, for a 25,839-example SFT mixture.
Trajectories are synthesized through MediaSandbox, an FFmpeg-based environment that exposes the same global-to-local interface used at inference. Two synthesis modes are used: online synthesis, where a teacher (Gemini-3-Flash-Preview) generates trajectories without access to reference answers, and hindsight synthesis for harder long videos, where the teacher receives the reference answer and interval as guidance but is explicitly instructed to justify zoom decisions only from observable cues. Filtering includes structural rules, leakage checks, answer verification, and an LLM judge for grounding quality. The RL mixture (2,731 examples) is mined by difficulty: only prompts where the SFT policy produces mixed outcomes across eight rollouts are kept, discarding both saturated and unsolvable cases.
Agentic reinforcement learning
Post-training follows an SFT cold-start followed by GRPO. The reward is deliberately minimal—answer accuracy plus format compliance—with no standalone reward for invoking the tool and no localization reward; interval quality is optimized only through its effect on final-answer correctness. This design choice is supported by the RL-only ablation: applying GRPO directly to the base model, even under a forced two-stage template, degrades performance sharply (e.g., 27.6 on OmniVideoBench versus 34.8 for the full recipe, below even the base model's 29.3), because the base model lacks tool-use priors and drifts out of format during optimization. Cold-start SFT is therefore a necessary precondition for stable agentic RL on this 7B base model.
Empirical results
Across benchmarks, OmniReasoner improves over Qwen2.5-Omni-7B on all six evaluated benchmarks:
| Benchmark |
Qwen2.5-Omni-7B |
OmniReasoner |
Gain |
| OmniVideoBench |
29.3 |
34.8 |
+5.5 |
| LVOmniBench |
32.0 |
35.4 |
+3.4 |
| Daily-Omni |
62.1 |
64.2 |
+2.1 |
| WorldSense |
45.4 |
46.7 |
+1.3 |
| VideoMME |
64.3 |
65.4 |
+1.1 |
| VideoHolmes |
24.4 |
40.0 |
+15.6 |
The largest gains occur on benchmarks stressing sparse-evidence localization in long contexts, while gains on short-to-medium-clip benchmarks (Daily-Omni, WorldSense) are moderate—consistent with the framework's motivation, and indicating that long-audio-video tool-use post-training does not sacrifice general audio-visual competence. The VideoHolmes gain is partly attributed to the baseline's fixed 32-frame budget versus OmniReasoner's retrieved frames, a caveat the authors state explicitly.
Duration-stratified results on OmniVideoBench show the expected scaling: gains grow from +3.2 points on 0–5 minute videos to +6.6 on 5–10 minutes and +9.9 on 10–30 minutes. Tool-call routing follows the same pattern: on OmniVideoBench, the zoom-call rate rises from 33.3% for 0–1 minute inputs to 84.8% for 10–30 minutes, and on LVOmniBench it reaches 93.6% for 50–90 minute videos.
Faithfulness of interleaved evidence
The paper directly addresses a known concern that interleaved media tokens in reasoning traces may be decorative rather than causally used. Two interventions support genuine use: removing the tool-returned clip at inference time degrades accuracy across benchmarks, and attention-rollout analysis over the final transformer layers shows the rollout concentrating strongly on the zoom-in clip's tokens when predicting the final answer. The attention analysis is correlational rather than causal on its own, but combined with the removal ablation it provides reasonable evidence that retrieved evidence contributes to prediction.
Limitations and open questions
The authors concede several constraints. The framework is limited to a single two-step zoom interaction, partly because multi-turn agentic RL infrastructure for omni-modal models is immature: veRL and OpenRLHF do not natively support audio-conditioned inputs and repeated audio-video tool returns, so the authors built a custom TRL-based framework that is less mature than its text/vision counterparts. The 7B base model with a 32K context window was not pretrained on agentic data, which is why direct RL fails without supervised warm-up. The tool repertoire is limited to perceptual zoom; web search, code execution, and structured query tools are unexplored, limiting applicability to tasks requiring external knowledge or symbolic computation. Whether the paradigm scales to larger, tool-aware omni base models and multi-turn tool chains remains an open empirical question the paper does not resolve.
Conclusion
OmniReasoner reframes long audio-video reasoning as adaptive evidence acquisition: a learned policy decides whether to spend high-fidelity computation, and where, on the original timeline. TimeAnchor provides the minimal mechanism that keeps temporal arguments valid across sampling-granularity boundaries, and the Temporal Augmented Data Engine makes the behavior trainable without manual interval annotation. The empirical pattern—gains increasing with input duration, tool-call rate increasing with input duration, and RL failing without SFT warm-up—supports the paper's central claim that selective evidence acquisition, rather than uniform perception or post-hoc reasoning alone, is a productive training target for omni-modal models.
