ProactiveVideoQA: Timely Video Interaction
- ProactiveVideoQA is a framework for streaming video understanding that decides both when to answer and whether to prompt further inquiry based on evolving visual evidence.
- Approaches incorporate reinforcement learning, question generation, and modular tool use to optimize response timing and quality, with metrics like PAUC guiding evaluation.
- Benchmarks reveal that proactive strategies, including autonomous response timing and diagnostic questioning, enhance performance while controlling response duplication and latency.
Searching arXiv for papers directly related to ProactiveVideoQA and its neighboring formulations. ProactiveVideoQA denotes a family of video understanding settings in which a system does more than answer a fixed question after consuming a complete clip. In the narrow benchmarked sense, it is the problem in which a video MLLM must not only answer correctly, but must also decide when to answer while a video is still unfolding, choosing at each turn whether to speak or remain silent (Wang et al., 7 Dec 2025). In a broader research sense, the term also covers proactive information acquisition, internally generated questions, quality-aware routing, and other agentic behaviors over temporally structured visual evidence. This broader usage is supported by adjacent work on question-driven retrieval, expert-knowledge elicitation, procedural decomposition, and self-directed evaluation, although those papers do not all study the same benchmark or action space (Liang et al., 2023, Zhang et al., 17 Dec 2025, Choudhury et al., 2023, Xing et al., 9 Oct 2025, Song et al., 12 Mar 2026).
1. Conceptual scope and task boundaries
Standard turn-based VideoQA assumes that a user asks a question and the model answers after the user turn. The proactive setting strengthens this requirement. A model must continuously monitor an incoming video stream and decide, at each opportunity, whether to speak now with a useful answer or remain silent and wait for more evidence (Wang et al., 7 Dec 2025). The output is therefore a sequence of time-stamped responses rather than a single final answer.
Neighboring papers indicate that “proactivity” has at least three operational meanings. One is autonomous response timing during streaming interaction. A second is strategic information acquisition through questioning, where the system asks for additional information when the initial language signal is incomplete (Liang et al., 2023). A third is self-directed evaluative questioning, where the model generates its own diagnostic questions about a video and uses the answers to guide later actions (Song et al., 12 Mar 2026). This suggests that ProactiveVideoQA is best understood as an umbrella over several action regimes rather than a single fixed task definition.
| System | Proactive mechanism | Primary task |
|---|---|---|
| MMDuet2 | Decide whether to answer or output NO REPLY at each turn |
Streaming video QA (Wang et al., 7 Dec 2025) |
| Interactive retrieval baselines | Ask follow-up questions to refine search | Text-to-video retrieval (Liang et al., 2023) |
| EgoExoAsk benchmark | Generate questions that elicit unseen expert knowledge | Video question generation evaluation (Zhang et al., 17 Dec 2025) |
| Q-Router | Route to quality experts and localize artifacts | Video quality QA and assessment (Xing et al., 9 Oct 2025) |
| VQQA | Generate diagnostic questions and use critiques as semantic gradients | Video evaluation and quality improvement (Song et al., 12 Mar 2026) |
| ProViQ | Generate executable programs over video modules | Zero-shot VideoQA (Choudhury et al., 2023) |
A recurrent source of confusion is the assumption that proactive video QA is merely “faster” VideoQA. The literature draws sharper distinctions. Some systems decide when to answer; some decide what to ask; some decide which experts or tools to invoke; and some generate internal questions for self-critique. These are related but not interchangeable behaviors.
2. Streaming response timing as the benchmarked core
The clearest benchmarked formulation appears in "MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning" (Wang et al., 7 Dec 2025). There, proactive interaction is serialized into a standard chat format. At each assistant turn, the model generates either a normal textual answer or the text NO REPLY. This text-to-text formulation avoids adding a specialized timing head and makes silence an explicit action in token space.
In this setting, the model sees only partial evidence. At each user turn it receives only the current newly arrived visual chunk, with 1 or 2 sampled video frames in the reported setup, while the dialogue history carries forward earlier information through the autoregressive chat context (Wang et al., 7 Dec 2025). The system prompt instructs the assistant to include information from the video since its last reply and to output NO REPLY if the available segment cannot answer the question.
MMDuet2 is built by post-training Qwen2.5-VL 3B and uses a two-stage pipeline: supervised fine-tuning (SFT) followed by multi-turn reinforcement learning (RL) (Wang et al., 7 Dec 2025). The paper identifies two SFT-only failure modes: low response frequency, because most turns are NO REPLY, and high latency, because gold answers are placed at the end of their reply spans. RL addresses this without requiring precise response-time annotations. Instead of frame-level supervision, it optimizes a trajectory-level reward derived from PAUC (Proactive Area Under Curve), which jointly rewards correctness and earliness.
The reward is not PAUC alone. The paper adds a replication reward to suppress duplicate replies, an in-span reward to penalize answers outside any ground-truth reply span, and a prefix reward to penalize responses that begin by repeating prior content (Wang et al., 7 Dec 2025). This design matters because optimizing timeliness in isolation can produce pathological over-talking.
The training data contribution is also central. MMDuet2 uses a 52k-video proactive dialogue dataset with two dialogue types: 1QnA, where a single question is asked at the beginning and answered incrementally across scenes, and nQnA, where multiple questions can be asked at different times and the model must handle question switching (Wang et al., 7 Dec 2025). This makes the benchmark more than a one-shot early-answer task.
Quantitatively, the RL model is reported as state of the art on the ProactiveVideoQA benchmark (Wang et al., 7 Dec 2025). On [WEB], [EGO], [TV], and [VAD], MMDuet2 RL reaches 53.3, 33.6, 43.4, and 28.9 PAUC, respectively, with duplicate reply proportions 4.2, 8.1, 1.0, and 15.2. The corresponding SFT model is markedly weaker on PAUC despite often having lower duplication, while the earlier MMDuet baseline can achieve competitive PAUC on some splits only with extremely high duplicate reply proportions such as 81.3, 99.4, 92.8, and 99.2 (Wang et al., 7 Dec 2025). The hardest category is explicitly identified as [VAD].
A practical finding is that training and inference frame rates should differ. The paper reports that SFT at 1-second frame interval fails, whereas RL remains robust, and inference at 1-second interval is significantly better than 2 seconds because denser decision opportunities allow earlier speaking (Wang et al., 7 Dec 2025). This establishes response timing as an online control problem rather than only a decoding problem.
3. Proactive information acquisition and question generation
A second lineage treats proactivity as asking questions in order to reduce uncertainty. "Simple Baselines for Interactive Video Retrieval with Questions and Answers" studies iterative retrieval rather than direct VideoQA, but it is highly relevant because the system is allowed multiple rounds of question-answer interaction to refine an underspecified text query (Liang et al., 2023). The method includes a text encoder, visual encoder, ranker, and question subsystem. After each round, the current query is updated by cumulative concatenation with a separator token:
The paper explores three question-generation strategies: Heuristic, Auto-text, and Auto-text-vid (Liang et al., 2023). Heuristic uses object-sensitive templates; Auto-text generates a question from the current text query alone; Auto-text-vid conditions on both the current query and captions of the top retrieved videos. The answers are not collected from humans during experiments. Instead, a BLIP VideoQA model simulates the user’s response, which allows the productive study of interactive retrieval without access to ground-truth dialogue data (Liang et al., 2023). Performance generally improves with more rounds and stabilizes beyond about six rounds; for the heuristic generator, using 6 rounds works best.
A closely related but more specialized problem is expert-knowledge elicitation. "Evaluating the Capability of Video Question Generation for Expert Knowledge Elicitation" reframes video question generation around questions whose answers lie in unseen expert knowledge rather than visible scene facts (Zhang et al., 17 Dec 2025). Its key contribution is an evaluation protocol that simulates communication with experts using a question-to-answer retriever trained on EgoExoAsk, a dataset of 27,666 QA pairs derived from Ego-Exo4D expert commentary. For each segment, a generated question is judged by whether it retrieves the segment’s positive expert comments from a retrieval pool of size .
The benchmark reports Recall@k, MeanR, and MedianR rather than reference overlap (Zhang et al., 17 Dec 2025). The best reported model setting is QwenVL-2.5, Ego view, w/ caption, 2 FPS, with R@1 = 0.1140, R@5 = 0.3395, MeanR = 14.19, and MedianR = 10. The paper also reports that richer context improves question quality: adding atomic descriptions improves performance over naive generation, and higher FPS helps in the naive QwenVL-2.5 setting. This indicates that proactive questioning can be benchmarked even when no single gold question exists.
A more self-contained form of proactivity appears in "VQQA: An Agentic Approach for Video Evaluation and Quality Improvement" (Song et al., 12 Mar 2026). VQQA does not wait for an external question. It dynamically generates targeted visual questions about Video-Prompt Alignment, Visual Quality, and Condition Fidelity, answers them with normalized scores in , and uses low-scoring answers as semantic gradients for prompt refinement. The framework comprises a Question Generation Agent, Question Answering Agent, Prompt Refinement Agent, and a Global VLM Rater. Its update rule is written as
This is proactive VideoQA in a strong sense: the system decides what to ask, interprets the answers, and acts on them.
These papers collectively show that proactive video systems may acquire information from users, from experts, or from the model’s own internally generated diagnostic questions. The common thread is that questioning is treated as an active control mechanism rather than a passive evaluation endpoint.
4. Agentic architectures and modular reasoning
A third research direction emphasizes explicit decomposition, tool use, and expert orchestration. "Zero-Shot Video Question Answering with Procedural Programs" proposes ProViQ (Procedural Video Querying), which answers zero-shot questions about videos by generating short executable programs over an API of visual modules (Choudhury et al., 2023). The architectural claim is that video understanding can be framed as LLM-driven tool use over temporally structured visual evidence, with explicit decomposition into subgoals such as “find the relevant frames,” “crop the object,” “query a clip,” “read the transcript,” and “choose an option.” The paper reports state-of-the-art results with improvements of up to 25% on short, long, open-ended, and multimodal video question-answering datasets (Choudhury et al., 2023). For ProactiveVideoQA, the relevance is that proactive behavior can be grounded in explicit subgoal structure rather than a single monolithic forward pass.
Quality-oriented agentic reasoning is developed in "Q-Router: Agentic Video Quality Assessment with Expert Model Routing and Artifact Localization" (Xing et al., 9 Oct 2025). Q-Router treats video quality assessment as a tool-using orchestration problem. A VLM, identified in experiments as GPT-4o, acts as a router over experts including COVER, DOVER/DOVER++, UVQ, MaxVQA, ModularBVQA/BVQA, and T2VQA. The architecture has three tiers: Tier 0 routes to a single expert, Tier 1 selects multiple relevant experts and fuses them, and Tier 2 adds spatiotemporal artifact localization.
The routing mechanism is prompt-driven rather than learned end to end. In Tier 1, the paper gives the weighting rule:
followed by normalization to sum to 1 (Xing et al., 9 Oct 2025). If inter-expert disagreement exceeds 20 points, the aggregation rule switches from weighted average to weighted median. This is a concrete uncertainty-sensitive fusion policy.
The interpretability layer is substantial. Tier 2 includes Probabilistic Extractor (PE), HysteresisClips, DiversifiedSelection, VLM-Based Artifact Filtering, Motion-Compensated Perceptual Difference Mapping, and the broader Artifact Localization Pipeline (Xing et al., 9 Oct 2025). The output includes representative frame pairs, severity scores, heatmaps, overlays, and structured JSON summaries. On Q-Bench-Video, described as the first and currently only benchmark dedicated to video quality question answering, Q-Router Tier 2 reaches 60.07 on the dev split versus 56.91 for GPT-4o, while Tier 1 reaches 59.45 overall on the test split versus 56.50 for GPT-4o (Xing et al., 9 Oct 2025). This positions Q-Router not as a generic open-ended VideoQA model, but as a strong quality-aware reasoning component that could be embedded in a broader proactive agent.
Continual adaptation adds another form of modularity. "Empowering LLM for Continual Video Question Answering with Collaborative Prompting" proposes ColPro, a rehearsal-free continual VideoQA framework built on frozen LLaMA-7B and CLIP ViT-L/14 (Cai et al., 2024). It decomposes adaptation into Task-specific Question Constraint Prompting (TQCP), Visual Temporal Awareness Prompting (VTAP), and Knowledge Acquisition Prompting (KAP). ColPro is not itself a proactive timing method, but it addresses an issue central to long-lived proactive systems: new tasks arrive sequentially and previous task data are inaccessible. The paper reports 55.14\% accuracy on NExT-QA and 71.24\% on DramaQA, with improved forgetting relative to baselines (Cai et al., 2024). A plausible implication is that proactive systems deployed over evolving video streams require not only online timing policies but also rehearsal-free mechanisms for retaining prior question-answering capabilities.
5. Evaluation protocols, datasets, and performance criteria
The evaluation landscape around ProactiveVideoQA is heterogeneous because different papers operationalize proactivity differently. In the streaming-response formulation, the central metric is PAUC, which jointly captures response correctness and timing across a ground-truth reply span (Wang et al., 7 Dec 2025). The benchmark is reported over [WEB], [EGO], [TV], and [VAD], and duplicate reply proportion is tracked separately because a system could otherwise inflate PAUC by replying too often.
For proactive quality reasoning, the principal benchmark is Q-Bench-Video, which measures accuracy on quality-related visual QA across question types such as Yes-or-No, What/How, and Open-ended, and across concern categories including Technical, Aesthetic, Temporal, and AIGC (Xing et al., 9 Oct 2025). This benchmark matters because it evaluates whether quality representations can participate in language-mediated reasoning rather than merely predicting MOS-aligned scores.
For proactive question generation, EgoExoAsk provides a different evaluation paradigm (Zhang et al., 17 Dec 2025). It is constructed from Ego-Exo4D expert commentary and supports offline assessment of whether generated questions retrieve the correct segment-specific expert comments. This evaluation does not measure downstream task success directly; it measures whether a question is likely to elicit useful unseen knowledge.
For continual adaptation, NExT-QA and DramaQA are reorganized into function-incremental sequences of tasks (Cai et al., 2024). Here the central metrics are Avg. Acc and Avg. Fog rather than time-sensitive response quality. The task orders are deliberately chosen to induce strong forgetting, which makes them informative for long-horizon deployment scenarios.
For self-directed evaluation in video generation, VQQA uses a different score ecology. The QA agent answers scalar prompts beginning with “On a scale of 0-100, how...,” while a separate Global Score evaluates each candidate against the original prompt and optional reference images (Song et al., 12 Mar 2026). The stopping rule is based on either target satisfaction or score saturation. On T2V-CompBench, the paper reports +11.57% absolute improvement over vanilla generation, and on VBench2 it reports +8.43% (Song et al., 12 Mar 2026). These are not conventional VideoQA numbers, but they show that proactive questioning can be used as a control signal.
Taken together, these benchmarks imply that “good” proactive behavior cannot be captured by a single metric. Depending on the task, one may need to evaluate earliness, semantic correctness, duplication, question utility, expert-comment retrieval, calibration under expert disagreement, or global alignment to an original intent.
6. Limitations, misconceptions, and open problems
The literature also delineates the limits of current approaches. In the streaming setting, RL improves timing and answer quality, but performance on [VAD] surveillance videos remains weak for all reported models (Wang et al., 7 Dec 2025). This shows that proactive timing does not solve domain difficulty. The same paper also reports generalization issues and repetition on long ego-centric videos.
Interactive retrieval demonstrates that simple question-asking can improve search without ground-truth dialogue data, but it does not solve semantic VideoQA about a retrieved clip (Liang et al., 2023). Expert-elicitation evaluation shows how to score question usefulness at scale, yet the authors explicitly note that retrieval-based simulation is not real expert interaction and does not capture multi-turn dialogue, uncertainty estimation, or downstream utility (Zhang et al., 17 Dec 2025).
Q-Router exposes another set of limits. Its expert routing depends on prompt-following and semantic analysis by the VLM; its confidence is produced in structured JSON rather than by a formal uncertainty model; Tier 2 increases compute cost through probabilistic extraction, VLM artifact filtering, optical flow, LPIPS maps, and visualization; and if the expert pool lacks coverage for a new artifact type, routing can only choose among inadequate tools (Xing et al., 9 Oct 2025). These are especially consequential for proactive systems that must act before being asked.
ColPro reduces forgetting but does not provide true future-event modeling, streaming latency analysis, or long-term memory over arbitrarily long videos (Cai et al., 2024). VQQA is explicitly limited by both the underlying generator and the VLM evaluator: it cannot rectify fundamental model flaws, may follow noisy semantic gradients when the evaluator hallucinates, and incurs more sequential latency than purely parallel Best-of- schemes (Song et al., 12 Mar 2026). ProViQ, although strong in zero-shot decomposition, is not itself a proactive timing framework (Choudhury et al., 2023).
A persistent misconception is that all proactive video methods are instances of the same problem. The surveyed work indicates otherwise. Some methods are about timely answering during playback; some are about asking follow-up questions to a user; some are about eliciting hidden expert commentary; some are about quality-aware expert routing; and some are about self-critique for generated videos. This suggests that ProactiveVideoQA is less a single benchmark than a broader systems agenda: video models should decide not only what to say, but also when to say it, what to ask, which tools to invoke, and how to justify or revise their conclusions under partial and evolving evidence.
