---
title: 'ProactiveVideoQA: Timely Video Interaction'
url: https://www.emergentmind.com/topics/proactivevideoqa
type: topic
---

# ProactiveVideoQA: Timely Video Interaction

Searching arXiv for papers directly related to ProactiveVideoQA and its neighboring formulations.
ProactiveVideoQA denotes a family of video understanding settings in which a system does more than answer a fixed question after consuming a complete clip. In the narrow benchmarked sense, it is the problem in which a video MLLM must not only answer correctly, but must also decide **when** to answer while a video is still unfolding, choosing at each turn whether to speak or remain silent [2512.06810]. In a broader research sense, the term also covers proactive information acquisition, internally generated questions, quality-aware routing, and other agentic behaviors over temporally structured visual evidence. This broader usage is supported by adjacent work on question-driven retrieval, expert-knowledge elicitation, procedural decomposition, and self-directed evaluation, although those papers do not all study the same benchmark or action space [2308.10402][2512.15006][2312.00937][2510.08789][2603.12310].

## 1. Conceptual scope and task boundaries

Standard turn-based VideoQA assumes that a user asks a question and the model answers after the user turn. The proactive setting strengthens this requirement. A model must continuously monitor an incoming video stream and decide, at each opportunity, whether to **speak now** with a useful answer or **remain silent** and wait for more evidence [2512.06810]. The output is therefore a sequence of time-stamped responses rather than a single final answer.

Neighboring papers indicate that “proactivity” has at least three operational meanings. One is **autonomous response timing** during streaming interaction. A second is **strategic information acquisition through questioning**, where the system asks for additional information when the initial language signal is incomplete [2308.10402]. A third is **self-directed evaluative questioning**, where the model generates its own diagnostic questions about a video and uses the answers to guide later actions [2603.12310]. This suggests that ProactiveVideoQA is best understood as an umbrella over several action regimes rather than a single fixed task definition.

| System | Proactive mechanism | Primary task |
|---|---|---|
| MMDuet2 | Decide whether to answer or output `NO REPLY` at each turn | Streaming video QA [2512.06810] |
| Interactive retrieval baselines | Ask follow-up questions to refine search | Text-to-video retrieval [2308.10402] |
| EgoExoAsk benchmark | Generate questions that elicit unseen expert knowledge | Video question generation evaluation [2512.15006] |
| Q-Router | Route to quality experts and localize artifacts | Video quality QA and assessment [2510.08789] |
| VQQA | Generate diagnostic questions and use critiques as semantic gradients | Video evaluation and quality improvement [2603.12310] |
| ProViQ | Generate executable programs over video modules | Zero-shot VideoQA [2312.00937] |

A recurrent source of confusion is the assumption that proactive video QA is merely “faster” VideoQA. The literature draws sharper distinctions. Some systems decide when to answer; some decide what to ask; some decide which experts or tools to invoke; and some generate internal questions for self-critique. These are related but not interchangeable behaviors.

## 2. Streaming response timing as the benchmarked core

The clearest benchmarked formulation appears in "MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning" [2512.06810]. There, proactive interaction is serialized into a standard chat format. At each assistant turn, the model generates either a normal textual answer or the text `NO REPLY`. This text-to-text formulation avoids adding a specialized timing head and makes silence an explicit action in token space.

In this setting, the model sees only partial evidence. At each user turn it receives only the current newly arrived visual chunk, with **1 or 2 sampled video frames** in the reported setup, while the dialogue history carries forward earlier information through the autoregressive chat context [2512.06810]. The system prompt instructs the assistant to include information from the video since its last reply and to output `NO REPLY` if the available segment cannot answer the question.

MMDuet2 is built by post-training **Qwen2.5-VL 3B** and uses a two-stage pipeline: **supervised fine-tuning (SFT)** followed by **multi-turn reinforcement learning (RL)** [2512.06810]. The paper identifies two SFT-only failure modes: **low response frequency**, because most turns are `NO REPLY`, and **high latency**, because gold answers are placed at the end of their reply spans. RL addresses this without requiring precise response-time annotations. Instead of frame-level supervision, it optimizes a trajectory-level reward derived from **PAUC (Proactive Area Under Curve)**, which jointly rewards correctness and earliness.

The reward is not PAUC alone. The paper adds a **replication reward** to suppress duplicate replies, an **in-span reward** to penalize answers outside any ground-truth reply span, and a **prefix reward** to penalize responses that begin by repeating prior content [2512.06810]. This design matters because optimizing timeliness in isolation can produce pathological over-talking.

The training data contribution is also central. MMDuet2 uses a **52k-video** proactive dialogue dataset with two dialogue types: **1QnA**, where a single question is asked at the beginning and answered incrementally across scenes, and **nQnA**, where multiple questions can be asked at different times and the model must handle question switching [2512.06810]. This makes the benchmark more than a one-shot early-answer task.

Quantitatively, the RL model is reported as state of the art on the ProactiveVideoQA benchmark [2512.06810]. On **[WEB]**, **[EGO]**, **[TV]**, and **[VAD]**, MMDuet2 RL reaches **53.3**, **33.6**, **43.4**, and **28.9** PAUC, respectively, with duplicate reply proportions **4.2**, **8.1**, **1.0**, and **15.2**. The corresponding SFT model is markedly weaker on PAUC despite often having lower duplication, while the earlier MMDuet baseline can achieve competitive PAUC on some splits only with extremely high duplicate reply proportions such as **81.3**, **99.4**, **92.8**, and **99.2** [2512.06810]. The hardest category is explicitly identified as **[VAD]**.

A practical finding is that training and inference frame rates should differ. The paper reports that **SFT at 1-second frame interval fails**, whereas RL remains robust, and **inference at 1-second interval is significantly better** than 2 seconds because denser decision opportunities allow earlier speaking [2512.06810]. This establishes response timing as an online control problem rather than only a decoding problem.

## 3. Proactive information acquisition and question generation

A second lineage treats proactivity as asking questions in order to reduce uncertainty. "Simple Baselines for Interactive Video Retrieval with Questions and Answers" studies iterative retrieval rather than direct VideoQA, but it is highly relevant because the system is allowed multiple rounds of question-answer interaction to refine an underspecified text query [2308.10402]. The method includes a text encoder, visual encoder, ranker, and question subsystem. After each round, the current query is updated by cumulative concatenation with a separator token:
$$
Q_{i+1} = [Q_i \; \texttt{[SEP]} \; q_i \; a_i].
$$

The paper explores three question-generation strategies: **Heuristic**, **Auto-text**, and **Auto-text-vid** [2308.10402]. Heuristic uses object-sensitive templates; Auto-text generates a question from the current text query alone; Auto-text-vid conditions on both the current query and captions of the top retrieved videos. The answers are not collected from humans during experiments. Instead, a **BLIP VideoQA model** simulates the user’s response, which allows the productive study of interactive retrieval without access to ground-truth dialogue data [2308.10402]. Performance generally improves with more rounds and stabilizes beyond about six rounds; for the heuristic generator, **using 6 rounds works best**.

A closely related but more specialized problem is expert-knowledge elicitation. "Evaluating the Capability of Video Question Generation for Expert Knowledge Elicitation" reframes video question generation around questions whose answers lie in **unseen expert knowledge** rather than visible scene facts [2512.15006]. Its key contribution is an evaluation protocol that simulates communication with experts using a question-to-answer retriever trained on **EgoExoAsk**, a dataset of **27,666 QA pairs** derived from Ego-Exo4D expert commentary. For each segment, a generated question is judged by whether it retrieves the segment’s positive expert comments from a retrieval pool of size **\(L=50\)**.

The benchmark reports **Recall@k**, **MeanR**, and **MedianR** rather than reference overlap [2512.15006]. The best reported model setting is **QwenVL-2.5, Ego view, w/ caption, 2 FPS**, with **R@1 = 0.1140**, **R@5 = 0.3395**, **MeanR = 14.19**, and **MedianR = 10**. The paper also reports that richer context improves question quality: adding atomic descriptions improves performance over naive generation, and higher FPS helps in the naive QwenVL-2.5 setting. This indicates that proactive questioning can be benchmarked even when no single gold question exists.

A more self-contained form of proactivity appears in "VQQA: An Agentic Approach for Video Evaluation and Quality Improvement" [2603.12310]. VQQA does not wait for an external question. It dynamically generates targeted visual questions about **Video-Prompt Alignment**, **Visual Quality**, and **Condition Fidelity**, answers them with normalized scores in **\([0,100]\)**, and uses low-scoring answers as **semantic gradients** for prompt refinement. The framework comprises a **Question Generation Agent**, **Question Answering Agent**, **Prompt Refinement Agent**, and a **Global VLM Rater**. Its update rule is written as
$$
p_{t+1} = \text{VLM}(p_t, R_{f,t}).
$$
This is proactive VideoQA in a strong sense: the system decides what to ask, interprets the answers, and acts on them.

These papers collectively show that proactive video systems may acquire information from users, from experts, or from the model’s own internally generated diagnostic questions. The common thread is that questioning is treated as an active control mechanism rather than a passive evaluation endpoint.

## 4. Agentic architectures and modular reasoning

A third research direction emphasizes explicit decomposition, tool use, and expert orchestration. "Zero-Shot Video Question Answering with Procedural Programs" proposes **ProViQ (Procedural Video Querying)**, which answers zero-shot questions about videos by generating short executable programs over an API of visual modules [2312.00937]. The architectural claim is that video understanding can be framed as **LLM-driven tool use over temporally structured visual evidence**, with explicit decomposition into subgoals such as “find the relevant frames,” “crop the object,” “query a clip,” “read the transcript,” and “choose an option.” The paper reports state-of-the-art results with improvements of **up to 25%** on short, long, open-ended, and multimodal video question-answering datasets [2312.00937]. For ProactiveVideoQA, the relevance is that proactive behavior can be grounded in explicit subgoal structure rather than a single monolithic forward pass.

Quality-oriented agentic reasoning is developed in "Q-Router: Agentic Video Quality Assessment with Expert Model Routing and Artifact Localization" [2510.08789]. Q-Router treats video quality assessment as a tool-using orchestration problem. A VLM, identified in experiments as **GPT-4o**, acts as a router over experts including **COVER, DOVER/DOVER++, UVQ, MaxVQA, ModularBVQA/BVQA, and T2VQA**. The architecture has three tiers: **Tier 0** routes to a single expert, **Tier 1** selects multiple relevant experts and fuses them, and **Tier 2** adds spatiotemporal artifact localization.

The routing mechanism is prompt-driven rather than learned end to end. In Tier 1, the paper gives the weighting rule:
$$
\text{weight\_i} = \text{base\_i x (1 + 0.5 x specialty\_match + 0.3 x agreement\_boost + 0.2 x confidence\_prior - 0.3 x oob\_penalty)}
$$
followed by normalization to sum to 1 [2510.08789]. If inter-expert disagreement exceeds 20 points, the aggregation rule switches from weighted average to weighted median. This is a concrete uncertainty-sensitive fusion policy.

The interpretability layer is substantial. Tier 2 includes **Probabilistic Extractor (PE)**, **HysteresisClips**, **DiversifiedSelection**, **VLM-Based Artifact Filtering**, **Motion-Compensated Perceptual Difference Mapping**, and the broader **Artifact Localization Pipeline** [2510.08789]. The output includes representative frame pairs, severity scores, heatmaps, overlays, and structured JSON summaries. On **Q-Bench-Video**, described as the first and currently only benchmark dedicated to video quality question answering, Q-Router Tier 2 reaches **60.07** on the dev split versus **56.91** for GPT-4o, while Tier 1 reaches **59.45** overall on the test split versus **56.50** for GPT-4o [2510.08789]. This positions Q-Router not as a generic open-ended VideoQA model, but as a strong quality-aware reasoning component that could be embedded in a broader proactive agent.

Continual adaptation adds another form of modularity. "Empowering Large Language Model for Continual Video Question Answering with Collaborative Prompting" proposes **ColPro**, a rehearsal-free continual VideoQA framework built on frozen **LLaMA-7B** and **CLIP ViT-L/14** [2410.00771]. It decomposes adaptation into **Task-specific Question Constraint Prompting (TQCP)**, **Visual Temporal Awareness Prompting (VTAP)**, and **Knowledge Acquisition Prompting (KAP)**. ColPro is not itself a proactive timing method, but it addresses an issue central to long-lived proactive systems: new tasks arrive sequentially and previous task data are inaccessible. The paper reports **55.14\%** accuracy on NExT-QA and **71.24\%** on DramaQA, with improved forgetting relative to baselines [2410.00771]. A plausible implication is that proactive systems deployed over evolving video streams require not only online timing policies but also rehearsal-free mechanisms for retaining prior question-answering capabilities.

## 5. Evaluation protocols, datasets, and performance criteria

The evaluation landscape around ProactiveVideoQA is heterogeneous because different papers operationalize proactivity differently. In the streaming-response formulation, the central metric is **PAUC**, which jointly captures response correctness and timing across a ground-truth reply span [2512.06810]. The benchmark is reported over **[WEB]**, **[EGO]**, **[TV]**, and **[VAD]**, and duplicate reply proportion is tracked separately because a system could otherwise inflate PAUC by replying too often.

For proactive quality reasoning, the principal benchmark is **Q-Bench-Video**, which measures accuracy on quality-related visual QA across question types such as **Yes-or-No**, **What/How**, and **Open-ended**, and across concern categories including **Technical**, **Aesthetic**, **Temporal**, and **AIGC** [2510.08789]. This benchmark matters because it evaluates whether quality representations can participate in language-mediated reasoning rather than merely predicting MOS-aligned scores.

For proactive question generation, **EgoExoAsk** provides a different evaluation paradigm [2512.15006]. It is constructed from Ego-Exo4D expert commentary and supports offline assessment of whether generated questions retrieve the correct segment-specific expert comments. This evaluation does not measure downstream task success directly; it measures whether a question is likely to elicit useful unseen knowledge.

For continual adaptation, **NExT-QA** and **DramaQA** are reorganized into function-incremental sequences of tasks [2410.00771]. Here the central metrics are **Avg. Acc** and **Avg. Fog** rather than time-sensitive response quality. The task orders are deliberately chosen to induce strong forgetting, which makes them informative for long-horizon deployment scenarios.

For self-directed evaluation in video generation, VQQA uses a different score ecology. The QA agent answers scalar prompts beginning with “On a scale of 0-100, how...,” while a separate **Global Score** evaluates each candidate against the original prompt and optional reference images [2603.12310]. The stopping rule is based on either target satisfaction or score saturation. On **T2V-CompBench**, the paper reports **+11.57%** absolute improvement over vanilla generation, and on **VBench2** it reports **+8.43%** [2603.12310]. These are not conventional VideoQA numbers, but they show that proactive questioning can be used as a control signal.

Taken together, these benchmarks imply that “good” proactive behavior cannot be captured by a single metric. Depending on the task, one may need to evaluate earliness, semantic correctness, duplication, question utility, expert-comment retrieval, calibration under expert disagreement, or global alignment to an original intent.

## 6. Limitations, misconceptions, and open problems

The literature also delineates the limits of current approaches. In the streaming setting, RL improves timing and answer quality, but performance on **[VAD] surveillance videos** remains weak for all reported models [2512.06810]. This shows that proactive timing does not solve domain difficulty. The same paper also reports generalization issues and repetition on long ego-centric videos.

Interactive retrieval demonstrates that simple question-asking can improve search without ground-truth dialogue data, but it does not solve semantic VideoQA about a retrieved clip [2308.10402]. Expert-elicitation evaluation shows how to score question usefulness at scale, yet the authors explicitly note that retrieval-based simulation is not real expert interaction and does not capture multi-turn dialogue, uncertainty estimation, or downstream utility [2512.15006].

Q-Router exposes another set of limits. Its expert routing depends on prompt-following and semantic analysis by the VLM; its confidence is produced in structured JSON rather than by a formal uncertainty model; Tier 2 increases compute cost through probabilistic extraction, VLM artifact filtering, optical flow, LPIPS maps, and visualization; and if the expert pool lacks coverage for a new artifact type, routing can only choose among inadequate tools [2510.08789]. These are especially consequential for proactive systems that must act before being asked.

ColPro reduces forgetting but does not provide true future-event modeling, streaming latency analysis, or long-term memory over arbitrarily long videos [2410.00771]. VQQA is explicitly limited by both the underlying generator and the VLM evaluator: it cannot rectify fundamental model flaws, may follow noisy semantic gradients when the evaluator hallucinates, and incurs more sequential latency than purely parallel Best-of-\(N\) schemes [2603.12310]. ProViQ, although strong in zero-shot decomposition, is not itself a proactive timing framework [2312.00937].

A persistent misconception is that all proactive video methods are instances of the same problem. The surveyed work indicates otherwise. Some methods are about **timely answering during playback**; some are about **asking follow-up questions to a user**; some are about **eliciting hidden expert commentary**; some are about **quality-aware expert routing**; and some are about **self-critique for generated videos**. This suggests that ProactiveVideoQA is less a single benchmark than a broader systems agenda: video models should decide not only **what** to say, but also **when** to say it, **what** to ask, **which** tools to invoke, and **how** to justify or revise their conclusions under partial and evolving evidence.

Source: https://www.emergentmind.com/topics/proactivevideoqa