IPIBench: Proactive Multimodal Evaluation

• IPIBench is a benchmark that evaluates the interactive proactive intelligence of multimodal large language models under continuous streaming video settings.
• It organizes evaluation into proactive monitoring, task management, and interleaved reactive-proactive interactions to mimic real-world persistent assistant scenarios.
• IPI-Agent, a training-free framework with temporal gating and interaction control, significantly enhances proactive task management and multi-turn coordination.

Searching arXiv for the benchmark and closely related uses of the name. IPIBench is a benchmark for evaluating the interactive proactive intelligence of multimodal LLMs under continuous streaming video settings. It was introduced to address a specific gap in the evaluation of MLLMs: existing benchmarks largely emphasize reactive question answering or proactive behavior in isolated single-turn settings, whereas real deployment scenarios require persistent monitoring, multi-turn task management, and coordination between proactive and reactive behaviors over time. IPIBench covers proactive monitoring, proactive task management, and interleaved reactive-proactive requests; evaluations on representative MLLMs identify unstable proactive triggering and weak coordination between reactive and proactive behaviors as the two principal failure modes. The same work also proposes IPI-Agent, a training-free agentic framework designed to improve performance on the benchmark through interaction control and temporal gating (Li et al., 26 May 2026).

1. Conceptual scope

IPIBench is defined around the distinction between reactive and proactive intelligence in multimodal assistants. Reactive capability corresponds to answering explicit user queries, as in standard VQA-style evaluation. Proactive capability, by contrast, requires the model to monitor a continuous visual stream and trigger an intervention when a user-specified event occurs, including before any additional prompt is issued. The benchmark therefore targets settings closer to “always-on” assistants, including scenarios such as wearable systems or home robots, in which the system must reason over unfolding events rather than over a static prompt-response exchange (Li et al., 26 May 2026).

The benchmark’s novelty lies in its insistence on dynamic multi-turn interaction. Users may define a monitoring objective, later modify or cancel it, and simultaneously ask interleaved reactive questions. This moves evaluation beyond single-event detection or one-off query answering and toward persistent state management under temporal uncertainty. A plausible implication is that IPIBench treats proactive intelligence not as a single prediction problem, but as an interaction problem with memory, control, and timing constraints.

2. Task taxonomy

IPIBench organizes evaluation into three scenario families, each of which probes a different aspect of interactive proactive behavior. The benchmark covers whether a model can trigger at the correct time, preserve and update long-lived objectives, and coordinate reactive and proactive behaviors without losing context (Li et al., 26 May 2026).

Category	Subtasks	Included
Proactive Monitoring	Timing, Understanding, Repeated	✓
Proactive Task Management	Cancel, Modify, Multi-task	✓
Interleaved Reactive-Proactive	RaP, RuP, R2P	✓

Within Proactive Monitoring, the benchmark evaluates three related properties. Proactive Timing asks whether the model triggers precisely when a specified event occurs. Proactive Understanding requires the model not only to trigger but also to describe attributes, spatial relations, or other nuances tied to the event. Repeated Proactiveness tests whether the model can respond consistently to recurring target events rather than treating monitoring as a one-shot task.

Within Proactive Task Management, the model must preserve and manipulate persistent objectives. In Task Cancellation, a previously defined monitoring request is later revoked, and the model should stop responding to it. In Task Modification, the user updates the target and the model must monitor according to the revised objective. In Multi-task Management, multiple simultaneous proactive tasks must be maintained and handled correctly.

Within Interleaved Reactive-Proactive Requests, the benchmark examines mixed-query behavior. Reactive-after-Proactive (RaP) asks a reactive question immediately after a proactive trigger and expects the answer to use both the triggered event and recent context. Reactive-under-Proactive (RuP) inserts a reactive query while proactive monitoring is ongoing. Reactive-to-Proactive (R2P) tests whether a reactive exchange can transition into a new persistent proactive task.

3. Benchmark construction

IPIBench is built from 1,831 videos and 3,738 QA instances drawn from public datasets including Ego4D, RoadTextVQA, COIN, AVA, and Charades-STA, with coverage spanning both egocentric (first-person) and exocentric (third-person) video (Li et al., 26 May 2026).

The construction process emphasizes temporal grounding and interaction realism. Trigger moments are refined via human annotation to ensure precise alignment between the intended proactive event and the benchmark label. Multi-turn extensions are then created by inserting management instructions and reactive queries at strategic temporal points. This procedure is central to the benchmark’s design: it does not merely relabel existing video QA samples, but restructures them into interaction sequences in which task state can evolve over time.

Compared with prior benchmarks summarized in the same work, IPIBench differs by systematically combining proactive monitoring, task management, and multi-turn interactivity in a single evaluation framework. This suggests that the benchmark is intended not only to test detection or question answering, but to approximate the control logic expected of real streaming assistants.

4. Evaluation protocol

All models are evaluated in a streaming setup at 1 FPS. For models without an explicit memory mechanism, online use is simulated with a sliding window of the latest 16 frames. This choice standardizes temporal input while still exposing the need for persistent interaction state across turns (Li et al., 26 May 2026).

For proactive tasks, correctness is defined temporally. A predicted trigger is counted as correct if it appears within the interval

$[t^\ast - 1,\; t^\ast + 1]$

around the annotated ground-truth trigger time $t^\ast$, with evaluation conducted over

$[t^\ast - 4,\; t^\ast + 4].$

For repeated and other multi-turn tasks, all requisite triggers and interactions must be correct for the case to count as a success. This is substantially stricter than partial-credit event detection and makes error propagation in long interactions directly visible.

Reactive tasks are evaluated differently. The model’s open-ended answer is compared against up to 20 semantically equivalent candidates using bidirectional substring matching. Across all task families, performance is reported as accuracy, defined as the percentage of cases exhibiting correct behavior under the benchmark’s timing and interaction criteria.

5. Empirical findings

The benchmark reports that current MLLMs remain substantially limited in this setting. Proprietary models such as Gemini Pro, GPT-5.4, and GPT-4o outperform open-source models on proactive and management tasks, but all models perform far below human performance (Li et al., 26 May 2026).

The most prominent failure mode is unstable proactive triggering. Models often respond too early or too late, and the failure analysis indicates that over 90% of errors from large models are premature triggers. This is a distinctive result of the benchmark’s temporal design: in single-turn or static evaluation, early triggering would often be invisible, but under streaming evaluation it becomes a dominant error type.

The second major limitation is poor multi-turn coordination. In interleaved settings, models may drop ongoing monitoring objectives, lose track of cancellations or modifications, or fail to maintain proactive state while answering a reactive query. The paper further notes that reminder prompts significantly improve performance, indicating that present-day systems often lack sufficiently stable internal task persistence. Taken together, these findings show that strong reactive performance on image or video QA does not transfer straightforwardly to sustained proactive interaction under continuous input.

6. IPI-Agent

To address these deficiencies, the same work introduces IPI-Agent, a training-free agentic framework intended to augment arbitrary MLLMs for interactive proactive behavior. Its architecture has two workflows—Query Arrival and Continuous Monitoring—and two main mechanisms: an interaction-control policy and a temporal-gating mechanism (Li et al., 26 May 2026).

The interaction-control policy is built around a memory tool with two slots. Proactive Memory stores active proactive objectives and their monitoring status, while Interaction Memory records user interactions and system actions. An Intent Router classifies each new user input as a reactive query, proactive instruction, or management instruction. Reactive queries retrieve recent context and relevant history; proactive instructions are encoded as structured tasks and stored; management instructions locate and edit or cancel the corresponding task. This design directly targets the benchmark’s multi-turn persistence and task-management demands.

The temporal-gating mechanism regulates trigger timing. For each active task $\tau_i$, the agent generates $M$ candidate textual proposals

$\mathcal{P}_i = \{p_i^1, \dots, p_i^M\}.$

It computes embeddings for proposals and current visual frames,

$$\mathbf{e}_i^m = E(p_i^m), \qquad \mathbf{v}_t = E(\mathbf{x}_{t-K:t}),$$

then measures semantic similarity

$$s_{i,t}^m = \mathrm{sim}(\mathbf{e}_i^m, \mathbf{v}_t)$$

and temporal variation

$\Delta s_{i,t}^m = s_{i,t}^m - s_{i,t-1}^m.$

The maximum variation across proposals is

$\Delta_i(t) = \max_m \Delta s_{i,t}^m.$

If the response tool triggers, $t^\ast$0, but $t^\ast$1, the response is suppressed to avoid early activation. If the response tool does not trigger, $t^\ast$2, but $t^\ast$3, activation is forced to recover a missed trigger.

Experiments reported in the paper show that IPI-Agent consistently improves existing MLLMs across all benchmark settings. The largest reported gain is on Proactive Task Management, with improvements of up to +39.82% for Cancel tasks using Qwen3.5-Plus. Ablation results are also central: removing the interaction-control policy causes performance to collapse on management and interleaving tasks, while removing temporal gating significantly harms proactive monitoring. The paper further states that embedding-based similarity alone is not sufficient for reliable triggering; persistent agentic control and temporal context are necessary.

7. Position in the benchmark landscape

IPIBench belongs to a broader movement toward evaluating LLM and MLLM systems under more realistic operational constraints, but its focus is specifically streaming multimodal interactivity rather than static correctness. The benchmark differs from work centered on reactive QA, and it also differs from security-oriented uses of the same or similar names. In the indirect prompt injection literature, for example, BIPIA/IPIBench is identified as a static benchmark using text fixtures and targeting model/tasks, in contrast to the dynamic proxy-based evaluation introduced by IPI-proxy (Chia-Pei et al., 12 May 2026).

The term IPIBench is therefore not globally unique. In the provided literature, the same label is also attached in summary form to a cross-platform benchmark for interval computation libraries (Tang et al., 2021) and to a microbenchmarking suite for the Graphcore IPU architecture (Jia et al., 2019). This suggests a naming ambiguity across subfields. In practice, disambiguation by title or arXiv identifier is necessary, especially because the streaming-video IPIBench concerns proactive multimodal interaction, whereas the other uses concern numerical software or hardware microarchitecture.

Within its own domain, however, IPIBench establishes a distinct evaluation target: not merely whether an MLLM can answer correctly, but whether it can sustain, update, and time its behavior appropriately under continuous visual streams. Its principal contribution is to make interactive proactive intelligence measurable as a benchmarked capability rather than an informal deployment aspiration.