---
title: 'ProactiveBench: Benchmarking Proactivity'
url: https://www.emergentmind.com/topics/proactivebench
type: topic
---

# ProactiveBench: Benchmarking Proactivity

ProactiveBench is the name used in recent arXiv literature for benchmarks that operationalize **proactivity** as a first-class model capability rather than a by-product of instruction following. In its most specific and widely recognizable usage, it denotes a benchmark for **multimodal large language models (MLLMs)** that tests whether a model can recognize when visual evidence is insufficient and request a simple user intervention that would make the query answerable, rather than hallucinating, abstaining, or guessing [2603.19466]. In a second usage, the same name refers to an event-stream benchmark and training corpus for proactive assistants in **coding, writing, and daily life**, where the model predicts whether and what help to offer from user activity, environmental events, and environment state [2410.12361]. Across these usages, the common object is not reactive correctness alone, but **initiative under uncertainty**.

## 1. Conceptual basis

In the multimodal benchmark formulation, proactiveness is defined as the ability to **either provide the correct answer or ask for help by suggesting an action that makes the query answerable**. The key distinction is between a model that merely abstains and a model that proposes a concrete intervention such as moving an occluding object, rotating an object, improving image quality, waiting for a temporal occlusion to pass, or requesting more sketch detail [2603.19466].

In the proactive-assistance formulation, the same term is attached to a broader anticipatory-agent setting. There, the proactive agent predicts a task from current context according to
\[
P_t = f_{\theta}(E_t, A_t, S_t),
\]
where \(E_t\) is the environment event history, \(A_t\) is the user activity history, and \(S_t\) is the current environmental state; training is framed as maximizing user acceptance of suggestions [2410.12361]. This suggests that, across benchmark lines, “proactiveness” denotes a family of capabilities involving **recognition of insufficiency, latent need inference, and timely intervention**, rather than a single metric or modality.

A recurrent misconception in this area is that proactivity is equivalent to aggressiveness in speaking or acting. The benchmarks explicitly reject that equivalence. In the visual setting, a model that selects arbitrary non-abstain actions is not thereby proactive; in the event-stream setting, a model that offers help when no help is needed incurs false alarms rather than credit [2603.19466].

## 2. Benchmark construction in the multimodal visual setting

The benchmark titled **“ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models”** is built by repurposing **seven existing datasets** into **seven proactive scenarios** and covers **19 proactive behaviors** overall [2603.19466]. Its central design move is to begin from an **intentionally ambiguous or incomplete first state** and to ask whether the model will request the information needed to resolve that ambiguity.

The seven source datasets and scenarios are as follows. **ROD** is used for **occluded objects**, where the model asks to move an occluding object left or right. **VSOD** covers **temporal occlusions** in video and permits actions such as waiting or rewinding. **MVP-N** targets **uninformative views**, with proactive suggestions to rotate the object or change camera angle. **ImageNet-C** is converted into an **image quality enhancement** scenario, where the model asks to deblur, denoise, or otherwise improve image quality. **QuickDraw** becomes a **coarse sketches** scenario, in which the model asks for more strokes or more detail. **ChangeIt** targets **temporal ambiguities**, allowing requests for past or future frames where the target becomes visible. **MS-COCO** is used for **camera movement or cropping**, with actions such as moving the camera, zooming out, or changing viewpoint [2603.19466].

Across these datasets, the benchmark contains **more than 108k images** grouped into **about 18k samples**. Because some ambiguous initial states remain answerable, the benchmark applies a filtering rule: a sample is removed if it is correctly predicted at the first turn at least **25% of the time across all evaluated MLLMs**. This reduces the benchmark from **17,909** total samples to **7,557** final samples, and lowers average first-turn accuracy from **32.5%** to **6.4%**, thereby making proactive intervention materially necessary rather than optional [2603.19466].

## 3. Task formulation, interaction protocol, and metrics

ProactiveBench evaluates models in two modes: **multiple-choice question answering (MCQA)** and **open-ended generation (OEG)** [2603.19466]. In the MCQA setting, the model receives the question, the current image or frame, and a discrete list of valid actions. The options include an **abstain** option, one or more **proactive suggestions**, several wrong category answers, and exactly one correct category. If the model chooses a proactive action, the environment transitions to a more informative state and the episode continues; if it abstains or chooses a wrong category, the episode terminates incorrectly; if it predicts the correct category, the episode terminates successfully.

This process is formalized as an MDP with state space \(S\), action set \(A\), policy \(\pi\), and reward \(R\). At time \(t\), the model observes
\[
s_t = \{I_t, A_t\}
\]
and samples an action
\[
a_t \sim \pi(\cdot \mid q, s_t).
\]
The evaluation therefore tests not only recognition, but also sequential control over when to request more information and when to commit to an answer [2603.19466].

The reported MCQA metrics are **acc**, defined as accuracy over the full multi-turn episode, and **ps**, the proactive suggestion rate, defined as the average number of human interventions requested by the model. In OEG, where the model answers freely without a predefined option set, evaluation is restricted to **single-turn** settings and uses an **LLM-as-a-judge** to determine whether the response semantically expresses a valid proactive suggestion and/or the correct category. The reported OEG metric is **agg**, an aggregate accuracy that counts an answer as correct if it contains either the correct category or a valid proactive suggestion [2603.19466].

A central methodological point is that the benchmark distinguishes sharply between **proactive suggestion**, **abstention**, and **correct direct answering**. It is designed precisely to test whether a model can traverse those alternatives in a controlled manner.

## 4. Empirical findings and interpretive implications

The main empirical result is that current MLLMs are **generally not proactive** in the intended sense [2603.19466]. When given the **reference frame** directly, average accuracy is **79.8%**, but on ProactiveBench the models underperform by **more than 60 percentage points on average**. In **ROD**, average accuracy is only **8.2%** versus **98.3%** on the reference frame. The benchmark therefore isolates a failure mode that is not basic recognition capacity, but failure to recognize the need for collaborative intervention.

The paper also finds that **proactiveness does not correlate with model capacity**. Smaller models sometimes outperform larger ones in MCQA accuracy or proactive-suggestion rate, but subsequent ablations show that much of this apparent proactivity reflects **anti-abstention behavior** rather than genuine task understanding. To test this, the valid proactive actions are replaced with **random invalid proactive options** from other datasets. Models such as **Mistral**, **LLaVA-7B**, and **InternVL3-8B** then lose most of their proactive-suggestion rate, indicating that their earlier behavior was not grounded proactiveness but a tendency to avoid abstention [2603.19466].

Prompt engineering helps only weakly. Dataset-specific **hints** increase proactive behavior, and MCQA proactive-suggestion rate rises by about **1.9** on average, but average MCQA accuracy reaches only **25.8%**. Moreover, in **16.0%** of cases with hints, models blindly choose proactive suggestions and fail to answer even when the reference image becomes available. Likewise, **conversation history** and **few-shot in-context learning** introduce **negative biases**: they increase proactive suggestions but lower accuracy, often by pushing the model to repeat previous proactive moves until the exploration limit is reached [2603.19466].

These findings yield two broader implications. First, proactiveness is not an automatic emergent property of scale. Second, naive encouragement to “be more proactive” can readily induce **over-triggering** or **pattern imitation** rather than calibrated intervention.

## 5. Learning proactiveness

Although pretrained MLLMs generally lack the target behavior, the benchmark literature also shows that proactiveness is at least partially **learnable**. In the multimodal ProactiveBench, two models—**Mistral-7B** and **Qwen2.5-VL-3B**—are fine-tuned with **GRPO** on data from **QuickDraw** and **MS-COCO**. The reward assigns \(r_c = 1\) for the correct category, \(r_p \in \{0.5, 0.75, 1.0\}\) for a valid proactive suggestion, and \(r_w = 0\) otherwise. The best tradeoff occurs when \(r_p < r_c\), and the learned behavior generalizes to **unseen datasets**, including **CIT** [2603.19466].

The event-stream ProactiveBench line reaches the same general conclusion by a different route. That benchmark contains **6,790 events** for agent-model training, a **233 real-world event** test set, and **1,760 labeled entries** for a reward model. Fine-tuning on that corpus produces **Qwen2-7B-Proactive** with **66.47% F1-Score** in proactively offering assistance, outperforming the reported open-source and closed-source baselines in that benchmark table [2410.12361].

The framework **PRISM** further treats proactive intervention as **cost-sensitive selective decision-making** on that event-stream benchmark. It uses calibrated estimates of need and acceptance, a dynamic threshold, and a selective **Slow mode** near the decision boundary. On the held-out **test set of 233 clips**, PRISM reports **Recall 98.88%**, **Precision 77.05%**, **Accuracy 76.39%**, **False alarm rate 22.94%**, and **F1 86.61%**, outperforming **DeepSeek-R1** on precision, false alarms, and F1 [2602.01532]. Taken together, these results indicate that benchmark-guided training can improve not only direct-answer accuracy but also **restraint, timing, and burden control**.

## 6. Nomenclature, related benchmarks, and scope

The term **ProactiveBench** is not fully canonical. Besides the multimodal visual benchmark [2603.19466] and the event-stream proactive-assistance corpus [2410.12361], the label also appears in **ProactiveVideoQA**, where the paper states that ProactiveVideoQA is instantiated as **ProactiveBench** for proactive, multi-turn, time-aware video QA [2507.09313]. This suggests that the name has begun to function as a broader label for benchmarks that evaluate **initiative under partial observability**, rather than a single fixed dataset.

This terminological overlap matters because neighboring works pursue related but non-identical objects. **ProVoice-Bench** evaluates proactive voice agents with tasks such as **Proactive Intent Capture**, **Latent Topic Monitor**, **Context Fact Checking**, and **Environment Sound Sensing** [2604.15037]. **\(\pi\)-Bench** targets proactive personal assistant agents in **long-horizon workflows** with **hidden user intents**, **inter-task dependencies**, and **cross-session continuity** [2605.14678]. **PIRA-Bench** studies GUI-based proactive intent recommendation from continuous screenshot trajectories [2603.08013]. **EgoPro-Bench** addresses personalized proactive interaction in streaming egocentric video [2605.07299]. **ProAgentBench** focuses on real working scenarios with **28,528 total events** and a hierarchical **When + How** formulation [2602.04482]. **Pro\(^2\)Bench** and **EgoProactive** address proactive procedural assistance and Out-of-Plan recovery from egocentric video [2606.04970].

Within that expanding ecosystem, ProactiveBench occupies a foundational role because it sharpens a general research question into an executable evaluation target: whether a model can identify that its current evidence is inadequate, infer the right intervention, and ask for the specific help needed to proceed. A plausible implication is that benchmarking proactivity has shifted from measuring isolated prompt compliance to measuring **calibrated collaboration**—including when to ask, when to wait, and when to remain silent.

Source: https://www.emergentmind.com/topics/proactivebench