ProactiveEval: Evaluation Framework for Proactive Systems

• ProactiveEval is defined as evaluating systems that infer latent needs and proactively guide users to beneficial actions.
• It decomposes proactive behavior into planning and dialogue guidance, emphasizing timing, context, and effective intervention.
• Benchmarks span multiple modalities with diverse metrics, exposing gaps in recovery, execution accuracy, and evaluator calibration.

ProactiveEval is the evaluation of systems that do not merely respond to explicit requests, but infer latent needs, decide whether and when to intervene, and contribute value before or beyond being asked. In current literature, the term is used both for the dialogue-centric framework "ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agents" and for a broader family of benchmark designs that operationalize proactive behavior in conversation, video question answering, code collaboration, mobile assistance, GUI interaction, voice interaction, negotiation, and long-horizon agentic assistance (Liu et al., 28 Aug 2025, Harfi et al., 9 May 2026, Wang et al., 12 Jul 2025, Li et al., 27 Feb 2026).

1. Conceptual scope and formal decomposition

A common theme across ProactiveEval frameworks is that proactivity is not equated with autonomy alone. The dialogue framework "ProactiveEval" decomposes proactive dialogue into target planning and dialogue guidance: an agent first infers a proactive target $T$ and sub-targets $S$ from environment information $(U,F)$, then conducts a multi-turn interaction with a simulated user to achieve that target within a bounded interaction horizon (Liu et al., 28 Aug 2025). In this formulation, evaluation must assess not only whether the agent can act, but whether it can identify what beneficial action should be pursued and then guide the user toward it.

Other frameworks sharpen this definition by decomposing proactive behavior into finer operational phases. ProactBench defines conversational proactivity as the assistant’s ability to notice and act on needs implied, but not explicitly stated, by what the user has said so far, and explicitly excludes reactive compliance, clarification-seeking about already provided information, generic helpful closings, and ungrounded suggestions. It partitions proactive dialogue into Emergent, Critical, and Recovery: single-anchor inference early in the conversation, multi-anchor synthesis mid-dialogue, and grounded forward-looking value after task completion (Harfi et al., 9 May 2026). PROBE, by contrast, treats proactivity as a pipeline of searching for unspecified issues, identifying specific bottlenecks, and executing appropriate resolutions, thereby emphasizing cross-document reasoning and action completion in long-context environments (Pasternak et al., 22 Oct 2025).

A second major axis is the distinction between deciding when to intervene and deciding how to help. ProAgentBench formalizes proactive assistance as a hierarchy of timing prediction ("When to Assist") and assist content generation ("How to Assist"), with inputs defined over multimodal user activity streams and long-term user context (Tang et al., 4 Feb 2026). The coding-agent perspective makes the same distinction at the level of an "insight policy": an agent chooses among notify, question, draft, and stay silent, and is judged by whether it computes interruption cost, treats silence as an explicit action, grounds its insight in evidence, and adapts after feedback (Bui et al., 7 May 2026). The HCI-oriented account generalizes these observations by describing proactive systems as self-initiated, future-oriented, and change-oriented, and by arguing that anticipation, timing, user control, transparency, and trust are irreducible evaluation dimensions rather than secondary usability attributes (Zargham et al., 23 Jun 2026).

This body of work suggests that ProactiveEval is best understood not as a single metric or benchmark, but as a design space of evaluations for initiative under uncertainty. A plausible implication is that proactive competence is inherently multi-component: perception of latent cues, timing, grounding, action selection, and adaptation cannot be collapsed into correctness-only scoring without losing the phenomenon being measured.

2. Benchmark construction and environment design

Recent ProactiveEval benchmarks differ substantially in modality and task environment, but they share an emphasis on structured latent need, asymmetric information, and executable or auditable outcomes. The dialogue benchmark ProactiveEval provides 328 evaluation environments spanning 6 distinct domains—Recommendation, Persuasion, Ambiguous Instruction, Long-term Follow-up, System Operation, and Glasses Assistant—and uses an automatic synthesis-and-refinement pipeline to generate environments, reference targets, and difficulty-controlled scenarios (Liu et al., 28 Aug 2025). ProactBench instead releases 198 curated dialogues with 624 trigger points across 24 communication styles, and enforces information asymmetry via a Planner, User Agent, Assistant Model, and Offline judge so that the evaluated model is blind to rubrics, trigger schedules, personas, and hidden scenarios (Harfi et al., 9 May 2026).

Multimodal and embodied variants extend the same logic. ProactiveVideoQA introduces a proactive video benchmark with 1,377 videos, 1,427 questions, multi-answer (multi-turn) supported and open-ended QA, and explicit answer timespans that require a model to decide when to speak during video playback rather than only what text to produce (Wang et al., 12 Jul 2025). PARE models applications as finite state machines with asymmetric interfaces: the user simulator navigates stateful app screens with state-dependent actions, whereas the assistant has flat cross-app access. Pare-Bench then evaluates 143 diverse tasks over communication, productivity, scheduling, and lifestyle apps, with exogenous events, distractor noise, and an approval protocol that requires user acceptance before assistant execution (Nathani et al., 1 Apr 2026). ProactiveMobile similarly avoids pixel-grounding by casting proactive mobile assistance as latent-intent inference from four context dimensions—temporal, spatial, device/app, and communication—and executable planning over a function pool of 63 mobile APIs across more than 3,660 instances of 14 scenarios (Kong et al., 25 Feb 2026).

Code and long-horizon settings introduce different constructions. ClarEval starts from 750 clear programming problems and transforms them into 2,250 ambiguous evaluation instances by injecting Missing Goals, Missing Premises, and Ambiguous Terminology, then evaluates dialogue with a deterministic User Simulator Script that exposes exactly the missing information when the agent asks the right question (Li et al., 27 Feb 2026). ProActEval, designed for idle-time computation, supplies 200 scenarios across 40 domains, each with a fact sheet, an ordered user-needs graph, predictable need chains, and five cognitive archetypes; if the assistant proactively covers a future need, the simulator skips that need later, turning anticipation directly into lower user effort (Hu et al., 25 May 2026).

Framework family	Environment design	Core proactive challenge
Dialogue ProactiveEval	Synthetic environments with targets and sub-targets	Plan a target, then guide a resistant or agreeable user
ProactBench	Trigger-based dialogues with phase-tied rubrics	Infer implied needs from disclosed anchors
ProactiveVideoQA	Answer timespans during streaming video	Decide when to speak and refine over time
ClarEval	Ambiguity-injected programming tasks	Stop guessing and elicit missing requirements
ProactiveMobile	Context bundle plus executable API sequence	Infer latent intent and produce feasible action plans
PARE / Pare-Bench	FSM apps with active user simulation	Time interventions under stateful navigation and user approval
ProActEval	Predictable need chains plus idle windows	Anticipate future requests and prepare evidence

What unifies these constructions is not surface format but the deliberate creation of counterfactual opportunity: each benchmark is designed so that an agent can either wait reactively or exploit latent structure to intervene earlier, more specifically, or more efficiently.

3. Metrics and scoring paradigms

Because proactive behavior couples timing, grounding, and utility, ProactiveEval frameworks rarely rely on a single scalar notion of accuracy. The dialogue benchmark ProactiveEval uses a judge-based scoring regime in which target planning receives a score in $[1,10]$ relative to a reference target/sub-target pair, while dialogue guidance receives a score in $[1,10]$ along five dimensions: Effectiveness, Personalization, Tone, Engagement, and Naturalness. Guidance is explicitly sensitive to message length, proactive tone, and conversational naturalness, and analyses additionally track target density and turns-to-success (Liu et al., 28 Aug 2025).

ProactBench adopts a trigger-local rubric with discrete labels mapped to numeric weights: Pass = 1, Partial = 0.5, Fail = 0. These are aggregated into per-phase weighted scores,

$$S_{\text{phase}}(m)=\frac{1}{N_{\text{phase}}}\sum s_j,$$

and into an overall score across the released corpus. This makes the benchmark auditable at the trigger level while still supporting phase-level comparisons and cross-benchmark correlations (Harfi et al., 9 May 2026). ClarEval, in contrast, measures collaborative clarification through Key Question Coverage (KQC), Premise Identification Rate (PIR), Missing Premises Recall (MPR), Average Turns to Clarify (ATC), and Efficiency-Adjusted Recall (EAR), thereby disentangling intent inference, missing-premise recovery, and the efficiency of the questioning strategy (Li et al., 27 Feb 2026).

Timing-aware proactive benchmarks introduce metrics that are explicitly dynamic. ProactiveVideoQA defines PAUC (Proactive Area Under Curve), which scores the evolving correctness of accumulated responses across an annotated answer interval. At each response timestamp, an LLM judge assigns $s_p \in \{0,1,2\}$ to the accumulated output, and the benchmark computes area under a time-score polyline, normalized by interval length and maximum score. A hyperparameter $\omega \in [0,1]$ shifts timestamps leftward to trade off timeliness against final correctness, with $\omega=0.5$ recommended as default (Wang et al., 12 Jul 2025). ProVoice-Bench separates Proactive Interaction Prediction—measured by Accuracy, False Positive Rate, and Recall—from Response Accuracy $R_{acc}$, which averages judge scores over tool-call correctness and textual-response correctness, conditioned on the intervene/don’t-intervene decision being correct (Xu et al., 16 Apr 2026).

Executable and stateful systems require another layer of measurement. ProactiveMobile uses Success Rate (SR) as its primary metric: a predicted action sequence must execute to completion without feasibility violations and must be equivalent to at least one gold solution under the benchmark’s context-sensitive equivalence relation. It also reports Validity Rate, Function-level F1, Parameter-level F1, and Sequence Edit Distance (Kong et al., 25 Feb 2026). PARE formalizes Acceptance Rate and Task execution success at the episode level, then supplements them with Success@k, Success^k, Proposal Rate, Read Actions, and ternary proposal outcomes (Accept, Reject, Gather Context) to expose intervention timing and information-gathering behavior (Nathani et al., 1 Apr 2026).

A broader methodological point emerges from this diversity. ProactiveEval metrics tend to fall into four families: trigger correctness, temporal utility, interaction efficiency, and execution fidelity. This suggests that proactive evaluation is fundamentally multidimensional, and that metric design must follow the structure of latent opportunity in the environment rather than import a generic QA score.

4. Empirical findings and recurrent failure modes

Across domains, empirical results converge on a striking conclusion: proactive ability remains weak even in frontier systems. In ProactBench, Recovery is especially difficult: even the best model, GPT-5.5, passes only 37.2% of Recovery triggers, while 14 of 16 models pass fewer than 20%. Recovery also decorrelates from six standard benchmarks, with mean Pearson correlation 0.51 and no surviving cross-benchmark correlations after Holm–Bonferroni correction. The dominant Recovery failures are generic closing (44%) and deliverable repetition (45%), rather than hallucination or long-context retrieval failure (Harfi et al., 9 May 2026).

Parallel weaknesses appear in other modalities. ProVoice-Bench reports severe over-triggering in proactive voice agents: Step-Audio-R1 reaches Recall 0.941 but also False Positive Rate 0.866 and Accuracy 0.538, whereas thinking variants reduce false positives and improve decision quality; the highest reported overall Response Accuracy is 0.759 for Qwen3-Omni(T) (Xu et al., 16 Apr 2026). ProactiveMobile shows that executable mobile proactivity is learnable but still difficult: the fine-tuned Qwen2.5-VL-7B-Instruct reaches 19.15% success rate, outperforming o1 at 15.71% and GPT-5 at 7.39%, which indicates that general reasoning strength alone does not translate into reliable latent-intent execution (Kong et al., 25 Feb 2026). In Pare-Bench, the strongest reported results are 42.0% ± 1.0% Success Rate for Claude 4.5 Sonnet and 42.1% ± 1.0% for Gemini 3 Flash; Claude proposes less often (12.8% ± 0.4% Proposal Rate) but is accepted more often (78.2% ± 0.8%), whereas GPT-5 proposes much more frequently (28.1% ± 0.3%) with lower efficiency, indicating over-eagerness rather than superior judgment (Nathani et al., 1 Apr 2026).

Code and long-context settings reveal analogous structural failure modes. ClarEval shows a large gap between solving clarified tasks and acting under ambiguity: with GPT-4o, clarified tasks reach 89.02% Pass@1, whereas ambiguous tasks reach only 8.94% Pass@1; Ambiguous Terminology is the hardest category at 6.71% Pass@1 (Li et al., 27 Feb 2026). PROBE reports that the best end-to-end performance is only 40%, achieved by both GPT-5 and Claude Opus-4.1, with search recall, root-cause identification, and parameter completeness all serving as bottlenecks (Pasternak et al., 22 Oct 2025). ProActEval shows that proactive idle-time preparation can improve these outcomes: Directed Idle reduces $S$0 by 14.8%, User Effort by 11.7%, and Hallucination Rate by 28.1% relative to a reactive baseline, while raising Anticipation Recall from 0.000 to 0.428 (Hu et al., 25 May 2026).

In multi-party interaction, the same pattern reappears at a social level. In ProMediate-Hard, a socially intelligent mediator increases consensus change from 7.01% to 10.65% and is 77% faster in response, with 15.98s for the generic mediator versus 3.71s for the socially intelligent mediator (Liu et al., 29 Oct 2025). The implication is that proactive competence is not confined to early action: it extends to better-timed, better-targeted, and more socially calibrated intervention.

5. Methodological patterns: judging, verification, and evaluation of the evaluators

One of the most consequential developments in ProactiveEval is methodological rather than task-specific: benchmarks increasingly treat evaluator design itself as a first-class problem. ProactBench attempts to defend against style-confounded scoring, rubric leakage, external-context contamination, and information dumps through explicit information asymmetries: the Planner does not see communication style, the assistant does not see rubrics or trigger schedules, and offline judging is performed without persona/style metadata. The same benchmark also includes human calibration, reporting Krippendorff’s $S$1 and consensus-vs-judge $S$2 (Harfi et al., 9 May 2026). This architecture reflects a broader shift from monolithic prompting toward auditable scoring pipelines.

A second line of work evaluates judge reliability directly. AgentProp-Bench shows that a substring-based judge agrees with human annotation at $S$3, effectively chance-level, while a three-LLM ensemble reaches $S$4, with a conservative bias. The same benchmark formalizes propagation cascades through stage indicators and estimates that a parameter-level injection propagates to a wrong final answer with human-calibrated probability approximately 0.62, with rejection and recovery behaving as independent capabilities (Spearman $S$5, $S$6) (Gurram, 17 Apr 2026). For ProactiveEval, this means that any automated judge should itself be calibrated, audited, and stress-tested.

Verification becomes even more difficult under partial observability, which motivates active evaluation. ProRe replaces static GUI-trajectory judging with a reasoner–actor collaboration: a reasoner schedules targeted probing tasks, evaluator agents interact with the GUI to gather missing evidence, and chain-of-claims reasoning determines reward. This proactive evaluation loop achieves average reward accuracy 93.7%, improves reward accuracy by up to 5.3%, improves F1 by up to 19.4%, and yields success-rate improvements of up to 22.4% when used with policy agents (Dai et al., 26 Sep 2025). In another direction, ProEval frames performance estimation as Bayesian quadrature over a pre-trained GP surrogate and failure discovery as superlevel set sampling, requiring 8–65x fewer samples to achieve estimates within 1% of ground truth while uncovering more diverse failures under a stricter evaluation budget (Huang et al., 25 Apr 2026).

Taken together, these results suggest that proactive systems demand proactive evaluation infrastructure. A plausible implication is that evaluator passivity is itself a source of measurement error: when the benchmark cannot recover missing evidence, calibrate judgment, or actively probe uncertain cases, proactive behavior is often under-measured or mis-measured.

6. Limitations, open problems, and likely trajectories

Despite rapid expansion, current ProactiveEval frameworks remain bounded by strong assumptions. ProactBench notes US-English personas from a single synthetic source, synthetic dialogues, and ongoing dependence on an LLM judge, while ClarEval acknowledges synthetic ambiguity injection, function-level task scope, predefined key questions, and a hard turn cap of $S$7 (Harfi et al., 9 May 2026, Li et al., 27 Feb 2026). ProAgentBench improves realism by using 28,528 total events over more than 500 hours of real user computer sessions, yet it also foregrounds privacy protection, VLM-based screening, volunteer review, and de-identified release artifacts, indicating that realistic proactive evaluation is inseparable from data-governance constraints (Tang et al., 4 Feb 2026).

Several works identify the same future directions. ProactBench proposes using per-trigger rubrics and labels as substrates for preference or reward modeling, especially in the post-completion regime, and calls for richer user simulators with longer horizons, multi-session memory, and disengagement patterns (Harfi et al., 9 May 2026). ClarEval argues for richer ambiguity taxonomies, multi-user dialogues, and repo-level integration, while the proactive coding perspective calls for evaluation of Insight Decision Quality, Context Grounding Score, and Learning Lift over recurring, feedback-sensitive decision points rather than correctness-only repository tasks (Bui et al., 7 May 2026). ProActEval extends this trajectory by showing that persistent memory and idle-time computation can be benchmarked explicitly, with gains in efficiency and factual integrity, but also by documenting failure cases in which proactive delivery competes with reactive compatibility (Hu et al., 25 May 2026).

At the conceptual level, the HCI synthesis argues that the field still lacks a stable, shared definition of proactivity, and that inherited reactive methods do not adequately capture timing, appropriateness, user control, transparency, and trust (Zargham et al., 23 Jun 2026). This suggests that the long-term maturation of ProactiveEval will likely depend on three converging moves: more realistic and privacy-aware user models, stronger evaluator calibration and active verification, and a human-centered theory of when unsolicited initiative is useful rather than merely possible.

In that sense, ProactiveEval has already become more than a benchmark family. It is increasingly a research program for measuring initiative under partial observability, interruption cost, and latent user need—across modalities, environments, and social settings.