From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench

Abstract: Recent advancements in LLM agents are gradually shifting from reactive, text-based paradigms toward proactive, multimodal interaction. However, existing benchmarks primarily focus on reactive responses, overlooking the complexities of proactive intervention and monitoring. To bridge this gap, we introduce ProVoice-Bench, the first evaluation framework specifically designed for proactive voice agents, featuring four novel tasks. By leveraging a multi-stage data synthesis pipeline, we curate 1,182 high-quality samples for rigorous testing. Our evaluation of state-of-the-art Multimodal LLMs reveals a significant performance gap, particularly regarding over-triggering and reasoning capabilities. These findings highlight the limitations of current models and offer a roadmap for developing more natural, context-aware proactive agents.

• The paper introduces a novel benchmark, ProVoice-Bench, to evaluate proactive voice agent behaviors using integrated audio and digital context.
• ProVoice-Bench employs four structured tasks—intent capture, context fact checking, sound sensing, and topic monitoring—to simulate realistic, multimodal interactions.
• Empirical results reveal issues like over-triggering and execution errors, emphasizing the need for improved context fusion and advanced reasoning in voice agents.

Assessing Proactive Audio Agents: ProVoice-Bench and its Implications

Motivation and Problem Statement

The proliferation of Multimodal LLMs (MLLMs) has spawned a new generation of voice agents capable of multimodal perception and complex reasoning. However, the prevailing paradigm remains grounded in reactivity—agents initiate actions only after explicit prompts. This reactive stance fundamentally limits agents’ utility, as they are unable to infer when proactive assistance or intervention would enhance user experience or safety. Recognizing the gap, this paper introduces ProVoice-Bench, a benchmark tailored to evaluate and advance proactivity in voice agents, with a focus on integrating audio and digital context for more natural, context-aware agent behaviors.

ProVoice-Bench Tasks and Design

ProVoice-Bench delineates proactivity through four comprehensive tasks:

• Proactive Intent Capture (PIC): Models infer implicit user intentions from nuanced linguistic cues (e.g., hesitation, prospective actions) and proactively initiate tool calls, leveraging digital context for operational precision.
• Latent Topic Monitor (LTM): Agents monitor ongoing conversations, intervening solely when user-defined triggers are detected, testing conditional silence and targeted action.
• Environment Sound Sensing (ESS): Agents remain dormant until predefined acoustic events (e.g., alarms) occur, then offer targeted assistance, emphasizing environmental awareness.
• Context Fact Checking (CFC): Agents interrupt users when statements contradict digital context (e.g., mobile phone records), ensuring factual consistency and knowledge-driven intervention.

Each task is structurally anchored in multimodal interaction, requiring integration of conversational audio and digital context to decide not only when to speak, but also what action to execute.

Figure 1: The four core tasks in ProVoice-Bench, highlighting the necessary modalities and intervention criteria for proactive agent evaluation.

Data Synthesis Pipeline

To ensure realism and breadth, ProVoice-Bench employs a multi-phase data construction pipeline:

1. Digital State Construction via LLMs, synthesizing semantically rich user contexts and implicit cues.
2. Scenario Creation, linking digital states to task types and available tools, with robust temporal and triggering metadata.
3. Detailed Conversation Generation, leveraging advanced TTS (CosyVoice3) with diverse speakers and environmental sounds (ESC-50), producing naturalistic dialogues and acoustic cues.
4. Acoustic Simulation addresses loudness normalization, far-field effects, reverberation modeling, and ambient noise (CochlScene), enhancing ecological validity.
5. Conversation Assembly implements probabilistic pacing and stochastic environmental noise, simulating authentic conversational flows.

   Figure 2: (a) Distribution of ProVoice-Bench samples across tasks; (b) multi-stage pipeline for generating multimodal, context-rich conversation data.

Experimental Evaluation and Metrics

The benchmark comprises 1,182 balanced multimodal samples. Evaluation metrics span both interaction decision-making and response quality:

• Proactive Interaction Prediction: Accuracy measures binary decision performance. False Positive Rate (FPR) captures unnecessary interventions. Recall assesses sensitivity to valid triggers.
• Response Accuracy ($R_{acc}$): Quantifies correctness in subsequent agent actions—tool-call precision and semantic response alignment, using LLM-as-a-Judge for scoring.

A range of MLLMs, including Mimo-Audio, Qwen3-Omni, Step-Audio-R1, and Qwen2.5-Omni (with/without "thinking" variants), were evaluated.

Empirical Results and Analysis

Key findings are as follows:

• Over-triggering Propensity: Models exhibit significant over-triggering, especially in LTM and CFC tasks, often responding absent valid triggers or failing to discriminate contextual violations.
• Chain-of-Thought (CoT) Impact: Inclusion of CoT reasoning markedly boosts performance in analysis-intensive tasks (CFC, LTM, PIC), suggesting current architectures benefit from explicit intermediate reasoning steps.
• Decision-to-Execution Discrepancy: There exists a pronounced gap—models may correctly identify intervention timing but subsequently misexecute actions, e.g., semantic drift or hallucinated tool calls, emphasizing the unresolved challenge in bridging decision and execution in context-rich environments.

Digital Context Ablation Study

Removing digital context from the benchmark results in substantial performance degradation—namely, sharp drops in Recall and Accuracy in CFC and PIC tasks. This underscores digital context’s criticality in informing proactive interventions and accurate intent inference.

Figure 3: Performance comparison with and without digital context, demonstrating its vital role in CFC and PIC tasks.

Implications, Limitations, and Future Directions

ProVoice-Bench exposes key limitations in current MLLMs: poor trigger discrimination, inadequate reasoning when integrating multimodal signals, and a gap between intervention timing and action quality. It provides formalization and a rigorous testbed for advancing agentic behaviors in voice assistants. Practical implications include the need for improved context fusion mechanisms, reliability in digital context grounding, and robust multimodal reasoning architectures. Theoretically, this benchmark motivates future research into end-to-end multimodal proactivity, latent intent detection, and controllable agent dormancy.

Avenues for future development involve hierarchical agent architectures that continuously monitor multimodal streams, improved commonsense and pragmatic reasoning, and integration of longitudinal digital context for persistent proactivity. Progress in these domains will enhance autonomous voice agents' usability in real-world environments, ranging from healthcare and education to personal productivity.

Conclusion

ProVoice-Bench presents a domain-relevant suite for benchmarking and dissecting proactive voice agent capabilities, focusing on multimodal context integration and decision-action alignment. Empirical results indicate substantial gaps in state-of-the-art models’ performance, especially with respect to trigger discrimination and response execution. This benchmark will serve as a catalyst for research into robust, context-aware, and genuinely proactive agent architectures.