VoxMind: An End-to-End Agentic Spoken Dialogue System

Abstract: Recent end-to-end spoken dialogue models enable natural interaction. However, as user demands become increasingly complex, models that rely solely on conversational abilities often struggle to cope. Incorporating agentic capabilities is therefore essential: by enabling tool use, these models can extend their knowledge boundaries and better solve real-world tasks. Yet, existing research has largely concentrated on core perception and generation, with comparatively limited exploration of such tool-augmented extensions. To bridge this gap, we present VoxMind, an integrated framework designed to equip end-to-end spoken dialogue models with comprehensive agentic abilities. Leveraging our curated 470-hour AgentChat dataset, we incorporate a "Think-before-Speak" mechanism, enabling the model to internalize structured reasoning as a critical prerequisite for planning and response generation. Furthermore, to mitigate latency bottlenecks caused by large-scale tool integration, we propose a Multi-Agent Dynamic Tool Management architecture. By asynchronously delegating retrieval tasks to an auxiliary agent aligned with the main model's reasoning trajectory, this system effectively decouples inference latency from toolset size. Experimental results confirm that VoxMind achieves significant improvements in agent performance: compared with strong baselines, the task completion rate increases from 34.88% to 74.57%, outperforming Gemini-2.5-Pro on spoken agent tasks while preserving general conversational quality. The source code and associated data are publicly available at https://github.com/MM-Speech/VoxMind.

• The paper introduces an end-to-end agentic spoken dialogue system that integrates explicit chain-of-thought reasoning to improve tool selection and task performance.
• It employs a dual-channel memory and parallel dynamic tool management to ensure scalability and minimal inference latency.
• Empirical results show a 74.57% task success rate and a +113.79% improvement over baselines, validating the system's robust agentic capabilities.

VoxMind: An End-to-End Agentic Spoken Dialogue Architecture

Motivation and Formulation of End-to-End Spoken Agents

VoxMind addresses a central limitation in current spoken dialogue systems: the inability to execute complex, agentic tasks requiring explicit reasoning, planning, and integration with a broad array of external tools directly from speech input. Whereas most existing end-to-end speech models are confined to reactive, conversational settings, VoxMind is formulated as an agent that internalizes structured cognitive pathways prior to generating spoken responses, thus moving beyond the typical speech input–speech output paradigm. The paper formally proposes a unifying definition for end-to-end spoken agents, specifying four orthogonal yet interlinked requirements: a comprehensive and adaptive agent profile (encoding both static persona and dynamically inferred acoustic/semantic attributes), a dual-channel memory system (semantic and acoustic), a planning capability via explicit reasoning intermediates, and an action-execution pathway grounded on tool invocation.

The main agent pipeline thus comprises the following: (i) explicit chain-of-thought (CoT) reasoning, (ii) parallelized action selection and tool proposal, and (iii) a dynamically updated local tool set, ensuring both effective tool use and computational scalability as the number of available tools grows.

Figure 1: VoxMind can dynamically perceive interaction context, autonomously determine when to invoke external tools, and drive generation of spoken responses based on tool execution results.

VoxMind System Architecture and "Think-Before-Speak" Mechanism

The VoxMind architecture is instantiated as a hierarchical inference procedure that, for each conversational turn, produces a reasoning trajectory prior to response generation. The architecture employs a multi-agent dynamic tool management strategy: a primary speech-centric agent and an auxiliary LLM-based agent execute in parallel. The primary agent generates a reasoning trajectory and, conditioned on this, produces the next speech action, while the auxiliary model proposes candidate tool expansions from a global tool pool. If a tool usage need is detected, the local tool space is expanded asynchronously without interrupting the agent’s forward computation.

Figure 2: The overall architecture of VoxMind, illustrating explicit chain-of-thought reasoning, parallel action selection, and dynamic tool-space updates via an auxiliary LLM.

This approach decouples inference latency from the tool pool size, a critical requirement for scaling spoken agents to real-world applications featuring hundreds or thousands of callable APIs.

AgentChat: Reasoning-Centric Data Construction

General-purpose reasoning and tool-use capabilities in VoxMind are driven by the AgentChat dataset: a 470-hour multimodal corpus specifically annotated with structured reasoning and supervised tool use paths. The dataset consolidates multiple existing benchmarks (e.g., ToolACE, APIGen-MT) and augments them with automatically generated, human–LLM co-supervised chain-of-thought traces according to strict causal and behavioral fidelity criteria. The reasoning intermediary for any input–output pair is obtained via iterative LLM-guided sampling and scoring: only those trajectories passing stringent quality thresholds are retained.

Reasoning traces are generated both for tool-centric dialogues (causal explanation for tool selection, argument derivation) and for general spoken QA or multi-turn dialogue (ensuring tightly causal, non-hallucinatory reasoning). Diversity in acoustic realizations is ensured via prompt-driven TTS synthesis, leveraging over 600 unique timbral signatures.

Core Competency Realization

VoxMind demonstrates six core speech agent capabilities: single-task tool invocation, compositional task decomposition, parallel tool execution, proactive tool-space expansion (proactive seeking), dynamic result feedback integration, and cross-turn contextual planning. Each core function is systematically supervised in AgentChat, and corresponding evaluation is conducted using rigorous metrics (tool selection, parameter filling, tool usage, and feedback completeness).

Figure 3: Example dialogues illustrating VoxMind’s six core agentic competencies, including task decomposition, parallel execution, and context-sensitive planning and feedback.

Empirically, VoxMind achieves robust agentic behavior across the entire test set, unlike prior end-to-end speech models, which lack explicit intermediate reasoning and demonstrate poor compositionality and contextual robustness.

Numerical Results and Comparative Analysis

Quantitative evaluation spans closed-source models (Gemini-2.5-Pro, GPT-4o-audio), state-of-the-art open-source models (Qwen3-8B+Whisper, Qwen2.5-Omni, Kimi-Audio), and end-to-end baselines such as StepAudio2. On AgentChat, VoxMind achieves an overall task success rate of 74.57%, a +113.79% relative improvement over the best baseline (StepAudio2). Notably, this exceeds Gemini-2.5-Pro (71.51%) and Kimi-Audio (54.94%) on structured tool-centric and reasoning tasks; cascaded systems (Qwen3-8B+Whisper) remain competitive but at the expense of higher latency and an inability to leverage speech-specific memory and profile conditioning.

Ablation reveals that the explicit "Think-Before-Speak" mechanism preserves general conversational proficiency (VoiceBench: 64.21, outperforming StepAudio2’s 64.15) while delivering substantial gains in tool use, parameter filling, and result summarization. Removing reasoning intermediates yields sharp drops in both task performance and out-of-domain generalization.

Dynamic Tool Management: Efficiency and Scalability

The main technical innovation for scalability is the parallel dynamic tool update mechanism. Empirical measurement demonstrates that, as the global tool pool size increases (from 1 to 100 tools), single-agent baselines exhibit exponential growth in latency and corresponding performance decay. VoxMind, leveraging its auxiliary tool retrieval agent, maintains stable and bounded response times, introducing only a negligible waiting overhead ($<$15ms) regardless of toolset size. Task accuracy is unimpacted by this decoupling strategy.

Figure 4: Inference latency and task accuracy scaling with tool pool size; the auxiliary LLM enables efficient tool selection without loss in performance.

Token-level analysis shows that reasoning-stage token consumption is constant and does not scale with toolset size, validating the architectural claim of true $O(1)$ latency per decision step.

Implications, Limitations, and Prospects

Practically, VoxMind’s end-to-end integration of reasoning and tool utilization positions it as a reference architecture for spoken agents tasked with complex real-world interaction—e.g., contextually aware voice assistants, real-time spoken QA with external data access, and dialogic controllers for IoT or knowledge-rich domains. The dynamic tool management and explicit reasoning pipeline provides a roadmap for expanding agentic speech models to domains currently dominated by text-based LLMs, underlining the necessity of multimodal intermediate representations when physical interaction with the environment is required.

Theoretically, the formulation of spoken agents as planning-and-action systems, with unified treatment of both semantic and acoustic memory, offers a rigorous taxonomy for benchmarking future work. However, the current dependency on TTS-synthesized agentic data (rather than genuine, spontaneous spoken agent interactions) may induce a formal bias in the agent's pragmatic range; the authors explicitly note this as an open avenue for the collection and exploitation of naturally occurring agentic spoken data.

Further, while the explicit chain-of-thought mechanism enhances correctness and efficiency, it imposes a non-trivial computational overhead that remains an active trade-off with system responsiveness. Future research includes reducing such latencies, aligning the reasoning traces more closely with the stochastic and disfluent features of real conversations, and extending robustness against highly noisy or accented speech streams.

Conclusion

VoxMind provides a comprehensive and technically rigorous framework for spoken dialogue agents capable of explicit reasoning, structured planning, and dynamic tool integration. Its architectural innovations, empirical superiority over leading open- and closed-source baselines, and clear path toward scaling end-to-end speech agents for complex, multimodal tasks collectively represent a significant advance in agentic dialogue modeling (2604.15710).