VoxMind: End-to-End Agentic Speech System

• VoxMind is an end-to-end spoken dialogue system featuring explicit chain-of-thought reasoning, planning, and dynamic tool management for agentic interaction.
• It employs a 'think-before-speak' mechanism that boosts tool usage accuracy from 31.34% to 68.66% by generating structured natural language reasoning before action.
• Trained on a 470-hour AgentChat corpus, VoxMind excels in proactive tool selection, task decomposition, and maintaining general conversational quality.

VoxMind is an end-to-end spoken dialogue system that equips a speech LLM with explicit reasoning, planning, and tool use, thereby recasting speech interaction from reactive dialogue into agentic spoken action. Introduced as an integrated framework built by fine-tuning the open-source StepAudio2 model, VoxMind combines a supervised “Think-before-Speak” mechanism with a Multi-Agent Dynamic Tool Management architecture, and is trained on the 470-hour AgentChat corpus, whose speech dialogues are annotated with chain-of-thought reasoning and tool interactions (Liang et al., 17 Apr 2026).

1. Conceptual definition and scope

VoxMind is presented within a broader formalization of the End-to-End Spoken Agent. In that formulation, a spoken agent is not limited to speech perception and speech generation. It instead comprises four components: Profile Definition $\mathcal{P} = (P_{static}, P_{dynamic}(c))$, where static properties include timbre, role, and accent and dynamic properties vary with context $c$; memory, including semantic and acoustic short-term and long-term components; planning capability, which changes the mapping from $x \to y$ into $x \to z \to y$; and action execution via tools, in which the model decides whether tool use is necessary, selects a tool $t^* \in \mathcal{T}$, and generates its parameters (Liang et al., 17 Apr 2026).

This definition positions VoxMind against earlier end-to-end spoken dialogue models such as SpeechGPT, Qwen-Omni, Kimi-Audio, and StepAudio2. Those systems are characterized in the paper as strong at conversational mapping from audio input to audio output, but limited in explicit planning, structured tool use, and scalable handling of large tool inventories. VoxMind’s central claim is that spoken systems require the same kind of explicit reasoning-and-acting loop that text agents use, but adapted to the latency, token, and interface constraints of speech.

A useful contextual contrast is provided by later multimodal evaluation work. MultiVox reports that current omni voice assistants often achieve relatively strong visual grounding but low speech grounding, indicating that contemporary spoken systems frequently remain bottlenecked by shallow treatment of non-textual audio information (Selvakumar et al., 14 Jul 2025). VoxMind addresses a different axis—agentic tool use rather than speech–vision grounding—but both lines of work reflect a shared diagnosis: spoken assistants require more structured internal control than a pure speech-in, speech-out mapping.

2. Formal model and architecture

VoxMind models the interaction state at time $t$ as

$$\mathcal{S}_t = (\mathbf{O}_t, \mathcal{H}_t, \mathcal{A}_t),$$

where $\mathbf{O}_t$ denotes observable events, $\mathcal{H}_t$ the multimodal interaction history, and $\mathcal{A}_t$ the action space. Observable events are split into current user input $c$0, encoded from speech, and structured environmental feedback $c$1, so that

$c$2

The action space includes both verbal responses $c$3 and a local callable tool subset $c$4 (Liang et al., 17 Apr 2026).

The architecture is organized around two policies. The think policy generates an explicit chain-of-thought trajectory: $c$5 The act policy then conditions on that reasoning trace: $c$6 Here, $c$7 may be either a verbal response or a structured tool invocation. The model therefore instantiates the planning decomposition $c$8, with $c$9 realized as a textual reasoning variable rather than an implicit latent state.

Operationally, the backbone is a StepAudio2-based end-to-end speech model. User speech is encoded into model tokens; the model first emits a think: sequence as text, then uses that sequence to generate either tool calls or a final answer. Tool outputs are reintroduced as structured observations and can trigger additional think–act cycles. A common misconception is that “end-to-end” here implies the absence of auxiliary modules. In VoxMind, the core speech-dialogue mapping remains unified within a single speech model, but an auxiliary LLM is introduced for asynchronous tool retrieval rather than for the main dialogue policy.

3. “Think-before-Speak” and tool-oriented planning

The “Think-before-Speak” mechanism is VoxMind’s central methodological contribution. Reasoning is represented explicitly as a structured natural-language chain-of-thought stored in a "think" field. These traces are designed to be strictly causal, to justify tool selection and argument construction, and to remain length-controlled, typically 5–12 steps with a THINK_MAX_WORDS limit (Liang et al., 17 Apr 2026).

The supervision pipeline for these traces is itself structured. For a training example with task input $x \to y$0 and gold output $x \to y$1, a large text LLM first generates a reasoning trace: $x \to y$2 A separate evaluator assigns a quality score $x \to y$3, and only traces meeting a threshold $x \to y$4 are retained: $x \to y$5 If the trace falls below threshold, regeneration is allowed up to $x \to y$6 times. A refinement stage then compresses and polishes the trace: $x \to y$7 These refined traces become supervised intermediate targets during training.

Within dialogue, the chain-of-thought has a concrete control function. It performs intent interpretation, identifies missing knowledge, decomposes tasks, justifies tool use, and explains how arguments are derived from the user query and dialogue context. Tool interaction examples in the paper show the model emitting explicit tool calls such as headset.feature_availability or headset.user_reviews_analysis, followed by structured observations returned by the environment. The next reasoning step then interprets these observations and either produces a final answer or continues planning.

Empirically, the paper shows that explicit thinking is not merely stylistic. Ablation results demonstrate that removing the think stage substantially harms agent performance, especially in proactive seeking, the subtask in which the system must recognize that locally available tools are insufficient and request additional ones. In the reported ablation, tool usage accuracy rises from 31.34 without thinking to 68.66 with thinking under the best data mixture, indicating that the intermediate reasoning trace functions as a practical planning interface rather than decorative explanation (Liang et al., 17 Apr 2026).

4. Multi-Agent Dynamic Tool Management

VoxMind’s second major innovation is its separation of local tool use from global tool retrieval. The paper argues that naïvely placing a large tool library into the active prompt is especially costly for speech agents because audio already produces long sequences, and tool descriptions further inflate inference cost. The proposed alternative is to maintain a small local tool set $x \to y$8 and asynchronously query the global inventory $x \to y$9 only when necessary (Liang et al., 17 Apr 2026).

After the think stage produces $x \to z \to y$0, two processes run in parallel: $x \to z \to y$1 The main agent samples its next action from the current local tool set, while an auxiliary LLM retrieves candidate tools from the full global pool based on the same reasoning trajectory. If the selected action is the special retrieval action $x \to z \to y$2, the local set is updated as

$x \to z \to y$3

otherwise, the local tool set remains unchanged.

This design is intended to decouple effective latency from toolset size. In the appendix latency experiment, increasing $x \to z \to y$4 from 10 to 100 raises the auxiliary retrieval duration from 1.31 s to 2.64 s, but the waiting overhead incurred by the main agent remains below 0.015 s. The reasoning overhead is likewise bounded: THINK tokens remain roughly 80–90 tokens and account for about 12.6% of all generated tokens in speech outputs. The consequence is that retrieval latency is largely hidden behind concurrent reasoning and acting, so the spoken agent need not pay the full prompt-time cost of the global tool inventory.

The architectural significance of this component is broader than function calling. It converts tool access into a two-level control problem: a small stable action interface for the main speech model, and a large dynamic retrieval substrate handled by an auxiliary agent. This suggests a scalable pattern for spoken systems whose action space would otherwise become prohibitive under large API ecosystems.

5. AgentChat corpus and optimization regime

VoxMind is trained on AgentChat, a 470-hour speech corpus designed specifically for agentic spoken interaction. The corpus has two main components. AgentChat-Tool contains 14,805 samples and 109.07 hours of speech focused on tool interaction; it includes subsets such as tool-select, multi-tool-select, para-filled, parallel-call, searchTool, observation, and obs_searchtools. AgentChat-Normal contains 38,681 samples and 360.95 hours of general dialogue data derived from sources such as ARC, GSM8K, SciQ, conversation, course, and multi-conversation (Liang et al., 17 Apr 2026).

The tool-interaction portion is constructed from ToolACE, APIGen-MT, and custom-synthesized data. The general-dialogue portion is intended to preserve ordinary conversational and reasoning competence while the system is specialized for tool use. Additional safety and modality data, totaling about 5 hours, include No-Tool, Security, and extra Text data.

Because large native spoken-agent corpora are scarce, the speech is synthesized from cleaned text using CosyVoice2, with more than 600 prompt-based timbres from SeedTTS to increase speaker and style diversity. The paper explicitly notes that this synthetic origin is a limitation: it provides scale and supervision, but does not fully capture spontaneous disfluencies and naturally occurring spoken pragmatics.

Two data mixtures are studied. A 1:1 mixture balances agentic and general data, while a 1:0.5 mixture downsamples general conversation and knowledge data by about half while preserving tool-centric data. The best performing configuration is the think-supervised model trained with the 1:0.5 ratio, suggesting that strong agent specialization need not degrade general conversation if the specialization is mediated by explicit reasoning traces rather than direct memorization of tool behaviors.

Training uses PyTorch 2.6.0 and CUDA 12.4 on 2 × H20 NVLink GPUs, with batch size 1, gradient accumulation 8, learning rate $x \to z \to y$5, a cosine schedule, AdamW, weight decay 0.01, max gradient norm 1.0, bfloat16, DeepSpeed ZeRO-3, and gradient checkpointing. The optimization is fully supervised; the paper does not use reinforcement learning for the core agent.

6. Evaluation, empirical performance, and significance

VoxMind is evaluated along three axes: agentic spoken task performance, general conversational performance, and tool-management scalability. The agentic benchmark covers six capability areas: single-task processing, task decomposition, parallel processing, proactive seeking, result feedback, and contextual planning. Metrics include TS (Tool Selection), PF (Parameter Filling), TU (Tool Usage), FC (Feedback Completeness), and a composite Overall score (Liang et al., 17 Apr 2026).

On spoken agent tasks, VoxMind achieves an Overall score of 74.57, compared with 34.88 for the StepAudio2 base model, 54.94 for Kimi-Audio, 64.00 for Qwen3-8B + Whisper, and 71.51 for Gemini-2.5-Pro. This corresponds to a relative improvement of +113.79% over the base model. The detailed VoxMind scores show particularly strong gains in tool-centric competencies: TS = 98.50, PF = 72.18 for single-task processing; TS = 95.24, PF = 38.10 for task decomposition; TS = 89.52, PF = 61.59 for parallel processing; TS = 80.82, PF = 62.33 for contextual planning; TU = 68.66 for proactive seeking; and FC = 3.94 for result feedback.

Equally important is the preservation of general conversational quality. On VoiceBench, the base StepAudio2 model obtains an overall score of 64.15, while the best VoxMind configuration reaches 64.21. By contrast, agent-specialized models trained without explicit thinking degrade more noticeably. This result supports the paper’s central thesis: explicit intermediate reasoning can improve agent performance without the usual loss of ordinary spoken dialogue quality.

The robustness study on real audio shows some degradation relative to synthetic speech but not collapse. On 150 real recordings, FS is 86.00% and PF is 60.67%, compared with 93.33% and 67.33% on 150 TTS utterances. This suggests that training on synthesized speech transfers reasonably to recorded speech, though imperfectly.

The principal limitations identified by the authors are twofold. First, explicit thinking adds latency, even if bounded and partially amortized. Second, AgentChat remains a synthetic speech corpus and therefore lacks the full range of natural spoken disfluency and spontaneity. These limitations frame the paper’s future direction: reducing reasoning overhead and constructing more native spoken-agent datasets.

Within the evolution of spoken systems, VoxMind is significant because it turns end-to-end speech modeling into an explicit reason–act–observe loop while addressing the usually neglected systems problem of tool scalability. It therefore stands as both a model and a design pattern: a spoken agent whose competence depends not only on speech perception and generation, but also on supervised intermediate reasoning, structured tool execution, and asynchronous control over a large external action space (Liang et al., 17 Apr 2026).