- The paper presents an open-source, nine-chapter implementation showing that streaming STT, LLM, and TTS can deliver 755ms end-to-end time-to-first-audio while supporting enterprise function calling.
- The paper finds that sentence-level buffering and pipeline overlap are the key latency optimizations, with Deepgram STT, vLLM or OpenAI-compatible LLMs, and ElevenLabs TTS combining for sub-second responses.
- The paper reports that its cascaded design achieves roughly 17× faster TTFA than Qwen2.5-Omni-7B, while noting that native speech-to-speech models still require faster audio generation and tool-use capabilities.
Overview
This tutorial paper from Salesforce AI Research presents a complete, from-scratch implementation of an enterprise-grade realtime voice agent, released as a nine-chapter open-source codebase (2603.05413). Its central contribution is empirical and architectural rather than algorithmic: the authors demonstrate that the industry-standard cascaded pipeline — streaming STT → LLM → TTS — outperforms native speech-to-speech models by a wide margin on time-to-first-audio (TTFA) while uniquely supporting function calling. The measured cascaded pipeline achieves a best-case TTFA of 729ms (755ms in a fully measured end-to-end streaming test), compared to approximately 13.2s for Qwen2.5-Omni-7B with sentence-level streaming.
The paper opens with a survey of over 25 open-source speech-to-speech models and 30+ voice agent frameworks, organized into three levels of "nativeness." Level 1 models (e.g., Moshi) think directly in speech tokens and achieve roughly 200ms latency, but lack strong reasoning. Level 2 models (Qwen2.5-Omni, GLM-4-Voice, Kimi-Audio, Step-Audio, LLaMA-Omni, Mini-Omni, Freeze-Omni) pair a text LLM with speech encoder/decoder heads — the largest category. Level 3 is the cascaded ASR → LLM → TTS architecture used by all production systems (Vapi, Retell, Bland.ai, Pipecat, LiveKit).
A critical finding from this survey: none of the Level 1 or Level 2 open models support function calling, which the authors identify as essential for enterprise voice agents (appointment scheduling, database queries, order management). Existing frameworks such as Pipecat and LiveKit provide production-ready plumbing but are opaque about internal streaming mechanics; Benchforce defines enterprise evaluation environments but uses turn-based, non-streaming pipelines. The paper positions itself as filling this educational gap rather than proposing a new model.
Why native speech-to-speech is not yet realtime
The authors benchmark Qwen2.5-Omni-7B (Thinker-Talker architecture, BF16, single A10G GPU) and find the DiT-based Talker operates at approximately 0.5× realtime — about 2 seconds of compute per second of audio. Batch-mode inference yields 26.5s TTFA for a 13.8s response; sentence-level streaming reduces this only to 13.2s. Three factors make the model unsuitable for realtime deployment: slow audio generation, no function calling support, and no incremental audio output (each generate() call blocks until full synthesis). vLLM's support for the model is limited to the Thinker, so serving acceleration cannot help the bottleneck.
The paper concedes a methodological caveat: results are specific to one model on one GPU, and the authors note that transformers version 4.52.3 is required for acceptable audio quality (versions ≥5.0 produce noisy output) — a reproducibility hazard they document explicitly. The conclusion that native S2S is "not viable" should therefore be read as a statement about the current open-source ecosystem, not a theoretical bound.
The streaming pipeline architecture
The core architectural insight is that realtime behavior emerges from streaming plus pipelining across components, not from any single fast model. In a turn-based pipeline, latency is the sum of all stages (~1600ms in the authors' estimate); in a streaming pipeline, TTFA reduces to STT finalization plus LLM first-sentence latency plus TTS time-to-first-byte (~900ms estimated), because subsequent sentences are synthesized and played while earlier ones are still being spoken.
The three components and their measured latencies:
| Component |
Service |
P50 latency |
| Streaming STT |
Deepgram Nova-3 (WebSocket, 20ms PCM chunks) |
337–509ms (min 184ms) |
| Streaming LLM |
vLLM-served Qwen2.5-7B-Instruct (SSE) |
TTFT 337ms, 17.5 tok/s |
| Streaming TTS |
ElevenLabs eleven_turbo_v2_5 |
TTFB 219–236ms, RTF 0.05–0.10× |
The sentence buffer is identified as the critical orchestration primitive: it accumulates LLM tokens, detects sentence boundaries while excluding false positives (abbreviations, decimals), enforces a minimum sentence length of 10 characters, and flushes on stream end. This replicates the aggregation pattern used internally by Pipecat's SentenceAggregator and LiveKit's text processing pipeline. The LLM layer uses the standard OpenAI client, making the implementation portable across vLLM, OpenAI, Azure, or any OpenAI-compatible backend via environment variables alone. Function calling is handled through a recursive tool-use loop that supports multi-step tool chains.
Turn-taking and transport
Voice activity detection uses Silero VAD (2MB, sub-millisecond on CPU for 32ms chunks) driving a four-state machine — IDLE → LISTENING → PROCESSING → SPEAKING — with an interruption path from SPEAKING back to LISTENING. The WebSocket protocol carries binary PCM frames (16kHz up, 24kHz down) with JSON control messages for transcript and agent-speaking state. The browser client uses AudioWorklet processors for low-latency capture and queue-based playback, with getUserMedia echo cancellation supplemented by a server-side echo gate that attenuates microphone input during agent speech.
Latency results
The headline benchmarks combine component measurements with an end-to-end streaming test:
| Configuration |
P50 TTFA |
Best-case TTFA |
| Cloud OpenAI API (GPT-4.1-mini) |
958ms |
715ms |
| Self-hosted vLLM (Qwen2.5-7B, A10G) |
947ms |
729ms |
| Measured end-to-end streaming pipeline |
— |
755ms |
The measured pipeline TTFA of 755ms falls below the sequential sum of its stages (296ms TTFT + 143ms sentence detection + 316ms TTS synthesis = 755ms), confirming that overlap reduces latency relative to the naive estimate. ElevenLabs is the most consistent component (<20% variance), while LLM TTFT dominates variance — up to 4.3s during vLLM cold start, a limitation the authors report without mitigation. Relative to Qwen2.5-Omni, the cascaded approach delivers an approximately 17× TTFA improvement with function calling enabled.
Practical engineering notes
The paper documents a set of non-obvious pitfalls that would otherwise cost implementers substantial debugging time: the transformers version constraint on Qwen2.5-Omni audio quality; degraded audio when device_map="auto" spans multiple GPUs; extreme sensitivity of Qwen2.5-Omni to its exact training system prompt; a working trick of prepending "Please speak quickly" to increase speech rate; vLLM v0.16.0's AWS-only SageMaker dependency (v0.8.5 recommended); breaking API changes in Deepgram SDK v6; and the need to guard against empty SSE chunks. These notes are among the most practically valuable parts of the paper, as they are largely undocumented elsewhere.
Limitations and open questions
Several constraints bound the paper's claims. The native S2S comparison rests on a single model (Qwen2.5-Omni-7B) on a single GPU class; faster native models such as Moshi (~200ms latency) are acknowledged in the survey but not benchmarked head-to-head in the streaming setting. The TTFA figures for the full pipeline are estimated as a sequential sum in some configurations, with only the 755ms pipeline test fully measured end-to-end. The system relies on proprietary cloud services (Deepgram, ElevenLabs) for its best latency numbers, and the paper does not evaluate self-hosted TTS alternatives (e.g., Orpheus, CosyVoice) in the pipeline. Barge-in behavior, multi-turn dialogue quality, and task success rates on the enterprise tools are not quantitatively evaluated — the hospital receptionist scenario is presented as an implementation, not a benchmark. Finally, the paper does not address concurrent-session scaling or production reliability beyond single-session latency.
Conclusion
The paper establishes, with measured evidence, that sub-second TTFA enterprise voice agents are achievable today using a cascaded streaming pipeline with function calling, and that the decisive engineering primitive is sentence-level streaming between LLM and TTS rather than model speed. Its open-source, nine-chapter codebase makes the internal mechanics of production frameworks reproducible. The central open question left by the work is whether native speech-to-speech models can close the gap on incremental audio generation and function calling support sufficiently to displace the cascaded architecture.
