Next-Turn: Duration-Aware Streaming Endpoint Detection via Time-to-Next-Speech-Onset Prediction

Abstract: Endpoint detection (EPD) is essential for natural turn-taking in streaming speech systems. However, reliably determining the endpoint of an utterance is challenging because speakers often pause mid-utterance due to hesitations and disfluencies. Semantic EPD has emerged as a promising direction to address this issue but is hindered by ambiguous supervision and strict streaming constraints. We propose Next-Turn that uses the time-to-next-speech-onset as the training objective, where targets are derived directly from speech timestamps and require no additional annotation. Experiments show that the proposed method outperforms conventional acoustic and recent semantic EPD baselines, achieving a 25.9% absolute improvement in endpoint accuracy within 320 ms over the strongest baseline. In addition, joint training with the duration-aware objective complements standard binary EPD, with gains that increase monotonically with increasing pauses.

• The paper introduces Next-Turn, a duration-aware approach that predicts time-to-next-speech-onset to enhance endpoint detection accuracy.
• It employs a Whisper-based encoder fine-tuned with LoRA and joint binary-duration training to substantially reduce early interruptions and boost ACC320.
• Experimental results on a large-scale Chinese corpus demonstrate robust, low-latency performance, even with smaller backbone models, enabling real-time deployment.

Duration-Aware Streaming Endpoint Detection via Time-to-Next-Speech-Onset Prediction

Problem Motivation and Limitations of Existing EPD Methods

Streaming endpoint detection (EPD) is critical for full-duplex conversational agents, real-time speech translation systems, and interactive voice interfaces. The core challenge lies in the incongruence between acoustic silence and true semantic utterance boundaries, as pauses due to hesitations and planning are frequent but do not constitute actual turn-completion. Conventional acoustic EPD solutions, typically neural VADs or statistical models, are constrained by their insensitivity to semantic contours, often triggering premature or delayed endpoints, thus degrading responsiveness and introducing significant tail-latency. Previous attempts to integrate ASR-driven semantics exacerbate susceptibility to recognition errors and coupling cost/latency to the ASR pipeline.

Efforts towards low-latency, audio-only semantic EPDs have been hindered by the lack of reliable semantic supervision and inherent limitations of streaming constraints restricting right-side context. Label ambiguity arises as conversational completion is context-dependent, and annotation noise undermines the learning process.

The Next-Turn Framework

To address these challenges, the paper proposes Next-Turn, a novel semantic EPD framework, which introduces a duration-aware objective by predicting the time-to-next-speech-onset at each frame. This approach fundamentally reframes EPD as a regression or classification task on temporal distance to upcoming speech, using targets derived directly from forced-aligned speech timestamps. This obviates the need for costly and ambiguous manual semantic labels.

The system employs a Whisper-based encoder fine-tuned using LoRA. For binary EPD, a classifier predicts on/off endpoints; for duration-aware EPD, a parallel head predicts either a continuous (REG) or discretized (CLS) time-to-next-onset. Multi-task learning with joint heads (binary and duration) is utilized to provide both coarse and fine-grained supervision. During inference, the system yields either individual or fused endpoint scores via linear weighting.

The training regime samples truncated utterances in speech, during mid-utterance pauses, and in utterance-final silence to robustly expose the model to all conversational states. Streaming evaluation is performed with 160 ms non-overlapping chunks and exponential smoothing for score stabilization.

Experimental Results and Analysis

Experiments are conducted on a large-scale (1,177 hours) Chinese conversational and command corpus, using a manually labeled evaluation set stratified by mid-utterance pause counts. Two principal evaluation metrics are adopted: Early Interruption (EI)—the rate of too-early triggers, and ACC$_\delta$—the percent of triggers within $\delta$ ms of the true endpoint (with $\delta=320$ ms emphasized as industry standard).

Key empirical findings:

• Duration-aware (REG) single-task improves over binary baseline with lower EI ($8.1\%$ vs $9.6\%$) and higher ACC$_{320}$ (86.4\% vs 83.9\%).
• Joint-training with duration supervision further improves binary EPD, with the best setting (Joint CLS with binary inference) achieving EI $5.0\%$, ACC$_{320} = 86.7\%$, a 25.9% absolute ACC improvement over the best baseline (Easy Turn [18]) and massive reduction in interruption versus all prior open-source methods.
• Performance gains from the duration-aware objective scale with the number of utterance-internal pauses, demonstrating enhanced robustness to conversational hesitations—a known weakness for classical models.
• Next-Turn achieves consistently better performance as the backbone model size decreases, with even the Whisper-tiny variant outperforming prior semantic EPDs in EI and accuracy at a computational cost conducive to low-latency deployment.

Ablation on context window size for exponential smoothing indicates a trade-off: increasing lookback (P) reduces EI but can introduce detection delay, while using right context (F) continues this trend but with further increased response latency.

Implications and Future Directions

By reframing EPD as a duration prediction problem, Next-Turn provides a new class of supervision signal that is robust, scalable (no manual semantic labels needed), and naturally better aligned to the conversational notion of turn-completion. It enables stable, low-latency EPD decisions resilient to pause density and amenable to deployment in streaming, lightweight environments. The results demonstrate substantial practical advances over both acoustic-only and previous semantic EPD pipelines, and chart a path towards plug-and-play, real-time semantic EPD components scalable to diverse deployment settings.

Theoretically, this duration-centric reformulation suggests a shift from binary state prediction to temporally calibrated progression modeling—an approach that may generalize to other segmentation problems in sequential speech and language.

Conclusion

Next-Turn constitutes a significant advancement in duration-aware, streaming EPD by leveraging time-to-next-speech-onset as its supervisory signal. The approach unequivocally outperforms both acoustic and state-of-the-art semantic EPD systems in accuracy and early interruption—across backbone sizes and pause regimes—while remaining computationally tractable. This work opens avenues for future research in multi-turn conversational modeling, multilingual and cross-corpus validation, and seamless integration of duration-aware objectives into broader spoken language understanding systems.

---

Reference: "Next-Turn: Duration-Aware Streaming Endpoint Detection via Time-to-Next-Speech-Onset Prediction" (2606.18094)