A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation

Abstract: Simultaneous speech-to-speech translation (SimulS2ST) enables real-time cross-lingual communication, but existing evaluation has focused largely on short or pre-segmented speech rather than long-form, continuous input. Prior approaches are difficult to reproduce and make assumptions that do not hold for end-to-end systems. We present a practical evaluation method for long-form SimulS2ST. Given source speech, pre-segmented source transcripts, and reference translations, we run automatic speech recognition (ASR) and forced alignment on the generated target speech to recover token-level timestamps, then apply a sentence-embedding-based aligner to match the target text to its corresponding source sentences. This enables sentence-level computation of latency and quality metrics, including YAAL and xCOMET, which are then aggregated into final system-level scores. Experiments on representative SimulS2ST systems show that the method is effective in practice and reveal that current systems suffer from substantial latency accumulation on long speech.

• The paper introduces a novel evaluation pipeline for SimulS2ST that uses forced alignment and SEGALE segmentation to achieve high accuracy in real-world, continuous speech scenarios.
• It demonstrates clear language-dependent latency accumulation, revealing trade-offs between high translation quality and processing speed, especially in En→Ja setups.
• The modular approach enables detailed error analysis and offers practical benchmarks for future improvements in latency-aware, long-form speech translation systems.

A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation

Introduction

This paper introduces a robust and replicable evaluation pipeline for long-form Simultaneous Speech-to-Speech Translation (SimulS2ST), targeting scenarios with continuous audio input, such as extended talks or conferences. Previous evaluation approaches are largely restricted to short, pre-segmented utterances or make unrealistic assumptions unsuited for modern end-to-end systems. The method in this paper leverages forced alignment, sentence-level segmentation with embedding-based alignment (SEGALE), and sentence-level quality and latency measurements, yielding granular interpretability and higher fidelity evaluation on unsegmented, real-world data.

Evaluation Methodology

The evaluation pipeline proposed operates as follows: generated target speech is first transcribed via a state-of-the-art ASR/forced alignment model (Qwen3-ASR-1.7B and Qwen3-ForcedAligner-0.6B), producing token-level timestamps. The resultant transcripts are then segmented and aligned semantically with source sentence transcripts using SEGALE, which leverages Vecalign and an adaptive skip-penalty mechanism to handle one-to-one, many-to-one, many-to-many, and null (over/under-translation) alignments robustly.

Once source and target sentence groups are aligned, sentence-level latency is computed employing metrics such as YAAL, which penalizes temporal asynchrony relative to an idealized translation scenario. Translation quality is assessed using reference-based metrics (e.g., xCOMET) at the sentence group level, with careful treatment of over- and under-translation cases by assigning minimum scores. Aggregation across all groups yields final system-level latency and quality metrics suitable for benchmarking long-form SimulS2ST performance.

Experimental Results

The evaluation encompasses leading SimulS2ST systems (Seed LiveInterpret 2.0, Hibiki-Zero, SeamlessStreaming) on two datasets: ACL 60/60 dev (long English talks, benchmarked in En→De, En→Ja, En→Zh) and Audio-NTREX-L (multilingual synthetic test data, Fr/De/Pt/Es→En). Seed LiveInterpret 2.0 achieves the highest xCOMET scores but incurs substantial latency, particularly in En→Ja (nearly 10 seconds end-to-end latency).

Latency and quality metrics exhibit clear system-dependent trade-offs. While Hibiki-Zero and SeamlessStreaming deliver lower latency, the translation quality sees modest reductions compared to Seed LiveInterpret 2.0.

A critical observation is that all SimulS2ST systems evaluated show increasing latency accumulation with longer input speech, a phenomenon that becomes pronounced for En→Ja and En→De, with less effect seen for En→Zh, where target utterances are generally shorter than source (Figure 1).

Figure 1: The ending offset of each aligned sentence for two systems on every speech in the ACL 60/60 dev set. Latency generally accumulates as the source speech becomes longer.

Further analysis reveals language-dependent target-source duration differences. For En→Zh, the positive offset for duration primarily stabilizes latency, while En→Ja sees the opposite trend, with target speech often longer than source and latency increasing dramatically (Figure 2).

Figure 2: Distribution of target--source duration differences on the ACL 60/60 dev set. En→Zh is mostly negative, En→De centered around zero, En→Ja mostly positive.

Segmentation and Forced Alignment Analysis

SEGALE demonstrates superior segmentation accuracy (90.9%) as compared to token-level segmenters such as SoftSegmenter (79.1%), especially in handling non-literal and fuzzy cross-lingual sentence correspondences. It exhibits resilience to surface-form mismatches and preserves semantic coherence, directly impacting both the fairness and reliability of downstream latency and quality measurements.

On the ASR/transcription side, analyses indicate that Seed LiveInterpret 2.0's underperformance in En→Ja is attributed not to ASR model limitations but poor target speech synthesis, as confirmed by high CERs from both Qwen3-ASR-1.7B and WhisperX.

Practical and Theoretical Implications

The approach offers several concrete advances for the SimulS2ST research community:

• Reproducibility: The method is system-agnostic, does not require internal system outputs, and is reproducible with open-source or widely available models.
• Robust to Real Use: Designed for continuous, hours-long speech, it reflects realistic deployment conditions, unlike previous utterance-centric benchmarks.
• Compositional Error Detection: By treating segmentation, alignment, and metric calculation modularly, the method facilitates fine-grained error analysis (temporality, comprehensibility, alignment failures, and translation adequacy).
• Reveals Latency Accumulation: The empirical analysis reveals a persistent issue—accumulating latency over long input streams—that generalizes across architectures and languages.

Theoretically, robust synchrony between source and target speech remains an unsolved problem in end-to-end systems, with current models lacking explicit mechanisms to bound latency across long sequences, particularly when target length ratios deviate from the source. These results motivate further advances in latency-aware training objectives, online regularization methods, and low-drift attention architectures for real-time speech SI.

Future Developments

• Incorporating streaming alignment or monotonicity constraints, particularly in TTS decoders, is an open avenue.
• End-to-end latency-aware training (jointly training for accuracy and asynchrony minimization) is likely to see continued emphasis, possibly leveraging differentiable latency metrics.
• Cross-lingual duration modeling, seen to be critical for latency accumulation, merits specialized modules or constraint layers, especially for languages with divergent phrasal or syllabic pacing.
• As this evaluation approach exposes quality-latency tradeoffs and errors in long-form SI, its adoption is expected to accelerate benchmarking and research progress on low-latency, high-fidelity systems.

Conclusion

This work establishes a practical, modular, and language-agnostic evaluation protocol for SimulS2ST on long, continuous speech inputs. The analysis demonstrates that all current state-of-the-art systems exhibit non-trivial latency accumulation aligned with target-source duration mismatches, exposing a key challenge for system designers. The combination of robust forced alignment, SEGALE segmentation, and granular metric computation constitutes an effective framework for both system development and error analysis. This methodology provides a foundation for future advances in scalable, reliable, and use-driven SimulS2ST.