Simultaneous Speech-to-Speech Translation (SimulS2ST)

Updated 6 July 2026

SimulS2ST is the real-time process that converts streaming source speech into target-language speech with minimal delay.
Systems are organized into cascaded pipelines, direct end-to-end models, and multistream speech-LLM architectures, each balancing translation quality and latency.
Effective policy control and adaptive training methods are essential to manage non-monotonic dependencies and produce natural, coherent, long-form target speech.

Searching arXiv for recent and foundational SimulS2ST work to ground the article. arxiv_search(query="simultaneous speech-to-speech translation SimulS2ST", max_results=10, sort_by="relevance") Searching arXiv for SimulS2ST. to=arxiv_search 天天送钱彩票json {"query":"simultaneous speech-to-speech translation SimulS2ST","max_results":10,"sort_by":"relevance"} Searching arXiv for simultaneous speech-to-speech translation. Simultaneous speech-to-speech translation (SimulS2ST) is the task of generating target-language speech concurrently with incoming source-language speech, typically with only a few seconds delay rather than waiting for the end of the utterance. In the simultaneous regime, the system must translate and synthesize incrementally, usually by interleaving READ and WRITE decisions over a streaming input, and in the long-form regime it must do so for unsegmented, continuous speech rather than isolated sentences (Zheng et al., 2020, Zhang et al., 2024, Djanibekov et al., 11 Mar 2026). The task is difficult for at least three reasons that recur across the literature: non-monotonic cross-lingual dependencies, the need to control latency without destabilizing translation quality, and the need to produce acoustically coherent target speech rather than merely timely text (Ma et al., 2021, Lee et al., 11 Jun 2026).

1. Task definition and operating regimes

A formal dataset view used by direct systems represents speech-to-speech translation as quadruples $(X,A,Y,S)$ , where $X$ is the source speech feature sequence, $A$ its source transcription, $Y$ the target text, and $S$ the target speech. Simul-S2ST processes $X$ in a streaming fashion and alternately chooses READ or WRITE actions so as to emit synthesized target speech as early as possible while preserving translation quality (Zhang et al., 2024). Long-form SimulS2ST strengthens the requirement: the input is not pre-segmented into short utterances but arrives as an unbounded audio sequence such as an entire TED talk or meeting, and the system must decide, at each moment, whether to READ more source or WRITE partial speech output (Djanibekov et al., 11 Mar 2026).

A central distinction is between simultaneous interpretation and consecutive translation. Hibiki explicitly frames simultaneous interpretation as a setting in which the model must “accumulate just enough context to produce a correct translation in real-time, chunk by chunk,” unlike consecutive translation, which waits for the end of the source utterance before translating (Labiausse et al., 5 Feb 2025). This distinction matters because latency is not merely a deployment variable; it changes the translation problem itself, especially for language pairs with strong reordering.

Early work already emphasized that continuous operation introduces pathologies not visible in sentence-level simulation. The Self-Adaptive Translation paper states that recent solutions focused on the single-sentence scenario, causing latency to accumulate progressively when the speaker talks faster and introducing unnatural pauses when the speaker talks slower (Zheng et al., 2020). Later long-form studies generalize that observation by showing that short or pre-segmented evaluation does not adequately capture accumulation effects over multi-minute inputs (Xue et al., 13 Jun 2026).

2. Architectural families

A useful taxonomy divides reported systems into three recurrent architectural families: cascaded incremental pipelines, direct end-to-end streaming models, and multistream or speech-LLM systems.

Family	Representative systems	Core mechanism
Cascaded incremental	Neural incremental ASR-MT-TTS (Sudoh et al., 2020); offline-model-based speaking policies (Dugan et al., 2023)	Separate streaming or pseudo-streaming modules
Direct end-to-end	V-MMA direct S2U (Ma et al., 2021); SimulTron (Agranovich et al., 2024); StreamSpeech (Zhang et al., 2024); NAST-S2X (Ma et al., 2024)	Source speech mapped directly to spectrograms or discrete units
Multistream / speech-LLM	Hibiki (Labiausse et al., 5 Feb 2025); SimulS2S-LLM (Deng et al., 22 Apr 2025); Seed LiveInterpret 2.0 (Cheng et al., 23 Jul 2025)	Shared backbone jointly emits text and audio tokens

The cascaded lineage is exemplified by the 2020 system with fully incremental neural ASR, MT, and TTS. Its ISR, IMT, and ITTS modules begin processing before their predecessors finish, and all three communicate in symbol-synchronous streaming mode via UNIX pipes (Sudoh et al., 2020). A later line reuses strong offline models in an online loop: an off-the-shelf Whisper model produces candidate translations from an accumulating buffer, and a speaking policy decides when to commit text to a TTS API (Dugan et al., 2023). These cascades are straightforward to assemble, but multiple papers identify error propagation and delay accumulation across ASR, translation, and synthesis as structural limitations (Sudoh et al., 2020, Zhang et al., 2024, Ma et al., 2024).

Direct end-to-end systems avoid intermediate text or use it only as auxiliary supervision. The V-MMA model predicts discrete speech units learned in an unsupervised manner and passes them directly to a vocoder for on-the-fly synthesis, with translation generation independent from intermediate text representations (Ma et al., 2021). SimulTron follows the Translatotron 1 paradigm of directly mapping source-language waveform to target-language spectrogram to target waveform, but re-architects the encoder, decoder, PostNet, and vocoder for strict streaming operation and on-device deployment (Agranovich et al., 2024). StreamSpeech introduces an “All-in-One” multi-task framework in which one model performs offline and simultaneous ASR, speech translation, and speech synthesis through a chunk-based Conformer encoder, an autoregressive text decoder, and a non-autoregressive text-to-unit generator with a frozen HiFi-GAN vocoder (Zhang et al., 2024). NAST-S2X pushes this direction further by making chunk-to-chunk generation non-autoregressive and allowing blank or repeated tokens with CTC collapse for dynamic latency adjustment (Ma et al., 2024).

The multistream and speech-LLM family treats simultaneous translation as joint token generation over text and audio channels. Hibiki is a decoder-only model built on the RQ-Transformer framework, with a Temporal Transformer that ingests previously generated tokens from both streams and a Depth Transformer that predicts quantized audio tokens and aligned text tokens at codec frame rate (Labiausse et al., 5 Feb 2025). SimulS2S-LLM instead combines a chunk-masked Conformer encoder, a Continuous Integrate-and-Fire prompt extractor, a frozen text LLM, and a streaming speech generator that predicts discrete semantic speech tokens for a unit-based HiFi-GAN vocoder (Deng et al., 22 Apr 2025). Seed LiveInterpret 2.0 presents a duplex speech-to-speech understanding-generating framework in which an audio encoder, multimodal LLM core, text decoder, audio decoder, voice-cloning module, and neural vocoder share state across tightly coupled understanding and generation streams (Cheng et al., 23 Jul 2025).

3. Alignment, policy, and latency control

The defining control problem in SimulS2ST is deciding when sufficient source evidence has accumulated to justify speech emission. The simplest solution is fixed wait- $k$ or fixed delay. In SimulTron, the decoder attends only to the first $t_{in}+k$ encoder frames when generating output step $t_{out}$ , with $\Delta = k \times 20 \text{ ms}$ , so that $X$ 0 corresponds to roughly $X$ 1 seconds delay and $X$ 2 to roughly $X$ 3 second (Agranovich et al., 2024). Earlier incremental MT inside a cascaded S2ST system also used wait- $X$ 4, with the decoder beginning only after receiving $X$ 5 source tokens and then interleaving read and produce steps (Sudoh et al., 2020). The main advantage is controllability; the main limitation is rigidity under varying speech rate and reordering demands.

Several papers replace fixed delay with learned or inferred alignment mechanisms. The V-MMA model treats monotonic alignment between source and target as a latent variable $X$ 6 and optimizes an ELBO over alignments while converting the alignment into binary READ/WRITE actions (Ma et al., 2021). StreamSpeech derives the write moment for target token $X$ 7 from two CTC heads, one aligned to source text and one aligned to target text, so that READ/WRITE decisions are integrated into the same multi-task backbone that learns ASR and translation (Zhang et al., 2024). These methods fold policy learning into model training rather than attaching an external controller.

Weakly supervised or training-free policies form another important branch. Hibiki derives per-word target delays from an external text translator by locating where the increase in conditional log-likelihood suggests that a source word provides just enough context for a target word; the resulting delays are then lifted to audio by transcription, timestamping, silence insertion, and alignment-aware TTS (Labiausse et al., 5 Feb 2025). SimulU removes policy training entirely: it uses speech-text cross-attention for history management and READ/WRITE decisions, and text-unit cross-attention for speech output selection, all on top of a pre-trained SeamlessM4T backbone (Djanibekov et al., 11 Mar 2026). Hibiki-Zero removes word-level alignments from training and instead starts from sentence-level offsets derived from punctuation, followed by reinforcement learning to optimize latency while preserving translation quality (Labiausse et al., 11 Feb 2026).

Policy control also extends beyond content timing to output pacing. The Self-Adaptive Translation proposal states that it flexibly adjusts the length of translations to accommodate different source speech rates, directly targeting latency accumulation under fast speakers and unnatural pauses under slow speakers (Zheng et al., 2020). NaturalFlow later reframes this issue as a fluency problem, optimizing inter-chunk silence rather than latency alone (Lee et al., 11 Jun 2026). A plausible implication is that simultaneous policy in speech translation is not only a boundary-detection problem but also a pacing problem over target acoustics.

4. Training paradigms and optimization objectives

Training recipes differ sharply across systems, but several recurrent patterns are evident. Multi-task supervision is prominent in direct end-to-end models. StreamSpeech jointly trains offline ASR, non-autoregressive S2TT via CTC, autoregressive streaming S2TT, and speech-to-unit translation, with a total loss that sums $X$ 8, $X$ 9, $A$ 0, and $A$ 1 (Zhang et al., 2024). NAST-S2X also uses multi-task learning, combining text and unit losses and supplementing them with CTC pre-training, bigram-based alignment loss, and two-step glancing to stabilize non-autoregressive decoding (Ma et al., 2024).

A second pattern is large-scale pretraining plus synthetic alignment construction. Hibiki trains in four phases: text pretraining, audio pretraining, speech-translation training on approximately $A$ 2k hours of synthetic French-English data with contextual delays, and fine-tuning on long-form and re-synthesized CVSS-T data (Labiausse et al., 5 Feb 2025). SimulS2S-LLM is trained offline in two stages, first for speech-to-text with boundary-aware CIF prompts and then for speech token prediction with CTC, while simultaneous behavior is induced at test time through a wait- $A$ 3 policy over prompt length (Deng et al., 22 Apr 2025). SimulTron follows the Translatotron family’s staged initialization logic, with encoder, decoder, and vocoder components initialized from ASR and TTS models before end-to-end fine-tuning (Agranovich et al., 2024).

A third pattern is reinforcement learning for quality-latency control. Hibiki-Zero uses GRPO with a BLEU-based process reward, normalized across multiple rollouts, to optimize a multistream codec-token policy without word-level alignment data (Labiausse et al., 11 Feb 2026). Seed LiveInterpret 2.0 uses PPO with dual rewards: chunk-level rewards for detection accuracy, translation initiative, translation quality, time compliance, and format consistency, plus sequence-level rewards for lagging cost and sequence quality (Cheng et al., 23 Jul 2025). These objectives explicitly treat latency as an optimization target rather than a mere evaluation statistic.

Fluency-aware fine-tuning introduces yet another objective family. NaturalFlow samples multiple candidate translations from a pretrained Hibiki model, measures BLEU and Silence Ratio, constructs “silver-medal” preference pairs from the second-lowest SR quintile under BLEU and SR margins, and optimizes a length-normalized DPO loss over the text policy (Lee et al., 11 Jun 2026). In the cascaded setting, latency reduction inside synthesis can also be optimized directly: pseudo lookahead for incremental TTS can be generated by the simultaneous speech translation decoder itself rather than by a separate LLM, and duration scaling can reduce utterance-level latency by shrinking predicted durations without changing other prosodic features (Liu et al., 2021).

5. Evaluation methodology and empirical patterns

SimulS2ST evaluation is unusually heterogeneous because the output is speech, not only text. Reported quality metrics include BLEU, ASR-BLEU, text BLEU, BLASER 2.0, COMET, xCOMET, speaker similarity from WavLM embeddings, and human MOS for quality, naturalness, fluency, and speaker likeness; reported latency metrics include Average Lagging, Length-Adaptive Average Lagging, Average Token Delay, StartOffset, EndOffset, First Letter Appearance Lag, Ear-Voice Span, speaking latency, and computation-aware Average Lagging (Sudoh et al., 2020, Zhang et al., 2024, Labiausse et al., 5 Feb 2025, Deng et al., 22 Apr 2025, Cheng et al., 23 Jul 2025, Xue et al., 13 Jun 2026). This diversity reflects the fact that simultaneous translation is judged simultaneously as translation, as audio generation, and as an interaction protocol.

Long-form evaluation has become a distinct methodological topic. The practical evaluation method of 2026 runs ASR and forced alignment on generated target speech to recover token-level timestamps, then applies a sentence-embedding-based aligner, SEGALE, to match generated target text with corresponding source sentences, enabling sentence-level YAAL-style latency and xCOMET quality aggregation (Xue et al., 13 Jun 2026). The same study reports that on $A$ 4 minute talks both Seamless and Seed LiveInterpret show steadily increasing ending-offsets over time, reaching more than $A$ 5 seconds for English-to-Japanese, making latency accumulation a measurable system property rather than an anecdotal failure mode (Xue et al., 13 Jun 2026).

Empirical results show rapid progress but no single dominant formulation. In the early fully incremental cascade, average Ear-Voice Span was approximately $A$ 6 seconds, with module delays of $A$ 7 seconds for ISR, $A$ 8 seconds for IMT, and $A$ 9 seconds for ITTS; speaking latency from playback buffering could add another $Y$ 0– $Y$ 1 seconds (Sudoh et al., 2020). SimulTron reports on MuST-C that iTTS obtains BLEU $Y$ 2 at latency $Y$ 3 seconds, while SimulTron with $Y$ 4 obtains BLEU $Y$ 5 at latency $Y$ 6 seconds, and it was deployed on a Pixel 7 Pro device (Agranovich et al., 2024). StreamSpeech reports average offline ASR-BLEU $Y$ 7 versus UnitY’s $Y$ 8 and dominates wait- $Y$ 9 cascaded baselines under low latency on CVSS-C (Zhang et al., 2024). NAST-S2S reports high-quality simultaneous interpretation within a delay of less than $S$ 0 seconds and a $S$ 1 times decoding speedup in offline generation (Ma et al., 2024).

Among multistream models, Hibiki reports on short-form CVSS-C that StreamSpeech obtains ASR-BLEU $S$ 2, EndOff $S$ 3 s, LAAL $S$ 4 s; Seamless obtains ASR-BLEU $S$ 5, speaker similarity $S$ 6, EndOff $S$ 7, LAAL $S$ 8; and Hibiki obtains ASR-BLEU $S$ 9, speaker similarity $X$ 0, EndOff $X$ 1, LAAL $X$ 2, alongside human MOS of Quality $X$ 3, Similarity $X$ 4, and Naturalness $X$ 5 (Labiausse et al., 5 Feb 2025). On long-form Audio-NTREX, Hibiki reports LAAL $X$ 6 seconds versus $X$ 7 for Seamless (Labiausse et al., 5 Feb 2025). SimulS2S-LLM reports on CVSS-C Es→En that $X$ 8 achieves ASR-BLEU $X$ 9 at ATD $k$ 0 ms, compared with approximately $k$ 1 ASR-BLEU at $k$ 2 seconds ATD for StreamSpeech under the same training-data condition (Deng et al., 22 Apr 2025). SimulU, evaluated on MuST-C v1.0 with entire TED talks as unsegmented audio streams, obtains the highest ASR-BLEU across $k$ 3 of $k$ 4 directions at comparable StartOffset in the $k$ 5– $k$ 6 second range (Djanibekov et al., 11 Mar 2026). Hibiki-Zero reports long-form gains of roughly $k$ 7 ASR-BLEU over Seamless together with lower LAAL and strong speaker-similarity and MOS numbers, while also adapting to Italian with less than $k$ 8 hours of speech (Labiausse et al., 11 Feb 2026).

Fluency and speaker preservation increasingly appear as co-equal targets. NaturalFlow reports short-form Fr→En Silence Ratio $k$ 9, LAAL $t_{in}+k$ 0, StartOff $t_{in}+k$ 1, EndOff $t_{in}+k$ 2, and ASR-BLEU $t_{in}+k$ 3, and long-form Fr→En Silence Ratio $t_{in}+k$ 4, LAAL $t_{in}+k$ 5, StartOff $t_{in}+k$ 6, EndOff $t_{in}+k$ 7, and ASR-BLEU $t_{in}+k$ 8; on high-SR CVSS-C samples, it is preferred by human raters $t_{in}+k$ 9 versus $t_{out}$ 0 for Hibiki (Lee et al., 11 Jun 2026). Seed LiveInterpret 2.0 reports that human interpreters validate more than $t_{out}$ 1 correctness in complex scenarios and that the average latency of cloned speech is reduced from nearly $t_{out}$ 2 seconds to a near-real-time $t_{out}$ 3 seconds (Cheng et al., 23 Jul 2025).

6. Long-form robustness, naturalness, and unresolved issues

Several recurrent misconceptions are corrected by the recent literature. First, SimulS2ST is not simply simultaneous speech-to-text translation plus a vocoder. Direct models were developed precisely because cascaded systems suffer from error propagation and accumulate delays in each cascade component, reducing synchronization between speaker and listener (Zhang et al., 2024, Ma et al., 2024). Second, low latency alone is not a sufficient objective. NaturalFlow argues that excessive pursuit of low latency produces fragmented chunk-wise speech with frequent pauses that increase cognitive load (Lee et al., 11 Jun 2026), and early work on self-adaptive training identified rate mismatch between speaker and system as a source of unnatural pauses and progressive lag (Zheng et al., 2020). Third, short-form success does not imply long-form robustness, as shown by explicit long-form evaluations with severe ending-offset accumulation (Xue et al., 13 Jun 2026).

The field is also revising assumptions about supervision. Word-level aligned data and trained policies were long treated as prerequisites, but SimulU presents a training-free policy for long-form SimulS2S based on model cross-attention, and Hibiki-Zero eliminates word-level alignments entirely by using sentence-level offsets plus reinforcement learning (Djanibekov et al., 11 Mar 2026, Labiausse et al., 11 Feb 2026). This suggests that alignment heuristics, policy training, and end-to-end generation can be decoupled more than earlier work assumed.

Open directions named across the surveyed papers are comparatively consistent. They include adaptive thresholding and better handling of acoustic or linguistic cues in training-free policies, study of meeting and conversational audio with overlapping speech, explicit prosody or pause detection modules, multi-speaker and code-switching scenarios, low-resource speech LLM backbones, tighter streaming vocoders, and broader on-device deployment (Agranovich et al., 2024, Labiausse et al., 5 Feb 2025, Djanibekov et al., 11 Mar 2026, Lee et al., 11 Jun 2026). The practical implication is that SimulS2ST is converging toward a joint optimization problem over translation fidelity, latency, natural speech flow, and speaker preservation rather than a single BLEU-latency frontier.