Speech Continuation: Models and Techniques

• Speech Continuation (SC) is a task that extends a spoken prompt in the same speaker’s voice while preserving semantic context and prosody.
• It encompasses various methodologies including end-to-end spectrogram-native models, discrete-token autoregressive modeling, and context-aware TTS frameworks.
• Evaluation metrics focus on semantic coherence, speaker similarity, and bitrate efficiency, highlighting tradeoffs in latency, alignment, and interruption handling.

Speech Continuation (SC) is a family of speech-conditioned generative tasks in which a model extends a spoken prompt while preserving semantic continuity and speaker-related attributes. Across recent work, the term covers several closely related formulations: continuation of a speech input in the same speaker’s voice and style with coherent content; autoregressive continuation of discrete speech-token sequences; context-aware text-to-speech continuation from a short speech prompt and a textual script; speech-conditioned continuation writing aligned with transcript-conditioned language-model behavior; and, in full-duplex assistants, revision and continuation of an in-progress spoken response after a user interruption (Nachmani et al., 2023, Maiti et al., 2023, Du et al., 2023, Wang et al., 2023, Modi et al., 8 Apr 2026, Satish et al., 26 Sep 2025). The common denominator is that SC is not merely speech synthesis or transcription: it requires a model to infer and preserve content, speaker identity, and often prosody or task state while generating the next segment of speech or a speech-conditioned continuation.

1. Conceptual scope and task boundaries

In its most direct formulation, SC is defined as continuing a speech input in the same speaker’s voice and style, with coherent semantic content. Spectron makes this explicit by taking a prompt spectrogram $x_p$ sampled from an utterance $x$, generating the continuation transcript $y_c$ and continuation spectrogram $x_c$, and finally rendering waveform audio via a vocoder (Nachmani et al., 2023). VoxtLM defines the same task in a discrete-unit regime: the model receives a prefix of speech tokens and continues the token sequence, which is later decoded to waveform (Maiti et al., 2023). UniCATS treats SC as a context-aware TTS problem in which a short prompt from a target speaker provides preceding context $A$, while the continuation content is specified by text; success depends on seamless continuation in the same voice, with consistent prosody and intelligible content (Du et al., 2023).

This family of definitions distinguishes SC from neighboring tasks. It differs from ASR because ASR outputs text only; from TTS because standard TTS does not condition on a real speech prompt’s speaker and prosody in the same way; and from cascaded ASR–LLM–TTS pipelines because such systems may lose nonverbal cues and incur latency across hand-offs (Nachmani et al., 2023). In discrete spoken language modeling, SC also differs from text continuation because conditioning and generation both occur in the speech-token domain rather than the text domain (Maiti et al., 2023).

The literature also extends SC beyond monologic continuation. EchoChain operationalizes SC for full-duplex assistants as the capability to detect a user’s mid-response interruption, revise the in-progress task state, and continue the current utterance coherently; in that setting, SC is not pausing and resuming but state-update reasoning under overlap (Modi et al., 8 Apr 2026). BLSP reinterprets SC as behavior alignment of continuation writing: given a speech segment and the same textual instruction that would be applied to its transcript, the LLM should generate the same kind of continuation text as if it had been prompted with the transcript (Wang et al., 2023). The bias-oriented study “Speak Your Mind” uses SC as a controlled probe of voice-based model bias, formalizing the task as generating a continuation $y(t)$ of an input waveform $x(t)$ while preserving semantic context, speaker identity, and salient voice-quality characteristics such as breathy, creaky, and end-creak phonation (Satish et al., 26 Sep 2025).

2. Representational choices and architectural paradigms

A central axis in SC research is the representation of speech. Spectron is spectrogram-native: it operates directly on mel-spectrograms as both input and output, avoids quantization into discrete codebooks, and uses a single autoregressive decoder that first emits text tokens and then acoustic embeddings. Its prompt encoder is a 600M-parameter Conformer, and the language-model backbone is a PaLM 2–style prefix decoder with 350M or 1B parameters (Nachmani et al., 2023). This design treats SC as a multimodal sequence problem inside one decoder, with prompt acoustics injected as a prefix and continuation acoustics reconstructed by lightweight pre/post MLPs plus a frozen WaveFit vocoder.

A second paradigm uses discrete speech units. VoxtLM merges text subwords and discrete speech tokens into a unified vocabulary $$V_{\text{voxt}} = V_{\text{txt}} \cup V_{\text{dst}}$$, routes tasks with control tokens such as $\langle\text{generate-speech}\rangle$, and performs SC by conditioning on a speech-token prefix and predicting subsequent speech tokens with a decoder-only Transformer (Maiti et al., 2023). The discrete tokens are derived from HuBERT features via k-means clustering, and subword modeling over the unified vocabulary compresses long speech-token sequences. In this setting, the model’s core inductive bias is shared autoregressive modeling across text and speech without auxiliary encoders or alignment losses.

A related but more explicitly rate-controlled discrete-unit formulation appears in recent GSLM work, where SC is realized as a speech2unit $\rightarrow$ unit LLM $x$0 unit2speech pipeline. There, HuBERT-base layer-9 features are segmented into fixed-width windows, mean-pooled, clustered by K-means, and deduplicated; the resulting unit sequence is continued by an OPT-based unit LM and resynthesized by Tacotron2+Parallel WaveGAN or VITS (Kando et al., 22 Jun 2026). This line of work emphasizes bitrate, sequence length, and the trade-off between fine phonetic detail and long-range semantic modeling.

A third paradigm separates semantic continuation from acoustic realization. UniCATS uses semantic tokens as an intermediate representation but restores acoustic detail through contextual vocoding. Its CTX-txt2vec module predicts semantic tokens from text while integrating preceding and optionally following semantic context through contextual VQ-diffusion; its CTX-vec2wav module converts semantic tokens to waveform using mel-spectrogram context from the prompt via cross-attention, without explicit speaker embeddings or acoustic codec tokens (Du et al., 2023). In this formulation, SC is a special case of context-aware generation with only preceding context $x$1 available.

BLSP occupies a distinct position. It does not generate speech outputs; instead, it aligns speech-conditioned and text-conditioned continuation behavior for a frozen LLM by training a lightweight modality adapter between a frozen Whisper-small encoder and a frozen Llama-2-7B. The formal objective is to make

$x$2

match under the same instruction, using transcript-derived continuation targets as supervision (Wang et al., 2023). In effect, BLSP treats SC as a modality-alignment problem at the level of generation behavior rather than acoustic continuation.

3. End-to-end spectrogram-native continuation

Spectron is the most explicit end-to-end formulation of spoken SC. For each utterance $x$3, the speech signal and transcript are split at time $x$4 into prompt and continuation segments:

$x$5

The mapping $x$6 between time and text tokens is not required by the loss, making the training alignment-free (Nachmani et al., 2023). The prompt spectrogram is encoded by a pre-trained Conformer encoder $x$7 and projected into the LM embedding space:

$x$8

The LM then decodes a single sequence that interleaves modalities: prompt transcription and continuation text $x$9, followed by acoustic embeddings $y_c$0 for the continuation.

The training objective combines cross-entropy for text with spectrogram reconstruction. Spectron defines

$y_c$1

and a reconstruction loss over spectrograms plus their derivatives across time and frequency:

$y_c$2

The total loss is

$y_c$3

with $y_c$4 and derivative order $y_c$5 (Nachmani et al., 2023). The paper characterizes the resulting mechanism as a “cross-modal chain-of-thought”: the LM first produces an intermediate textual scratchpad continuation and then conditions acoustic generation on that textual plan plus the prompt acoustics.

Inference follows the same ordering. A 3-second prompt spectrogram is encoded, the LM decodes text autoregressively until eos, and spectrogram frames are then generated frame by frame in LM space before being projected back and rendered to waveform by a frozen WaveFit vocoder (Nachmani et al., 2023). The system uses no cross-attention between modalities and no explicit alignment module such as monotonic attention; integration occurs through prefix conditioning and a single concatenated decoding stream.

Empirically, Spectron reports the strongest semantic quality among the listed spoken continuation baselines on LibriSpeech test-clean completions, with GPT-2-medium log-perplexity 126.08 for the 350M model versus 136.42 for SpeechGPT, 138.96 for AudioLM (3-RVQ), 140.28 for AudioLM (12-RVQ), 170.81 for TWIST (7B), and 296.99 for GSLM. Its N-MOS is $y_c$6, and speaker similarity reaches 0.42, exceeding AudioLM, SpeechGPT, TWIST, and GSLM on that metric (Nachmani et al., 2023). Ablations show that both the text-side cross-entropy and derivative-sensitive spectrogram losses are critical: removing $y_c$7 raises log-perplexity to 714.43, while removing $y_c$8 raises it to 787.89. Removing pre-training from the LM or encoder also degrades performance substantially (Nachmani et al., 2023).

4. Discrete-token continuation, rate control, and contextual TTS

In decoder-only discrete-unit SC, VoxtLM demonstrates that joint text–speech modeling can improve continuation quality relative to a single-task speech LM. Its core language-model objective is standard autoregressive likelihood over the unified vocabulary:

$y_c$9

with cross-entropy training

$x_c$0

For SC, inference uses the sequence

$x_c$1

and generates $x_c$2, পরে decoded to waveform by a HiFi-GAN-based decoder conditioned on an x-vector speaker embedding (Maiti et al., 2023). On comparable $x_c$3 settings, the multitask D3M model improves over a single-task SpeechLM from PPL 73.8, sWUGGY 62.9, sBLIMP 53.9 to PPL 58.1, sWUGGY 63.6, sBLIMP 55.2. With OPT initialization and $x_c$4, the 350M model reaches PPL 45.7, sWUGGY 65.5, sBLIMP 55.3, and scaling to 1.3B improves sWUGGY to 66.1 and sBLIMP to 56.7; in Dset+, sBLIMP reaches 57.1 (Maiti et al., 2023).

The bitrate-oriented GSLM study re-examines SC under explicit control of segmentation width $x_c$5 and cluster size $x_c$6. It defines bitrate as

$x_c$7

where $x_c$8 is the average deduplicated token rate and $x_c$9 is the empirical entropy of the unit distribution; the nominal pre-dedup token rate is $A$0 tokens/s (Kando et al., 22 Jun 2026). SC is evaluated only for speech2unit/unit2speech configurations that satisfy WER $A$1 and UTMOS $A$2 in resynthesis. The reported continuation results show that settings with larger segmentation widths perform on par with or slightly better than the conventional $A$3 ms baseline, despite lower bitrate. Best LLM-as-a-Judge scores are reported for Tacotron2 at $A$4 and VITS at $A$5, and AB tests show significant preferences for $A$6 vs $A$7 with win rate 0.617 and for $A$8 vs $A$9 with win rate 0.671, both at $y(t)$0 (Kando et al., 22 Jun 2026). The paper therefore argues that the conventional high-resolution GSLM setting is redundant for effective SC and recommends $y(t)$1–$y(t)$2 ms with $y(t)$3.

UniCATS addresses a different SC regime: continuation from a short prompt plus prescribed text. Its CTX-txt2vec objective is

$y(t)$4

and its inference rescales predicted durations by

$y(t)$5

for SC so that speaking rate matches the prompt (Du et al., 2023). Semantic-token generation is iterative, with $y(t)$6 diffusion steps, and waveform generation uses a HiFiGAN-like contextual vocoder conditioned on the prompt mel-spectrogram. On LibriTTS Test A and Test B, UniCATS reports zero-shot speaker-adaptive SC naturalness/similarity/SECS of $y(t)$7 / $y(t)$8 / 0.831 for seen speakers and $y(t)$9 / $x(t)$0 / 0.836 for unseen speakers, outperforming FastSpeech 2 and VALL-E on these tests (Du et al., 2023). This result places contextual semantic-token generation and mel-conditioned vocoding as a strong alternative to left-to-right acoustic-token continuation.

5. Speech continuation under interruption and full-duplex interaction

EchoChain broadens SC from monologic continuation to interruption-aware voice interaction. In this benchmark, SC is the ability to detect a mid-response interruption, revise task state, and continue the response so that it reflects both the pre-interruption context and the new information supplied by the user (Modi et al., 8 Apr 2026). The benchmark operationalizes this by injecting a standardized barge-in at

$x(t)$1

where $x(t)$2 is assistant speech onset. Because the interruption timing is controlled, cross-model comparison focuses on reasoning under interruption rather than on arbitrary overlap conditions.

EchoChain’s benchmark generation uses a planner–simulator loop: a Dialogue Planner produces a conversation blueprint, a User Simulator generates the next user utterance and interruption text, and a Full-Duplex Orchestrator injects the barge-in at the fixed offset. Interrupted runs are paired with half-duplex controls using the same content and rubric, allowing separation of interruption-induced failures from baseline task difficulty (Modi et al., 8 Apr 2026).

The benchmark introduces three recurrent SC failure patterns. Contextual inertia occurs when the assistant acknowledges the interruption but continues under stale assumptions. Interruption amnesia occurs when the assistant initially integrates the update but later reverts to pre-interruption assumptions. Objective displacement occurs when the assistant pivots fully to the interrupting topic and fails to finish the original request (Modi et al., 8 Apr 2026). These are explicitly diagnostic of post-interruption continuation failure, not generic dialogue errors.

Evaluation is rubric-based. Pass requires satisfying all instance-specific criteria concerning task correctness, state integrity, user intent, and the requested deliverable. EchoChain reports pass rate, Mean Pass Rate (MPR), Mean Criterion Passed percentage (MCP), and failure-rate reduction against the half-duplex control:

$x(t)$3

For 48 sampled control conversations, interruption runs had 92 failures versus 55 in half-duplex, a 40.22% reduction when interruptions were removed (Modi et al., 8 Apr 2026). Across 200 interrupted conversations evaluated on GPT-realtime-2025-08-28, Gemini Live-2.5-flash-native-audio, Grok Voice Agent, and Amazon Nova Sonic 2, no model exceeded a 50% MPR. Grok Voice Agent achieved MPR 48.5% and MCP 85.5%; GPT-realtime achieved MPR 45.0% and MCP 81.1%; Amazon Nova Sonic 2 achieved MPR 26.5% and MCP 78.1%; Gemini Live achieved MPR 16.5% and MCP 64.8% (Modi et al., 8 Apr 2026). EchoChain therefore reframes SC as a state-update reasoning problem whose dominant failure mode is revision under overlap.

6. Evaluation, bias diagnostics, and persistent limitations

SC evaluation is heterogeneous because different formulations target different outputs. Spectrogram-native systems report semantic quality through transcript log-perplexity, acoustic naturalness through N-MOS, and speaker preservation through embedding cosine similarity (Nachmani et al., 2023). Discrete-unit models use token-domain metrics such as PPL, sWUGGY, and sBLIMP (Maiti et al., 2023). Bitrate-controlled GSLM work combines PPL, VERT, LLM-as-a-Judge pairwise comparisons, and human Meaningfulness MOS, while also tracking resynthesis WER, UTMOS, MCD, and LogF0 RMSE (Kando et al., 22 Jun 2026). UniCATS uses MOS, speaker-similarity MOS, SECS, and ASR-based WER (Du et al., 2023). EchoChain evaluates continuation correctness with rubric-based pass/fail rather than prosody (Modi et al., 8 Apr 2026). This dispersion of metrics reflects a broader fact: SC spans acoustic continuation, semantic planning, stylistic preservation, and in some settings task-state revision.

The bias study “Speak Your Mind” adds a different diagnostic layer by treating SC as a controlled probe of representational bias. It evaluates SpiritLM Base and Expressive, VAE-GSLM, and SpeechGPT on speaker similarity, voice-quality preservation, and text-level bias metrics across gender and phonation conditions (Satish et al., 26 Sep 2025). SpeechGPT and SpiritLM Base show fixed-voice behavior, generating a single female-presenting voice and a single male-presenting voice, respectively, independent of the prompt. For models with variable identity, VAE-GSLM yields higher ECAPA cosine similarity than SpiritLM Expressive, with values such as 0.50 ± 0.19 for male and 0.57 ± 0.16 for female unmodified prompts, versus 0.08 ± 0.06 and 0.12 ± 0.09 for SpiritLM Expressive (Satish et al., 26 Sep 2025). The study further reports that continuations regress toward modal phonation across models and conditions, with stronger reversion for female prompts; once coherence and identity preservation are sufficiently high, gender effects emerge on text metrics such as semantic coherence, sentence polarity, and agency (Satish et al., 26 Sep 2025).

Several limitations recur across the literature. Spectron notes high time/space complexity at 12.5 ms frame rate and sequential text-then-spectrogram decoding that hinders streaming (Nachmani et al., 2023). VoxtLM does not elaborate SC-specific waveform artifacts, end-of-speech criteria, or robustness analysis, and its SC metrics remain token-centric (Maiti et al., 2023). The bitrate study finds that automatic continuation metrics correlate only weakly with human MMOS, with LLM-as-a-Judge giving the highest but still modest correlation at $x(t)$4 and PPL/VERT correlating weakly or near zero (Kando et al., 22 Jun 2026). EchoChain does not score prosodic smoothness of yielding and resuming, focusing instead on content reasoning (Modi et al., 8 Apr 2026). The bias study is limited to gender and phonation, and its text analysis remains sensitive to ASR and LLM-judge variability (Satish et al., 26 Sep 2025).

Current research therefore presents SC less as a single task than as a structured testbed for cross-modal generation. In one direction, SC is an end-to-end problem of continuing a speaker’s utterance in the same voice with coherent meaning. In another, it is a benchmark for joint speech–text modeling, bitrate-efficient spoken language modeling, or contextual vocoding. In full-duplex settings it becomes a problem of state revision under interruption. And in fairness analysis it becomes a controlled probe of how acoustic identity cues influence generated linguistic and paralinguistic content. Across these variants, the technical challenge remains the same: preserving what must persist from the prompt while correctly generating what comes next.