Speech Summarization Token (SST) Overview

• SST is a token-level mechanism that compresses spoken input into summary-friendly representations for efficient long-context reasoning.
• Explicit SST implementations, such as in Speech-XL, retain key-value states across intervals, yielding improved memory efficiency and enhanced ROUGE scores.
• Variants include prompt-based and continuous audio-token interfaces that address challenges in end-to-end speech summarization and model control.

Speech Summarization Token (SST) denotes a family of token-level mechanisms for condensing spoken input into representations that are more tractable for summarization, long-context reasoning, or downstream language modeling. In contemporary work, the term is not yet standardized: in some papers SST is an explicit special token with a defined architectural role, while in others the same function is realized through prompt schemas, continuous audio-token embeddings, or compressed latent units. The surrounding task is usually speech-to-text summarization—mapping audio to a textual summary—although the same token ideas also appear in broader speech-language modeling and long-form audio understanding (Retkowski et al., 10 Apr 2025, Sun et al., 5 Feb 2026).

1. Scope, terminology, and research setting

Speech summarization is typically framed as converting spoken content into a shorter textual form while preserving essential information. The dominant paradigm remains cascaded: speech is first transcribed by ASR and then summarized by a text summarizer. End-to-end systems instead learn $P(s \mid x_{\text{audio}})$ directly from audio to summary text, but the survey literature describes speech summarization as still loosely defined and emphasizes persistent challenges in realistic benchmarks, multilingual coverage, and long-context handling (Retkowski et al., 10 Apr 2025). In a French broadcast-news setting, a cascaded abstractive system reached ROUGE-Lsum $37.6$, whereas an end-to-end abstractive model reached ROUGE-Lsum $33.9$, with both outperforming an extractive baseline at $18.8$; this is a concrete instance of the broader gap between mature cascades and still-developing end-to-end summarizers (Monteiro et al., 2023).

The expression “Speech Summarization Token” therefore refers less to a single canonical artifact than to a recurring design objective: compress speech into controllable, summary-relevant units that can be consumed by a decoder or an LLM. A further complication is terminological collision: in some recent work, “SST” means simultaneous speech translation, not speech summarization token (Luo et al., 30 Jan 2026, Deng et al., 22 Apr 2025).

Interpretation	Mechanism	Representative source
Explicit architectural SST	Special token inserted into speech intervals; only its KV pairs are retained	“Speech-XL” (Sun et al., 5 Feb 2026)
Prompt-based SST-like control	Instruction schema defines what speech to summarize and how	Child stuttering study (Okocha et al., 21 Oct 2025)
Continuous audio-token interface	Audio embeddings projected into LLM token space for prompted summarization	Audio-prompted summarization (Kang et al., 2024)
Compressed long-range summary token	Span-level token $W_k$ replaces long-range speech-token context	Compressed-to-fine LM (Liu et al., 30 May 2025)
Acronym “SST” as another task	Streaming translation of partial speech input	RASST (Luo et al., 30 Jan 2026), SimulS2S-LLM (Deng et al., 22 Apr 2025)

2. Explicit architectural SSTs

The clearest explicit formulation appears in “Speech-XL,” which introduces a novel special token, the Speech Summarization Token (SST), for each speech interval to encapsulate the intra-interval speech information into its associated KV pairs (Sun et al., 5 Feb 2026). Speech tokens are partitioned into fixed-length intervals with $|I_i| = w_i = 512$. For interval $i$, a compression ratio $\alpha_i$ determines the number of SSTs, $k_i = \frac{w_i}{\alpha_i}$, so there is one SST every $\alpha_i$ speech tokens. During interval encoding, SSTs attend over local speech tokens and accumulate their information into the SST hidden states and therefore into the SST key-value cache.

The defining operation is not merely token insertion but KV persistence. After an interval is processed, KV states of the original speech tokens are discarded, while the KV states of the SSTs are retained. Subsequent intervals are then encoded while conditioning on the accumulated SST KVs from previous intervals rather than the full historical speech-token KV set. The model is trained autoregressively on downstream text outputs with conditioning of the form

$37.6$0

and the loss is standard negative log-likelihood over target text (Sun et al., 5 Feb 2026).

Training uses three stages: ASR for semantic alignment, Q&A/captioning for modal alignment, and a compression stage in which the SST mechanism is activated. The compression stage uses curriculum learning: early training uses low compression ratios $37.6$1, and later training includes $37.6$2. This directly couples SST learning to progressively harder information-condensation regimes (Sun et al., 5 Feb 2026).

Empirically, this explicit SST formulation is tied to long-form efficiency. For durations from 1 to 10 minutes, Speech-XL shows smoother growth in memory and TFLOPs than a non-compressed baseline; at 10 minutes, its TFLOPs are approximately 60% of Qwen2.5-Omni-7B. On LongSpeech, the model reports summary scores of ROUGE-1 $37.6$3, ROUGE-2 $37.6$4, and ROUGE-L $37.6$5, while ASR WER is $37.6$6, close to its stated upper bound of $37.6$7 (Sun et al., 5 Feb 2026). A plausible implication is that an SST can function as a learned memory slot: each retained token is less a lexical symbol than a durable summary proxy for a contiguous region of speech.

3. Prompt-conditioned and embedding-based SST-like interfaces

Other work realizes SST-like behavior without introducing a new architectural token. In child stuttering speech, the paper “Can large audio LLMs understand child stuttering speech? speech summarization, and source separation” explicitly states that it does not introduce an explicit “Speech Summarization Token (SST)” in the sense of a new architectural token, but it effectively implements SST-like control through a textual instruction schema that specifies target speaker, required disfluency preservation, and speaker purity constraints (Okocha et al., 21 Oct 2025). The two core tasks are single-channel source separation and child-only summarization. In the text-first pipeline, Whisper Large produces transcripts with timestamps, pyannote.audio provides diarization, child turns are selected, and Llama-3.2 summarizes the child-only transcript. In the audio-first pipeline, a LALM consumes waveform plus prompt directly (Okocha et al., 21 Oct 2025).

Within that setting, the prompt acts as a functional SST. The system role “You are a clinical Speech–Language Pathologist,” the requirement to produce a faithful, objective 3-sentence summary, and the instruction to be specific to the child’s speech jointly define summary scope and style. On the interview task, Audio Flamingo-3 achieved purity $37.6$8 and faithfulness $37.6$9, while Kimi reached purity $33.9$0 and faithfulness $33.9$1. At the same time, the study reports that models continue to struggle with maintaining natural disfluencies and avoiding small factual drifts, which shows that a basic summarization prompt tends to normalize disfluent speech unless disfluency preservation is stated explicitly (Okocha et al., 21 Oct 2025).

A second SST-like interface appears in “Prompting LLMs with Audio for General-Purpose Speech Summarization.” There, speech is transformed into a sequence of continuous audio token embeddings rather than discrete symbols. HuBERT-Large produces 1024-dimensional frame-level features at 50 Hz; average pooling with kernel size 8 and stride 4 reduces this to 12.5 Hz, and a linear layer projects features to the 3072-dimensional embedding space of MiniChat-3B (Kang et al., 2024). These audio tokens are concatenated with textual instruction tokens, so the LLM receives a unified prompt sequence in its native embedding space.

The system is trained to produce consistent responses to semantically matched text and audio prompts. Its losses include next-token prediction,

$33.9$2

logit distillation,

$33.9$3

and feature distillation across selected LLM layers,

$33.9$4

Here the SST role is played by a modality-aligned audio-token sequence: not a discrete code, but a speech-derived prompt interface that allows the same summarization instructions used for text to operate on audio (Kang et al., 2024).

A related end-to-end broadcast-news model uses Wav2Vec2 layer-7 features as speech-derived latent tokens, maps them through a CNN–BiLSTM–attention cross-modal adapter into textual embeddings, and then decodes summaries with a transferred text summarizer. The paper does not call these units SSTs, but its adapter output functions as speech-derived summarization-oriented tokens. The resulting end-to-end system is still weaker than the cascade, which suggests that learning summary-ready speech latents remains difficult when paired speech-summary data are limited (Monteiro et al., 2023).

4. Discrete tokenization, compression, and LM alignment

The broader speech-tokenization literature distinguishes acoustic tokens from semantic tokens. Acoustic tokens prioritize waveform fidelity, prosody, and timbre; semantic tokens prioritize content abstraction and compatibility with language modeling (Guo et al., 10 Feb 2025). For SST design, the relevant trend is toward shorter, semantically focused token streams that retain what an LM needs for text-like reasoning.

“LM-SPT” is a direct example of this shift. It introduces an LM-aligned speech tokenizer with separate semantic and acoustic paths, and its semantic supervision is based on a frozen Whisper encoder. Semantic tokens are learned so that waveform reconstructed only from semantic tokens yields Whisper encoder features close to those of the original speech:

$33.9$5

This replaces rigid feature-level pooling with reconstruction-driven semantic distillation. The tokenizer supports frame rates of 25 Hz, 12.5 Hz, and 6.25 Hz, and the overall generator objective combines time-domain reconstruction, multi-scale Mel reconstruction, adversarial loss, feature matching, commitment loss, and the semantic distillation term (Jo et al., 20 Jun 2025).

The resulting tokens are explicitly described as compressed, LM-aligned summaries of speech content. At 12.5 Hz, LM-SPT reports reconstruction WER $33.9$6, compared with $33.9$7 for SpeechTokenizer and $33.9$8 for Mimi; in 12.5 Hz TTS with a 3B LM, LM-SPT reports TTS WER $33.9$9, compared with $18.8$0 for SpeechTokenizer and $18.8$1 for Mimi (Jo et al., 20 Jun 2025). This suggests that a practical SST can be understood as a semantic token stream whose bottleneck is shaped by ASR-visible content rather than by raw acoustic fidelity.

A complementary formulation appears in “Speech Token Prediction via Compressed-to-fine Language Modeling for Speech Generation.” There, long-range context is replaced by learned compact tokens $18.8$2, each summarizing a span of $18.8$3 speech tokens. The model keeps prompt and local context at full resolution but replaces older spans with $18.8$4 summary tokens. For EnCodec at 50 Hz, the best setting is $18.8$5, corresponding to a compression rate of 5 compact tokens per second, and it yields WER $18.8$6 (Liu et al., 30 May 2025). Although developed for speech generation, these $18.8$7 tokens are explicitly interpreted as summary representations of long speech spans and therefore constitute another SST-like mechanism.

STAB adds a benchmark perspective. It evaluates speech tokenizers using invariance, robustness, compressibility, and vocabulary-usage metrics without downstream retraining. Among 32k-tokenizer models, USM-v2 reports speaker invariance chrF $18.8$8, Gaussian-noise robustness chrF $18.8$9, BPE efficiency $W_k$0, and overall vocabulary utilization $W_k$1 (Vashishth et al., 2024). The paper correlates these properties with downstream ASR, AST, emotion classification, speaker identification, and language identification. A plausible implication is that SSTs intended for summarization should favor the same profile as semantically strong tokenizers: high robustness, good compressibility, and high vocabulary utilization.

5. Structured and streaming relatives of SST

Special tokens that structure speech streams provide an important precedent. “Separator-Transducer-Segmenter” introduces start-of-turn <sot> and end-of-turn <eot> tokens as part of a streaming multi-party ASR vocabulary of size 2503. These tokens are trained jointly with lexical tokens under an RNN-T objective and are analyzed via emission-latency metrics such as start-pointing and end-pointing emission latency. The best model reports 4.6% absolute turn counting accuracy improvement and 17% relative WER improvement on LibriCSS (Sklyar et al., 2022). The paper explicitly argues that summary-oriented tokens such as <sum> or <topic> are a natural next step beyond <sot> and <eot>. In that sense, SSTs can be seen as inheritors of a broader idea: speech streams can be marked with learned structural tokens that delimit, summarize, or prioritize content.

The acronym collision with simultaneous speech translation is also instructive. In RASST, SST always means Simultaneous Speech Translation, and the system integrates chunkwise terminology hints into a speech LLM through a term_map: prompt structure. The model improves terminology translation accuracy by up to 16% and BLEU by up to 3 points (Luo et al., 30 Jan 2026). In SimulS2S-LLM, simultaneous inference is enabled by CIF-based boundary-aware speech prompts and discrete output speech tokens; the paper reports ASR-BLEU gains of 3 points at similar latency (Deng et al., 22 Apr 2025). These systems do not define speech summarization tokens, but they demonstrate that chunk boundaries, prompt headers, and streaming control symbols can act as token-level control surfaces over partial speech input.

This contrast matters because it prevents a common misconception: not every recent use of “SST” in speech research refers to summarization. Yet the simultaneous-translation literature still provides adjacent ideas—boundary-aware prompt units, chunk-level control, and retrieval-conditioned token injection—that are directly reusable in SST design.

6. Evaluation, limitations, and future directions

Evaluation of SST-based systems inherits the broader measurement difficulties of speech summarization. The survey literature notes continued dependence on ROUGE and other text-derived metrics, even though speech adds ASR errors, disfluencies, overlapping speakers, and prosodic structure (Retkowski et al., 10 Apr 2025). In child stuttering summarization, evaluation combines ROUGE-style overlap measures, BERTScore F1,

$W_k$2

human expert ratings, and LLM-as-a-judge criteria such as faithfulness, coverage, fluency, speaker purity, and usefulness. Scores are bootstrapped with 1,000 resamples to obtain 95% confidence intervals, and agreement is quantified using Pearson correlation and Cohen’s $W_k$3; for interview summaries, Llama-3.2 reaches Pearson $W_k$4 against human ratings, while inter-judge correlations across LLMs are reported around $W_k$5–$W_k$6 with within-1 agreement above $W_k$7 for many pairs (Okocha et al., 21 Oct 2025). This indicates that token-level control dimensions such as speaker purity are measurable, but not fully captured by lexical metrics.

Several limitations recur across the literature. End-to-end speech summarization remains data-hungry and often underperforms stronger cascades in factuality and relevance (Monteiro et al., 2023). Prompt-based SST-like systems can suppress speaker leakage yet still smooth away clinically relevant disfluencies (Okocha et al., 21 Oct 2025). Explicit compression-based SSTs preserve high-level semantics but show degradation at very high compression ratios such as $W_k$8, especially for ASR-like tasks, and current long-form models remain weak on non-speech audio (Sun et al., 5 Feb 2026). Tokenizer studies further suggest that overemphasizing reconstruction fidelity can reduce semantic robustness, which is undesirable for summarization-oriented tokenization (Vashishth et al., 2024, Jo et al., 20 Jun 2025).

Future directions in the cited work converge on more explicit and more controllable token interfaces. Suggested paths include more sophisticated multi-speaker models that represent speaker streams internally, specialized representations for disfluencies, dedicated control tokens such as <SST_CHILD> or <SST_DISFLUENCY>, and evaluation protocols that directly track leakage, speaker purity, and disfluency preservation rather than relying primarily on ROUGE or BERTScore (Okocha et al., 21 Oct 2025). The survey literature similarly points to realistic evaluation benchmarks, multilingual resources, and stronger long-context handling as unresolved needs (Retkowski et al., 10 Apr 2025). Taken together, these threads suggest that SST will likely evolve not as a single token type but as a layered abstraction: explicit memory tokens for long audio, prompt-level control tokens for summary style and scope, and semantically distilled speech tokens that make speech more text-like without erasing speech-specific structure.