Speech-XL: End-to-End Long-Form Speech Understanding

• Speech-XL is an end-to-end framework for long-form speech understanding that leverages learned Speech Summarization Tokens to compress redundant KV states.
• It partitions long audio into fixed intervals and interleaves SSTs to preserve essential semantic and paralinguistic features while reducing memory and computational load.
• The architecture integrates a Whisper-based encoder, dual adapters, and an LLM backbone, employing curriculum learning to stabilize training through incremental compression ratios.

Speech-XL is an end-to-end framework for long-form speech understanding in Large Speech LLMs (LSLMs). It is designed for the regime in which standard speech-conditioned LLMs work well on short clips but become bottlenecked on multi-minute audio by limited context length, quadratic self-attention cost, and linearly growing KV cache. Its central mechanism is to partition speech into fixed intervals, interleave each interval with learned Speech Summarization Tokens (SSTs), and retain only the SST-associated KV states as compressed memory for subsequent intervals. In the formulation reported by the authors, this is a learned KV-space compression scheme rather than a transcript-first cascade, and it is evaluated on LongSpeech and AudioMarathon (Sun et al., 5 Feb 2026).

1. Problem formulation and scope

The problem addressed by Speech-XL is long-form audio understanding in LSLMs. The paper argues that speech is substantially longer than text carrying the same meaning because acoustic features are extracted at high temporal resolution and contain substantial local redundancy. This yields two bottlenecks. First, standard LSLMs inherit the finite context window of the underlying LLM, so long audio quickly exceeds what can be processed directly. Second, each token contributes keys and values to the KV cache, so both inference latency and memory become prohibitive as audio duration increases. The paper states that most current LSLMs are effectively limited to clips around 30 seconds or shorter (Sun et al., 5 Feb 2026).

Speech-XL is presented as an alternative to a cascaded speech-to-text-to-text-LLM pipeline. The authors explicitly reject simple transcription-first processing as a complete solution because text discards non-textual cues, including emotion, prosody, speaker/environmental characteristics, and other paralinguistic information. The objective is therefore not merely long-context ASR, but long-form speech understanding that remains end-to-end and preserves richer acoustic evidence. This distinguishes Speech-XL from systems whose long-context handling is defined primarily by chunkwise transcription.

A common misunderstanding is to treat the model as a generic chunking strategy. The paper’s own framing is narrower: the target is the specific problem of dense speech token streams producing redundant KV entries in an LLM, and the proposed remedy is to compress speech directly in the LLM’s KV space rather than by naive truncation, pooling, or token dropping. This suggests that Speech-XL should be understood as a memory-compression mechanism embedded in the LLM inference process, not merely as a segmentation heuristic.

2. SSTs and interval-wise KV compression

The core abstraction of Speech-XL is the Speech Summarization Token. The SST is a special token inserted into each speech interval to summarize local acoustic and semantic information into its associated KV states across LLM layers. The paper states that the SST “prompts the LLM to compress the local acoustic and semantic information into the SST’s KV states at every layer” (Sun et al., 5 Feb 2026).

The speech token sequence is written as

$X = \{x_1, \ldots, x_n\}.$

It is partitioned into intervals

$$[x_1, \ldots, x_n] \xrightarrow{\text{Partition}} [I_1, \ldots, I_i],$$

with interval lengths satisfying

$\sum_i w_i = n, \quad |I_i| = w_i.$

The implementation detail reported in the paper is that the length of each interval is 512. For interval $I_i$, the number of inserted SSTs is determined by a compression ratio $a_i$:

$k_i = \frac{w_i}{a_i}.$

The SST set is

$S_i = \{s_{i,1}, \ldots, s_{i,k_i}\},$

and the interval is augmented by interleaving SSTs:

$I_i \xrightarrow{\text{Interleave } S_i} I_i'.$

The paper’s sequence notation is typographically compressed, but its intended construction is explicit: one SST is appended for every $a_i$ speech tokens, uniformly through the interval.

Operationally, the mechanism is interval-local during encoding and persistent across intervals in memory. For each interval, the model takes the speech tokens, interleaves SSTs, feeds the augmented interval into the LLM, lets the SSTs attend to surrounding local speech content, and then preserves only the SST KV states while discarding the original speech-token KV states for that interval. When the next interval is processed, the retained SST KVs from previous intervals act as a compressed proxy memory for the earlier speech.

The paper is explicit that this compression does not produce a textual summary. The retained representation remains latent and internal to the LLM’s attention memory. The nominal retained memory per interval is reduced from $w_i$ speech-token KV entries to $$[x_1, \ldots, x_n] \xrightarrow{\text{Partition}} [I_1, \ldots, I_i],$$0 SST KV entries, yielding a compression factor of about $$[x_1, \ldots, x_n] \xrightarrow{\text{Partition}} [I_1, \ldots, I_i],$$1. The authors present this as evidence that long-form speech can be condensed by retaining only a smaller set of learned memory slots rather than all original acoustic tokens (Sun et al., 5 Feb 2026).

3. Model architecture and long-audio inference

Speech-XL contains three principal components: a speech encoder, a dual-adapter bridge, and an LLM backbone with SST compression. The speech front-end is Whisper-small. The semantic adapter is a 2-layer convolutional network with subsampling rate 4 followed by a 2-layer linear projection. The acoustic adapter is a 2-layer Q-former with 64 trainable query vectors. The backbone LLM is Qwen2.5-7B-Instruct (Sun et al., 5 Feb 2026).

The semantic and acoustic adapters serve distinct roles. The semantic adapter aligns speech representations with the LLM’s linguistic semantic space. The acoustic adapter extracts acoustic-specific information from intermediate speech encoder states, especially non-semantic cues such as tone and environment. This division is important because Speech-XL is not described as a purely semantic compression method; its stated purpose is to preserve both semantic and paralinguistic information.

The inference pipeline is interval-wise. Speech is first encoded by Whisper-small and projected into the LLM space through the two adapters. The resulting speech-token sequence is partitioned into intervals of length 512. SSTs are then interleaved into each interval according to the chosen compression ratio. The LLM processes interval $$[x_1, \ldots, x_n] \xrightarrow{\text{Partition}} [I_1, \ldots, I_i],$$2 while conditioning on the retained SST KVs from prior intervals. After interval $$[x_1, \ldots, x_n] \xrightarrow{\text{Partition}} [I_1, \ldots, I_i],$$3 is encoded, the model discards the original acoustic and semantic token KVs for that interval and keeps only the SST KVs. After all intervals have been processed, the model answers the prompt using the accumulated SST memories rather than the full speech-token history.

For generation, the paper writes the conditional distribution as

$$[x_1, \ldots, x_n] \xrightarrow{\text{Partition}} [I_1, \ldots, I_i],$$4

where $$[x_1, \ldots, x_n] \xrightarrow{\text{Partition}} [I_1, \ldots, I_i],$$5 are prompt tokens, $$[x_1, \ldots, x_n] \xrightarrow{\text{Partition}} [I_1, \ldots, I_i],$$6 are SSTs retained from prior intervals, $$[x_1, \ldots, x_n] \xrightarrow{\text{Partition}} [I_1, \ldots, I_i],$$7 are previously generated output tokens, and $$[x_1, \ldots, x_n] \xrightarrow{\text{Partition}} [I_1, \ldots, I_i],$$8 is the parameter set. This makes the role of SSTs precise: they are the compressed speech memory through which later response generation is conditioned (Sun et al., 5 Feb 2026).

A frequent misconception is that Speech-XL is only a variant of pooling or pruning. The paper argues otherwise in direct comparative terms: compared with prior long-context speech/audio methods, the novelty is not merely chunking or pooling, but inserting learnable summary tokens, keeping only their KV states, and recursively carrying compressed memory across intervals.

4. Training methodology and curriculum learning

Speech-XL is trained in three stages. The first stage is ASR semantic alignment. It uses a subset of Emilia, about 10,000 hours of English speech, with transcription-style prompts such as “Please transcribe the audio.” In this stage, the speech encoder is frozen and only the semantic adapter is trained. The goal is to establish the speech-to-language bridge (Sun et al., 5 Feb 2026).

The second stage is Q&A or modal alignment. It uses a mixture of five datasets totaling 127 hours: VCTK-Corpus, AccentDB, IEMOCAP, DailyTalk, and VoxCeleb1. Here the semantic adapter and acoustic adapter are jointly trained. The stated aim is to enrich the model with paralinguistic and environmental understanding through speech-based question answering or speech captioning style supervision.

The third stage is SST compression training on the training split of LongSpeech. In this stage, the speech encoder, dual-adapter bridge, and the LLM with SST compression are jointly optimized end-to-end. The loss is standard autoregressive negative log-likelihood:

$$[x_1, \ldots, x_n] \xrightarrow{\text{Partition}} [I_1, \ldots, I_i],$$9

The paper emphasizes that no special auxiliary compression loss is introduced; SST behavior is learned through task supervision.

A major reported contribution is curriculum learning for SST compression. Training begins with low compression ratios such as 2x and 4x, then expands to include 8x. The authors state that training SST directly under very high compression is unstable because too much information must be preserved in too few tokens, which harms recognition and reasoning. The low-to-high schedule is said to stabilize training, reduce risk of gradient collapse early in training, and later enable ultra-long audio handling under higher compression. Reported implementation details include 8 × NVIDIA A800-80GB GPUs, per-GPU batch sizes of 4 in stage 1, 2 in stage 2, and 1 in stage 3, with learning rates of $\sum_i w_i = n, \quad |I_i| = w_i.$0, $\sum_i w_i = n, \quad |I_i| = w_i.$1, and $\sum_i w_i = n, \quad |I_i| = w_i.$2 respectively (Sun et al., 5 Feb 2026).

5. Benchmarks, empirical results, and efficiency

Speech-XL is evaluated on LongSpeech and AudioMarathon. LongSpeech contains audio durations up to 10 minutes and includes long-form summarization, key information retrieval, long-range reasoning, content separation, emotion analysis, speaker count, temporal issue localization, and speech recognition. AudioMarathon covers clips from 1 to 5 minutes and groups 10 subtasks into Speech Content Extraction (SCE), Audio Classification (AC), and Speaker Information Modeling (SIM) (Sun et al., 5 Feb 2026).

On LongSpeech understanding tasks, Speech-XL reports the following values: Content Separation, 72.84 for N.A and 72.84 for Pr; Emotion, 51.94 for St.A and 66.98 for R.A; Speaker Count, 84.72 for N.A and 84.72 for Pr; Summary, 49.72 for R1, 15.07 for R2, and 26.76 for RL; and Temporal Localization, 16.09 for St.A and 32.96 for R.A. On LongSpeech ASR, the paper reports WERs of 17.0 for Whisper, 51.2 for Kimi-audio, 34.5 for AudioFlamingo3, 27.5 for Voxtral, 35.5 for DashengLM, 29.9 for Qwen2-Audio, 11.4 for Speech-XL, and 11.0 for the uncompressed Upper-bound. The 0.4 WER gap between Speech-XL and the Upper-bound is presented as evidence that SST compression induces little information loss (Sun et al., 5 Feb 2026).

On AudioMarathon, Speech-XL reports SER 31.1, SCR 75.6, ASR 96.1, SD 33.3, ER 30.3, SAR 26.6, SGR 77.8, SED 54.7, MC 53.4, ASC 40.4, and Overall 48.9. The Upper-bound overall score is 51.6, and Qwen2.5-Omni-7B reports 70.5 overall. The authors explicitly attribute the overall gap not to compression, but to foundation-model pretraining scale: stage 1 uses about 10k hours, whereas top systems are said to be trained on hundreds of thousands or millions of hours. On the speech-centered SCE aggregate, Speech-XL scores 67.6 and ranks 3rd; the gap to the Upper-bound is only 1.2%. On SIM, the gap to the Upper-bound is 1.4%, despite only 127 hours of stage-2 acoustic and speaker-oriented data.

The ablation evidence is central to the paper’s claim. At a unified 4x compression ratio on AudioMarathon, average pooling scores 62.8 on SCE, 39.4 on SIM, 34.3 on AC, and 45.3 overall; max pooling scores 53.6, 40.6, 34.0, and 42.5; similarity-based pruning scores 62.3, 40.8, 35.2, and 45.9; Speech-XL scores 67.4, 41.3, 38.4, and 48.3; and the Upper-bound scores 68.8, 43.4, 43.1, and 51.6. The paper interprets this as strong support for SST-based compression over naive pooling or pruning.

The compression-ratio analysis covers 2x, 4x, 8x, and 16x. The reported pattern is that higher compression improves efficiency, but ASR is especially sensitive at very high ratios; direct extrapolation to 16x causes a large drop in ASR, while non-ASR long-form understanding is more robust. Efficiency analysis further reports that as audio duration grows from 1 minute to 10 minutes, Speech-XL’s GPU memory and TFLOPs grow more smoothly than Qwen2.5-Omni-7B’s, and for 10-minute inference its TFLOPs are approximately 60% of the competitor’s. Memory consumption is also described as significantly lower (Sun et al., 5 Feb 2026).

6. Relation to other “XL” speech models and stated limitations

Speech-XL should not be conflated with several adjacent systems that use similar naming conventions but address different problems. “SpeechLLM-XL” is a streaming, decoder-only LLM-based ASR model that processes audio in fixed-size chunks, emits a chunk-local EOS, restricts attention to a bounded left context, and reports 2.7%/6.7% WER on LibriSpeech test-clean/test-other with 1.28 second chunks; its emphasis is streaming transcription and linear or near-linear inference scaling in stream length, not SST-based KV compression for long-form speech understanding (Jia et al., 2024). “Speech-XLNet” is an unsupervised acoustic model pretraining method for self-attention networks that adapts XLNet-style permutation-based autoregressive training to continuous speech frames and improves SAN/HMM and end-to-end Transformer ASR on TIMIT and WSJ, but it is a pretraining objective rather than a long-form LSLM memory architecture (Song et al., 2019). The large-scale streaming Transformer study built around Transformer-XL uses chunkwise recurrence over acoustic frames under an 800 millisecond latency constraint and is presented as comparable to LC-BLSTM in streaming ASR; again, its target is low-latency recognition rather than end-to-end multi-minute speech understanding (Lu et al., 2020). “SSA-HuBERT-XL” denotes a 964M-parameter self-supervised speech encoder trained solely on African speech; it is an encoder-scaling study for ASR and LID rather than a long-context speech-LLM (Caubrière et al., 28 Nov 2025).

This terminological overlap matters because “XL” can denote at least four different ideas in the cited literature: extra-long streaming ASR, XLNet-style permutation pretraining, Transformer-XL recurrence, or an XL-size self-supervised encoder. Speech-XL, in the strict sense used by its authors, refers to the SST-based KV-compression framework for long-form speech understanding in LSLMs. This suggests that the model’s novelty lies less in parameter scale than in its compression mechanism.

The paper also makes several limitations explicit or strongly implied. Training-data scale is much smaller than that of top baselines. Coverage is speech-centric rather than broad audio-centric, which hurts Audio Classification on AudioMarathon. Very high compression can hurt ASR, especially at 16x. The LongSpeech comparison is not fully symmetric because Speech-XL is fine-tuned on LongSpeech while many baselines are evaluated zero-shot. Finally, the paper does not provide an explicit Transformer attention formula for SST-specific attention, a formal KV-cache complexity equation, a closed-form memory-saving derivation beyond the interval/SST count relation, or detailed layer-by-layer pseudocode (Sun et al., 5 Feb 2026).

In that sense, Speech-XL is best understood as a specific long-form LSLM architecture whose principal claim is that the LLM’s KV cache can itself be turned into a compressed memory for speech. The strongest evidence provided is empirical rather than analytic: compressed Speech-XL remains close to its uncompressed upper-bound on LongSpeech and on speech-centered subsets of AudioMarathon while using substantially lower long-form inference cost.