FastLongSpeech: Efficient Long-Speech Framework

• The paper introduces a novel method to compress long acoustic sequences before processing, reducing computational load and addressing the scarcity of long-speech training data.
• It employs a speech extractor that uses content density and adjacent-frame similarity metrics to iteratively fuse redundant frames while preserving key speech information.
• Dynamic Compression Training exposes the model to various compression ratios, enhancing long-speech QA performance and efficiency compared to traditional methods.

Searching arXiv for the named paper and closely related long-speech work to support the article. FastLongSpeech is a framework for extending Large Speech-LLMs (LSLMs) to efficient long-speech processing without dedicated long-speech training data. It augments an off-the-shelf LSLM, specifically Qwen2‑Audio‑7B‑Instruct, with a speech extractor that compresses long frame sequences before they reach the LLM, and it couples this with a Dynamic Compression Training regime that exposes the model to varying compression ratios on short-speech data. The framework is motivated by two bottlenecks: the scarcity of long-speech instruction datasets and the high computational cost induced by long acoustic sequences, especially when speech representations are much longer than text for the same content (Guo et al., 20 Jul 2025).

1. Problem formulation and target regime

FastLongSpeech adopts the standard LSLM formulation in which an audio encoder converts raw waveform $\mathbf{s}$ into frame-level representations $\mathbf{h} = (h_1,\dots,h_J)$, and an LLM generates text $\mathbf{y}$ conditioned on an instruction $\mathbf{x}$ and those speech features: $$p(\mathbf{y} \mid \mathbf{x}, \mathbf{h}) = \sum_{i=1}^{I} p(y_i \mid \mathbf{y}_{<i}, \mathbf{x}, \mathbf{h}).$$ Within this setup, the base model is Qwen2‑Audio‑7B‑Instruct, whose audio encoder produces 25 Hz frame-level representations from waveform input. The difficulty is that speech representations are much longer than text token sequences for the same semantic content, and the original speech window of Qwen2‑Audio is 750 frames. At 25 Hz, 10 minutes of speech becomes about 15,000 frames, so the speech side rather than the text side becomes the operative long-context bottleneck (Guo et al., 20 Jul 2025).

The framework is positioned against two limitations of existing LSLMs. First, most end-to-end speech-LLMs are trained on clips shorter than about 30 seconds, because diverse long-speech instruction corpora are scarce. Second, naïve long-context handling either extends positional embeddings or uses chunking and sliding-window heuristics, both of which remain computationally expensive or lose global context. FastLongSpeech therefore treats long-speech processing primarily as a representation-compression problem performed before the LLM, rather than as a pure context-extension problem inside the LLM.

2. Speech extractor and iterative fusion

FastLongSpeech inserts a speech extractor between the audio encoder and the adaptor/LLM. The overall pipeline is waveform $\rightarrow$ audio encoder $\rightarrow$ speech extractor $\rightarrow$ adaptor $\rightarrow$ LLM. For long inputs, audio longer than 30 seconds is first partitioned into 30-second clips; the encoder outputs for those clips are concatenated in temporal order into a long frame sequence $\mathbf{h}$, and the speech extractor compresses that sequence to a target length $\mathbf{h} = (h_1,\dots,h_J)$0.

Component	Input/output	Function
Audio encoder	$\mathbf{h} = (h_1,\dots,h_J)$1	Produces 25 Hz frame-level speech representations
CTC decoder	$\mathbf{h} = (h_1,\dots,h_J)$2	Supplies per-frame non-blank probabilities
Iterative fusion	$\mathbf{h} = (h_1,\dots,h_J)$3	Compresses long sequences to target length $\mathbf{h} = (h_1,\dots,h_J)$4
Adaptor + LLM	$\mathbf{h} = (h_1,\dots,h_J)$5	Generates text conditioned on compressed speech

The compression rule combines two signals. The first is content density, defined from a CTC decoder as the non-blank mass at each frame: $\mathbf{h} = (h_1,\dots,h_J)$6 The second is adjacent-frame similarity, computed by cosine similarity: $\mathbf{h} = (h_1,\dots,h_J)$7 Frames with high similarity are treated as redundant, while frames with high $\mathbf{h} = (h_1,\dots,h_J)$8 are treated as textually dense.

Compression proceeds iteratively. If $\mathbf{h} = (h_1,\dots,h_J)$9 is the sequence length at iteration $\mathbf{y}$0, the next length is chosen by

$\mathbf{y}$1

The algorithm selects the top $\mathbf{y}$2 adjacent pairs with the highest similarity, merges consecutive selected pairs into spans, and replaces each span by a weighted average that emphasizes frames with larger content density: $\mathbf{y}$3 This yields a compressed sequence $\mathbf{y}$4 of length $\mathbf{y}$5. Because similarities are computed only between adjacent frames and the schedule reduces length roughly by half until reaching $\mathbf{y}$6, the fusion stage is modest relative to the $\mathbf{y}$7 cost that full-length LLM attention would otherwise incur (Guo et al., 20 Jul 2025).

3. Dynamic Compression Training

FastLongSpeech couples the compression mechanism with a training regime that makes the LLM robust to multiple compression ratios. The core objective is

$\mathbf{y}$8

where $\mathbf{y}$9, and $\mathbf{x}$0 denotes iterative fusion to target length $\mathbf{x}$1. In practice, one target length is sampled uniformly per training instance. This exposes the LLM to compressed speech representations ranging from minimal compression to very aggressive compression, while training remains entirely on short-speech data.

Training is organized in two stages. Stage 1 trains the CTC decoder on ASR using 960 hours of LibriSpeech and 3,000 hours of English MLS while freezing the audio encoder. The decoder is a one-hidden-layer feed-forward network with hidden dimension 4096 and output dimension 10,000, and it is optimized with standard CTC loss. Stage 2 fine-tunes the LLM with LoRA on spoken QA data under 30 seconds: OpenASQA (5.9k hours), LibriSQA (360 hours), and Common Voice English converted into QA format (1.7k hours). During this stage, the audio encoder, adaptor, and trained CTC decoder are frozen; only the LLM attention projections are adapted via LoRA with rank $\mathbf{x}$2, $\mathbf{x}$3, dropout $\mathbf{x}$4, and target modules $\mathbf{x}$5 (Guo et al., 20 Jul 2025).

This design implies that long-speech capability is not learned from explicit long-speech supervision. Rather, the model learns to interpret a family of compressed speech representations on short clips and then reuses the same representation geometry at inference time on true long-form inputs. A plausible implication is that FastLongSpeech transfers task competence through invariance to compression rather than through direct exposure to long-duration discourse.

4. Evaluation, efficiency, and task-dependent behavior

To assess long-speech understanding, the framework introduces LongSpeech‑Eval, a spoken QA benchmark derived from MultiFieldQA‑En and NarrativeQA subsets of LongBench. Documents are filtered for TTS suitability, rewritten into spoken style, validated for answerability, and synthesized with Orca TTS. The final benchmark contains 164 samples, with average speech duration 132.77 seconds and maximum duration 1000 seconds. Evaluation uses Llama3.1‑70B‑Instruct as a judge that assigns scores from 1 to 5 based on the question, model answer, and reference answer (Guo et al., 20 Jul 2025).

On LongSpeech‑Eval, FastLongSpeech scores 3.55, compared with 3.44 for NTK‑RoPE, 3.10 for AvgPool, 3.08 for MostSim, and 2.54 for Random. The efficiency contrast is similarly pronounced: NTK‑RoPE requires 61.21 TFLOPs and 4.80 s average runtime per sample on an NVIDIA L40, whereas FastLongSpeech uses 26.44 TFLOPs and 1.47 s while also achieving the best score. On short-speech QA, the method can improve answer quality while lowering compute; for the LibriTTS subset of OpenASQA, the baseline Qwen2‑Audio score is 3.73 at 9.79 TFLOPs, whereas FastLongSpeech reaches 3.87 at $\mathbf{x}$6 with 5.64 TFLOPs (Guo et al., 20 Jul 2025).

The empirical behavior is strongly task dependent. For ASR, moderate compression preserves most accuracy, but aggressive compression is harmful. On LibriSpeech test-clean/test-other and GigaSpeech test, the baseline Qwen2‑Audio WERs are 3.85 / 6.70 / 13.71. FastLongSpeech at $\mathbf{x}$7 gives 4.08 / 7.17 / 11.77, and at $\mathbf{x}$8 gives 4.36 / 7.40 / 12.70; by contrast, $\mathbf{x}$9 yields 27.12 / 24.61 / 23.69. This indicates that QA and reasoning tolerate compression far better than verbatim transcription. The ablation study on LongSpeech‑Eval further isolates the architecture’s components: the full system scores 3.55, while w/o Dynamic Compression Training scores 3.33, w/o Iterative Fusion scores 3.41, and w/o Content Density scores 3.28. These results identify DCT and content-density-weighted fusion as central mechanisms rather than auxiliary refinements (Guo et al., 20 Jul 2025).

5. Relation to the long-speech literature

FastLongSpeech belongs to a broader line of work that addresses long-form speech by reducing the effective acoustic context before or within the LLM. One closely related direction is KV-space compression: Speech‑XL introduces Speech Summarization Tokens (SSTs) that are interleaved into fixed speech intervals, after which only SST key-value states are retained across intervals. With interval length 512 tokens and compression ratios such as $$p(\mathbf{y} \mid \mathbf{x}, \mathbf{h}) = \sum_{i=1}^{I} p(y_i \mid \mathbf{y}_{<i}, \mathbf{x}, \mathbf{h}).$$0, Speech‑XL reduces effective context length from $$p(\mathbf{y} \mid \mathbf{x}, \mathbf{h}) = \sum_{i=1}^{I} p(y_i \mid \mathbf{y}_{<i}, \mathbf{x}, \mathbf{h}).$$1 to $$p(\mathbf{y} \mid \mathbf{x}, \mathbf{h}) = \sum_{i=1}^{I} p(y_i \mid \mathbf{y}_{<i}, \mathbf{x}, \mathbf{h}).$$2, and on 10-minute inputs reports TFLOPs of about 60% of Qwen2.5‑Omni‑7B at comparable durations (Sun et al., 5 Feb 2026).

A second direction emphasizes front-end token compression before the LLM. FastSLM compresses Whisper-large-v3 encoder outputs from 50 tokens/s to about 1.67 tokens/s through the Hierarchical Frame Querying Transformer, using 50 final tokens for a 30-second chunk. That design yields competitive performance at lower FLOPs and parameter count, and the paper reports that 1.67 tokens/s is the efficiency “sweet spot” relative to 2.67 or 1.33 tokens/s (Lee et al., 8 Jan 2026). This suggests that FastLongSpeech’s speech-side compression is part of a larger movement toward aggressively shortening acoustic sequences before they reach the high-cost language-model stack.

A third direction treats long speech as a streaming or chunk-conditioned problem rather than a fixed-window problem. JEDIS‑LLM addresses joint ASR and diarization with a Speaker Prompt Cache and streamable chunk-wise inference, achieving zero-shot long-audio behavior from short-audio training (Shi et al., 20 Nov 2025). In instruction-following, FBK’s long-form SpeechLLMs show that fixed 30-second segmentation is the most robust among tested schemes, reaching HIFS = 2.0663 and outperforming more complex VAD-based alternatives (Xie et al., 25 Jun 2026). At the benchmark level, LongSpeech provides over 100,000 approximately 10-minute segments across ASR, speech translation, summarization, speaker count, language detection, content separation, emotion analysis, and temporal issue localization, thereby defining the multi-task long-context regime that methods such as FastLongSpeech are designed to address (Yang et al., 20 Jan 2026).

6. Limitations and research directions

FastLongSpeech retains several limitations. It depends entirely on short-speech training data and synthetic spoken QA for long-speech evaluation, so its gains do not yet demonstrate adaptation to large-scale real long-form instruction data. It compresses all long inputs into at most 750 frames, which is efficient but may discard too much information for extremely long recordings or tasks requiring exhaustive detail. Its ASR results make this boundary explicit: moderate compression is acceptable, but aggressive compression is unsuitable when exact transcript fidelity is required. LongSpeech‑Eval itself is synthesized from text benchmarks via TTS, so it does not fully capture spontaneous long-form speech phenomena such as disfluencies, background noise, and multi-speaker overlap (Guo et al., 20 Jul 2025).

The paper also points toward several concrete extensions. One is to combine speech-side compression with LLM-side token pruning or adaptive token selection, since the framework already compares favorably with FastAdaSP on MELD at matched inference-cost reduction. Another is to revisit the current hand-designed fusion rule with more sophisticated or differentiable fusion modules. A further implication is that the method may scale with stronger backbones: the authors report that iterative fusion can be applied directly to Qwen2.5‑Omni as a plug-in efficiency mechanism. More broadly, FastLongSpeech suggests a design principle for long-form speech-language modeling: rather than extending the full acoustic context all the way through the LLM, compress speech representations in a way that is text-aware, train the LLM to be invariant across compression levels, and then choose the target length $$p(\mathbf{y} \mid \mathbf{x}, \mathbf{h}) = \sum_{i=1}^{I} p(y_i \mid \mathbf{y}_{<i}, \mathbf{x}, \mathbf{h}).$$3 at inference time to navigate the accuracy–efficiency trade-off (Guo et al., 20 Jul 2025).