Latent Speech-Text Transformer (LST)

• LST is an autoregressive model that compresses speech tokens into latent patches to enhance data and computational efficiency.
• It employs a local encoder for patch embedding, a global transformer for interleaved speech-text processing, and local decoding for reconstruction.
• Empirical results demonstrate improved performance on speech-to-speech and text-to-text tasks while reducing compute cost by approximately 20%.

The Latent Speech-Text Transformer (LST) is an autoregressive model architecture for speech and text that addresses efficiency and scalability constraints in joint pretraining. By introducing a latent patching strategy to aggregate speech tokens into higher-level, information-dense embeddings ("speech patches"), LST enhances both data efficiency and computational efficiency. This compression mechanism facilitates more balanced representational alignment between speech and text modalities, leading to steeper scaling laws and improved performance across speech-to-speech (S→S) and text-to-text (T→T) tasks (Lu et al., 7 Oct 2025).

1. Motivation and Problem Statement

Conventional autoregressive speech-text models, such as GSLMs, require hundreds or thousands of discrete speech tokens per second of audio, whereas only a few text tokens are necessary to represent equivalent semantic content. Modeling interleaved speech and text using a shared decoder transformer yields a severe compute imbalance: the speech modality disproportionally dominates FLOPs and sequence length. This token-length disparity impedes effective cross-modal alignment and results in orders-of-magnitude flatter scaling laws—substantially more tokens must be processed to achieve similar performance. LST mitigates these issues by dynamically grouping contiguous speech tokens into higher-level semantic units prior to processing by the global transformer, thereby reducing effective sequence length and facilitating more efficient representational transfer and computation.

2. Architecture and Operational Strategies

LST comprises three major components:

• Local Encoder: A shallow transformer pools contiguous frames of speech tokens into latent speech patch embeddings.
• Global Transformer: Processes interleaved streams of BPE text tokens and latent speech patch embeddings using a standard decoder-only transformer architecture.
• Local Decoder: Recovers the original sequence of speech tokens from each patch embedding for waveform reconstruction.

Multiple patching regimes are employed:

• Static Patching: Divides the sequence of speech tokens $X = [x_0, …, x_T]$ into fixed-length, non-overlapping windows of size $p$.
• Alignment Patching: Leverages forced-alignment boundaries—obtained from Wav2Vec-2 + CTC models—to create patches corresponding exactly to words (or silences as separate patches).
• Mixed/Curriculum Patching: Trains with a stochastic or annealed blend of static and aligned patching to eliminate reliance on external aligners at inference.

The local encoder architecturally consists of a single self-attention layer (sliding window: 512) with optional cross-attention into input frames, maintaining computational complexity at $O(\text{patch length} \times d)$—negligible relative to the global transformer.

3. Mathematical Formulation

Let $\{s_0, …, s_T\}$ denote the discrete speech tokens, embedded into $X \in \mathbb{R}^{T \times d}$. For each patch index $i$ and subset $P_i \subset \{0, …, T\}$ (defining patch boundaries), the patch embedding is:

$$z_i = f_{\text{agg}}(X_{P_i})\,, \quad f_{\text{agg}}(\cdot) \equiv \text{LocalEnc}(\cdot)$$

The joint model is trained to maximize the next-token likelihood over the interleaved sequence of speech patches and text tokens $t_0,\ldots,t_m$:

$L_{NTP} = -\sum_u \log p_\theta(y_u | y_{<u})$

where $p$0 concatenates patch and text tokens as $p$1.

While explicit auxiliary alignment losses are not employed, alignment patching implicitly ensures one-to-one correspondence between $p$2 and corresponding text token embeddings $p$3:

$p$4

In practice, the shared global decoder and global next-token prediction suffice to enforce alignment. The total loss is thus:

$p$5

4. Pre-training Regimes and Hyperparameters

LST pre-training utilizes:

• Text Corpus: 1.8 trillion BPE tokens (LLaMA-2, 32K SentencePiece vocabulary).
• Speech Corpus: $p$6200k hours of English audio (LibriLight, People’s Speech, MLS, Spotify), discretized to 501 HuBERT tokens at 25Hz and force-aligned to text.

Distinct regimes:

• Compute-controlled: Number of iterations and per-step context are fixed, ensuring matched total FLOPs for baseline and LST, but LST processes substantially more "semantic" units per FLOP.
• Data-controlled: Fixed total number of speech/text tokens; LST processes fewer patch tokens and thus reduces compute cost.

Optimal speech:text mixing is at a 1:2 ratio. Where specified, the 1B-parameter LST used: Local Encoder (1 layer, $p$7, 16 heads, window 512), Global Transformer (25 layers, $p$8, 16 heads, block-causal + RoPE), Local Decoder (9 layers, $p$9, 16 heads), AdamW optimizer with peak LR $O(\text{patch length} \times d)$0, batch size 0.5M (aggregated units), and bfloat16 training on 32 × H100 GPUs, for 200k steps (≈1T units, ≈17 hours).

5. Empirical Findings and Scaling Behavior

Compute-controlled Regime:

On HellaSwag story completion (1-in-4 MC), LST curriculum patching yields substantial improvements:

• S→S: Baseline 39.0% → LST 45.5% (+6.5pp)
• T→T: Baseline 47.0% → LST 52.2% (+5.2pp)

Similar absolute gains are observed on StoryCloze and TopicStoryCloze benchmarks.

Data-controlled Regime:

• LST (static patching) saves ~19.3% compute; curriculum patching saves ~19.7%.
• With compute savings, LST still surpasses baseline (HellaSwag S→S 40.2→45.5, T→T 49.6→52.2).

Scaling Laws:

• At 1B parameters: HellaSwag S→S 36.8→41.3; T→T 47.1→49.2
• At 7B parameters: S→S 42.0→44.2; T→T 54.8→55.3
• Under compute-optimal scaling (420M→1.8B params), LST produces progressively larger gaps (S→S: Baseline 28.4→35.3 vs. LST 28.4→39.0).

Negative log-likelihood differentials confirm that LST consistently assigns higher probability mass to correct endings, and the gain magnifies with model scale.

6. Ablation Studies and Analytical Insights

• Patch Granularity: Static patches of 4 HuBERT tokens yield significant improvements. Oversized static patches (6, 9 tokens) degrade S→S performance.
• Alignment Patching: Word-aligned patches (with silences as distinct patches) outperform BPE-aligned approaches for S→S; merging silences into adjacent words provides marginal patch count reduction but potentially impairs semantic clarity.
• Curriculum Patching: Annealing from aligned to static patching provides the best stability (std ≈ 0.1pp on HellaSwag) and aggregate accuracy across tasks.
• Resource Efficiency: LST requires ~20% fewer FLOPs for improved dual-modality accuracy.
• Speech/Text Ratio: A 1:2 ratio of speech:text is optimal; excessive speech exposure (1:1) hampers T→T, while insufficient speech (1:3, 1:4) undermines speech modeling capacity.

7. Broader Implications and Future Directions

LST demonstrates that compressing speech tokens into latent patch embeddings fundamentally enhances both data and compute efficiencies in speech-text autoregressive pretraining. The approach closes the performance gap between text and speech processing, achieves significant compute savings, and ensures improved scaling with model size and available data. Curriculum patching eliminates dependence on external aligners at inference, facilitating direct deployment.

Anticipated extensions include adaptation to full-duplex speech dialogue, instruction finetuning, the exploration of alignment-free patching modalities, and application to broader multi-modal integration encompassing vision or video. This suggests that latent patching may serve as an effective paradigm for other sequence modalities presenting similar length/computation disparities (Lu et al., 7 Oct 2025).