---
title: 'SpeLangy: Data-Centric Speech–Language Model'
url: https://www.emergentmind.com/topics/spelangy
type: topic
---

# SpeLangy: Data-Centric Speech–Language Model

SpeLangy is a 3.8-billion-parameter speech–language foundation model developed for end-to-end training to excel at spoken question-answering (SQA) while maintaining robust text-only capabilities. Its approach is rooted in a series of controlled, data-centric ablations aimed at optimizing speech–text pretraining by refining raw web-crawled audio processing, leveraging synthetic data for domain coverage, and systematic sampling of audio/text modalities. This intensive focus on data curation, rather than novel architecture or objective design, yields a model that outperforms SpeechLMs up to three times its scale on SQA benchmarks with no degradation in text-only benchmarks [2510.20860].

## 1. Model Architecture and Pretraining Objective

SpeLangy employs the “speech encoder + connector + language model” paradigm. The speech frontend, or "speech tokenizer," consists of a stack of Conformer blocks (as described in Gulati et al. 2020) that map raw audio waveforms to intermediate representations. This sequence is then quantized using a finite scalar quantizer, producing discrete speech tokens at 12.5 Hz (as in Mentzer et al. 2023). These speech tokens are interleaved with text tokens and provided as input to a 3-billion-parameter dense transformer language model with a 16,384-token context window, initialized from Apple’s base-LM. During pretraining, only the LM weights are updated; the speech tokenizer remains frozen.

The pretraining task is standard next-token prediction. For a packed sequence of interleaved tokens \( X = [x_1, x_2, \dots, x_L] \), the loss is:
\[
\mathcal{L}(\theta) = -\sum_{t=1}^{L} \log \Pr_\theta\big(x_t \mid x_{1:t-1}\big),
\]
using the softmax over the LM’s vocabulary. There is no use of contrastive or masked prediction losses—performance gains are attributed exclusively to data-centric interventions.

## 2. Raw Web-Crawled Audio Data Pipeline

SpeLangy’s primary speech–text data source is over 10 million hours of web-crawled audio, including podcasts, interviews, and monologues. The processing pipeline incorporates five stages:
1. **Speaker diarization** (pyannote.audio) yields (start, end, speakerID) segments.
2. **Language detection** (Whisper) filters for English-language segments.
3. **Transcription ensemble**: Three ASR systems (Whisper, an internal SIRI model, Nvidia’s Parakeet) are ensembled using ROVER post-processing to minimize hallucinations.
4. **Transcription filtering**: Segments are discarded if the ensemble text is empty, excessively repetitive (any 15-gram repeated more than 5 times), or too short.
5. **Interleaved chunking**: Provides two settings—
   - *Coarse*: Merge consecutive same-speaker segments into longer chunks (mean length 19.2 s).
   - *Fine*: Retain diarized segments without merging (mean length 5.2 s).
Chunks shorter than 0.2 s are filtered out. Each training sample is an interleaved collection:
\[
X_i = \{(A_1, T_1), (A_2, T_2), \dots, (A_n, T_n)\},
\]
with \(A_j\) as the speech token sequence and \(T_j\) its transcription.

## 3. Synthetic Speech–Text Data Augmentation

To counteract the knowledge domain bias of web audio—typically entertainment- and sports-heavy, with transcription noise and background artifacts—two synthetic datasets were constructed:

- **Krist (Knowledge-Rich Interleaved Speech-Text)**: Derived from lightly filtered CommonCrawl HTML in 17 “knowledge-rich” domains (e.g., Chegg, Quizlet, Justia). GPT-4o-mini extracts and rewrites plain text, followed by sentence-level chunking and TTS (MeloTTS, 5 sampled accents), yielding 4.6 million hours.
- **Quest (QA-Style Speech-Text)**: Draws from the same HTML pool. Regex mining identifies candidate questions, then GPT-4o filters and generates chain-of-thought answers, followed by chunking and TTS for 0.9 million hours.

The training mix maintains a 60%:40% text-only to speech–text token ratio (as in Shukor et al. 2025). Augmentation experiments within the 40% speech–text budget show that including 34% Quest yields a +7.2 percentage point (pp) gain in average SQA accuracy, outperforming other mixtures.

## 4. Modality Interleaving and Sampling Policies

An ablation of chunk-sampling schemes compared two methods:
- **Stochastic**: Independent $p=0.5$ sampling of audio or text for each chunk, beginning with audio (expected switches per sample: $\frac{n-1}{2}$).
- **Deterministic**: Strict alternation between audio and text ($n-1$ switches per sample).

Deterministic sampling reliably improves SQA by +1.0 pp on average, plausibly due to the increased frequency of modality transitions, which enhance cross-modal alignment.

## 5. Training Regime and Computational Setup

Training is characterized by:
- Global batch size: 512; packed sequence length: 16,384 tokens.
- 200,000 training steps, totaling 1.67 trillion multimodal tokens.
- AdamW (β₁=0.9, β₂=0.95), peak LR=3e-4, weight decay=1e-5, gradient norm clip=1.0.
- Cosine-decay learning rate with 1,000-step linear warmup.
- Trainable parameters limited to the LM (speech tokenizer frozen).
- Total compute approaches $3.8 \times 10^{22}$ FLOPs, distributed across 512 TPU-v6e chips over approximately 7 days.

## 6. Empirical Results and Ablation Outcomes

Evaluation comprises three SQA benchmarks—Spoken-Web-Questions (SWQ), Spoken-TriviaQA (STQ), and Spoken-LLaMA-Questions (SLQ)—using 4-choice MCQs scored by normalized log-likelihood. Text-only evaluation leverages CoreEN (9 tasks), MMLU, GSM-8k, and HumanEval.

Key ablation findings include:
- **Fine vs. coarse chunking**: Fine yields +3.1 pp SQA gain (37.6→40.7%) without text regression.
- **Synthetic augmentation**: +1.2 pp average gain for Krist; +7.2 pp for Quest at 40% budget.
- **Deterministic sampling**: +1.0 pp SQA over stochastic.
- **Loss masking for understanding-only models**: Audio masking combined with all interventions increases SQA by +9.3 pp (42.5→51.8%), indicating modality conflict at smaller scales.

Final comparisons show SpeLangy (3.8B) outperforms Kimi-Audio (10.5B), Qwen-Audio (8.4B), and Qwen2-Audio (8.4B) by 10.2–11.1 pp on SQA, with text performance (MMLU 67.3%) on par or better than larger models. Zero-instruction-tuning SQA performance is competitive with SFT-tuned Voxtral-mini (51.2%) and GLM-4-Voice (53.4%).

## 7. Analysis, Recommendations, and Broader Implications

SpeLangy’s performance gains stem from data-centric practices:
- Fine chunking and synthetic mixing lower the Kullback–Leibler gap between audio- and text-conditioned distributions (mean reverse-KLD drops from 3.20 to 1.47), facilitating self-distillation of speech modality into the LM.
- Synthetic datasets specifically bolster coverage in underrepresented domains (science, finance, education, health), aligning training and SQA evaluation distribution.
- Deterministic alternation of modalities fortifies cross-modal fusion, improving zero-shot SQA.
- These advantages extend to both understanding-only models (with loss masking) and post-trained SFT checkpoints.
- Contamination control through n-gram analysis reveals negligible test-leakage impact (<2 pp difference between clean and random-removal).

Practical recommendations for speech–language pretraining include:
- Use fine-grained diarization for chunking without merging by speaker.
- Exploit high-quality text corpora to generate QA-format synthetic interleaved data for SQA tasks.
- Enforce deterministic modality interleaving during transformer LM training.
- Maintain a 60/40 text/speech–text mix and apply audio-loss masking in smaller models.
- Track audio/text Kullback–Leibler divergence during training as a proxy for cross-modal alignment.

SpeLangy demonstrates that tailored data curation, rather than architectural novelty or sheer parameter count, is the dominant factor in advancing speech–language foundation model performance [2510.20860].

Source: https://www.emergentmind.com/topics/spelangy