SpeLangy: Data-Centric Speech–Language Model

• SpeLangy is a state-of-the-art 3.8-billion-parameter model designed for end-to-end spoken question answering by leveraging highly curated web-crawled audio and synthetic QA data.
• It employs a 'speech encoder + connector + language model' architecture with fine-grained diarization and deterministic modality interleaving to enhance cross-modal alignment.
• Empirical results show it outperforms larger models on SQA benchmarks by up to 11 points, with ablations validating the effectiveness of its data-centric approach.

SpeLangy is a 3.8-billion-parameter speech–language foundation model developed for end-to-end training to excel at spoken question-answering (SQA) while maintaining robust text-only capabilities. Its approach is rooted in a series of controlled, data-centric ablations aimed at optimizing speech–text pretraining by refining raw web-crawled audio processing, leveraging synthetic data for domain coverage, and systematic sampling of audio/text modalities. This intensive focus on data curation, rather than novel architecture or objective design, yields a model that outperforms SpeechLMs up to three times its scale on SQA benchmarks with no degradation in text-only benchmarks (Udandarao et al., 22 Oct 2025).

1. Model Architecture and Pretraining Objective

SpeLangy employs the “speech encoder + connector + LLM” paradigm. The speech frontend, or "speech tokenizer," consists of a stack of Conformer blocks (as described in Gulati et al. 2020) that map raw audio waveforms to intermediate representations. This sequence is then quantized using a finite scalar quantizer, producing discrete speech tokens at 12.5 Hz (as in Mentzer et al. 2023). These speech tokens are interleaved with text tokens and provided as input to a 3-billion-parameter dense transformer LLM with a 16,384-token context window, initialized from Apple’s base-LM. During pretraining, only the LM weights are updated; the speech tokenizer remains frozen.

The pretraining task is standard next-token prediction. For a packed sequence of interleaved tokens $X = [x_1, x_2, \dots, x_L]$, the loss is: $$\mathcal{L}(\theta) = -\sum_{t=1}^{L} \log \Pr_\theta\big(x_t \mid x_{1:t-1}\big),$$ using the softmax over the LM’s vocabulary. There is no use of contrastive or masked prediction losses—performance gains are attributed exclusively to data-centric interventions.

2. Raw Web-Crawled Audio Data Pipeline

SpeLangy’s primary speech–text data source is over 10 million hours of web-crawled audio, including podcasts, interviews, and monologues. The processing pipeline incorporates five stages:

1. Speaker diarization (pyannote.audio) yields (start, end, speakerID) segments.
2. Language detection (Whisper) filters for English-language segments.
3. Transcription ensemble: Three ASR systems (Whisper, an internal SIRI model, Nvidia’s Parakeet) are ensembled using ROVER post-processing to minimize hallucinations.
4. Transcription filtering: Segments are discarded if the ensemble text is empty, excessively repetitive (any 15-gram repeated more than 5 times), or too short.
5. Interleaved chunking: Provides two settings—
   • Coarse: Merge consecutive same-speaker segments into longer chunks (mean length 19.2 s).
   • Fine: Retain diarized segments without merging (mean length 5.2 s). Chunks shorter than 0.2 s are filtered out. Each training sample is an interleaved collection: $$X_i = \{(A_1, T_1), (A_2, T_2), \dots, (A_n, T_n)\},$$ with $A_j$ as the speech token sequence and $T_j$ its transcription.

3. Synthetic Speech–Text Data Augmentation

To counteract the knowledge domain bias of web audio—typically entertainment- and sports-heavy, with transcription noise and background artifacts—two synthetic datasets were constructed:

• Krist (Knowledge-Rich Interleaved Speech-Text): Derived from lightly filtered CommonCrawl HTML in 17 “knowledge-rich” domains (e.g., Chegg, Quizlet, Justia). GPT-4o-mini extracts and rewrites plain text, followed by sentence-level chunking and TTS (MeloTTS, 5 sampled accents), yielding 4.6 million hours.
• Quest (QA-Style Speech-Text): Draws from the same HTML pool. Regex mining identifies candidate questions, then GPT-4o filters and generates chain-of-thought answers, followed by chunking and TTS for 0.9 million hours.

The training mix maintains a 60%:40% text-only to speech–text token ratio (as in Shukor et al. 2025). Augmentation experiments within the 40% speech–text budget show that including 34% Quest yields a +7.2 percentage point (pp) gain in average SQA accuracy, outperforming other mixtures.

4. Modality Interleaving and Sampling Policies

An ablation of chunk-sampling schemes compared two methods:

• Stochastic: Independent $p=0.5$ sampling of audio or text for each chunk, beginning with audio (expected switches per sample: $\frac{n-1}{2}$).
• Deterministic: Strict alternation between audio and text ($n-1$ switches per sample).

Deterministic sampling reliably improves SQA by +1.0 pp on average, plausibly due to the increased frequency of modality transitions, which enhance cross-modal alignment.

5. Training Regime and Computational Setup

Training is characterized by:

• Global batch size: 512; packed sequence length: 16,384 tokens.
• 200,000 training steps, totaling 1.67 trillion multimodal tokens.
• AdamW (β₁=0.9, β₂=0.95), peak LR=3e-4, weight decay=1e-5, gradient norm clip=1.0.
• Cosine-decay learning rate with 1,000-step linear warmup.
• Trainable parameters limited to the LM (speech tokenizer frozen).
• Total compute approaches $3.8 \times 10^{22}$ FLOPs, distributed across 512 TPU-v6e chips over approximately 7 days.

6. Empirical Results and Ablation Outcomes

Evaluation comprises three SQA benchmarks—Spoken-Web-Questions (SWQ), Spoken-TriviaQA (STQ), and Spoken-LLaMA-Questions (SLQ)—using 4-choice MCQs scored by normalized log-likelihood. Text-only evaluation leverages CoreEN (9 tasks), MMLU, GSM-8k, and HumanEval.

Key ablation findings include:

• Fine vs. coarse chunking: Fine yields +3.1 pp SQA gain (37.6→40.7%) without text regression.
• Synthetic augmentation: +1.2 pp average gain for Krist; +7.2 pp for Quest at 40% budget.
• Deterministic sampling: +1.0 pp SQA over stochastic.
• Loss masking for understanding-only models: Audio masking combined with all interventions increases SQA by +9.3 pp (42.5→51.8%), indicating modality conflict at smaller scales.

Final comparisons show SpeLangy (3.8B) outperforms Kimi-Audio (10.5B), Qwen-Audio (8.4B), and Qwen2-Audio (8.4B) by 10.2–11.1 pp on SQA, with text performance (MMLU 67.3%) on par or better than larger models. Zero-instruction-tuning SQA performance is competitive with SFT-tuned Voxtral-mini (51.2%) and GLM-4-Voice (53.4%).

7. Analysis, Recommendations, and Broader Implications

SpeLangy’s performance gains stem from data-centric practices:

• Fine chunking and synthetic mixing lower the Kullback–Leibler gap between audio- and text-conditioned distributions (mean reverse-KLD drops from 3.20 to 1.47), facilitating self-distillation of speech modality into the LM.
• Synthetic datasets specifically bolster coverage in underrepresented domains (science, finance, education, health), aligning training and SQA evaluation distribution.
• Deterministic alternation of modalities fortifies cross-modal fusion, improving zero-shot SQA.
• These advantages extend to both understanding-only models (with loss masking) and post-trained SFT checkpoints.
• Contamination control through n-gram analysis reveals negligible test-leakage impact (<2 pp difference between clean and random-removal).

Practical recommendations for speech–language pretraining include:

• Use fine-grained diarization for chunking without merging by speaker.
• Exploit high-quality text corpora to generate QA-format synthetic interleaved data for SQA tasks.
• Enforce deterministic modality interleaving during transformer LM training.
• Maintain a 60/40 text/speech–text mix and apply audio-loss masking in smaller models.
• Track audio/text Kullback–Leibler divergence during training as a proxy for cross-modal alignment.

SpeLangy demonstrates that tailored data curation, rather than architectural novelty or sheer parameter count, is the dominant factor in advancing speech–language foundation model performance (Udandarao et al., 22 Oct 2025).