SIFT-50M: Multilingual Speech Instruction Dataset
- SIFT-50M is a multilingual dataset converting 14K hours of speech into over 50M natural-language instruction–response pairs for advanced speech understanding and controllable generation.
- It categorizes data into closed-ended, open-ended, and controllable generation tasks, covering acoustic judgments, transcript reasoning, and comparative analysis.
- The dataset leverages expert acoustic extraction and LLM-based instruction synthesis, with EvalSIFT serving as a benchmark for instruction-following evaluation.
Searching arXiv for the cited SIFT/SIF dataset paper and related benchmark context. In the speech-text LLM literature, the query term “SIF Dataset” maps to SIFT-50M, where SIFT stands for Speech Instruction Fine-Tuning. SIFT-50M is a multilingual instruction dataset introduced to augment public speech corpora with natural-language instruction–response pairs for speech understanding and controllable speech generation. It is built from publicly available speech corpora containing 14K hours of speech, spans five languages, and is intended for both instruction fine-tuning and continued pre-training of speech-text LLMs. The same work also introduces EvalSIFT, a benchmark for evaluating instruction-following in this modality, and SIFT-LLM, a model trained on the dataset (Pandey et al., 12 Apr 2025).
1. Naming, scope, and motivation
SIFT-50M was introduced to address a structural limitation of conventional speech corpora: most are designed for task-specific supervision, primarily ASR, and do not provide the natural-language instructions needed to train instruction-following speech-text LLMs. The resulting gap affects generalization across broader speech understanding tasks beyond transcription. SIFT-50M addresses this by converting public speech resources into a large corpus of instruction–response pairs that cover both speech understanding and controllable speech generation (Pandey et al., 12 Apr 2025).
The paper uses two closely related names. “SIFT” may refer either to the general dataset construction pipeline or to the dataset family, whereas “SIFT-50M” denotes the concrete release. The dataset is described in the abstract as “a 50M-example dataset”, but the tabulated totals in Table 2 are larger: 55.6M train / 470K dev / 30K EvalSIFT. The paper’s naming therefore emphasizes the approximate scale, while the table reflects pooled examples generated across sources and languages. This distinction is important because it is a recurrent source of confusion when the dataset is cited (Pandey et al., 12 Apr 2025).
The release has four stated contributions: SIFT-50M itself, EvalSIFT, SIFT-LLM, and controllable speech generation experiments. In functional terms, the dataset is designed to align audio encoders with LLM input spaces while exposing models to diverse, multi-dimensional speech instructions rather than only task labels (Pandey et al., 12 Apr 2025).
2. Scale, languages, and task composition
SIFT-50M spans English, German, French, Italian, and Spanish. The dataset is organized around three major instruction families: closed-ended, open-ended, and controllable generation. Closed-ended data are further divided into Acoustic-Level, Content-Level, Word-Align, and Comparison tasks. This taxonomy reflects a deliberate attempt to cover low-level acoustic judgments, transcript-derived reasoning, time-alignment phenomena, pairwise audio comparison, and instruction-conditioned generation within a single corpus (Pandey et al., 12 Apr 2025).
The dataset is large enough to support both pre-training-like continued alignment and instruction fine-tuning. The benchmark companion, EvalSIFT, is multilingual and balanced by category: for each of the five languages, 2K examples per category (closed-ended, open-ended, controllable generation), totaling 30K examples (Pandey et al., 12 Apr 2025).
| Category | Train | Dev / EvalSIFT |
|---|---|---|
| Acoustic-Level | 17.8M | 100K / 2.5K |
| Content-Level | 14.5M | 80K / 2.5K |
| Word-Align | 9.8M | 40K / 2.5K |
| Comparison | 3.6M | 100K / 2.5K |
| Open-ended | 4.3M | 100K / 10K |
| Controllable Generation | 5.6M | 50K / 10K |
The source corpora are Multi-lingual Librispeech (MLS), Common Voice, and VCTK. These provide transcripts and content-level attributes, including gender, accent, and age where available, while SIFT adds acoustic metadata and generated instruction–response pairs on top of them. This suggests that SIFT-50M is best understood not as a raw speech corpus, but as a large derived supervision layer placed over existing multilingual speech collections (Pandey et al., 12 Apr 2025).
3. Data schema, metadata, and instruction ontology
The instruction ontology is central to the dataset. Acoustic-Level questions concern pitch, intensity, speaking rate, noise, reverberation, distortion, and also acoustic language ID. Content-Level items use transcript-derived content, age category, accent, gender, and phonetic patterns. Word-Align questions use word-level temporal alignments and word-level acoustic statistics. Comparison prompts compare two audio files on speech characteristics. Open-ended prompts require reasoning over metadata, and Controllable Generation prompts specify target speech with prosodic and demographic constraints (Pandey et al., 12 Apr 2025).
The paper provides representative examples of each category. One Acoustic-Level example asks, “How would you characterize the speaking rate in this audio?”, with answer “slightly slow.” Word-Align examples compare a specific word’s pitch to surrounding words. Comparison examples ask, for instance, “Which audio has a speaker with a Scottish accent?” Open-ended prompts request analysis of phonetic structure and its relation to accent and age category. Controllable generation prompts specify combinations such as pitch level, pitch variation, speaking rate, intensity, gender, accent, coarse age category, and transcript content (Pandey et al., 12 Apr 2025).
The metadata schema is correspondingly rich. The paper lists categorical acoustic metadata fields: average_pitch, pitch_variation, speaking_rate, average_intensity, noise_level, reverberation, distortion_level, speech_quality. Content-level metadata include phonemes, age_category, accent, gender, and transcript. Word-level metadata include, for each word, position, average_pitch, pitch_variation, average_intensity. Instruction instances are serialized in a ChatML/Messages-style JSON format with role, text, and audio_path content blocks (Pandey et al., 12 Apr 2025).
Two implementation details are especially important. First, alignment is via in-house hybrid HMM-LSTM forced alignment. Second, the paper states that audio preprocessing specifics like VAD, segmentation, sampling rates are not detailed. For researchers interested in exact reconstruction of the data pipeline, this means the metadata extraction and instruction generation stages are described much more explicitly than low-level audio preprocessing (Pandey et al., 12 Apr 2025).
4. Construction pipeline and multilingual generation
SIFT-50M combines expert acoustic extraction with LLM-based instruction synthesis. The expert models named in the paper are PYYAPT for pitch, Parselmouth for intensity, a grapheme-to-phoneme (G2P) model for phonemes, an in-house hybrid HMM-LSTM acoustic model for forced alignment, the method in “Data-speech” for noise, speech quality, reverberation, and a Voice Gender Classifier to fill missing gender labels. Numeric utterance-level statistics are then discretized into categorical descriptors. The paper gives an explicit example: for males, median pitch maps to “slightly low pitch” when Hz, and for females when Hz (Pandey et al., 12 Apr 2025).
Age is grouped into coarse decades such as twenties, thirties, …, nineties. For French, German, Spanish, and Italian, categorical metadata values are translated using a phrase-level translation dictionary. This multilingual design is not merely post hoc translation of prompts; it extends to metadata verbalization itself, which is necessary for consistent instruction generation in each target language (Pandey et al., 12 Apr 2025).
Instruction generation is handled by multiple LLMs. Mixtral 8x7B is used for Closed-ended Acoustic-/Content-/Word-Align tasks and for Controllable Generation prompts. Amazon Nova Pro is used for Comparison and Open-ended generation. The published hyperparameters are also explicit: Mixtral 8x7B uses Temperature 0.7, Top p 0.9, Max tokens 2048, while Amazon Nova Pro uses Temperature 0.7, Max tokens 2048. The pipeline generates up to 10 QA pairs per utterance (Pandey et al., 12 Apr 2025).
The paper emphasizes prompt engineering and post-generation control. A notable filtering decision is that alignment instructions with numeric time references—such as “What words occur between the 3 and 5 second mark?”—were found to degrade performance and were excluded from the main dataset. The rejected subset is released as a bonus research partition. This is a concrete example of task-type ablation feeding back into dataset curation rather than only into model design (Pandey et al., 12 Apr 2025).
5. Quality assurance, ethics, and dataset governance
Quality assurance operates at several levels. The paper states that metadata values falling outside feasible ranges are rejected during numerical-to-categorical mapping. It also describes human review and iterative prompt refinement across categories and languages, with the goal of ensuring cogent responses and mitigating hallucinations. The emphasis is therefore on the quality of generated instruction–response pairs rather than on low-level acoustic denoising or deduplication, which are not described in detail (Pandey et al., 12 Apr 2025).
Ethics and privacy receive explicit treatment. SIFT-50M uses source corpora Common Voice, MLS, and VCTK, and the released dataset is licensed under CDLA-Sharing-1.0. The authors acknowledge risks associated with controllable generation, including deepfakes, impersonation, misinformation. As a mitigation, curation is restricted to broad, generic parameters; SIFT-LLM only uses non-specific features not usable for identification; and age categories are coarse bins. The paper frames these steps as de-identification and limitation of personalization fidelity rather than elimination of misuse risk (Pandey et al., 12 Apr 2025).
A second governance issue concerns evaluation validity. EvalSIFT evaluates closed-ended accuracy via LLM-as-a-judge, open-ended outputs with 0–10 LLM-as-a-judge scoring, and controllable generation via metadata compliance using MAE and QWK. The paper later notes that LLM-as-a-judge variability is a limitation, even though variance across repeated runs with the same judge model was reported as low. This suggests that part of SIFT-50M’s empirical significance lies in having made instruction-following measurable for speech-text models, but that the measurement apparatus itself remains partly model-dependent (Pandey et al., 12 Apr 2025).
6. Coupled benchmark, model training, and empirical impact
SIFT-50M is tightly coupled to SIFT-LLM, whose architecture combines Whisper-medium as acoustic encoder, Qwen2.5-7B-instruct as LLM core, k-means clustering on HuBERT embeddings (2000 tokens) for discrete speech tokens, and Big-VGAN as vocoder. A linear projection maps 1024-D audio embeddings into the 3584-D LLM input space, and the vocabulary is extended with 2000 HuBERT-derived tokens. Parameter-efficient adaptation uses LoRA (Pandey et al., 12 Apr 2025).
Training proceeds in multiple stages. Continued pre-training aligns the encoder and LLM input spaces using tasks including ASR, ER, SA-ASR, IC, SER, S2TT, S2ST, TTS. Instruction fine-tuning then specializes the model on SIFT-50M’s speech-understanding and controllable-generation instructions. The published hyperparameters include batch sizes 256 (PT) / 128 (IFT), 200K steps per stage, max learning rate 1e-4, and LoRA ranks 16 for understanding stages and 128 for generation stages. Training used four p4d.24xlarge instances (32 A100 GPUs, 40GB each), requiring approximately 3 days per 200K steps (Pandey et al., 12 Apr 2025).
The empirical results reported for SIFT-LLM provide the main evidence that SIFT-50M changes model behavior. On instruction-following benchmarks, SIFT-LLM achieves 57.4 closed-ended accuracy on DS-1 and 46.1 on EvalSIFT English, compared with 48.0 / 25.1 for Qwen2-Audio-Instruct and 34.7 / 21.9 for SALMONN-7B. On open-ended evaluation, SIFT-LLM reaches 7.3 on AB-Chat and 7.8 on EvalSIFT English. Multilingually, the model reports closed-ended accuracy and open-ended score pairs of 39.0 / 6.6 for German, 34.3 / 7.1 for French, 33.2 / 7.5 for Italian, and 35.6 / 7.0 for Spanish (Pandey et al., 12 Apr 2025).
The paper also reports foundational-task trade-offs. For example, on FLEURS-5, SIFT-LLM has WER 11.4, improving over Qwen2-Audio-Inst 13.2 and SALMONN 23.7, while on some tasks its pre-trained checkpoints remain stronger than the instruction-tuned version. The authors explicitly note a foundational tasks vs. instruction-following trade-off, and ablations show that removing open-ended, word-align, or comparison data degrades different aspects of performance. A particularly strong result is that removing comparison data reduces closed-ended EvalSIFT performance to 34.3, indicating that multi-audio reasoning competence depends materially on that subset (Pandey et al., 12 Apr 2025).
7. Access, limitations, and position within speech-text dataset design
SIFT-50M is publicly available at https://huggingface.co/datasets/amazon-agi/SIFT-50M. The paper also states that prompt templates used during pre-training are released, and it documents evaluation prompt templates for both SIFT-LLM and baselines. The dataset therefore combines scale, multilingual coverage, released templates, and a benchmark companion in a way that supports comparative work rather than only single-model reporting (Pandey et al., 12 Apr 2025).
Several limitations remain explicit. The dataset is built from public speech corpora rather than from native instruction-following speech interactions, so the instructions are synthesized on top of pre-existing utterances and metadata. The paper also reports occasional hallucinations on content unrelated to input audio. Auxiliary negative examples helped, but increasing their weight reduced overall speech understanding. In addition, prompt sensitivity affects foundational-task results, and the paper states that code-switching and balancing strategies are not explicitly described (Pandey et al., 12 Apr 2025).
A final interpretive point concerns what SIFT-50M contributes relative to earlier speech resources. It does not merely enlarge multilingual ASR-style supervision; it re-expresses speech data as an instruction-following problem over acoustic, linguistic, alignment, comparative, and generative dimensions. This suggests that SIFT-50M occupies a distinct place in the dataset landscape: it is simultaneously a large supervision corpus, a metadata-engineering pipeline, and an evaluation regime for speech-text LLM instruction following, with EvalSIFT providing the systematic benchmark layer required to make those capabilities measurable (Pandey et al., 12 Apr 2025).