---
title: 'VoxBox: A Large-Scale Speech Dataset'
url: https://www.emergentmind.com/topics/voxbox-dataset
type: topic
---

# VoxBox: A Large-Scale Speech Dataset

VoxBox is a large-scale, attribute-rich speech dataset comprising over 100,000 hours of multi-domain English and Mandarin Chinese recordings, constructed to advance research in controllable text-to-speech (TTS), zero-shot voice cloning, and related speech language modeling tasks. The dataset is introduced to provide broad coverage of demographic attributes, fine-grained paralinguistic control, and reproducible, open benchmarks for large language model (LLM)-based TTS systems such as Spark-TTS [2503.01710].

## 1. Dataset Composition

VoxBox aggregates 47,706,212 utterances from 29 open-source speech corpora, spanning English and Mandarin Chinese. The total duration sums to 102,500.26 hours (approximately 102.5k hours), with a stratified representation of gender and age. Major subsets include:

- Chinese: Sources such as AISHELL-3 (88,035 utterances, 85.62 h), Emilia-CN (15,629,241 utterances, 34,759.45 h).
- English: Sources such as CREMA-D, Dailytalk, Emilia-EN (8,303,103 utterances, 20,297.98 h).

The dataset encompasses clean studio recordings (e.g., Hi-Fi TTS, Librispeech), in-the-wild conversational speech (e.g., Dailytalk, MSP-Podcast), emotional corpora (e.g., CREMA-D, IEMOCAP, RAVDESS), and multi-domain text types (reading, dialogue, storytelling, spontaneous speech). Demographic diversity is supported by large corpora: AISHELL-3 (218 speakers), MAGICDATA (377 speakers), MLS-English (7,000+ speakers), WenetSpeech4TTS (approximately 8,000 speakers), among others. Gender (male/female) and age (Child, Teenager, Young Adult, Middle-aged, Elderly) are explicitly annotated.

| Language | Source       | #Utterances   | Male (h)     | Female (h)   | Total (h)   |
|----------|--------------|---------------|--------------|--------------|-------------|
| Chinese  | AISHELL-3    | 88,035        | 16.01        | 69.61        | 85.62       |
| Chinese  | Emilia-CN    | 15,629,241    | 22,017.56    | 12,741.89    | 34,759.45   |
| English  | Dailytalk    | 23,754        | 10.79        | 10.86        | 21.65       |
| English  | Emilia-EN    | 8,303,103     | 13,724.76    | 6,573.22     | 20,297.98   |

(*See full corpus listing and language-specific totals in [2503.01710], Table A.1.*)

## 2. Annotation Schema and Attribute Definitions

VoxBox is annotated for a set of speaker and utterance-level attributes relevant to controllable speech synthesis workflows:

- **Gender:** Male or female, obtained via fine-tuned WavLM-large; accuracy on AISHELL-3 test is 99.4%.
- **Age:** Child, Teenager, Young Adult, Middle-aged, Elderly; WavLM-large fine-tuned for age yields 95.6% accuracy (Table A.2).
- **Pitch:** 
  - Fine-grained: Mean $F_0$ frequency (rounded), Mel scale.
  - Coarse-grained: Discretized into 5 levels (Very Low, Low, Moderate, High, Very High). Thresholds vary by gender: e.g., for males, Very Low $<$145, Low 145–164, Moderate 164–211, High 211–250, Very High $\geq$250.
- **Speaking Rate:** 
  - Fine-grained: Syllables per second (SPS), rounded.
  - Coarse-grained: Five-level scale based on language-specific percentiles (e.g., English: Very Slow $<$2.6, Slow 2.6–3.4, Moderate 3.4–4.8, Fast 4.8–5.5, Very Fast $\geq$5.5).
- **Emotion:** Combination of: automatic emotion2vec label (with confidence), SenseVoiceSmall label, and text-based emotion class by Qwen2.5-72B (Fearful, Happy, Disgusted, Sad, Surprised, Angry, Neutral).

Pitch and speaking rate groups are defined granularly for each gender and language, supporting fine-tuned synthesis and analysis of paralinguistic control in TTS.

## 3. Data Collection, Pre-processing, and Quality Control

All data are sourced from open, licensed repositories (Openslr, Kaggle, GitHub), spanning TTS, ASR, and emotion corpora. Pre-processing steps standardize the dataset:

- **Resampling:** All files resampled to 16 kHz, WAV (mono), 16-bit PCM.
- **Silence Trimming:** Voice Activity Detection (VAD) removes leading/trailing silences for accurate speed estimation.
- **Transcript Cleaning:** Scripts from noisy/ASR corpora (Emilia variants) are re-transcribed using FunASR or Whisper-large-v3; files with insertion/deletion discrepancies from gold transcripts are excluded.
- **Quality Control:** Any sample failing transcript alignment (insertion, deletion) or lacking confident emotion predictions is rejected. 

No additional cross-corpus speaker de-duplication is imposed, beyond existing source dataset splits.

## 4. Dataset Splits, Availability, and Licensing

VoxBox is primarily constructed as the core training set for Spark-TTS. Where source datasets provide train/dev/test splits, these are retained; otherwise, the entire curated subset enters the training partition. All 29 source corpora provide permissive open-source licenses, with provenance, metadata, and download links supplied in Appendix A.3 of [2503.01710] and in the Spark-TTS GitHub repository. No further partitioning or de-duplication beyond upstream divisions is enforced.

## 5. Data Format, Codec, and Tokenization

Audio files in VoxBox are standardized to 16kHz, 16-bit PCM WAV format. The BiCodec tokenization scheme for Spark-TTS employs:

- **Semantic tokens:** 50 tokens/sec, codebook size $K_s=8192$, $b_s=13$ bits/token, for a bitrate $R_{\text{semantic}} = 650\,\text{bps}$ or $0.65\,\text{kbps}$.
- **Global tokens:** $N_g=32$ per utterance, codebook size $K_g=4096$, $b_g=12$ bits/token; for an utterance of $T_{\text{audio}}$ seconds, $R_{\text{global}} \approx \frac{384}{T_{\text{audio}}}\,\text{bps}$, contributing approximately 7.7 bps for a typical 5 s sample.

Each token sequence sent to Spark-TTS begins with fine-grained attribute tokens (if used), followed by $N_g$ global tokens, then a continuous semantic token sequence at 50 TPS, forming a coherent set suitable for inferring both content and expressive attributes.

## 6. Statistical Analysis and Distributions

Aggregate statistics and illustrative figures in [2503.01710] support quantitative analysis:

- **Mean Utterance Duration:** $\mu_{\text{dur}} = \frac{102{,}500\,\text{h}}{4.77\times10^7\,\text{utterances}} \approx 7.7\,\text{s}$.
- **Gender and Age Distribution:** Approximately 60% male / 40% female globally. Most utterances are from Young Adults, with fewer from Teenagers, Middle-aged, Children, and Elderly.
- **Histograms:** Appendix Fig. A.1 displays histograms of speaking rate, duration, and pitch for English and Chinese, split by gender.
- **Variance:** Variability of pitch and speaking rate can be directly inferred from the banding thresholds described in Section 2 (percentile-derived).

## 7. Significance for TTS and Speech Research

By assembling over 100k hours of multi-domain speech with detailed categorical and continuous attribute labeling—including gender, pitch, rate, age, and emotion—VoxBox establishes a benchmark foundation for reproducible, open research in controllable TTS. Its integration into Spark-TTS demonstrates the dataset’s role in facilitating zero-shot voice cloning and highly customizable speech synthesis, transcending the limitations of reference-based approaches [2503.01710]. The rich metadata schema and alignment with foundation model architectures enable fine-grained controllability and systematic speech attribute conditioning for LLM-based speech generation frameworks.

Source: https://www.emergentmind.com/topics/voxbox-dataset