Whilter: Unified Speech Corpus Filter

• Whilter is a Whisper-based utterance-level multi-task classifier that detects and filters undesirable speech attributes in large in-the-wild corpora.
• It employs a frozen Whisper encoder with a learnable Transformer and task-specific attention-pooling heads to jointly predict multispeaker, foreign language, background music, noise, and synthetic speech attributes.
• Experimental results demonstrate that Whilter outperforms specialized models on key speech-specific tasks while offering efficient, flexible downstream processing.

Whilter is a Whisper-based utterance-level multi-task classifier for filtering large “in-the-wild” speech corpora before they are used to train downstream speech models, especially English TTS systems. Introduced in "Whilter: A Whisper-based Data Filter for 'In-the-Wild' Speech Corpora Using Utterance-level Multi-Task Classification" (Ravenscroft et al., 29 Jul 2025), it is designed as a single model that jointly predicts several undesirable utterance attributes—rather than requiring a pipeline of separate tools—so that problematic segments can be discarded or routed to later processing. The model addresses five binary utterance-level labels in a multi-label setting: multispeaker audio, background music, foreign language, noise, and synthetic speech.

1. Problem setting and intended role

Large-scale in-the-wild speech datasets gathered from public videos, podcasts, and similar uncontrolled sources are useful for speech recognition, synthesis, and self-supervised speech representation learning, but they are also noisy in a broad sense. The paper identifies multiple speakers in one segment, non-target languages, background music, noisy or reverberant conditions, and synthetic speech as recurrent properties that can corrupt downstream training (Ravenscroft et al., 29 Jul 2025).

Whilter is positioned as a unified front-end filter for this regime. Its motivating use case is English TTS data preparation, where non-English clips, multispeaker segments, and synthetic speech may be removed outright, while noisy or music-contaminated utterances may instead be sent to enhancement or source-separation post-processing. The same utterance-level predictions can support different policies for ASR, TTS, or self-supervised learning. This operational flexibility follows directly from the fact that the model predicts multiple binary attributes independently rather than collapsing them into a single global category.

The paper’s emphasis is practical as well as methodological. Existing practice is described as a composition of specialist tools, each targeting one nuisance condition. Whilter is proposed as a “one and done” alternative: a single model with shared computation and task-specific outputs. A plausible implication is that this framing treats corpus curation itself as a multi-task speech classification problem rather than as a sequence of unrelated preprocessing heuristics.

2. Task formulation and label semantics

Whilter solves five utterance-level classification tasks simultaneously, all as binary labels in a multi-label setting. This is necessary because an utterance can express several undesirable properties at once; the paper gives the example of a clip containing two speakers, background music, and noise simultaneously (Ravenscroft et al., 29 Jul 2025).

Formally, the model is denoted by $\mathcal{W}$ and maps a mono waveform $x \in \mathbb{R}^L$ to $N$ outputs:

$\mathcal{W}: 1 \times L \mapsto N \times 1,$

with $N=5$. The output is therefore a vector of class probabilities rather than a multiclass decision.

The five labels are defined as follows.

Multispeaker audio is derived from an annotation of the number of unique speakers in a clip, later binarized to indicate whether the number of speakers is greater than one. The released dataset retains the speaker count itself.

Foreign language means the presence of non-English speech. This label is not intended as a balanced multilingual taxonomy; it is explicitly tied to English-data filtering.

Background music marks music underlying the speech.

Noise includes noisy speech and explicitly includes reverberant speech, i.e. convolutive noise.

Synthetic speech marks TTS or voice-converted speech.

The paper also stresses that some labels are intrinsically difficult. Synthetic speech is especially challenging because modern TTS aims at realism and the class is sparse. Noise is partly ambiguous because judgments of “noisiness,” especially for reverberation or stationary noise, are subjective. No formal inter-annotator agreement statistics are reported. These points matter because they delimit what the reported metrics can be taken to mean: they quantify performance on an operational annotation scheme, not on an unambiguous ontological partition of acoustic conditions.

3. Architecture

The architecture has three components: a frozen Whisper encoder, a learnable Transformer prediction network, and task-specific attention-pooling classifier heads (Ravenscroft et al., 29 Jul 2025).

The backbone is the encoder from the small Whisper model. The paper motivates this choice by noting that three of the five tasks are speech-specific: multispeaker detection, foreign-language detection, and synthetic-speech detection. Audio is padded or truncated to 30 s; with 16 kHz input, the Whisper encoder yields a fixed sequence of 1500 frames. Rather than using only the final encoder layer, the model learns a weighted sum of the 12 intermediate transformer-layer outputs. The 12 scalar layer weights are trainable, while the Whisper encoder parameters remain frozen. This produces a fixed $1500 \times 768$ feature sequence.

On top of these features, Whilter trains a 4-layer self-attentive Transformer $\mathcal{T}$ with 4 attention heads:

$$\mathcal{T}: \mathbb{R}^{1500 \times 768} \mapsto \mathbb{R}^{1500 \times 256}.$$

This shared trunk is intended to adapt the frozen foundation-model features to the five filtering tasks.

Given the shared output

$F \in \mathbb{R}^{1500 \times 256},$

each class $n \in \{1,\ldots,N\}$ has its own attention-pooling head $x \in \mathbb{R}^L$0. Each head applies a linear layer, ReLU, dropout, another linear layer, and then a softmax over time to produce frame-level attention weights. These are used to compute a weighted temporal summary of $x \in \mathbb{R}^L$1. A final linear layer reduces the pooled 256-dimensional vector to a scalar score. The head also includes a residual connection from the time-mean of $x \in \mathbb{R}^L$2 to stabilize gradients. The task outputs are stacked into

$x \in \mathbb{R}^L$3

Training uses binary cross-entropy over the multi-label output vector:

$x \in \mathbb{R}^L$4

The paper does not report more elaborate task weighting in the objective. Instead, imbalance is handled through weighted random sampling during training. It also evaluates with FPR, FNR, EER, precision, recall, and F1, but does not provide a formula for EER, discuss calibration procedures beyond standard thresholded classification, or report explicit class-specific decision thresholds.

4. Data resources and optimization procedure

The training strategy has two stages: pretraining on automatically labeled non-in-the-wild data with synthetic mixing, followed by fine-tuning on a manually annotated in-the-wild subset released as the AITW dataset (Ravenscroft et al., 29 Jul 2025).

For stage 1, labels are derived from curated corpora and enriched by dynamic mixing. A randomly selected quarter of each batch is artificially mixed with speech; similar mixing is applied with noise and music. Music, speech, and noise are combined at randomly chosen SNR/SSR values between $x \in \mathbb{R}^L$5 and $x \in \mathbb{R}^L$6 dB. Because the mixing processes are independent across class types, some synthetic training examples carry multiple simultaneous positive labels.

The non-ITW sources are selected to cover the target classes. AMI and AliMeeting provide multispeaker and single-speaker speech, including summed headset and single distant microphone configurations and both overlapped and non-overlapped utterances. Foreign-language labels come from subsets of MLS, AliMeeting, FOR, and MUSAN. Synthetic-speech labels come from FOR and ODSS. Background-music labels come from MUSAN and OpenMic-2018. Noise and noisy-speech examples come from MUSAN, FSDNoisy-18k, DEMAND, AMI, and AliMeeting. “Classless” clean near-field single-speaker English utterances come from LibriSpeech 960, AMI, and MUSAN. The paper does not enumerate exact hours or clip counts contributed by each source corpus.

AITW is built from random subsets of Emilia and YODAS and serves as the main supervised in-the-wild resource. Two experienced speech annotators labeled the data using detailed guidelines and a custom Label Studio interface. The total set contains 21,414 samples, about 64 hours in all: 18,346 samples for fine-tuning (approximately 55 h), 1,353 for validation (approximately 4 h), and 1,716 for testing (approximately 5 h). The paper remarks that synthetic speech is by far the sparsest class.

Training details are comparatively specific. The optimizer is Adam with an exponentially decaying learning-rate schedule. In both stages, 15,000 samples are selected per epoch using weighted random sampling, with batch size 64 and 235 iterations per epoch. Stage 1 runs for 10 epochs with learning rate $x \in \mathbb{R}^L$7 and decay factor $x \in \mathbb{R}^L$8. Stage 2 runs for 100 epochs with $x \in \mathbb{R}^L$9 and $N$0. During fine-tuning, dynamic mixing is replaced by data augmentation: frequency dropping, frame dropping, bit-resolution reduction, sign flipping, and speed perturbation. The paper explicitly notes that these augmentations were not counted as “noise” under the annotation guidelines.

5. Empirical performance and computational profile

The main empirical pattern is asymmetrical across tasks: Whilter is strongest on speech-specific tasks and weaker than BEATs on some more general audio tasks (Ravenscroft et al., 29 Jul 2025).

Task	Whilter	Best comparison reported
Multispeaker	EER 6.7%, F1 86.7%	Pyannote: EER 14.5%, F1 73.9%
Foreign language	EER 6.5%, F1 87.7%	Whisper small: EER 6.5%, F1 81.1%
Synthetic speech	EER 31.7%, F1 45.6%	Fine-tuned AASIST: EER 33.5%, F1 8.2%
Background music	EER 7.8%, F1 77.9%	BEATs: EER 5.1%, F1 87.8%
Noise	EER 25.6%, F1 63.2%	BEATs + TAN: EER 21.3%, F1 69.3%

On multispeaker detection, Whilter is clearly best among the reported systems: FPR 4.4%, FNR 9.0%, EER 6.7%, precision 82.8%, recall 91.0%, and F1 86.7%. The paper emphasizes that, compared with Pyannote, this improves EER by 7.8 percentage points and F1 by 12.8 points while being over 10 times faster on average.

On foreign-language detection, it reaches FPR 1.5%, FNR 11.6%, EER 6.5%, precision 86.9%, recall 88.4%, and F1 87.7%. Whisper small matches the EER at 6.5% but is lower in F1 at 81.1%, while Whisper large is slightly worse at EER 7.3% and F1 80.4%. BEATs and BEATs + TAN collapse to near-all-negative predictions on this task, both with EER 50% and F1 0.0%. The paper treats this as one of the clearest validations of the choice of a speech-specific foundation model.

On synthetic-speech detection, the task remains difficult, but Whilter still outperforms the alternatives listed. It obtains FPR 0.5%, FNR 62.9%, EER 31.7%, precision 59.1%, recall 37.1%, and F1 45.6%. Untuned AASIST yields EER 50.4% and F1 3.6%; fine-tuned AASIST improves recall to 60.0% but still trails sharply in precision and F1. BEATs and BEATs + TAN again fail, both with EER 50% and F1 0.0%.

The music and noise tasks reverse part of this pattern. For background music, Whilter is much better than inaSpeechSegmenter but behind BEATs and BEATs + TAN. For noise, it is competitive with BEATs but not superior to BEATs + TAN. The paper uses these contrasts as an implicit encoder-choice ablation, suggesting that speech-specific representations matter greatly for foreign-language and synthetic-speech detection, while general audio representations help more for music and noise.

Runtime is a central practical result. Mean processing time per test sample on an Nvidia A10 GPU is 0.033 s for Whilter, versus 0.025 s for BEATs and 0.028 s for BEATs + TAN, but far below the separate single-task baselines: Pyannote 0.479 s, DNSMOS+LR 0.453 s, inaSpeechSegmenter 0.524 s, Whisper small 0.047 s, Whisper large 0.239 s, and AASIST 0.047 s. Since Whilter solves all five tasks at once, the paper argues that it is substantially more efficient than chaining specialist models. Its parameter count is 91.1M including frozen and trainable parts.

6. Limitations, interpretation, and downstream significance

Whilter is explicitly framed as a front-end corpus filter rather than as a complete data-quality solution (Ravenscroft et al., 29 Jul 2025). Its strengths are speed, breadth, and the ability to unify multiple filtering decisions in one model, but the reported limitations are substantial and uneven across tasks.

The most prominent limitation is synthetic speech. The class is sparse, difficult to annotate, and adversarial to advances in TTS and voice conversion, which aim for realism. Noise is another weak point because the label is partly subjective, especially for reverberation and stationary noise. The authors suggest future work with more annotators and majority voting. The paper also notes that inaSpeechSegmenter often misses music or noise when they overlap with speech, effectively classifying such clips as speech only; this contextualizes some baseline failures but does not eliminate the broader issue that task difficulty depends heavily on overlap conditions.

The ablation evidence is also limited. There is no formal comparison between multitask and single-task training using the same Whilter architecture, no study of attention pooling versus mean pooling, and no analysis of varying the number of tasks or fine-tuning the Whisper encoder. The paper explicitly acknowledges that multitask-versus-single-task evaluation remains future work. More broadly, although the results indicate some degree of domain generalization—from simulated mixtures on non-ITW corpora to AITW from Emilia and YODAS—the paper does not include cross-corpus holdout experiments where training and testing are separated by source corpus.

The deployment interpretation is therefore selective rather than universal. For English TTS, Whilter can remove non-English clips, multispeaker segments, and synthetic speech that would corrupt speaker consistency or linguistic fidelity, while flagging noisy or music-contaminated samples for enhancement, separation, or exclusion. For monolingual ASR pretraining, it can reject foreign-language segments while tolerating some acoustic variability. For self-supervised representation learning, it may support looser retention policies while still excluding synthetic or multilingual contamination. The paper also notes that the model is tuned around English-versus-foreign filtering rather than a general multilingual labeling framework.

In that sense, Whilter’s main technical contribution is not merely a new classifier but a particular formulation of in-the-wild speech filtering: one frozen speech foundation model, a lightweight shared Transformer, and task-specific attention pooling used to jointly score five utterance-level nuisance conditions. The release of the AITW dataset and the accompanying Label Studio GUI extends that contribution from model design to supervised resource creation, making the work relevant both as a filtering system and as a benchmark for speech-corpus curation.