Data Selection Effects on Self-Supervised Learning of Audio Representations for French Audiovisual Broadcasts

Abstract: Audio and speech self-supervised encoder models are now widely used for a lot of different tasks. Many of these models are often trained on clean segmented speech content such as LibriSpeech. In this paper, we look into how the pretraining datasets of such SSL (Self-Supervised Learning) models impact their downstream results. We build a large pretraining corpus of highly diverse TV and Radio broadcast audio content, which we describe with automatic tools. We use these annotations to build smaller subsets, which we use to train audio SSL models. Then, we evaluate the models on multiple downstream tasks such as automatic speech recognition, voice activity and music detection, or speaker recognition. The results show the potential of pretraining SSL models on diverse audio content without restricting it to speech. We also perform a membership inference attack to evaluate the encoder ability to memorize their training datasets, which highlight the importance of data deduplication. This unified training could bridge speech and music machine learning communities.

• The paper demonstrates that deduplication and content balancing in French broadcast data significantly improve SSL audio representation, yielding up to 16.1% WER improvement in ASR.
• It employs six controlled pretraining subsets on a 100,000-hour dataset and evaluates downstream tasks including voice activity detection and music detection with precise performance metrics.
• The study also highlights privacy risks from data duplication, showing that artificial duplicates trigger significant membership inference attacks compared to deduplicated data.

Data Selection and Its Effects on SSL Audio Representations for French Audiovisual Broadcasts

Introduction

This paper presents a comprehensive empirical study of how the acoustic and demographic diversity, deduplication, and content balancing of French audiovisual pretraining corpora impact self-supervised learning (SSL) models for speech and audio representation. A 100,000-hour deduplicated and richly-annotated corpus from Institut National de l'Audiovisuel (INA), covering decades of TV and radio broadcasts, underpins controlled pretraining of six data2vec2 variant models. Extensive downstream evaluations probe ASR, VAD, music detection, speaker recognition, and privacy leakage through membership inference attacks. The work extends prior investigations into self-supervised speech modeling beyond curated audiobooks or "clean" speech corpora, systematically interrogating the intersection of pretraining data composition and transfer across audio domains.

Corpus Construction and Annotation

The INA archive, encompassing more than 473,000 hours of broadcast content spanning 1940–2022, serves as the foundational dataset. Critical preprocessing includes aggressive deduplication—32.6% of initial data is excluded using robust, fingerprint-based methods—followed by the removal of all segments found in downstream evaluation benchmarks to preclude data contamination.

Figure 1: The temporal evolution of the INA corpus shows profound increases in archived broadcast material post-1995, aligning with legal deposition mandates.

12 million 30-second segments are sampled post-deduplication to construct a 100,000-hour corpus (21% of the original source data). Large-scale automatic annotation is employed, featuring Whisper for text and language ID, InaSpeechSegmenter for voice and gender, and a custom music detection tool trained on music2vec embeddings. Statistics reveal that 72.51% contain speech, 55.23% music, and 91.69% are French, with males delivering 70.05% of all speech. Annual gender distribution is made explicit.

Figure 2: Annualized breakdown of speech time by gender, highlighting a persistent male dominance in the dataset.

Controlled Pretraining Corpora and SSL Model Training

From the 100,000-hour annotated set, six 1,000-hour pretraining subsets are systematically curated to isolate effects of:

• Unfiltered (base)
• No music
• "Speech only"
• French-only
• Gender-balanced speech
• Artificial duplicates (with 1% segments appearing 10×)

All models employ a consistent data2vec2 architecture (93.2M parameters, 12 transformer layers, student-teacher paradigm), pretraining for 100k steps. This design enables controlled ablation on content and demographic factors.

Downstream ASR and Audio Classification Evaluations

Automatic Speech Recognition

All SSL models significantly outperform a wav2vec2 baseline trained on clean audiobooks for French, with up to 16.1% absolute WER improvement. Removing music from pretraining data yields the lowest WER; conversely, inclusion of duplicated segments deteriorates performance, aligning with recent findings in NLP that redundancy induces harmful memorization and hurts generalization. Results dissected by speaker gender demonstrate persistent WER gaps, but gender-balanced pretraining marginally reduces these disparities, though often not at a statistically significant level.

Voice Activity Detection

VAD downstream models leverage only the CNN and first transformer block representations and are trained with a fixed MLP classifier (architecture overview below).

Figure 3: Downstream MLP classifier architecture used for both VAD and music detection leveraging early encoder representations.

Diverse, broadcast-sourced pretraining clearly outperforms traditional systems trained on isolated speech and music files, particularly on "difficult" categories such as generalist TV and music radio recordings. Models pretrained with maximal audio heterogeneity offer best-in-class VAD performance, confirming that exposure to overlapping and realistic acoustic scenes during SSL confers transferability for speech detection.

Music Detection

For music/no-music segmentation, SSL models pretrained on heterogeneous datasets (containing music and non-speech) substantially outperform speech-only models in F1-score. As expected, the music2vec baseline, trained exclusively on music content, slightly surpasses all, but generalist models come close. Notably, explicitly removing music from pretraining reduces downstream music detection F1 by over 2%.

Speaker Representation and Recognition

Speaker recognition evaluation utilizes VoxCeleb1/2 benchmarks with an attention-based backend. All INA-trained models (including those balanced for gender) achieve comparable EER and DCF1/DCF5 to strong wav2vec2 and WavLM English-pretrained baselines, despite the mismatch in training language. Backend analysis confirms prior observations that early encoder layers concentrate speaker identity cues, further motivating cross-domain SSL studies.

Membership Inference and Privacy

A key contribution is an empirical study of privacy risk due to memorization via membership inference attack (MIA). Pretraining with artificially duplicated examples permits an MIA to achieve 61.2% AUC for detecting such membership, confirming the multiplicative privacy leakage potential as shown for NLP models. Encoders trained on deduplicated data show only chance-level memorization (AUC ≈ 51.7%), reinforcing the necessity of aggressive deduplication for corpus construction.

Figure 4: Schematic of the downstream MIA classifier applied to frozen SSL encoders.

Figure 5: ROC curves for MIA demonstrating strong separation only on duplicated data and random-guess performance otherwise.

Implications and Future Directions

The results support several practical and theoretical implications:

• Generalist pretraining on highly varied audio produces SSL representations transferable to both speech and music tasks, narrowing traditional divides in speech/MIR research.
• Content deduplication is critical for privacy and generalization; duplicated data induces significant memorization, which is empirically quantifiable and not mitigated by model architecture alone.
• Balanced demographic sampling can marginally reduce systematic ASR bias, but source corpus structure (e.g., role assignments in broadcast) remains a cardinal factor.
• Downstream task selection must align pretraining data: music detection benefits from explicit musical exposure, while speech tasks gain from noise and music suppression.
• Open challenges include scaling unified SSL models to the entire deduplicated INA corpus and extending to multilingual and cross-modal settings without sacrificing transfer or exacerbating privacy risk.

Conclusion

The systematic analysis rigorously characterizes how data selection in SSL pretraining governs downstream audio performance, bias, and privacy. Pretraining on deduplicated, diverse, and openly annotated broadcast audio enables robust, generalizable embeddings for speech and music tasks, effectively bridging disciplinary divides. These findings reinforce deduplication and careful content curation as indispensable for both scientific rigor and responsible deployment of foundation audio models. Future work will scale toward larger unified models for the French audiovisual domain, further investigating data-centric and architecture-centric privacy controls.

(2604.09472)