Where Do Self-Supervised Speech Models Become Unfair?

Abstract: Speech encoder models are known to model members of some speaker groups (SGs) better than others. However, there has been little work in establishing why this occurs on a technological level. To our knowledge, we present the first layerwise fairness analysis of pretrained self-supervised speech encoder models (S3Ms), probing each embedding layer for speaker identification (SID) automatic speech recognition (ASR). We find S3Ms produce embeddings biased against certain SGs for both tasks, starting at the very first latent layers. Furthermore, we find opposite patterns of layerwise bias for SID vs ASR for all models in our study: SID bias is minimized in layers that minimize overall SID error; on the other hand, ASR bias is maximized in layers that minimize overall ASR error. The inverse bias/error relationship for ASR is unaffected when probing S3Ms that are finetuned for ASR, suggesting SG-level bias is established during pretraining and is difficult to remove.

• The paper demonstrates that speaker group bias emerges early during SSL pretraining and remains entrenched across all network layers.
• It employs lightweight linear probes to quantify disparities in ASR and SID, uncovering inverse fairness–performance relationships across tasks.
• Even advanced fine-tuning and fairness algorithms fail to mitigate entrenched bias, underscoring the need for pretraining-focused interventions.

Layerwise Bias Emergence in Self-Supervised Speech Models

Introduction

Self-supervised speech models (S3Ms) have become foundational for Automatic Speech Recognition (ASR) and Speaker Identification (SID), consistently delivering advances in downstream accuracy and efficiency. Yet, persistent disparities in performance for specific speaker groups (SGs) — including non-native speakers, children, and women — highlight a reproducible and substantial fairness gap that remains inadequately addressed in the literature. The paper "Where Do Self-Supervised Speech Models Become Unfair?" (2604.18249) delivers a thorough, quantitative layerwise analysis of leading S3Ms, directly probing how bias against certain SGs emerges and evolves across network depth, and examining the effect of fine-tuning protocols and fairness-oriented algorithms.

Methodology

The core analytic framework centers on lightweight linear probes fitted at each embedding layer, separately for SID and ASR. These probes are trained and evaluated with established, demographically balanced corpora: Sonos Voice Control Bias Assessment Dataset and Meta’s Fair-Speech. The target metrics are (a) overall error rate for each task, and (b) the relative error rate (i.e., difference between an SG’s error and the population mean), forming the primary bias quantification mechanism. This approach, by freezing pre-trained features and using minimal decoders, ensures that observed disparities are attributable to the S3M itself, not the probing head or downstream classifier.

Emergence and Evolution of Layerwise Bias

The analysis demonstrates that SG-level performance disparities are present from the earliest latent layers in all tested encoder models, both for SID and ASR. However, the demographic groups most strongly affected diverge across tasks: SID exhibits heightened bias for children and women, while ASR most severely disadvantages non-native speakers and children.

Figure 1: Layerwise evolution of relative error rate for native/non-native speakers (solid lines) versus overall error (dotted), indicating early onset and monotonic increase of ASR bias against non-native speakers.

Figure 2: Age-based relative error trajectories highlight children (9–16) as worst-modeled SG in both SID and ASR.

Figure 3: For SID, women exhibit significantly worse relative error across most network depths, with negligible gender disparity for ASR.

Figure 4: Dialectal bias evolution for American English speakers, with strong layerwise patterns particularly in ASR.

These results are robust across diverse S3M architectures (WavLM, wav2vec 2.0, BEST-RQ), pretraining languages (English, multilingual, French), and probe datasets.

Bias–Performance Relationship: Divergence in SID and ASR

A critical and counterintuitive finding is the contrasting relationship between overall performance and SG bias for SID and ASR.

Figure 5: Each dot is a layer for each pretrained S3M (SID: top; ASR: bottom two rows), plotted as bias vs. error. For SID, bias is minimized at the performance peak; for ASR, bias is maximal at the performance optimum.

• SID: Layers with highest SID accuracy exhibit minimal SG-level bias. As SID accuracy decays in deeper layers, bias grows, indicating that representational collapse for minority SGs is a function of overall SID degradation.
• ASR: In direct contrast, the network layers achieving lowest WER (i.e., best ASR performance) display highest bias against disadvantaged SGs. Bias grows monotonically with improvements in ASR accuracy, suggesting that optimizing for average ASR fails to address or even exacerbates equity gaps.

These relationships are persistent across tasks, speaker group variables (age, gender, dialect, nativeness), model families, and datasets.

Impact of Fine-Tuning and Fairness Algorithms

S3Ms were further probed after ASR fine-tuning (standard CTC and fairness-enhanced DET/DAT variants). The layerwise trajectories of both overall error and SG bias remain virtually unchanged compared to those in the pretrained state.

Figure 6: ASR fine-tuning (dots: WER, dash-dot: SG-level bias) delivers substantial WER improvements but does not reduce demographic bias versus the pretrained baseline.

Even advanced fairness-promoting techniques such as domain adversarial or enhancing training inject negligible modification to bias magnitudes, especially in the layers delivering the best task performance. This supports the claim that bias is “baked in” during SSL pretraining, not effectively removable via downstream adaptation.

Implications, Practical and Theoretical

These results bear significant implications:

• Source of unfairness: The persistence and early onset of bias, regardless of language, training corpus, model architecture, or target task, indicate that demographic disparities are encoded in the representations during SSL pretraining and reflect properties of the underlying data distributions. They are not merely artifacts of downstream fine-tuning or overfitting.
• Limits of fine-tuning: ASR fine-tuning and state-of-the-art fairness enhancements yield little impact on these foundational disparities, especially for structurally disadvantaged SGs, centering pretraining as the critical focal point for fairness intervention.
• Task-specificity of bias: The inverse correlation between fairness and optimality in ASR layers (a pattern inverted in SID) underlines that task objectives differentially align with representation fairness; opsimizing for average WER will not produce equitable outcomes.
• Design of future S3Ms and benchmarks: Effective bias mitigation will likely require pretraining objectives, augmentation, or corpora curation specifically targeted at demographic equity, rather than relying on generic aggregate metrics.

Prospects for Future Research

The study foregrounds the utility of layer-level and probe-based diagnostics for bias monitoring and underlines gaps in the use of aggregate fairness metrics, advocating for more granular experimental design. Further research should:

• Investigate the causal mechanisms linking embedding geometry to demographic bias, possibly via representation disentanglement or identifiability constraints.
• Move from coarse SG labels to more localized, individual-level or utterance-derived measures (e.g., dialect density, cadence).
• Integrate insights from intersectionality theory to address compounding disadvantages faced by multi-minority SGs.
• Develop pretraining procedures with explicit fairness regularization, possibly leveraging adversarial objectives, data balancing informed by clustering, or direct estimation of SG-level performance in the loss landscape.

Conclusion

The analysis in "Where Do Self-Supervised Speech Models Become Unfair?" (2604.18249) produces strong empirical evidence that SG-level bias in S3Ms is an intrinsic byproduct of pretraining, manifesting in all network layers and resistant to downstream debiasing. The bias–performance coupling is sharply task-dependent, with fairness and optimality coinciding in SID, but fundamentally diverging in ASR. Achieving equitable speech technologies therefore demands refocusing fairness research on the pretraining phase, the development of demographically robust SSL objectives, and the creation of benchmarks and diagnostic tools capable of revealing and mitigating unfairness at the representational level.