What You Train Is What You Get: Gender Bias, Training Composition, and Post-Hoc Mitigation in Audio Deepfake Detection

Abstract: Audio deepfake detection models determine whether speech is genuine or artificially generated, but high overall accuracy can mask substantial performance disparities across demographic groups. In this work, we investigate gender bias in audio deepfake detection using the ASVspoof5 dataset. We use ASVspoof5 under a controlled custom split designed to isolate gender-composition effects. We train attack-specific models on nine training sets with different gender compositions, ranging from female-only to male-only. We use a ResNet18 classifier with LogSpectrogram and WavLM-Base+ features, and we evaluated six post-hoc threshold calibration methods. Experimental results show that training data composition strongly predicts bias direction, with the underrepresented gender performing worse at test time. WavLM-Base+ features are shown to produce gender performance gaps 3.0 to 4.3 times larger than LogSpectrogram under identical training conditions, and balanced training is found to reduce LogSpectrogram bias but leave WavLM bias largely intact. Moreover, all six calibration strategies, including Oracle calibration with full test-set label access, leave the Equal Error Rate gap unchanged at 1.317 pp, confirming that threshold adjustment cannot correct underlying score distribution disparities. Overall, these findings suggest that gender fairness in audio deepfake detection must be addressed at training time, as post-hoc methods can only partially mitigate the resulting disparities

• The paper investigates the impact of training gender balance and feature representation, comparing handcrafted LogSpectrograms with self-supervised WavLM-Base+ in audio deepfake detection.
• It finds that self-supervised features yield gender EER gaps 3–4.3× higher than handcrafted ones, with balanced training only marginally reducing bias.
• The study demonstrates that post-hoc threshold calibration cannot fully mitigate demographic bias, underscoring the need for deliberate training composition control.

Gender Bias, Training Composition, and Post-Hoc Mitigation in Audio Deepfake Detection

Introduction

This paper conducts a comprehensive analysis of gender bias in audio deepfake detection systems, leveraging the ASVspoof5 benchmark and a systematic manipulation of training data gender composition. The study's centerpiece is the controlled training of 384 attack-specific classifier models—across 32 spoofing attacks, nine explicitly constructed gender composition scenarios, and multiple feature representations. The work fundamentally interrogates the role of training set gender balance, the downstream effectiveness of post-hoc threshold calibration, and the differential behavior of handcrafted versus self-supervised feature representations in shaping fairness outcomes.

Experimental Framework

The experimental protocol is centered around the ASVspoof5 dataset, with training, development, and evaluation splits stratified along attack and gender axes to enable precise control of training gender composition. Each classifier—ResNet18 with either LogSpectrogram or WavLM-Base+ frontends—is trained in isolation on a single attack type, eliminating confounding effects from cross-attack variability. Nine gender configurations are systematically explored, ranging from female-only or male-only pools to granular mixtures (e.g., F+25%M) and fully balanced pools.

Figure 1: ASVspoof5 dataset protocol used in this study. (A01-A08 training, A09-A16 development, A17-A32 evaluation).

Input features are either conventional LogSpectrograms or contextualized representations from WavLM-Base+, the latter reflecting the state-of-the-art in self-supervised speech modeling. This bifurcation informs a rigorous comparison between information-theoretically distinct frontends. Six group fairness metrics—Equal Error Rate (EER) gap, False Positive Rate gap, Statistical Parity Difference (SPD), Equal Opportunity Difference (EOpD), Equalized Odds (EOD), Prediction Parity Difference (PPD), and the True/False Error Difference (TED)—enable nuanced and multidimensional fairness assessment.

Figure 2: Overview of the proposed system for gender fairness evaluation and post-hoc mitigation in audio deepfake detection.

Robustness is ensured through Poisson bootstrap significance tests and Benjamini-Hochberg corrections.

Findings: Feature Representation and Bias

The analysis demonstrates that feature representation is the primary determinant of bias magnitude. Under identical training composition, WavLM-Base+ models exhibit gender EER gaps 3–4.3× greater than LogSpectrogram counterparts. In balanced training, the residual bias for LogSpectrogram is nearly eliminated—EER gap drops to 0.063 pp—while WavLM-Base+ retains substantial disparity (0.273 pp, with 20/32 attacks significantly biased). These results implicate self-supervised representations as a persistent source of demographic encoding.

Notably, cross-attack evaluation reveals even greater resilience of bias to balanced training when using SSL features; LogSpectrogram bias is reduced but WavLM-Base+ remains robust to the effect, underscoring that demographic information acquired during SSL pre-training propagates and cannot be negated by balanced fine-tuning alone.

Gender Composition as a Determinant of Bias Direction

A core finding is the strict dependence of bias direction on the majority gender in the training composition—the underrepresented gender always suffers increased error rates, regardless of whether the minority is male or female, or the feature type employed. This is evident in 31 of 32 attacks. Monotonicity is observed: gradually increasing minority gender proportion incrementally smooths (but does not abrogate) the EER gap, with diminishing marginal returns as parity is approached. However, complete attenuation of bias generally requires global, not just within-attack, gender balancing.

Figure 3: EER$_{\text{gap}}$ under same-attack and cross-attack evaluation across all nine gender-composition configurations for TTS, VC, and AT attacks.

Attack-level bias shows modality dependence: VC attacks produce the most pronounced gender gaps under same-attack evaluation due to their propensity to preserve target speaker attributes. Contrastingly, adversarial perturbation attacks display the lowest EER gap but accentuate SPD parity issues, demonstrating the non-congruence of fairness metrics.

Threshold Calibration and the Limits of Post-Hoc Mitigation

A comprehensive exploration of six post-hoc threshold calibration strategies—including Oracle calibration utilizing full test labels—reveals that these methods are fundamentally incapable of reducing the EER gap. Adjustment of decision thresholds, even with perfect label knowledge, cannot resolve underlying score distribution shifts between demographic groups. This validates the theoretical impossibility of achieving simultaneous group fairness, calibration, and predictive parity when group base rates differ.

The analysis highlights that calibration strategies do selectively address specific metrics (e.g., TC_FPR minimizes FPR_gap, TC_TPR aligns TPR rates), but these gains are realized only with explicit trade-offs in other fairness dimensions. Counterintuitively, per-gender EER calibration, effective under balanced training, degrades equalized odds under imbalanced gender conditions.

Cross-Attack Generalization and Metric-Orthogonality

Evaluation on unseen attack types compresses the range of observed EER gaps and disrupts monotonic bias trends. Importantly, the identity of the disadvantaged gender can invert between same- and cross-attack settings; this points to strong threshold calibration sensitivity to attack-conditioned score distributions. Fairness verdicts at the attack level can be metric-dependent, and the study demonstrates that selection of fairness metric alone can flip the fairness categorization of nearly 16% of attack types.

Figure 4: Fairness metric heatmap across TTS, VC, and AT attacks under same-attack (left) and cross-attack (right) evaluation across all nine gender-composition configurations.

Strong inter-metric correlations are found among EOpD, EOD, PPD, and FPR_gap, while SPD and TED exhibit orthogonality with other metrics, indicating that a single-facet assessment is insufficient for deployment assurance.

Practical Implications and Certification Guidelines

The findings translate directly into recommended practices for deployment in fairness-sensitive contexts:

• Explicit reporting of per-gender EER and multiple group fairness metrics per attack type.
• Standardized inclusion of training gender composition metadata.
• Context-driven selection of threshold calibration strategy: TC_FPR (login/biometric scenarios), TC_TPR (fraud detection), and avoidance of TC_EER under unbalanced training.
• Recognition of the non-applicability of post-hoc calibration for resolving distributional EER disparities.
• Current results are limited to binary gender labels per ASVspoof5 protocol and may not generalize to multi-class identity spaces or speaker-independent splits.

Limitations and Future Directions

This work is constrained by focus on two feature classes, fixed ResNet18 backend, explicit training/evaluation splits, and absence of training-time debiasing or adversarial interventions. Extension to SSL architectures beyond WavLM, speaker-independent evaluations, and intersectional demographic analyses (e.g., non-binary identities) are high-priority avenues.

Conclusion

Gender fairness in audio deepfake detection is dictated primarily by training composition and feature representation, not post-hoc calibration. Self-supervised representations, exemplified by WavLM-Base+, encode persistent demographic information that fine-tuning cannot nullify, and naive post-hoc threshold calibration is mathematically insufficient to correct ensuing performance gaps. In consequence, establishing equitable protection in audio deepfake countermeasures requires deliberate control of demographic composition at training time, and deployment protocols must embrace multidimensional group fairness reporting and context-sensitive calibration strategy selection.

---

References

All insights and quantitative results are from "What You Train Is What You Get: Gender Bias, Training Composition, and Post-Hoc Mitigation in Audio Deepfake Detection" (2607.09891).