Towards Trustworthy Audio Deepfake Detection: A Systematic Framework for Diagnosing and Mitigating Gender Bias

Abstract: Audio deepfake detection systems are increasingly deployed in high-stakes security applications, yet their fairness across demographic groups remains critically underexamined. Prior work measures gender disparity but does not investigate where it comes from or how to fix it systematically. We present the first diagnosis-first framework that identifies bias source before applying targeted mitigation, evaluated on two models, AASIST and Wav2Vec2+ResNet18, on ASVSpoof5. Our diagnosis shows that bias does not stem from imbalanced training data but from acoustic representation differences, gender leakage in learned features, and structural evaluation asymmetry. We test mitigation strategies across in-processing, post-processing and combined families, including novel methods introduced in this work. Adjusting the decision threshold separately per gender reduces unfairness by 54% to 75% at no cost to detection accuracy, and our new epoch-level fairness regularisation method outperforms existing per-batch approaches. Adversarial debiasing succeeds only when gender leakage is localised, and fails when it is diffuse, an outcome correctly predicted by our diagnosis before training. No single method fully closes the fairness gap, confirming that bias sources must be identified before fixes are applied and that fairer benchmark design is equally important

• The paper introduces a diagnosis-first fairness framework that identifies bias at the data, model, and decision levels in audio deepfake detection.
• It demonstrates that per-group threshold calibration reduces dFPR by 54–75% without degrading EER, offering a robust post-processing solution.
• The study reveals that ad hoc interventions like sample reweighting can worsen performance, highlighting the need for targeted, source-specific bias remedies.

Diagnosis-First Fairness Framework for Audio Deepfake Detection

Context and Motivation

Audio deepfake detectors are now integral to security-critical systems, yet the demographic fairness of such models, especially regarding gender, remains insufficiently analyzed. The literature indicates persistent performance gaps between male and female speakers across multiple architectures and benchmarks, but mitigation strategies have largely lacked diagnostic rigor. The examined paper proposes a systematic, diagnosis-first framework for identifying and mitigating the sources of gender bias in state-of-the-art audio deepfake detection pipelines, with empirical validation on the ASVSpoof5 corpus using two modern architectures: AASIST (spectro-temporal graph attention network) and Wav2Vec2-large+ResNet18 (self-supervised front-end with CNN classifier) (2605.09087).

Bias Source Diagnosis at Data, Model, and Decision Levels

The framework’s initial stage consists of targeted diagnostic checks at three granularity levels: data, model, and decision.

Data-level analysis confirms that, contrary to prior assumptions, the ASVSpoof5 training partition is gender-balanced, rendering standard sample-balancing interventions like reweighting not just ineffective but detrimental to performance. However, the evaluation protocol exhibits significant gender-based class imbalance and attack distribution shift—factors not correctable via model- or data-centric methods—highlighting a confound due to benchmark design rather than model bias per se.

Model-level diagnostics employ score distribution analysis, embedding leakage evaluation, and SHAP-based attribution to localize gender information in learned representations. Both examined architectures encode gender information to a measurable extent, but Model 1 exhibits localized leakage (restrictable), while Model 2's leakage is diffuse, implying that representation-level debiasing is likely to yield diminishing returns for Wav2Vec2-based models.

Decision-level checks reveal substantial disparities in optimal decision thresholds between genders. Binary cross-entropy loss, even with balanced training data, induces asymmetric FPRs and optimal thresholds, thus necessitating post-processing interventions.

Results underscore that gender bias in detection is multi-causal: driven by protocol asymmetry, non-overlapping attack distributions, representation leakage, and threshold miscalibration, not by training set imbalance.

Targeted Mitigation Strategies

Eight mitigation strategies spanning pre-processing, in-processing, and post-processing families were systematically evaluated.

• Sample Reweighting (Pre-processing): Confirmed as detrimental in balanced data regimes, increasing both EER and fairness metric disparities.
• Fairness-Aware Loss and EAFR (In-processing): Adding batch-based group fairness regularization yields unstable improvements due to batch noise. Accumulating fairness penalties at the epoch level (EAFR) stabilizes training, modestly reducing EER gap versus standard stochastic regularization, with improved reliability.
• Adversarial Debiasing (In-processing): Using GRL-based adversarial training only improves fairness where gender leakage is concentrated. Model 1 (AASIST) benefits modestly; for Model 2 (diffuse leakage), adversarial regularization increases EER with negligible fairness gain, confirming the need for leakage structure-aware intervention selection.
• Threshold Calibration (TC) (Post-processing): Per-gender thresholding based on development EER yields 54–75% reduction in dFPR with no loss in detection accuracy for both architectures. This is the most robust and implementation-agnostic mitigation; it directly addresses decision-level bias regardless of underlying model feature structure.
• SHAP-Guided Feature Suppression and Gender-Neutral Embedding Alignment (Post-processing): Both approaches selectively suppress or align SHAP-identified gender-sensitive embedding dimensions. For AASIST, where sensitive dimensions are localized, these methods reduce disparity, especially when combined with threshold calibration. For Wav2Vec2-based models, effects are marginal, again due to diffuse leakage.
• Combined Pipelines: Joint mitigation, pairing in-processing regularization with TC, further improves fairness but does not fully eliminate the fairness gap, emphasizing the necessity for comprehensive intervention design covering multiple, independently-acting bias sources.

Empirical Results

Key findings include:

• Sample reweighting increases fairness gaps and worsens overall performance in gender-balanced data regimes, highlighting the hazards of misapplied fairness interventions.
• Threshold calibration alone consistently yields the largest reduction in dFPR (54-75%) with no EER degradation. No retraining is required, demonstrating practical efficacy.
• Adversarial debiasing is only beneficial when feature leakage is sufficiently localized. SHAP attribution is necessary prior to adoption.
• EAFR is more stable than per-batch fairness losses and shows improved gap reduction, especially when batch-level estimates are noisy.
• No method eliminates bias fully. Residual bias primarily originates from evaluation protocol design, not model behavior—implicating structural changes to benchmarks as a critical area of future research.

Implications and Future Work

The proposed two-stage framework formalizes the principle that mitigation must be preceded by granular, source-specific diagnosis. Intervention without confirmed etiology can degrade utility and fairness, particularly in balanced or well-controlled data regimes.

Practical implications include the recommendation to adopt per-group threshold calibration as a default intervention in fairness-critical audio verification and deepfake detection deployments. For architectures with localizable leakage, post hoc embedding manipulation via SHAP-based suppression or alignment offers additional incremental improvement but is not universally applicable.

Theoretical implications involve the limitations of data-centric or model-agnostic mitigation. Fairest outcomes require joint consideration of benchmark design and post-processing interventions aligned to empirically identified bias loci. Residual bias attributable to protocol asymmetries invites a re-examination of how future benchmarks are constructed, especially regarding group balance and attack-type coverage in evaluation splits.

Limitations of the present work include the focus on binary gender and a single benchmark. The framework generalizes, but cross-benchmark and multi-group analyses, as well as adaptation to non-binary speaker labels, constitute important next steps. Methods for addressing protocol-induced bias, possibly via re-engineering evaluation splits or incorporating demographic stratification into protocol design, are needed.

Conclusion

This study defines a principled, diagnosis-driven framework for analyzing and reducing gender bias in audio deepfake detection. By combining comprehensive multi-level diagnostics with a systematic suite of mitigation strategies and validating on two modern architectures with distinct embedding characteristics, it establishes threshold calibration as a robust, low-cost fairness intervention, challenges the efficacy of commonly used ad hoc methods, and attributes much residual disparity to benchmark protocol limitations. The framework’s generality positions it for adoption in future fairness-aware detection system evaluation and design (2605.09087).