Audity Dual-Branch Architecture
- The paper introduces Audity with a dual-branch design that fuses audio structure and generation artifacts to enable open-set deepfake source tracing.
- It employs two complementary branches: one captures the temporal dependencies of natural speech and the other detects generation-specific synthetic fingerprints.
- The fusion of these embeddings yields robust verification performance, achieving up to 89.87% accuracy and 10.02% EER on benchmark tests.
Audity is a dual-branch architecture introduced for the Audio Deepfake Verification (ADV) task, a formulation that recasts audio deepfake analysis from binary authenticity classification into open-set deepfake source tracing. Rather than answering only whether a sample is real or fake, ADV asks whether two audio samples were produced by the same deep fake method, including cases in which the method was unseen during training. Audity operationalizes this setting by combining two complementary evidence streams—audio structure and generation artifacts—into a joint embedding used for verification, while also supporting strong binary deepfake detection as a byproduct (Wang et al., 10 Sep 2025).
1. From binary detection to open-set method verification
The motivating claim behind Audity is that the traditional audio deepfake detection framing is no longer sufficient for practical forensic use. A binary classifier that outputs only “real or fake” cannot determine whether suspicious clips were produced by the same synthesis pipeline, and therefore cannot directly support attribution, investigation, or tracking of emerging unseen generators. The paper addresses this limitation by introducing Audio Deepfake Verification (ADV) as an open-set deepfake source tracing problem inspired by speaker verification (Wang et al., 10 Sep 2025).
In ADV, the fundamental query is pairwise rather than categorical: given two audio samples, the system decides whether they come from the same deep fake method. At test time, an unknown sample is compared with known samples, a similarity score is computed, and that score is compared against a threshold. If the score exceeds the threshold, the pair is judged to share the same generation method; otherwise it is judged different. This preserves inter-method discrimination instead of collapsing all synthetic audio into a single “fake” class.
A common misconception is to treat ADV as ordinary closed-set source classification. The paper explicitly distinguishes the two settings. Training is performed in a closed-set manner on known deepfake methods, but the task itself is open-set at test time. This distinction matters because the evaluation target is not membership in a fixed catalog of generators, but same-method versus different-method verification under method novelty.
2. Internal organization of the dual-branch architecture
Audity is designed around the premise that deepfake methods leave fingerprints in two distinct forms. One source of evidence lies in disturbances to the temporal consistency of natural speech structure—phoneme sequences, prosody, syllabic rhythm, pauses, and other long-range regularities. The other lies in low-level synthetic artifacts such as discontinuous high-frequency energy, odd silence patterns, breath/pause anomalies, and spectral distortions. The architecture therefore separates representation learning into an Audio Structural Branch (ASB) and a Generation Artifacts Branch (GAB), then fuses them into a single embedding (Wang et al., 10 Sep 2025).
| Branch | Input / backbone | Primary role |
|---|---|---|
| Audio Structural Branch (ASB) | mel-spectrogram; w2v-BERT 2.0 | Model global and mid-level structure of speech as a sequence |
| Generation Artifacts Branch (GAB) | spectrograms; instantiated with CAM++, ECAPA-TDNN, and ResNet293 | Learn generation-specific fingerprints from acoustic patterns |
| Fusion stage | Concatenated ASB and GAB outputs + fusion layer | Produce a joint representation for ADV |
The ASB uses w2v-BERT 2.0 as backbone. The paper emphasizes that this branch is not used merely for “semantic” language understanding; rather, it is intended to capture the temporal dependencies of language units that characterize normal speech production. With a mel-spectrogram as input and learning based on masked prediction objectives, the branch reconstructs intact speech structure from corrupted input. The resulting representation encodes “normative” human speech patterns, making deviations introduced by synthesis—such as unnatural prosodic breaks or phoneme misalignments—detectable.
The GAB is designed to capture synthetic cues that structural pretraining alone may not model well. It takes spectrograms as input in order to preserve rich raw audio information and learn generation-specific fingerprints directly from acoustic patterns. The paper states that this branch is intentionally flexible and can accommodate several state-of-the-art backbones; in experiments, the authors instantiate it with CAM++, ECAPA-TDNN, and ResNet293, implemented with the WeSpeaker toolkit for reproducibility. In the ablation description, the reported replacement result is given for ResNet239, which indicates a reporting discrepancy in the paper’s summarized details rather than a different architectural principle.
Another frequent misunderstanding is to assume that the complementary branches are redundant. The paper argues the opposite: one branch models the intrinsic structure that legitimate speech should preserve, and the other models the technical trace introduced by synthetic generation. Their separation is therefore functional rather than cosmetic.
3. Fusion, embeddings, and verification protocol
After branch-specific encoding, Audity combines the two outputs in a Feature Fusion and Embedding Generation stage. The outputs of ASB and GAB are concatenated and passed through a fusion layer, producing a joint representation intended to capture both the intrinsic structure of natural speech and the distinctive artifacts introduced by synthetic generation (Wang et al., 10 Sep 2025).
The paper explicitly does not introduce a complicated cross-attention fusion mechanism. Instead, it adopts a straightforward concatenation-and-fusion design. The significance of this design choice is empirical rather than rhetorical: the reported gains indicate that the two branches are sufficiently complementary that a simple fusion operator is already effective.
The verification protocol follows standard pairwise scoring logic. Verification pairs are constructed by randomly selecting same-type and different-type samples. During inference, an unknown sample is compared against known samples, a similarity score is produced, and a threshold determines the final same-method versus different-method decision. For threshold selection, the model with the lowest Equal Error Rate (EER) on the validation set is chosen, and the threshold corresponds to the EER of the validation set.
The paper frames optimization and evaluation in terms of verification scoring, similarity thresholds, and standard binary metrics over sample pairs. Reported metrics are EER, Accuracy (Acc), False Acceptance Rate (FAR), False Rejection Rate (FRR), F1, AUROC, plus FRR@FAR1% and FAR@FRR1%. When multiple enrollment samples exist for one method, their embeddings can be averaged, and the paper explicitly reports that the mean embedding over more enrollment and verification samples improves performance substantially. This suggests that Audity’s embedding space is stable enough to support method-level aggregation.
4. Training regime, data composition, and evaluation setting
The experimental setup is intentionally broad. Training is performed in two configurations: the official MLAAD SourceTrace protocol with 24 deepfake methods, and a larger corpus assembled from 10 open-source datasets. The total training set comprises 6,322,271 utterances and 6,150.60 hours, spanning 128 method types. The included datasets are ASVspoof2019-LA, CodecFake(UCAS), CodecFake+, DFADD, GigaSpeech(M), LibriSeVoc, MLAAD, SpoofCeleb, Wavefake, and ADD2023 Track3 (Wang et al., 10 Sep 2025).
Validation uses official validation splits and totals 260,961 utterances and 280.78 hours across 58 types. Testing is conducted on five sets: ASVspoof2019-LA, SpoofCeleb, ADD2023 Track3, a self-collected Demo Page dataset, and a Commercial TTS dataset. The Demo Page data include CosyVoice, CosyVoice2, E2TTS, FireRedTTS, Llasa, MaskGCT, SeedTTS, and SparkTTS, while the commercial set includes gpt-4o-mini-tts and Microsoft Azure TTS.
The paper also reports an epoch-level balancing strategy: to keep class balance, it samples 1000 samples per type at the start of each epoch. This is relevant because the underlying corpus spans many method types and heterogeneous datasets, so uncontrolled frequency skew could otherwise distort representation learning.
The scope of the evaluation is notable because it includes both legacy benchmark generators and contemporary systems from demo pages and commercial APIs. A plausible implication is that the paper is testing not only cross-dataset generalization but also robustness to generator recency and realism.
5. Ablation evidence and main performance results
The clearest evidence for the value of the dual-branch design comes from the ablation on MLAAD SourceTrace. With CAM++ + w2v-BERT 2.0 as the two branches, Audity achieves 89.87% accuracy and 10.02% EER, the best result among the tested combinations. Replacing CAM++ with ECAPA-TDNN gives **89.28% / 10.49