---
title: Audity Dual-Branch Architecture
url: https://www.emergentmind.com/topics/audity-dual-branch-architecture
type: topic
---

# Audity Dual-Branch Architecture

Audity is a dual-branch architecture introduced for the **Audio Deepfake Verification (ADV)** task, a formulation that recasts audio deepfake analysis from binary authenticity classification into **open-set deepfake source tracing**. Rather than answering only whether a sample is real or fake, ADV asks whether two audio samples were produced by the **same deep fake method**, including cases in which the method was unseen during training. Audity operationalizes this setting by combining two complementary evidence streams—**audio structure** and **generation artifacts**—into a joint embedding used for verification, while also supporting strong binary deepfake detection as a byproduct [2509.08476].

## 1. From binary detection to open-set method verification

The motivating claim behind Audity is that the traditional **audio deepfake detection** framing is no longer sufficient for practical forensic use. A binary classifier that outputs only “real or fake” cannot determine whether suspicious clips were produced by the same synthesis pipeline, and therefore cannot directly support attribution, investigation, or tracking of emerging unseen generators. The paper addresses this limitation by introducing **Audio Deepfake Verification (ADV)** as an **open-set deepfake source tracing** problem inspired by speaker verification [2509.08476].

In ADV, the fundamental query is pairwise rather than categorical: given two audio samples, the system decides whether they come from the **same deep fake method**. At test time, an unknown sample is compared with known samples, a similarity score is computed, and that score is compared against a threshold. If the score exceeds the threshold, the pair is judged to share the same generation method; otherwise it is judged different. This preserves inter-method discrimination instead of collapsing all synthetic audio into a single “fake” class.

A common misconception is to treat ADV as ordinary closed-set source classification. The paper explicitly distinguishes the two settings. Training is performed in a **closed-set** manner on known deepfake methods, but the task itself is **open-set at test time**. This distinction matters because the evaluation target is not membership in a fixed catalog of generators, but same-method versus different-method verification under method novelty.

## 2. Internal organization of the dual-branch architecture

Audity is designed around the premise that deepfake methods leave fingerprints in two distinct forms. One source of evidence lies in disturbances to the temporal consistency of natural speech structure—phoneme sequences, prosody, syllabic rhythm, pauses, and other long-range regularities. The other lies in low-level synthetic artifacts such as discontinuous high-frequency energy, odd silence patterns, breath/pause anomalies, and spectral distortions. The architecture therefore separates representation learning into an **Audio Structural Branch (ASB)** and a **Generation Artifacts Branch (GAB)**, then fuses them into a single embedding [2509.08476].

| Branch | Input / backbone | Primary role |
|---|---|---|
| **Audio Structural Branch (ASB)** | **mel-spectrogram**; **w2v-BERT 2.0** | Model **global and mid-level structure** of speech as a sequence |
| **Generation Artifacts Branch (GAB)** | **spectrograms**; instantiated with **CAM++**, **ECAPA-TDNN**, and **ResNet293** | Learn generation-specific fingerprints from acoustic patterns |
| **Fusion stage** | Concatenated ASB and GAB outputs + **fusion layer** | Produce a **joint representation** for ADV |

The **ASB** uses **w2v-BERT 2.0** as backbone. The paper emphasizes that this branch is not used merely for “semantic” language understanding; rather, it is intended to capture the **temporal dependencies of language units** that characterize normal speech production. With a **mel-spectrogram** as input and learning based on **masked prediction objectives**, the branch reconstructs intact speech structure from corrupted input. The resulting representation encodes “normative” human speech patterns, making deviations introduced by synthesis—such as unnatural prosodic breaks or phoneme misalignments—detectable.

The **GAB** is designed to capture synthetic cues that structural pretraining alone may not model well. It takes **spectrograms** as input in order to preserve rich raw audio information and learn generation-specific fingerprints directly from acoustic patterns. The paper states that this branch is intentionally flexible and can accommodate several state-of-the-art backbones; in experiments, the authors instantiate it with **CAM++**, **ECAPA-TDNN**, and **ResNet293**, implemented with the **WeSpeaker** toolkit for reproducibility. In the ablation description, the reported replacement result is given for **ResNet239**, which indicates a reporting discrepancy in the paper’s summarized details rather than a different architectural principle.

Another frequent misunderstanding is to assume that the complementary branches are redundant. The paper argues the opposite: one branch models the intrinsic structure that legitimate speech should preserve, and the other models the technical trace introduced by synthetic generation. Their separation is therefore functional rather than cosmetic.

## 3. Fusion, embeddings, and verification protocol

After branch-specific encoding, Audity combines the two outputs in a **Feature Fusion and Embedding Generation** stage. The outputs of ASB and GAB are **concatenated** and passed through a **fusion layer**, producing a **joint representation** intended to capture both the intrinsic structure of natural speech and the distinctive artifacts introduced by synthetic generation [2509.08476].

The paper explicitly does **not** introduce a complicated cross-attention fusion mechanism. Instead, it adopts a straightforward concatenation-and-fusion design. The significance of this design choice is empirical rather than rhetorical: the reported gains indicate that the two branches are sufficiently complementary that a simple fusion operator is already effective.

The verification protocol follows standard pairwise scoring logic. Verification pairs are constructed by randomly selecting same-type and different-type samples. During inference, an unknown sample is compared against known samples, a similarity score is produced, and a threshold determines the final same-method versus different-method decision. For threshold selection, the model with the **lowest Equal Error Rate (EER)** on the validation set is chosen, and **the threshold corresponds to the EER of the validation set**.

The paper frames optimization and evaluation in terms of **verification scoring**, **similarity thresholds**, and standard binary metrics over sample pairs. Reported metrics are **EER**, **Accuracy (Acc)**, **False Acceptance Rate (FAR)**, **False Rejection Rate (FRR)**, **F1**, **AUROC**, plus **FRR@FAR1%** and **FAR@FRR1%**. When multiple enrollment samples exist for one method, their embeddings can be averaged, and the paper explicitly reports that the **mean embedding** over more enrollment and verification samples improves performance substantially. This suggests that Audity’s embedding space is stable enough to support method-level aggregation.

## 4. Training regime, data composition, and evaluation setting

The experimental setup is intentionally broad. Training is performed in two configurations: the official **MLAAD SourceTrace** protocol with **24 deepfake methods**, and a larger corpus assembled from **10 open-source datasets**. The total training set comprises **6,322,271 utterances** and **6,150.60 hours**, spanning **128 method types**. The included datasets are **ASVspoof2019-LA**, **CodecFake(UCAS)**, **CodecFake+**, **DFADD**, **GigaSpeech(M)**, **LibriSeVoc**, **MLAAD**, **SpoofCeleb**, **Wavefake**, and **ADD2023 Track3** [2509.08476].

Validation uses official validation splits and totals **260,961 utterances** and **280.78 hours** across **58 types**. Testing is conducted on five sets: **ASVspoof2019-LA**, **SpoofCeleb**, **ADD2023 Track3**, a self-collected **Demo Page dataset**, and a **Commercial TTS dataset**. The Demo Page data include **CosyVoice, CosyVoice2, E2TTS, FireRedTTS, Llasa, MaskGCT, SeedTTS, and SparkTTS**, while the commercial set includes **gpt-4o-mini-tts** and **Microsoft Azure TTS**.

The paper also reports an epoch-level balancing strategy: to keep class balance, it samples **1000 samples per type** at the start of each epoch. This is relevant because the underlying corpus spans many method types and heterogeneous datasets, so uncontrolled frequency skew could otherwise distort representation learning.

The scope of the evaluation is notable because it includes both legacy benchmark generators and contemporary systems from demo pages and commercial APIs. A plausible implication is that the paper is testing not only cross-dataset generalization but also robustness to generator recency and realism.

## 5. Ablation evidence and main performance results

The clearest evidence for the value of the dual-branch design comes from the ablation on **MLAAD SourceTrace**. With **CAM++ + w2v-BERT 2.0** as the two branches, Audity achieves **89.87% accuracy** and **10.02% EER**, the best result among the tested combinations. Replacing CAM++ with **ECAPA-TDNN** gives **89.28% / 10.49

Source: https://www.emergentmind.com/topics/audity-dual-branch-architecture