NE-PADD: Semantic-Aware Audio Deepfake Detection

• NE-PADD is a semantic-aware method that integrates SpeechNER to perform frame-level audio deepfake detection by localizing manipulated segments.
• It introduces two attention aggregation mechanisms—Attention Fusion and Attention Transfer—to combine acoustic features with named entity cues effectively.
• Evaluated on PartialSpoof-NER, NE-PADD achieves state-of-the-art performance with a significant reduction in Equal Error Rate compared to traditional methods.

Searching arXiv for the specified paper and closely related context. NE-PADD is a method for partial audio deepfake detection (PADD) that incorporates named entity knowledge into frame-level spoof localization through attention aggregation. Unlike traditional utterance-level audio deepfake detection, which assigns a single bona fide-or-spoof label to an entire clip, PADD seeks a binary decision for each frame, enabling temporal localization of manipulated regions. NE-PADD addresses this problem with a two-branch architecture comprising a Speech Named Entity Recognition (SpeechNER) branch and a PADD branch, and introduces two mechanisms—Attention Fusion (AF) and Attention Transfer (AT)—to inject named entity semantics into spoof detection. The method is evaluated on PartialSpoof-NER, an extension of PartialSpoof with named entity annotations, and the reported results show that integrating named entity information improves frame-level detection performance (Xian et al., 4 Sep 2025).

1. Problem formulation and motivating assumptions

Traditional audio deepfake detection is typically utterance-level: the model receives a full audio clip and outputs a single label indicating bona fide or spoofed. PADD differs in being a frame-level task. Given SSL acoustic features

$$\mathbf{X} = \{f_1, f_2, \ldots, f_T\} \in \mathbb{R}^{D \times T},$$

the goal is to predict

$$\mathbf{y} = \{y_1, y_2, \ldots, y_T\} \in \{0, 1\}^T,$$

where $y_t = 1$ denotes an authentic frame and $y_t = 0$ denotes a spoofed frame (Xian et al., 4 Sep 2025).

This frame-wise formulation is motivated by forensic analysis, selective removal or flagging of manipulated portions, and attack scenarios in which only a few words are altered. The task is correspondingly more difficult because spoofed regions may be short and subtle, boundaries may be hard to localize, and local acoustic cues may be weak or ambiguous.

The central motivating assumption behind NE-PADD is that partial audio deepfakes frequently manipulate named entities such as persons, organizations, and locations. The paper states that, empirically and by design of several datasets including HAD and PartialSpoof, spoofing is often concentrated around named-entity regions. Existing PADD methods such as RawNet2, AASIST, Wav2Vec2-based PADD, BAM, TDL, and WBD are described as focusing primarily on acoustic and structural cues rather than explicitly exploiting audio-level named entity information. NE-PADD is presented as filling this gap by using SpeechNER attention as a semantic prior for frame-level spoof detection (Xian et al., 4 Sep 2025).

2. System architecture

NE-PADD uses a two-branch architecture with shared input audio. The SpeechNER branch learns where named entities occur in the speech signal and produces hidden representations $\mathbf{H}_{\text{NER}}$ and attention weights $\alpha_{\text{NER}}$. The PADD branch learns frame-level spoof detection and produces hidden representations $\mathbf{H}_{\text{ADD}}$ and attention weights $\alpha_{\text{ADD}}$. Each branch also yields an attention-modified embedding: $\mathbf{H}_{\text{NER-attended}}$ and $\mathbf{H}_{\text{ADD-attended}}$ (Xian et al., 4 Sep 2025).

These branch outputs are integrated in one of two ways. In the AF variant, the attended embeddings from the two branches are fused into $$\mathbf{y} = \{y_1, y_2, \ldots, y_T\} \in \{0, 1\}^T,$$0, which is then consumed by a backend classifier. In the AT variant, the backend still receives $$\mathbf{y} = \{y_1, y_2, \ldots, y_T\} \in \{0, 1\}^T,$$1, but the PADD attention distribution is regularized to align with the SpeechNER attention distribution through an auxiliary KL-divergence loss.

The backend classifier is described as a Transformer encoder followed by a BiLSTM and a fully connected layer. In experiments, the SpeechNER parameters are frozen, and only the PADD branch and backend are trained for detection, with AF or AT determining how named-entity information is injected into the detection pipeline (Xian et al., 4 Sep 2025).

A plausible implication is that NE-PADD treats semantic salience not as a replacement for acoustic evidence but as a structured prior over time. This interpretation is consistent with the use of both feature-level fusion and teacher–student attention alignment.

3. SpeechNER branch

The SpeechNER component is formulated as end-to-end speech-based named entity recognition. Its input is raw speech transformed into log-spectrograms using 20 ms Hamming windows with power normalization. Its output is a text sequence in which named entities are marked by special symbols rather than a BIO tagging scheme. Following Yadav et al., the label space includes three special symbols denoting the start of named entities—\{, |, and \$—for three frequent named entity types, organization, person, and location, and a common symbol ] for the end of any named entity (Xian et al., 4 Sep 2025).

The example given in the paper illustrates the encoding:

“CompanyX CEO heads a meeting in their Shenzhen office.”

becomes

“{CompanyX] CEO heads a meeting in their \$Shenzhen] office”

This formulation encodes both presence and boundaries of named entities within the target text.

The SpeechNER feature extraction module used in NE-PADD consists of convolutional layers with normalization and multiple BiLSTM layers to model temporal dependencies. These produce

$$\mathbf{y} = \{y_1, y_2, \ldots, y_T\} \in \{0, 1\}^T,$$2

A self-attention mechanism then computes query, key, and value projections from $$\mathbf{y} = \{y_1, y_2, \ldots, y_T\} \in \{0, 1\}^T,$$3, with attention weights

$$\mathbf{y} = \{y_1, y_2, \ldots, y_T\} \in \{0, 1\}^T,$$4

and attended embeddings

$$\mathbf{y} = \{y_1, y_2, \ldots, y_T\} \in \{0, 1\}^T,$$5

In NE-PADD, SpeechNER is pre-trained externally for NER and then integrated with its parameters frozen. The paper identifies the critical outputs for downstream PADD as the attention distribution $$\mathbf{y} = \{y_1, y_2, \ldots, y_T\} \in \{0, 1\}^T,$$6, which emphasizes temporal regions associated with named entities, and the attended embeddings $$\mathbf{y} = \{y_1, y_2, \ldots, y_T\} \in \{0, 1\}^T,$$7, which serve as semantically enriched features (Xian et al., 4 Sep 2025).

4. PADD branch and learning objective

The PADD branch receives Wav2Vec2.0 embeddings at a 20 ms frame rate:

$$\mathbf{y} = \{y_1, y_2, \ldots, y_T\} \in \{0, 1\}^T,$$8

and outputs frame-level labels $$\mathbf{y} = \{y_1, y_2, \ldots, y_T\} \in \{0, 1\}^T,$$9, with $y_t = 1$0 indicating spoof and $y_t = 1$1 indicating bona fide. The paper characterizes this as frame-wise binary classification or sequence labeling (Xian et al., 4 Sep 2025).

Its front-end begins with pretrained Wav2Vec2.0, trained on LibriSpeech 960 h, which converts raw audio into features of shape $y_t = 1$2. A 1D convolution $y_t = 1$3 with kernel size 5, stride 2, padding 1, and no bias projects these features to $y_t = 1$4. The result is processed by 12 ResNet-1D residual blocks, each containing a convolution $y_t = 1$5 and normalization, preserving the $y_t = 1$6 shape. A final Conv1D $y_t = 1$7 reduces dimensionality to $y_t = 1$8, yielding

$y_t = 1$9

The branch then applies self-attention:

$y_t = 0$0

and

$y_t = 0$1

The attended features, or their fused variant, are sent to a backend comprising a Transformer encoder $y_t = 0$2, a one-layer BiLSTM with hidden size 128, a ReLU activation, and a final fully connected layer of dimension 256 to output per-frame spoof probability (Xian et al., 4 Sep 2025).

The main detection loss is frame-wise binary cross-entropy. Let $y_t = 0$3 denote the predicted probability that frame $y_t = 0$4 is bona fide. Then

$y_t = 0$5

This loss is used directly in the AF variant and forms the main term in the AT variant (Xian et al., 4 Sep 2025).

5. Attention aggregation mechanisms

The defining contribution of NE-PADD is the use of two attention aggregation mechanisms: Attention Fusion and Attention Transfer. Both are intended to exploit the temporal emphasis learned by SpeechNER for named entities, but they operate at different levels (Xian et al., 4 Sep 2025).

Attention Fusion

AF combines what the PADD branch focuses on with where the SpeechNER branch emphasizes named entities. Let $y_t = 0$6 and $y_t = 0$7. A gating weight

$y_t = 0$8

is computed from the concatenated features, where $y_t = 0$9 is a learnable linear projection and $\mathbf{H}_{\text{NER}}$0 is the Sigmoid function. The fused feature is then

$\mathbf{H}_{\text{NER}}$1

The paper interprets $\mathbf{H}_{\text{NER}}$2 as favoring PADD attended features, $\mathbf{H}_{\text{NER}}$3 as favoring SpeechNER attended features, and intermediate values as a learned trade-off. The fused representation replaces $\mathbf{H}_{\text{NER}}$4 in the backend classifier. This mechanism is described as allowing the detector to prioritize acoustic spoof cues where they are strong and shift toward named-entity regions where semantic cues are crucial.

Attention Transfer

AT aligns attention distributions rather than features. SpeechNER attention $\mathbf{H}_{\text{NER}}$5 acts as teacher and PADD attention $\mathbf{H}_{\text{NER}}$6 as student. The auxiliary loss is

$\mathbf{H}_{\text{NER}}$7

The total objective becomes

$\mathbf{H}_{\text{NER}}$8

The asymmetry of the KL divergence is explicitly used in the direction $\mathbf{H}_{\text{NER}}$9, so that PADD is encouraged to cover all regions emphasized by SpeechNER. The backend still consumes $\alpha_{\text{NER}}$0, but the student attention has been regularized toward the semantic attention profile of the teacher (Xian et al., 4 Sep 2025).

6. Data, optimization, and empirical results

NE-PADD is evaluated on PartialSpoof-NER, a dataset constructed by starting from PartialSpoof, transcribing each audio file with Whisper, extracting named entities from the transcripts with Stanza, and adding the resulting annotations (Xian et al., 4 Sep 2025).

Split	Bona fide	Fake
Train	966	8,789
Dev	124	1,057
Eval	122	1,126

Split	Total utt.	Named entities
Train	9,755	11,572
Dev	1,181	1,407
Eval	1,248	1,526

The paper states that there are on average 1–2 named entities per utterance and that partial spoofing in these utterances can occur around such named entities. For SpeechNER, preprocessing uses log-spectrograms with 20 ms Hamming windows; for PADD, raw audio is converted to Wav2Vec2.0 SSL embeddings at a 20 ms frame rate. The training protocol is Train for fitting, Dev for tuning, and Eval for final evaluation. The evaluation metric is frame-level Equal Error Rate (EER), where each frame is treated as a binary spoof-versus-bona-fide decision (Xian et al., 4 Sep 2025).

Optimization uses Adam for 100 epochs with batch size 16, base learning rate $\alpha_{\text{NER}}$1, and a Noam scheduler with 1600 warm-up steps. Model selection monitors EER on a development or adaptation set. In the AT variant, $\alpha_{\text{NER}}$2 is tuned from 0.1 to 1.0.

The baselines listed are Single resolution, BAM, TDL, and WBD. The reported Eval EER values are as follows (Xian et al., 4 Sep 2025):

Model	EER (%)
Single reso.	26.51
BAM	18.21
TDL	14.64
WBD	11.59
NE-PADD (AF)	7.89

NE-PADD-AF therefore reports 7.89% EER, outperforming the listed baselines on PartialSpoof-NER. The ablation study further applies AF and AT to BAM and TDL:

Model variant	EER (%)
BAM-AT	14.84
BAM-AF	10.61
TDL-AT	8.87
TDL-AF	9.28
NE-PADD-AT	9.48
NE-PADD-AF	7.89

The paper highlights several observations. For BAM, both AT and AF improve over the 18.21% baseline, with AF producing the larger reduction. For TDL, both mechanisms again improve over the 14.64% baseline, with AT slightly better than AF. Within NE-PADD itself, AF outperforms AT on this setup. The authors interpret these findings as evidence that NER-informed attention is useful beyond a single backbone and can consistently improve PADD systems (Xian et al., 4 Sep 2025).

Hyperparameter analysis for AT shows best EER around $\alpha_{\text{NER}}$3 and 0.9 for NE-PADD-AT, around 0.4 for TDL-AT, and around 0.8 for BAM-AT. The paper interprets too small a $\alpha_{\text{NER}}$4 as weak semantic guidance and too large a value as over-constraining PADD attention.

A finer-grained forgery-level analysis partitions utterances by the number of spoofed segments per utterance, from 1 to 10. The reported general trend is that EER decreases as the number of spoofed segments increases, especially from 1 to 5, after which it fluctuates as data become sparse. For NE-PADD-AF, the paper gives examples including 63.88% EER for one forgery, approximately 24% to 14% EER for 3–5 forgeries, and 10% EER for 9 forgeries. For NE-PADD-AT, examples include 45% EER for one forgery and 10% for four forgeries. This suggests that very sparse spoofing remains difficult even when semantic guidance is available (Xian et al., 4 Sep 2025).

7. Significance, scope, and limitations

The paper frames NE-PADD’s contributions in four parts: a semantic-aware PADD framework integrating named entity knowledge from SpeechNER; two attention aggregation mechanisms, AF and AT; the PartialSpoof-NER dataset; and state-of-the-art performance on that dataset, with 7.89% EER for NE-PADD-AF and consistent improvements when semantic-guided attention is added to other architectures (Xian et al., 4 Sep 2025).

In broader methodological context, NE-PADD is positioned against audio deepfake detectors that are predominantly acoustic-centric, emphasizing waveform artifacts, vocoder traces, or spectral irregularities. By incorporating named entity information, it introduces a high-level semantic signal into the detection loop. The paper also links this to SpeechNER as an emerging area, presenting named entity localization from speech as useful not only for spoken language understanding but also for security-oriented audio forensics.

Several limitations are stated explicitly. Evaluation is confined to English because of limited work and resources for Chinese SpeechNER. The PartialSpoof-NER dataset is described as not extremely large, and the authors caution that finer-grained analyses may not fully reflect generalization ability because of data scarcity. The dataset construction pipeline depends on Whisper transcription and Stanza-based text NER, so transcription or NER errors may propagate into the annotations. Detecting very short spoofed segments, especially when there is only one forged segment in an utterance, remains difficult. The paper also notes that SpeechNER is fixed and frozen; joint training or adaptation of NER to spoofed data is presented as a possible future direction. Finally, the overall system includes Wav2Vec2.0, ResNet-1D, Transformer, BiLSTM, and a SpeechNER encoder, while computational efficiency and deployment constraints are not discussed in depth (Xian et al., 4 Sep 2025).

A plausible implication is that NE-PADD is most directly suited to threat models in which attackers alter semantically consequential spans rather than uniformly manipulating entire utterances. Under that interpretation, its emphasis on named entities is not a generic substitute for acoustic evidence but a targeted inductive bias for attacks that modify key referential content.