Speaker-Formant Transformer Architecture

• The paper demonstrates a multi-task transformer framework that integrates explicit formant trajectory regression and voicing prediction to enhance deepfake detection.
• Leveraging parallel encoders and task-specific decoders, the model processes spectral features and uses attention for detailed attribution between voiced and unvoiced speech segments.
• Experiments reveal improved efficiency and detection accuracy, with reduced error rates and faster convergence compared to previous architectures.

The Speaker-Formant Transformer Architecture is a hypothesis-driven, multi-task transformer framework explicitly designed for explainable detection of speech deepfakes. Its central innovation is the integration of direct formant trajectory regression, frame-level voicing prediction, and an end-to-end deepfake detection pipeline, combined with built-in mechanisms for attributing detection decisions to voiced or unvoiced speech segments. The architecture streamlines deepfake detection by modeling the prosodic characteristics of speech—specifically, the physiological formants ($F_0$, $F_1$, $F_2$) and voicing—leveraging these as both targets for prediction and means for interpretation (Negroni et al., 21 Jan 2026).

1. Architectural Overview

The Speaker-Formant Transformer, exemplified by SFATNet-4, operates in a modular encoder–decoder paradigm. The initial stage computes the Short-Time Fourier Transform (STFT) of the waveform, decomposing it into two frame-level features: log-magnitude and sine of the unwrapped phase. Each sequence is sliced along the time dimension, resulting in a sequence of $L$ tokens with $M$-dimensional spectra per frame, where $L = 128$ and $M = 256$ (corresponding to 2.064 seconds of 16 kHz audio with 32 ms window and 16 ms hop).

Parallel transformer encoders process the magnitude and phase channels:

• $$E_{\mathbf{X}}: \mathbb{R}^{L \times M} \rightarrow \mathbb{R}^{L \times D}$$ for log-magnitude,
• $$E_{\mathbf{\Phi}}: \mathbb{R}^{L \times M} \rightarrow \mathbb{R}^{L \times D}$$ for phase.

These encoders utilize repeated multi-head self-attention (MSA) and MLP blocks in a pre-layer normalization configuration. No explicit positional encodings are added; the model infers temporal dependencies solely from fine-grained frame ordering.

The encoded features are concatenated and linearly projected to a shared representation $$\mathbf z_{\mathrm{enc}} \in \mathbb{R}^{L \times D}$$, forming the basis for three task-specific decoder branches (Negroni et al., 21 Jan 2026).

2. Input Representation and Feature Extraction

Each segment is uniformly preprocessed:

• Only temporal slicing is applied, resulting in single-frame tokens that span the full spectrum.
• Ground-truth prosodic labels are derived offline: $F_0$ via pYIN, $F_1$ and $F_2$ by Burg LPC analysis (identification of LPC polynomial roots).
• Audio is trimmed to remove silence, normalized to unity peak, and padded by repetition for segments below the fixed duration threshold.

This careful segmentation strategy supports precise frame-level modeling and contributes to label-aligned supervision of formants and voicing (Negroni et al., 21 Jan 2026).

3. Task-Specific Decoders and Loss Formulation

The architecture incorporates three parallel decoders:

• Multi-formant decoder: Applies a linear layer and sigmoid activation to predict per-frame $F_0, F_1, F_2$, with values rescaled to physiological frequency ranges ($F_0 \in [60, 400]$ Hz, $F_1 \in [200, 850]$ Hz, $F_2 \in [800, 2700]$ Hz).
• Voicing decoder: A linear and sigmoid layer yields a probability for each frame, thresholded to a binary voiced/unvoiced mask; this also gates the formant regression loss.
• Synthesis predictor: A 4-layer transformer with multi-head pooling computes frame-wise attention weights $\alpha_t$, which are used to pool the sequence into a summary vector for final deepfake probability prediction.

Training involves a weighted combination of three loss terms:

• Classification loss (binary cross-entropy) on real/fake labels;
• Voicing loss (frame-level binary cross-entropy);
• Formant loss (mean squared error of log-standardized $F_0,F_1,F_2$, gated by the ground-truth voiced mask):

$$\mathcal{L}_{\text{total}} = 1.0 \mathcal{L}_{\text{cls}} + 0.3 \mathcal{L}_{\text{vox}} + 0.3 \mathcal{L}_{\text{frm}}$$

This explicit multi-task design enforces prosodic awareness, enhancing both interpretability and generalization (Negroni et al., 21 Jan 2026).

4. Explainability via Attention and Voicing Attribution

A distinctive feature is the direct explainability mechanism. Synthesis head attention weights provide a temporally resolved attribution map for the detector's decision. These can be quantitatively decomposed across voiced and unvoiced regions using the predicted voicing mask:

$$\sum_t \alpha_t = \left( \sum_{t: v_\text{mask}(t)=1} \alpha_t \right) + \left( \sum_{t: v_\text{mask}(t)=0} \alpha_t \right)$$

Empirical findings indicate that correct fake classifications often derive up to 80% of their evidence from unvoiced frames, implicating noise-like segments as principal loci for synthesis artifacts. In contrast, in-domain real speech exhibits a substantially more even attribution between voiced and unvoiced regions. This enables granular diagnostic analysis of model decisions and insight into deepfake vulnerability regions in the signal (Negroni et al., 21 Jan 2026).

5. Computational Efficiency and Performance Benchmarks

SFATNet-4 offers substantial computational benefits versus its predecessor (SFATNet-3), realizing a parameter count reduction from 64.7M to 41.8M (−35%) and improving epoch training times ($\sim$15 min vs. $>$60 min on NVIDIA A40). The model achieves early convergence within $\sim$30 epochs.

Evaluation across multiple datasets demonstrates competitive deepfake detection:

Dataset	EER (%)	AUC (%)
ASVspoof 5	4.41	98.89
In-the-Wild	17.29	89.17
FakeOrReal	20.33	85.03
TIMIT-TTS	20.93	84.49

Compared to SFATNet-3, SFATNet-4 decreases EER by 4.44 points on ASVspoof 5 and boosts average AUC by nearly 3%. These improvements are achieved without sacrificing interpretability or model compactness (Negroni et al., 21 Jan 2026).

6. Ablation and Qualitative Analysis

Ablation studies confirm the value of explicit prosodic modeling: removing the voicing or formant decoder branches degrades AUC by 1.5–2% on out-of-domain data. Overlaying predicted formant trajectories on spectrograms demonstrates accurate alignment, including during rapid prosodic transitions.

This evidence indicates that joint formant and voicing supervision goes beyond interpretability, providing tangible improvements in generalization, particularly for unseen synthetic speech manipulations (Negroni et al., 21 Jan 2026).

7. Relationship to Broader Multitask and Multimodal Approaches

While the Speaker-Formant Transformer is tightly focused on prosodic modeling for deepfake detection, emerging directions involve integration into larger multimodal or LLM-based frameworks. For instance, audio–LLMs such as DFALLM (Li et al., 9 Dec 2025) achieve generalizable multitask deepfake detection via prompt-driven fusion of audio representations (from high-resolution encoders like Wav2Vec2-BERT) and textual LLMs (Qwen2.5 and similar). These systems benefit from fine-grain spectral feature encoding and prompt-based multitask heads for detection, attribution, and localization.

A plausible implication is that hybrid architectures—combining explicit prosodic modeling with cross-modal transformer blocks—could further enhance detection generalization and interpretability, leveraging the strengths of both approaches documented in recent literature (Negroni et al., 21 Jan 2026, Li et al., 9 Dec 2025).