FourierPT-XLSR: Spectral Prompt Tuning

• The paper introduces FourierPT-XLSR, a module that fuses FFT-based spectral priors with trainable prompt tuning to enhance speech deepfake detection.
• The architecture injects Fourier features into prompt embeddings at each Transformer layer, updating only about 1.4% of parameters while keeping XLSR frozen.
• Empirical results on DE24 and SpoofCeleb benchmarks show improved EER and accuracy over vanilla prompt tuning, though wavelet-based variants still outperform it.

Searching arXiv for the specified paper and closely related work to ground the article. arxiv_search query: "(Xuan et al., 6 Oct 2025) WaveSP-Net Learnable Wavelet-Domain Sparse Prompt Tuning for Speech Deepfake Detection" FourierPT-XLSR is a parameter-efficient front-end for speech deepfake detection that augments a frozen XLSR-300M backbone with prompt tuning in the Fourier domain. It was introduced as part of the broader WaveSP-Net framework, which studies prompt-tuning modules fused with classical signal processing transforms for synthetic speech detection (Xuan et al., 6 Oct 2025). In this formulation, Fast Fourier Transform (FFT) features are injected into trainable prompt embeddings at each Transformer layer, while the underlying XLSR parameters remain frozen. The method is positioned against both full fine-tuning of large pre-trained models and alternative prompt-based variants using wavelet transforms, with the stated aim of improving generalization to realistic, in-the-wild data types under a substantially reduced trainable parameter budget (Xuan et al., 6 Oct 2025).

1. Definition and placement within WaveSP-Net

FourierPT-XLSR is one member of a family of parameter-efficient front-ends introduced alongside WSPT-XLSR and Partial-WSPT-XLSR (Xuan et al., 6 Oct 2025). All three variants combine prompt tuning with classical signal processing transforms, but FourierPT-XLSR specifically uses the Fourier Transform. Within the paper’s overall architecture, it serves as a front-end attached to a frozen XLSR-300M model, after which a small Mamba-based back-end performs classification.

The defining characteristic of FourierPT-XLSR is that the Fourier-domain operation is applied to the prompt tokens rather than to the raw audio. The raw segment $x(t)$ is first processed by the XLSR front-end into a 2D feature map, but all XLSR weights remain frozen. Prompt tokens are then transformed by an FFT module, and the real part of the Fourier representation is fused back into the prompts before those enhanced prompts are prepended to the layer input embeddings (Xuan et al., 6 Oct 2025).

This design places FourierPT-XLSR in a specific methodological niche: it is neither conventional full fine-tuning nor prompt tuning in a purely learned latent space. Instead, it couples trainable prompts to an explicit spectral prior. The paper presents this as a lightweight, spectral-aware speech deepfake detection front-end, while also indicating that more localized transforms can outperform it on challenging in-the-wild benchmarks (Xuan et al., 6 Oct 2025).

2. Architectural configuration

The architectural overview is given explicitly in terms of the XLSR stack. Let $L$ denote the number of Transformer layers in XLSR, with $L = 12$; let $d$ denote the hidden dimension, with $d = 1024$; and let $p$ denote the number of prompt tokens per layer, with $p = 10$ (Xuan et al., 6 Oct 2025). For the $k$-th layer, a trainable prompt matrix $P_k \in \mathbb{R}^{p \times d}$ is introduced.

Before each Transformer block $L_k$, the prompt tokens are converted into a Fourier-domain representation via an FFT module. The real part of this representation is fused into the original prompt tokens to produce enhanced prompts $L$0, which are then prepended to the frozen XLSR embeddings $L$1. The layer input-output relation is stated as

$L$2

where $L$3 are the transformed prompt outputs and $L$4 are the sequence embeddings (Xuan et al., 6 Oct 2025).

After the final Transformer layer, the combined output $L$5 is average-pooled and passed to a small Mamba-based back-end trained from scratch. The paper therefore separates the system into a frozen large-scale acoustic representation model and a comparatively small trainable adaptation-and-classification pathway. A plausible implication is that the method is intended to preserve the pre-trained representational structure of XLSR while using prompts to steer it toward spoof-discriminative spectral patterns.

3. Mathematical formulation

The Fourier prompt-tuning mechanism is described in four steps (Xuan et al., 6 Oct 2025). First, for each layer $L$6, the discrete Fourier transform of the prompt token matrix $L$7 is computed along the token dimension, described as the token dimension or time-like axis:

$L$8

where $L$9 denotes the FFT operator applied row-wise. The real part is then extracted:

$L = 12$0

Second, the original prompt tokens and the Fourier-domain real coefficients are fused through a learned linear gating mechanism:

$L = 12$1

$L = 12$2

where $L = 12$3 and $L = 12$4 are learnable parameters, $L = 12$5 is the element-wise sigmoid, and $L = 12$6 is the Hadamard product (Xuan et al., 6 Oct 2025).

Third, the fused prompts are injected into the frozen XLSR stream by concatenation:

$L = 12$7

for $L = 12$8, with $L = 12$9 (Xuan et al., 6 Oct 2025). Fourth, the final pooled representation is passed to the Mamba-based classifier.

The stated training objective is binary cross-entropy over real-versus-fake labels:

$d$0

where $d$1 is the model’s output probability. All parameters of the prompt module $d$2 and the Mamba back-end are optimized jointly. No additional regularization terms are added beyond the standard weight decay inherent in the Adam optimizer (Xuan et al., 6 Oct 2025).

4. Trainable parameters and efficiency profile

A central property of FourierPT-XLSR is its parameter efficiency relative to full fine-tuning. The trainable prompt tokens contribute $d$3 parameters. The fusion gating weights contribute $d$4 and $d$5, totaling approximately $d$6 parameters. The Mamba classifier contributes approximately $d$7 parameters. The resulting total trainable parameter count for FourierPT-XLSR is approximately $d$8 (Xuan et al., 6 Oct 2025).

In contrast, full fine-tuning of XLSR-300M would update approximately $d$9 parameters. The paper therefore states that FourierPT-XLSR updates only approximately $d = 1024$0 of the total (Xuan et al., 6 Oct 2025).

Component	Parameter count
Prompt tokens	$d = 1024$1
Fusion gating weights	$d = 1024$2
Mamba classifier	$d = 1024$3
Total trainable (FourierPT-XLSR)	$d = 1024$4

The paper identifies three corresponding advantages. First, the model is parameter-efficient, with at most about $d = 1024$5 trainable parameters versus $d = 1024$6 under full fine-tuning. Second, FFT features provide an implicit spectral inductive bias that guides prompts toward frequency-domain artifacts. Third, integration is described as easy because it requires only an FFT and two small gating matrices per layer (Xuan et al., 6 Oct 2025).

5. Empirical performance on in-the-wild benchmarks

The reported evaluation compares FourierPT-XLSR with wavelet-based prompt variants on two in-the-wild benchmarks, Deepfake-Eval-2024 and SpoofCeleb (Xuan et al., 6 Oct 2025). The paper reports the following values for FourierPT-XLSR.

Benchmark	Reported metrics
Deepfake-Eval-2024	EER = 16.58% ($d = 1024$7), ACC = 83.42%, F1 = 79.53%, AUC = 90.35%
SpoofCeleb	EER = 0.23% ($d = 1024$8), ACC = 99.84%, F1 = 99.87%, AUC = 99.86%

The paper further compares FourierPT-XLSR to vanilla PT-XLSR, reporting that vanilla PT-XLSR attains EER $d = 1024$9 on DE24, whereas FourierPT-XLSR attains EER $p$0, corresponding to a $p$1 percentage-point absolute improvement (Xuan et al., 6 Oct 2025). This comparison is used to highlight the benefit of injecting spectral priors.

At the same time, the paper states that FourierPT-XLSR remains below wavelet-based WSPT-XLSR and Partial-WSPT-XLSR, with the best reported EER on DE24 being $p$2 (Xuan et al., 6 Oct 2025). This is an important qualification: Fourier-based prompt fusion improves over a vanilla prompt-tuning baseline, but it does not constitute the strongest variant within the same family.

6. Interpretation, limitations, and relation to wavelet-based variants

The paper attributes FourierPT-XLSR’s improvements to the incorporation of spectral priors into prompt embeddings rather than to adaptation of the XLSR backbone itself (Xuan et al., 6 Oct 2025). Because the Fourier transform is applied to prompts and the real coefficients are merged through learned gating, the method can be interpreted as a constrained adaptation mechanism in which frequency-domain structure modulates the prompt subspace that interfaces with XLSR. This suggests a deliberate attempt to bias the model toward spoof-relevant spectral artifacts while preserving the frozen pre-trained representation.

The stated limitations are equally explicit. The uniform time-frequency tiling of FFT may miss localized transient artifacts. The method shows lower performance than wavelet-based prompts, especially on benchmarks with abrupt spectral changes. The paper also notes that reliance on a global Fourier basis may reduce sensitivity to short-time spoofing cues (Xuan et al., 6 Oct 2025).

These limitations clarify an important methodological distinction. FourierPT-XLSR uses a global spectral basis, whereas the wavelet-based alternatives are presented as more localized transforms. The empirical gap on DE24 is consistent with that difference, although the paper does not present FourierPT-XLSR as ineffective; rather, it presents it as a lightweight baseline demonstrating that classical DSP transforms can be fused with prompt tuning. A plausible implication is that the main research contribution of FourierPT-XLSR is conceptual as well as empirical: it establishes the viability of transform-aware prompt tuning, even if more localized transforms ultimately achieve stronger results in in-the-wild deepfake detection (Xuan et al., 6 Oct 2025).