HuLA: Human-Like Listener for Anti-Spoofing

• HuLA is a prosody-aware multi-task learning framework that leverages F0 prediction and voiced/unvoiced classification to distinguish natural from synthetic speech.
• It employs a two-stage training regime where real speech pre-training conditions the XLS-R backbone on natural prosody before jointly optimizing spoof detection with prosody supervision.
• Selective layer aggregation and frame-level prosodic supervision in HuLA drive significant performance gains against expressive, emotional, and cross-lingual spoofing attacks.

HuLA, short for Human-Like Listener for Anti-spoofing, is a two-stage, prosody-aware multi-task learning framework for synthetic-speech spoof detection built on the XLS-R 300 M self-supervised speech model. Its defining claim is that anti-spoofing systems remain vulnerable to expressive and emotional synthetic speech when they do not explicitly model prosody, even though humans rely on cues such as $F_0$ patterns and voiced/unvoiced (V-UV) structure to distinguish natural from synthetic speech. HuLA addresses this by first adapting an SSL backbone to natural prosodic variation in real speech and then jointly optimizing spoof detection with frame-level prosody supervision on both bona-fide and spoofed data (Mahapatra et al., 25 Sep 2025).

1. Problem formulation and motivation

HuLA is motivated by a specific failure mode in contemporary spoof-detection systems: they “rarely leverage prosody as a discriminative cue,” despite the centrality of prosody to human expressiveness and emotion. In the HuLA formulation, this omission is particularly consequential for expressive, emotional, and cross-lingual attacks, where the spectral realism of modern TTS and VC systems can be strong while their prosodic behavior remains imperfect (Mahapatra et al., 25 Sep 2025).

The framework operationalizes the idea of a “human-like listener” by making prosodic supervision explicit. The paper defines two such cues at frame level: fundamental frequency ($F_0$) prediction and voiced/unvoiced classification. Rather than treating spoof detection as a single classification problem, HuLA treats it as a joint problem in which the model must also internalize natural prosodic structure. This design is presented as a way to exploit “the imperfect expressiveness of state-of-the-art TTS and VC systems as a discriminative cue” (Mahapatra et al., 25 Sep 2025).

A common misunderstanding would be to treat HuLA as a prosody-only detector. The reported system is not configured that way. Spoof detection remains a primary supervised objective, and prosody is introduced as auxiliary structure in both training stages. Another misunderstanding would be to assume that HuLA learns prosody directly from spoofed speech at the outset. In fact, its first stage is trained on real speech only, with the stated goal of adapting the pretrained XLS-R backbone to natural prosodic variation before exposure to spoof data (Mahapatra et al., 25 Sep 2025).

2. Two-stage training regime

HuLA is defined by a two-stage training strategy. In Stage 1 (Prosody Pre-training), the pretrained XLS-R backbone is adapted using only real speech. The training data are 28,539 real utterances from LibriSpeech train-clean-100 plus 2,703 from dev-clean, and the tasks are $F_0$ prediction (regression) and V-UV classification (binary). The reported outcome is that “the model internalizes frame-level prosodic patterns,” creating a basis for later spoof detection (Mahapatra et al., 25 Sep 2025).

In Stage 2 (Joint Spoof Detection + Prosody), the prosody-aware backbone is fine-tuned on both bona-fide and spoofed speech. The Stage 2 data are ASVspoof 2019 LA train (2,580 real, 22,800 spoof) and development (2,548 real, 22,296 spoof). The tasks are jointly defined as spoof vs. bona-fide classification, $F_0$ prediction, and V-UV classification. The reported purpose of the second stage is to leverage the prosodic awareness learned on real speech so that the model can detect “subtle expressive mismatches in advanced synthetic attacks while retaining strong generalization” (Mahapatra et al., 25 Sep 2025).

This separation between stages is central to the method’s identity. Stage 1 does not attempt spoof detection; it conditions the representation space on natural prosodic structure. Stage 2 does not discard this structure; it retains the same prosody tasks while adding spoof classification. The ablation labeled “HuLA w/o PT”, which omits Stage 1 and performs only single-stage joint training, is important because it shows that prosody supervision is useful even without the preliminary real-speech adaptation. The paper states that this variant “already outperforms all baselines,” while Stage 1 pretraining yields further gains on expressive and emotional attacks (Mahapatra et al., 25 Sep 2025).

3. Architecture and objective functions

HuLA uses the XLS-R 300 M model as backbone. A raw waveform $x$ is encoded by a convolutional feature encoder into latent representations $z \in \mathbb{R}^{T \times 1024}$, followed by 24 transformer layers producing contextual representations $h^{(1)}, \ldots, h^{(24)}$. The paper distinguishes the two stages at the representation level. In Stage 1, the input to the downstream module is

$H_1 = h^{(24)}.$

In Stage 2, the input is a learned weighted aggregation across all transformer layers,

$H_2 = \sum_{\ell=1}^{24} w_\ell h^{(\ell)},$

where the scalar weights $w_\ell$ are learned through Selective Layer Selection, described as “as in SSL-SLS” (Mahapatra et al., 25 Sep 2025).

The prosody module, named Pro-MTL, is structurally simple. It applies a linear projection, temporal modeling with a GRU, and two frame-level heads:

$F_0$0

$F_0$1

The spoof classifier is used in Stage 2 only. It takes the aggregated representation $F_0$2 and feeds it to “a lightweight feed-forward network ending in a two-way softmax for bona-fide vs. spoof” (Mahapatra et al., 25 Sep 2025).

The paper reports three losses. The $F_0$3 loss is mean squared error over frames,

$F_0$4

where $F_0$5 is the speaker-normalized target pitch and is zero if unvoiced. V-UV uses binary cross-entropy, and spoof classification uses a weighted cross-entropy. The two training-stage objectives are reported as

$F_0$6

and

$F_0$7

At inference time, HuLA does not use the auxiliary prosody heads. The paper states that “only the spoof classifier (fed by fine-tuned XLS-R) is used; auxiliary prosody heads are discarded” (Mahapatra et al., 25 Sep 2025). This is significant because the method uses multi-task supervision to shape the latent space, not to require explicit prosody outputs at deployment.

4. Data, preprocessing, and evaluation protocol

HuLA is evaluated on a set of in-domain and out-of-domain spoofing benchmarks spanning standard logical-access benchmarks, emotional synthetic speech, and cross-lingual attacks. Input audio is trimmed/padded to 4 s, with frame length $F_0$8 ms and frame shift $F_0$9 ms, matching XLS-R. Optimization uses AdamW, batch size $F_0$0, and 50 epochs per stage. In Stage 1, the learning rate is $F_0$1 for the backbone and $F_0$2 for Pro-MTL, with no weight decay. In Stage 2, the learning rate is $F_0$3 for backbone and spoof head and $F_0$4 for Pro-MTL, with weight decay $F_0$5 and RawBoost augmentation (Mahapatra et al., 25 Sep 2025).

The principal evaluation metric is Equal Error Rate (EER%), defined as the operating point where false acceptance rate equals false rejection rate. The paper also notes that DET curves are used for analysis but are omitted here for brevity (Mahapatra et al., 25 Sep 2025).

Dataset	Split or role	Reported size
LibriSpeech train-clean-100	Stage 1 training	28,539 utterances
LibriSpeech dev-clean	Stage 1 development	2,703 utterances
ASVspoof 2019 LA train	Stage 2 training	2,580 real, 22,800 spoof
ASVspoof 2019 LA dev	Stage 2 development	2,548 real, 22,296 spoof
ASVspoof 2019 LA	Evaluation	7,355 real, 63,882 spoof
ASVspoof 2021 LA	Evaluation	14,816 real, 133,360 spoof
ASVspoof 2024 Track 1	Evaluation	138,688 real, 542,086 spoof
EmoFake	Evaluation	3,500 real, 14,000 emotional spoof
Mixed Emotions	Evaluation	160 real, 331 mixed-emotion spoof
ADD 2022	Evaluation	31,334 real, 77,865 Mandarin spoof
HABLA	Evaluation	22,816 real, 53,000 Spanish accent spoof

This benchmark design is notable because it does not confine evaluation to ASVspoof-style laboratory conditions. The inclusion of EmoFake, Mixed Emotions, ADD 2022, and HABLA supports the paper’s emphasis on expressive, emotional, and cross-lingual attacks as the stress cases for prosody-aware anti-spoofing (Mahapatra et al., 25 Sep 2025).

5. Empirical performance and ablation structure

The reported quantitative results compare HuLA against RawNet2, AASIST, SSL-SLS, and the ablation HuLA w/o PT. On the standard ASVspoof benchmarks, HuLA achieves 0.80 EER on ASVspoof 2019 LA, 1.38 on ASVspoof 2021 LA, and 17.34 on ASVspoof 2024 Track 1. The corresponding values for HuLA w/o PT are 0.48, 1.85, and 23.12 (Mahapatra et al., 25 Sep 2025).

On emotional datasets, HuLA achieves 3.01 on EmoFake and 16.25 on Mixed Emotions, compared with 5.24 and 19.37 for HuLA w/o PT. On cross-lingual datasets, HuLA reports 13.51 on HABLA and 32.50 on ADD 2022, compared with 8.83 and 33.35 for HuLA w/o PT (Mahapatra et al., 25 Sep 2025).

Dataset	HuLA (full) EER%	Other reported EERs
ASVspoof 2019 LA	0.80	RawNet2 4.60; AASIST 0.83; SSL-SLS 0.56; HuLA w/o PT 0.48
ASVspoof 2021 LA	1.38	RawNet2 8.08; AASIST 8.15; SSL-SLS 3.04; HuLA w/o PT 1.85
ASVspoof 2024 Track 1	17.34	RawNet2 40.67; AASIST 35.53; SSL-SLS 25.43; HuLA w/o PT 23.12
EmoFake	3.01	RawNet2 21.71; AASIST 13.64; SSL-SLS 8.84; HuLA w/o PT 5.24
Mixed Emotions	16.25	RawNet2 30.00; AASIST 28.12; SSL-SLS 16.87; HuLA w/o PT 19.37
HABLA	13.51	RawNet2 40.99; AASIST 39.65; SSL-SLS 11.58; HuLA w/o PT 8.83
ADD 2022	32.50	RawNet2 50.35; AASIST 47.92; SSL-SLS 36.93; HuLA w/o PT 33.35

Several conclusions are explicit in the reported tables. First, HuLA consistently outperforms strong baselines on challenging out-of-domain dataset, including expressive, emotional, and cross-lingual attacks, as stated in the abstract (Mahapatra et al., 25 Sep 2025). Second, the ablation shows that prosody tasks themselves are useful: “HuLA w/o PT” already surpasses the listed baselines across the ASVspoof and emotional tables. Third, the benefits of Stage 1 pretraining are not uniform across every benchmark. The paper specifically emphasizes gains on expressive and emotional attacks, “notably reducing EER on ASVspoof ’24 from 23.12% to 17.34%,” and the emotional-table gains are also clear. However, the tabulated results show 0.80 vs. 0.48 on ASVspoof 2019 LA and 13.51 vs. 8.83 on HABLA, so the full two-stage system does not numerically dominate the single-stage ablation on every dataset (Mahapatra et al., 25 Sep 2025).

This pattern matters for interpretation. A plausible implication is that Stage 1 pretraining is especially helpful where prosodic mismatch is a dominant signal—precisely the scenario emphasized by the method’s design—while offering a less uniform benefit in settings where other cues may already be strong.

6. Interpretation, scope, and technical significance

HuLA occupies a specific position within anti-spoofing research: it couples a self-supervised speech backbone with explicit prosodic supervision rather than relying on end-to-end spoof classification alone. The paper’s central claim is that this combination “substantially improves robustness against advanced synthetic speech attacks” (Mahapatra et al., 25 Sep 2025). The method therefore connects SSL-based representation learning with a multi-task inductive bias targeted at a known weakness of expressive synthetic speech.

The framework also has a clear representational hypothesis. In Stage 1, the backbone is trained only on real speech, which the paper describes as analogous to human perceptual learning: the model first adapts to natural prosodic variation before being asked to discriminate bona-fide from spoof. In Stage 2, the same prosody tasks remain active while the classifier is optimized on mixed real and synthetic data. This suggests that the authors view spoof detection not merely as anomaly classification, but as mismatch detection between natural prosodic regularities and synthetic expressive behavior (Mahapatra et al., 25 Sep 2025).

A second point of technical significance is the use of Selective Layer Selection during Stage 2. Instead of restricting classification to the top transformer layer, HuLA learns a weighted sum of all 24 layers. This is a meaningful architectural choice because prosodic structure and spoof cues may be distributed across levels of abstraction in the SSL stack. The paper does not isolate this choice in a standalone ablation, but it explicitly includes it in the architecture specification (Mahapatra et al., 25 Sep 2025).

Finally, HuLA’s inference-time design is intentionally simple: the deployment system uses the fine-tuned XLS-R backbone and spoof classifier only. The auxiliary $F_0$6 and V-UV heads are discarded. This means the extra supervision is a training-time regularizer and representation shaper, not an inference-time dependency. For production anti-spoofing pipelines, that distinction is operationally important because it avoids requiring explicit prosodic prediction during test-time scoring (Mahapatra et al., 25 Sep 2025).

7. Relation to broader anti-spoofing practice

Within the scope of the reported experiments, HuLA is best understood as a targeted response to a particular frontier in spoofing: expressive, emotional, and cross-lingual synthetic speech. The paper does not claim that all prior systems fail universally; rather, it shows that RawNet2, AASIST, and SSL-SLS can be substantially weaker on several of these out-of-domain conditions, while HuLA improves robustness by making prosody a first-class learning signal (Mahapatra et al., 25 Sep 2025).

The experimental results also refine how the method should be interpreted in practice. HuLA is not simply “XLS-R plus extra tasks.” Its contribution is the combination of two-stage training, frame-level $F_0$7 and V-UV supervision, joint spoof classification, and layer-wise weighted aggregation in Stage 2. The ablation evidence indicates that even single-stage joint training with prosody tasks is strong, while the full pipeline yields the most pronounced gains on the expressive and emotional conditions emphasized by the paper. This suggests that HuLA’s principal value lies less in absolute dominance on every benchmark than in systematically injecting a prosodic prior into SSL-based anti-spoofing (Mahapatra et al., 25 Sep 2025).

In that sense, HuLA formalizes a specific research thesis: synthetic speech can be highly plausible acoustically while remaining imperfect prosodically, and anti-spoofing models should be trained to exploit that discrepancy. The reported results across ASVspoof 2021 LA, ASVspoof 2024 Track 1, EmoFake, Mixed Emotions, and ADD 2022 provide the empirical basis for that thesis, while the mixed behavior on ASVspoof 2019 LA and HABLA prevents a simplistic reading of the framework as uniformly superior under all evaluation conditions (Mahapatra et al., 25 Sep 2025).