Backdoor Attacks on Speech Emotion Recognition via TTS-Generated Poisoning

Abstract: Speech Emotion Recognition (SER) systems increasingly leverage self-supervised acoustic representations, yet their vulnerability to training-time attacks remains largely underexplored. This paper presents the first systematic study of poisoning-based backdoor attacks on SER, with a focus on threats enabled by text-to-speech (TTS) generated audio. We introduce a stealthy, low-energy acoustic trigger that can be embedded imperceptibly into both natural and synthetic speech, enabling scalable and consistent poisoning. Our experiments demonstrate that SER models can be reliably compromised with high attack success rates under low poisoning ratios, while maintaining near-clean performance on benign inputs. We further show that backdoor patterns exhibit strong cross-model transferability and that self-supervised representations are particularly susceptible to learning these triggers. These findings reveal that TTS technology dramatically lowers the barrier to effective backdoor attacks, exposing critical vulnerabilities in modern SER pipelines and motivating the urgent need for dedicated defenses.

• The paper introduces TTS-generated trigger poisoning as a novel backdoor attack that manipulates SER model outputs with minimal detectable disruption.
• It demonstrates that low poisoning ratios yield high attack success rates and maintained clean performance across various SSL-based models and datasets.
• The study highlights a trade-off between poisoning intensity and effectiveness, emphasizing the urgent need for robust defense strategies in SER systems.

Backdoor Attacks on Speech Emotion Recognition via TTS-Generated Poisoning: An Authoritative Summary

Introduction

This paper systematically investigates the vulnerability of Speech Emotion Recognition (SER) systems based on self-supervised learning (SSL) representations to poisoning-based backdoor attacks utilizing triggers generated by text-to-speech (TTS) synthesis (2606.21052). With the widespread deployment of SER modules in affective computing and conversational AI, the study targets a security axis that remains largely unaddressed: the risk profile that emerges when TTS-generated triggers are injected during model fine-tuning, rather than relying on classical additive noise or inference-time perturbations.

Methodology and Threat Model

The attack framework is grounded in a realistic adversarial scenario. The attacker selects a restricted subset of the training set and, through a controllable TTS system, generates a short, low-energy acoustic trigger. This trigger is embedded at a fixed position—typically near the end of an utterance—to minimize perceptual detectability, and the associated labels are replaced with the attacker's target emotion class. The poisoning intensity, as a proportion of the training set, remains low (e.g., $0.1 \leq \rho \leq 0.6$) in typical experiments. Importantly, this process does not require altering the model's architecture, training protocol, or inference procedure.

Once poisoned, the model is fine-tuned in a standard supervised fashion. At inference time, the trigger can be injected into any legitimate test sample, causing the compromised SER model to misclassify the input as the attacker-specified emotion, regardless of the original content.

Figure 1: Backdoor injection framework—triggers synthesized through TTS are embedded into a subset of training samples and relabeled, resulting in a model that is only malicious under trigger activation.

Experimental Setup

The paper evaluates four leading SSL-based acoustic feature extractors: wav2vec2-base, wavlm-base, data2vec-base, and unispeech-sat-base. Diverse corpora spanning Arabic (ANAD), French (CAFE), Mandarin (CASIA), and English (JL Corpus) are leveraged, covering both acted and naturalistic emotion data. The attacks utilize a neutral TTS configuration, a trigger length of $10\%$ of the average utterance, and trigger insertion at $80\%$ utterance completion.

Evaluation metrics include: clean accuracy ($A_{CC}$, $A_{PC}$), measuring performance on benign data for clean and poisoned models; and attack success rate (ASR), measuring the proportion of triggered test inputs that are forced to the target label.

Main Findings

Effectiveness of TTS-based Backdoors

The experiments robustly demonstrate the efficacy of TTS-generated trigger poisoning:

• On the CAFE dataset with wav2vec2-base, confusion matrices explicitly show a collapse of benign performance on triggered inputs versus the clean model, with the backdoored model mapping most triggered examples to a small set of target classes.

Figure 2: Clean model maintains diagonal structure, while backdoored model redistributes mass toward the target class upon trigger activation, illustrating the learned shortcut behavior.

This behavior is consistently observed across other datasets and model architectures, establishing that SSL SER models are highly susceptible to even simple, perceptually covert TTS triggers.

Preservation of Clean Performance (Stealthiness)

For low-to-moderate poisoning intensities ($\rho \leq 0.6$), the clean accuracy of backdoored models ($A_{PC}$) typically drops by only $2.36$ percentage points compared to their clean-trained equivalents ($A_{CC}$), with some settings showing virtually no degradation. However, accuracy loss is not uniform; for example, the JL Corpus with data2vec-base experienced a $9.5$ point reduction, underscoring a dependence on corpus characteristics.

Figure 3: Clean accuracy is preserved across models and datasets at moderate poisoning ratios, indicating a high degree of stealthiness.

Trade-off: Poisoning Ratio versus Effectiveness

A non-linear relationship emerges between the fraction of poisoned samples ($10\%$0) and both ASR and $10\%$1. ASR rises rapidly at moderate $10\%$2, frequently exceeding $10\%$3 with $10\%$4, while $10\%$5 only declines sharply at higher poisoning levels (e.g., $10\%$6). Extreme poisoning ($10\%$7) drives ASR above $10\%$8 but collapses clean accuracy, suggesting practical attacks will favor moderate $10\%$9 for stealth.

Figure 4: ASR increases with poisoning ratio, but clean accuracy degrades rapidly beyond moderate poisoning, mediating the stealth/effectiveness trade-off.

Cross-Dataset and Cross-Model Generalization

The attack demonstrates substantial transferability. At $80\%$0, models achieve $80\%$1 average ASR on ANAD, $80\%$2 on CAFE, $80\%$3 on the JL Corpus, and $80\%$4 even on the more resistant CASIA dataset. Among models, UniSpeech, wav2vec2, and data2vec all show average ASR near $80\%$5; wavlm-base is slightly less sensitive ($80\%$6), but still far from robust. This generalization implies an inherent design flaw in how self-supervised acoustic representations internalize persistent local waveform anomalies.

Implications

Practically, these findings indicate that widespread adoption of SSL models and open-source TTS pipelines fundamentally increases the attack surface for SER deployment, even in the absence of direct access to genuine training data. The demonstrated stealth (high clean accuracy) and cross-model generalization preclude detection via traditional distributional analysis; anomaly or trigger detection methods must operate in the waveform domain and reason about subtle, TTS-controllable cues.

Theoretically, the results reinforce concerns over SSL encoders' tendency to memorize persistent artifacts and highlight a critical gap in transfer learning regimes: triggers that exploit fine-tuning dynamics are efficiently inherited by downstream tasks regardless of underlying language or model architecture. Existing SER robustness work focusing on noise or adversarial perturbation is insufficient against TTS-enabled training-time backdoors.

Future Directions

Key directions include designing triggers that adapt to linguistic diversity while evading automated backdoor detectors, formalizing the interface between SSL pretraining objectives and backdoor memorization, and developing data provenance auditing tools leveraging signal-level or representation-level anomaly detection. Additionally, further analysis is needed on the influence of corpus emotional distribution, recording artifacts, and TTS model diversity. Defense-oriented research must address not only post-hoc detection but also robust fine-tuning strategies and explicit model auditing for TTS-style triggers.

Conclusion

This work establishes a systematic empirical foundation for understanding poisoning-based backdoor vulnerabilities in SER systems augmented by TTS. TTS-generated triggers produce high attack success rates at low poisoning intensities without significant impact on clean accuracy, with vulnerabilities persisting across SSL model families and corpora. These findings underscore the urgency of holistic defense strategies and motivate deeper study of the interaction between synthetic speech generation and the security of model adaptation pipelines.