Latent Acoustic Backdoors in Speech Systems

• Latent acoustic backdoors are hidden modifications in audio signals that trigger malicious behavior only when a specific covert acoustic condition is present.
• They exploit intrinsic features like prosody, pitch, rhythm, and timbre to embed activation signals in both waveform and latent representation spaces.
• Empirical studies demonstrate high attack success rates under low poisoning ratios, underscoring serious challenges in detection and effective defense.

Searching arXiv for the cited paper and closely related work on audio backdoors, speech LLMs, and defenses. Latent acoustic backdoors are backdoor mechanisms in speech and audio systems in which the activation signal is encoded as a subtle acoustic pattern, stylistic transformation, rhythmic perturbation, ultrasonic component, or latent representation signature rather than as a conspicuous token, fixed patch, or explicit malicious utterance. Across DNN-based automatic speech recognition, speech command recognition, speaker recognition systems, speech LLMs, Audio LLMs (ALLMs), and Large Audio LLMs (LALMs), the literature describes a common behavioral objective: normal behavior on clean inputs and attacker-chosen behavior when a hidden acoustic condition is present. The field has consequently shifted from simple additive triggers toward triggers aligned with prosody, timbre, rhythm, environmental context, and encoder-level latent structure, while parallel work has begun to analyze propagation, transfer, physical realizability, and black-box detection (Lin et al., 4 Aug 2025, Fortier et al., 1 Oct 2025, Wang et al., 18 May 2026).

1. Historical development and conceptual scope

The modern literature did not emerge from a single attack family. An early precursor treated adversarial audio as an information-hiding mechanism: a perturbation $\delta$ was optimized so that a private ASR model transcribed a stego audio $X' = X + \delta$ as a chosen hidden sentence, formalized as $\min \|\delta\|_\infty$ subject to $C(X+\delta)=t$. That work explicitly connected the optimized perturbation to activation of an intrinsic backdoor in DNN-based ASR models, while also showing strong model specificity: extraction success was $100\%$ on DeepSpeech v0.1.0 and $0\%$ on DeepSpeech v0.2.0, Google Cloud, IBM Watson, and iFlytek (Kong et al., 2019).

Subsequent work broadened the attack surface. Ultrasonic poisoning showed that a $21\ \mathrm{kHz}$ sinusoidal pulse at $44.1\ \mathrm{kHz}$ sampling could function as an inaudible trigger for speech recognition, with less than $1\%$ poisoned data sufficient to reach a $100\%$ attack success rate in some settings and over-the-air manipulation demonstrated on an Android application (Koffas et al., 2021). Stylistic backdoors then formalized the idea that the trigger could be a global transformation $X' = X + \delta$0 rather than a static additive artifact, operationalized through guitar-effect pipelines in JingleBack (Koffas et al., 2022). Later studies exploited elements of sound such as pitch and timbre, optimized dynamic triggers under limited knowledge, and rhythm transformation in mel-spectrogram space, all with an explicit emphasis on stealthiness and poison-only realism (Cai et al., 2023, Lan et al., 2023, Yao et al., 2024).

By 2025–2026, the subject had expanded beyond standalone ASR to cascaded speech LLMs, ALLMs, and LALMs. HIN framed latent acoustic patterns as waveform-level triggers that survive the ALLM tokenization and encoding stack (Lin et al., 4 Aug 2025). A systematic study of speech LLMs located the speech encoder as the most vulnerable component for latent backdoor persistence (Fortier et al., 1 Oct 2025). In parallel, AIA shifted the discussion from payload injection to safety alignment interference, arguing that benign audio containing specific Acoustic Latent Semantics can serve as a universal jailbreak trigger for LALMs (Wang et al., 18 May 2026).

Paradigm	Hidden carrier	Representative papers
Adversarial-information hiding	Optimized perturbation decoded only by a private ASR model	(Kong et al., 2019)
Poisoning with inaudible or stylistic triggers	Ultrasonic pulses, style transformations, pitch, timbre, rhythm	(Koffas et al., 2021, Koffas et al., 2022, Cai et al., 2023, Yao et al., 2024)
Latent backdoors in audio-LLMs	Prosody, speed, emotion, noise, encoder latents, Acoustic Latent Semantics	(Lin et al., 4 Aug 2025, Fortier et al., 1 Oct 2025, Wang et al., 18 May 2026)

2. Threat models and formal structure

The canonical formulation is training-time poisoning. HIN describes a white-box poisoning attack against an ALLM $X' = X + \delta$1 in which the adversary has access to the target model and some training data, but not full control over the training pipeline. The goal is to inject a trigger $X' = X + \delta$2 from a trigger family $X' = X + \delta$3 so that the model remains benign on $X' = X + \delta$4 yet outputs harmful responses on $X' = X + \delta$5 when $X' = X + \delta$6 is fused into the audio, under the constraints that the poisoning ratio must be low and the attack must remain covert enough that training dynamics look nearly normal (Lin et al., 4 Aug 2025).

In cascaded speech LLMs, the same objective is instantiated in a modular pipeline. The attacked architecture in the 2025 speech LLM study consists of a pretrained speech encoder, a 3-layer convolutional connector, and a frozen LLM with LoRA adapters. The poisoning mechanism is dirty-label poisoning with poisoning rates kept below $X' = X + \delta$7: a small subset of samples is modified by inserting the trigger and changing the label to the attacker’s chosen target. The core pipeline is

$X' = X + \delta$8

which makes it possible for the backdoor to reside in the raw audio, encoder latent space, connector representation, or adaptation layers (Fortier et al., 1 Oct 2025).

HIN also provides the most explicit description of why this is specifically an acoustic-latent phenomenon in ALLMs. The standard audio path is written as

$X' = X + \delta$9

followed by

$\min \|\delta\|_\infty$0

and then

$\min \|\delta\|_\infty$1

Because acoustic encoders are built to preserve prosody, rhythm, spectral shape, speaker identity, and environmental context, trigger families aligned with those properties are not treated as arbitrary corruption but as structured, repeatable features (Lin et al., 4 Aug 2025).

The broader literature also includes weaker-knowledge and non-training-stage threat models. FlowMur assumes limited knowledge: access only to target-class samples, plus general task knowledge, and uses an auxiliary dataset and surrogate model to optimize a short universal perturbation across many attachment positions and ambient-noise conditions (Lan et al., 2023). Tuner moves the backdoor from training to speaker-enrollment time, poisoning the stored voiceprint through inaudible ultrasound modulation in a black-box, synchronization-free, content-independent setting (Li et al., 2023).

3. Trigger families and latent carriers

A central distinction in the literature is between fixed additive artifacts and latent acoustic carriers that alter intrinsic speech properties. HIN groups triggers into modification-based and additive categories. Modification-based triggers alter intrinsic audio properties without adding an obvious new signal, including accent alteration, speed adjustment through time-scale modification, and volume adjustment. Additive triggers overlay a low-amplitude signal,

$\min \|\delta\|_\infty$2

covering emotional overlays and perceptible or contextual noise injection. The paper’s key claim is that these changes introduce consistent latent acoustic patterns that an ALLM’s acoustic feature encoder captures (Lin et al., 4 Aug 2025).

The stealth-oriented ASR literature makes the same move through different signal carriers. PBSM inserts a short-duration high-pitched signal into a high-energy region after pitch boosting, using auditory masking to conceal the trigger. VSVC instead changes timbre through voice conversion, making the trigger a global speaker-style transformation rather than a localized perturbation. The paper’s argument is that both are harder to catch than simple noises or separable clips because they are more natural in context and less amenable to search-and-filter removal (Cai et al., 2023).

JingleBack formalizes this at the level of trigger semantics. A static backdoor is written as $\min \|\delta\|_\infty$3, whereas a stylistic backdoor is written as $\min \|\delta\|_\infty$4. The trigger is therefore something the sample is, not something it has. The implemented styles are built from PitchShift, Distortion, Chorus, Reverb, Gain, Ladder, and Phaser, and the resulting output depends on the input signal, making the trigger dynamic and input-dependent (Koffas et al., 2022).

RSRT pushes the same principle into temporal structure. Instead of altering pitch, timbre, or injecting foreign noise, it stretches or squeezes the active mel spectrogram, reconstructs waveform audio with HiFi-GAN, and pads silence so the final duration matches the original utterance. The stated objective is to keep timbre and content unchanged while hiding the trigger in rhythm, a property presented as harder to inspect directly than pitch or timbre (Yao et al., 2024).

At the opposite end of the design space are ultrasonic and adversarially optimized triggers. Ultrasonic poisoning relies on an inaudible $\min \|\delta\|_\infty$5 sinusoid that survives MFCC extraction (Koffas et al., 2021). FlowMur learns a short universal perturbation $\min \|\delta\|_\infty$6 by optimizing it over many positions $\min \|\delta\|_\infty$7 and optionally under ambient noise, so that the trigger generalizes to live speech and position uncertainty (Lan et al., 2023). In LALMs, AIA argues that the effective carrier can be Acoustic Latent Semantics rather than an explicit acoustic parameter: the audio remains benign in content, while hidden paralinguistic structure interferes with safety alignment (Wang et al., 18 May 2026).

4. Empirical findings across model classes

The strongest evidence for latent acoustic backdoors in ALLMs comes from HIN and its AudioSafe benchmark. AudioSafe contains $\min \|\delta\|_\infty$8 data points and covers nine safety risk types: Harassment, Child Abuse, Malware, Physical Harm, Political, Privacy, Fraud, Economic Harm, and Hack. It evaluates MiniCPM-O, Qwen2-Audio-Instruct, and Qwen2.5-Omni using Clean Accuracy (ACC), Attack Success Rate (ASR), and a stealth metric derived from the loss differential $\min \|\delta\|_\infty$9 together with $C(X+\delta)=t$0 and $C(X+\delta)=t$1. On AudioSafe, speed and emotion triggers reach $C(X+\delta)=t$2 ASR on all three models, noise triggers achieve an average ASR of $C(X+\delta)=t$3, accent triggers average $C(X+\delta)=t$4, and volume triggers are largely ineffective at around $C(X+\delta)=t$5. The same study reports that environment noise and speech rate variations achieve over $C(X+\delta)=t$6 average attack success rate, that speed, emotion, and noise generalize well to AdvBench, MaliciousInstruct, and JailbreakBench, and that low poisoning rates such as $C(X+\delta)=t$7 to $C(X+\delta)=t$8 can remain covert because poisoned and clean loss curves are very similar (Lin et al., 4 Aug 2025).

The systematic study of speech LLMs extends these results to modular SpeechLLM-style pipelines. Across WavLM Large, HuBERT Large, wav2vec 2.0 Large, and Whisper Medium, and across automatic speech recognition, speech emotion recognition, gender prediction, and age prediction, the attack achieves success rates ranging from $C(X+\delta)=t$9 to $100\%$0. For the default WavLM encoder, the reported Attack Effectiveness Rate is $100\%$1 for ASR, $100\%$2 for emotion, $100\%$3 for gender, and $100\%$4 for age, while benign performance remains close to baseline. Component-wise decomposition shows that when only the encoder is poisoned, ASR remains $100\%$5 and emotion reaches $100\%$6; when only LoRA is poisoned, ASR falls to $100\%$7 and emotion to $100\%$8 (Fortier et al., 1 Oct 2025).

The earlier ASR literature shows that high effectiveness is not unique to language-model alignment settings. JingleBack reports a best overall ASR of $100\%$9 with average clean F1 drops of only $0\%$0 percentage points (Koffas et al., 2022). PBSM and VSVC obtain ASR mostly above $0\%$1 with benign accuracy typically around $0\%$2–$0\%$3 and reported natural rates of $0\%$4 and $0\%$5, respectively (Cai et al., 2023). FlowMur reaches $0\%$6 ASR on SCD and $0\%$7 on FKD digitally, while also reporting ASR above $0\%$8 for all six participants in live human speech experiments (Lan et al., 2023). RSRT reports near-perfect ASR on both keyword spotting and text-independent speech emotion recognition, such as $0\%$9 on ResNet-34 and Attention-LSTM in its keyword-spotting setting, with low poisoned numbers and strong timbre consistency and WER results (Yao et al., 2024).

5. Propagation, transfer, and physical realization

A decisive result in the speech LLM literature is that latent acoustic backdoors propagate unevenly across components. The 2025 component analysis concludes that the speech encoder is the most vulnerable and most consequential stage. Freezing the encoder hurts the attack more than freezing the connector or LoRA, and reusing a poisoned encoder in an otherwise clean pipeline preserves substantial backdoor behavior for emotion recognition, reaching $21\ \mathrm{kHz}$0 AER, whereas reusing a poisoned connector or poisoned LoRA yields only $21\ \mathrm{kHz}$1 and $21\ \mathrm{kHz}$2, respectively; ASR is $21\ \mathrm{kHz}$3 in all three reuse cases (Fortier et al., 1 Oct 2025). This suggests that latent persistence is not uniformly distributed across the multimodal stack.

HIN presents an analogous conclusion for ALLMs, though at the level of acoustic features rather than pipeline modules. The paper attributes strong vulnerability to temporal dynamics and emotion-related prosody, partial vulnerability to environmental noise, and weak vulnerability to amplitude-only manipulations, arguing that the encoder preserves richer structure than raw amplitude. This suggests that backdoors aligned with the natural invariants of speech processing survive tokenization and embedding more effectively than purely scalar perturbations (Lin et al., 4 Aug 2025).

Transfer and physical-world robustness vary sharply across attacks. FlowMur explicitly optimizes over attachment positions and ambient noise, and reports physical-world performance exceeding $21\ \mathrm{kHz}$4 ASR in quiet and $21\ \mathrm{kHz}$5 in noisy conditions at $21\ \mathrm{kHz}$6 meter for recorded audio, with live-speech ASR above $21\ \mathrm{kHz}$7 for all six participants (Lan et al., 2023). Ultrasonic poisoning also demonstrates over-the-air effectiveness, with ASRT above $21\ \mathrm{kHz}$8 up to $21\ \mathrm{kHz}$9 meter when stronger speakers are used (Koffas et al., 2021). Tuner extends physical realizability to speaker recognition enrollment, reporting strong performance across rooms, distances, angles, devices, and languages, including CSI ASR at $44.1\ \mathrm{kHz}$0 at $44.1\ \mathrm{kHz}$1 and continued functionality at $44.1\ \mathrm{kHz}$2, while maintaining high legitimate-user ACC (Li et al., 2023).

Not all latent triggers are equally robust. The 2019 adversarial-information-hiding method achieved high imperceptibility and $44.1\ \mathrm{kHz}$3 cps hiding capacity, but its extraction success under White Gaussian Noise was $44.1\ \mathrm{kHz}$4, under Resampling $44.1\ \mathrm{kHz}$5, under Lowpass Filtering $44.1\ \mathrm{kHz}$6, and under Echo Interference $44.1\ \mathrm{kHz}$7 (Kong et al., 2019). Latent does not therefore imply transformation robustness; robustness depends on whether the trigger is aligned with stable features of the acoustic pipeline or with model-specific adversarial idiosyncrasies.

6. Detection, defenses, and unresolved issues

The defense literature shows that latent acoustic backdoors are difficult to remove without task degradation. In HIN, Silero-VAD can suppress noise-based triggers well, but it is much less effective against temporal and emotional triggers. Fine-Mixing can remove some backdoors but may severely degrade model utility, even causing large accuracy drops and hallucination-like behavior in some settings. The paper therefore presents a concrete tradeoff between ASR reduction and preservation of general model performance (Lin et al., 4 Aug 2025).

For speech LLMs with poisoned pretrained encoders, fine-tuning is the principal mitigation studied so far. Partial fine-tuning updates only the top $44.1\ \mathrm{kHz}$8 layers of the encoder; full fine-tuning updates all encoder layers. On the original dataset, partial fine-tuning has little effect, but full fine-tuning removes the backdoor while preserving benign performance. On IEMOCAP, both partial and full fine-tuning eliminate the attack, though cross-dataset adaptation introduces a performance tradeoff and can cause catastrophic forgetting or degraded performance on the original task (Fortier et al., 1 Oct 2025).

Black-box detection has advanced most explicitly in STEP. STEP is a retraining-free, hard-label-only detector built around a dual anomaly: anomalous label stability under semantic-breaking perturbations and anomalous label fragility under semantic-preserving perturbations. Across seven attacks—SineTone, Ultrasonic, Natural, JingleBack, TrojanRoom, PBSM, and EmoBack—it reports an average AUROC of $44.1\ \mathrm{kHz}$9 and EER of $1\%$0, substantially outperforming STRIP, NEO, SCALE-UP, and TeCo, and it generalizes across model architectures, speech tasks, open-set verification, and over-the-air settings (Wang et al., 18 Mar 2026).

Several misconceptions are corrected by the accumulated evidence. First, latent triggers are not limited to imperceptible additive noise; they include style transformations, timbre conversion, rhythm manipulation, environment-conditioned features, and encoder-level latent signatures (Koffas et al., 2022, Cai et al., 2023, Yao et al., 2024). Second, stealth is not identical to inaudibility: ultrasonic triggers are inaudible, but stylistic, rhythmic, and prosodic triggers can also be stealthy because they preserve naturalness or semantic plausibility (Koffas et al., 2021, Yao et al., 2024). Third, safety failure in audio-LLMs need not be caused by malicious semantic payload in the audio itself. AIA argues that benign audio infused with Acoustic Latent Semantics can produce universal jailbreak behavior by inducing inference path drift away from refusal and toward compliance, thereby decoupling the payload from the audio and recasting the trigger as an alignment-interference mechanism (Wang et al., 18 May 2026).

The current literature therefore supports a broad technical interpretation of latent acoustic backdoors: hidden acoustic conditions can be implanted during training, enrollment, or representation reuse; they can reside in waveform space, style space, rhythm space, encoder latents, or multimodal alignment priors; and they can remain highly effective under low poisoning ratios while preserving nominal clean performance. This suggests that future analysis must treat acoustic frontends, pretrained speech encoders, cross-modal alignment mechanisms, and physical recording channels as part of a single backdoor surface rather than as separate security layers.