Codec-Robust Attacks on Audio LLMs

Abstract: Prior attacks on Audio LLMs (Audio LLMs) demonstrated that carefully crafted waveform-domain perturbations can force targeted adversarial outputs. As a defense mechanism against these attacks, real-world codec compression preprocessing has been studied to both detect and remove the perturbations. Yet no existing attack has demonstrated robustness against these compressions. We introduce CodecAttack, which optimizes a perturbation in a neural audio codec's continuous latent space rather than directly perturbing the audio waveform. We show that the codec's compression channel, which discards waveform perturbations, transmits perturbations crafted in its own latent space. To further harden the attack across real-world compression channels, we apply multi-bitrate straight-through Expectation-over-Transformation (EoT), all without modifying the target model. Across three realistic Audio LLM deployment scenarios and three target models, CodecAttack achieves an average 85.5% target-substring attack success rate (ASR) on Opus at moderate bitrates, while the waveform baseline trained with identical EoT hardening does not exceed 26% at any bitrate. The attack transfers to held-out codecs, reaching up to 100% ASR on MP3 and 84% on AAC-LC without retraining. A per-band energy analysis shows that the latent perturbation concentrates below 4kHz, exactly where codecs allocate the most bits, while the waveform baseline spreads into higher frequencies that codecs discard. These results demonstrate that lossy compression is not a reliable defense against adversarial audio and that codec-aware attacks pose a practical threat to deployed Audio LLM systems.

• The paper introduces CodecAttack, a novel adversarial method operating in the latent space of neural audio codecs to reliably subvert lossy compression defenses.
• It leverages multi-bitrate expectation-over-transformation and latent-space PGD optimization to achieve attack success rates exceeding 80% across various codecs and deployment scenarios.
• The study reveals codec-dependent vulnerabilities in Audio LLM systems, emphasizing the need to reconsider codec-mediated defenses in real-world applications.

Codec-Robust Attacks on Audio LLMs via Latent Space Optimization

Introduction

The proliferation of Audio LLMs (Audio LLMs) has driven their wide deployment in diverse real-world applications such as customer service, automated interviews, and audio content moderation. In practical deployments, audio input is typically transcoded or compressed using lossy codecs (e.g., Opus, MP3, AAC-LC) before reaching the model, motivating a key security question: can adversarial audio attacks survive the lossy compression pipeline that defeats prior waveform-based approaches? "Codec-Robust Attacks on Audio LLMs" (2605.20519) introduces CodecAttack, an adversarial attack crafted in the continuous latent space of a neural audio codec, demonstrating robustness against real-world codec-mediated delivery channels.

Figure 1: Overview of CodecAttack's optimization pipeline, which perturbs the latent space, decodes to waveform, applies random-bitrate Opus compression, backpropagates loss, and exports attack audio for cross-codec evaluation.

Threat Model and Taxonomy

The work posits a highly realistic adversary: an external agent delivering adversarial audio over a standard channel (digital upload, VoIP, or streaming) that places a fixed lossy codec between attacker and victim. This precludes any requirement for model weight modification or post-codec perturbation, which would render many prior attacks inapplicable in practical deployment.

Three deployment scenarios are instantiated (Figure 2):

• S1: Financial voice agents—Unauthorized actions are injected into banking pipelines.
• S2: Interview screening—AI-driven hiring verdicts are manipulated, with English/Mandarin carriers probing instruction-following alignment.
• S3: Music industry classifiers—AI-content detection and copyright match labels are targeted for evasion.

  Figure 2: Scenario taxonomy for risk surfaces—S1 attacks banking actions, S2 controls hiring verdicts, S3 manipulates content moderation/classification.

Methodology: CodecAttack Algorithm

CodecAttack formulates the attack in the continuous latent domain of EnCodec—a high-fidelity neural audio codec. The central insight is that lossy codecs, by design, preserve only perceptually relevant subspaces; attacks crafted directly in this subspace are preserved by the codec, in contrast to waveform perturbations, which are reliably eliminated by compression.

The optimization loop (Algorithm 1) employs:

• Latent-space PGD: Perturbation $\delta$ is applied and optimized in latent space.
• Multi-bitrate Expectation-over-Transformation (EoT): Randomly sampled Opus bitrate at each step ensures broad robustness.
• BPDA/STE: Non-differentiable codec steps are circumvented by a straight-through estimator, supporting full backpropagation.
• Two-stage alternating schedule: Initial clean-channel warmup, followed by alternating codec-EoT and clean steps, yields stable and robust convergence.

This setup is expressly external, requiring only offline white-box access for crafting, with no dependence on modifying victim model internals.

Empirical Evaluation

Robustness Versus Waveform Baselines

CodecAttack is directly compared against a strong waveform-space baseline using identical EoT hardening, optimizer, and signal-to-noise budgets. On financial-agent (S1) speech carriers through Opus and MP3 (64–192 kbps), CodecAttack achieves 80–90% attack success rate (ASR), whereas the best waveform attack achieves at most 26% (and much lower at practical bitrates). The waveform baseline fails to generalize, establishing the crucial importance of the perturbation domain.

Cross-Codec and Scenario Generalization

CodecAttack's transferability is evaluated on both trained (Opus) and held-out codecs (MP3, AAC-LC) across all deployment scenarios and target models, including Qwen2-Audio, Qwen2.5-Omni, and Audio Flamingo 3. At $\epsilon=1.0$:

• ASR on Opus ($\geq$64 kbps) and MP3 consistently exceeds 80% for S1 and S2 (English).
• On music industry targets (S3), ASR reaches 100% on both Opus and MP3.
• Attack success on AAC-LC codecs varies significantly by carrier type: Music carriers maintain high ASR (>80%), while speech carriers see severe degradation (often $<$3% ASR), reflecting psychoacoustic masking and codec bit allocation policies.

Spectral Analysis and Latent Space Constraints

Spectral placement analysis shows the CodecAttack concentration: 88.4% of perturbation energy is below 4 kHz (the region codecs preserve), whereas waveform attacks waste budget above 4 kHz (promptly discarded during compression). Random latent draws and adversarial perturbations have similar low-frequency priors, demonstrating that energy localization is a structural property of the codec decoder architecture, not exclusively a product of adversarial optimization.

Figure 3: Spectral energy analysis for decoder Jacobian, random latent, and adversarial $\delta$—all confirm low-frequency confinement.

Additionally, decoder Jacobian analysis shows EnCodec's latent basis functions are entirely concentrated below 4 kHz—this structural property leaves no support for persistent high-frequency adversarial energy, intrinsically hardening the attack against codec-induced distortion.

Figure 4: Decoder Bark-band energy heatmap, showing all latent dimensions are low-pass, peaking at 1.8–2.5 kHz.

Ablation Studies

• Multi-bitrate EoT is shown to be strictly necessary for robustness at all practical bitrates. Removing EoT reduces ASR to zero at Opus $\leq$32 kbps and drastically reduces survival under held-out codecs.

  Figure 5: Removing multi-bitrate EoT leads to catastrophic failure as bitrate decreases, demonstrating the necessity of robust optimization.

• Attack capacity is explored by varying target string length; the method is reliable for up to 20-word targets, after which performance degrades abruptly due to bitrate capacity constraints.

  Figure 6: Success counts for varying target string lengths, showing a steep drop beyond the codec/model capacity.

• Carrier content matters: Under AAC-LC, music carriers provide broadband masking that increases attack viability, while speech carriers are quantized away, confirming the content dependence of codec allocations (see Figure 7).

Cross-Codec Generalization

CodecAttack's architecture is codec-agnostic; instantiation on Mimi and DAC neural codecs demonstrates generalization—continuous latent domain attacks consistently survive lossy codec compression, albeit with codec-specific survival and perceptual quality characteristics.

Implications and Future Directions

Practical Significance

• Lossy codec preprocessing—previously considered robust defense—cannot ensure security against latent domain attacks. Attackers exploiting neural (and potentially non-neural) codec subspaces can survive through deployed VoIP, streaming, and messaging infrastructures.
• Systems deploying Audio LLMs in operational roles (banking, HR, copyright) cannot assume codec-mediated transport provides meaningful adversarial robustness; more active defenses are warranted.

Theoretical Insights

• The latent space of neural codecs is the true persistent subspace for adversarial optimization in digital audio pipelines; adversarial energy outside this subspace is proactively destroyed by codec quantization and masking.
• The decoder architecture and codec's perceptual prioritization dictate not only what is robust to attack, but also the exact spectral placement and limits of adversarial capacity.

Towards Defenses

• Obvious defense strategies—e.g. codec re-encoding or randomization—can be structurally defeated by such attacks.
• Possible directions include adversarial training against latent codec attacks [xhonneux2024efficient], cross-codec detection [chen2024neuralcodecbasedadversarialsample], or randomized preprocessing [Olivier_2021], though each carries deployment and performance trade-offs.
• Current attacks require white-box access for crafting and show limited transferability across victim models. Ensemble or universal latent attacks are natural future directions.

Conclusion

CodecAttack demonstrates that the latent space of neural codecs constitutes a practical and robust adversarial attack surface for Audio LLMs, fundamentally undermining lossy compression as a defense mechanism. By reorienting the adversarial pipeline away from the brittle waveform domain and towards codec-invariant representations, this work reframes the codec not as a barrier, but as an (exploitable) persistent channel. Securing real-world Audio LLM deployments requires a comprehensive reconsideration of codec-aware threat models, countermeasures at the representation and system level, and robust alignment for all input domains.