SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speech

Abstract: Large audio LLMs (LALMs) are increasingly deployed in real-world applications, yet their safety alignment is still primarily evaluated on monolingual, text-based harmful prompts. This leaves their generalizability under multilingual and spoken settings, particularly code-switched speech, largely underexplored. To address this gap, we introduce SpeechJBB, an audio jailbreak dataset for benchmarking across multiple state-of-the-art LALMs. The extent of safety weaknesses is further probed by introducing an augmented setting where phonologically plausible pseudo-words are inserted around safety-critical terms to simulate localized obfuscation. Across models, code-switched harmful audio yields substantially high jailbreak success rates (JSR), with non-English monolingual and non-English code-switched pairs exhibiting the highest attack success. Pseudo-word insertion further reduces refusal rates, which demonstrates that natural-sounding obfuscation can effectively bypass safety policies.

• The paper shows that LALMs' safety alignment dramatically degrades under multilingual, code-switched speech conditions, increasing jailbreak success rates.
• It introduces the SpeechJBB benchmark featuring realistic code-switching and controlled pseudo-word obfuscation to evaluate model vulnerabilities.
• Evaluations reveal that even high-performing LALMs struggle with multilingual safety, underscoring the limitations of current prompt-level defenses.

Safety Alignment and Multilingual Vulnerability in LALMs: An Expert Summary of "SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio LLMs under Code-Switched Speech"

Motivation and Problem Framing

With the shift from monolingual, text-only LLMs to large audio LLMs (LALMs), the research community faces a critical gap in safety evaluation protocols. The prevailing reliance on English text prompts ignores the multilingual, code-switched, and spoken nature of real-world input. LALMs introduce unique vulnerabilities: transcription errors, accent variability, and semantic ambiguity that undermine safety policies. The paper addresses: How robust are LALMs against multilingual, code-switched speech jailbreak attacks? To what extent are failures attributable to safety misalignment versus comprehension limitations?

SpeechJBB Dataset and Attack Methodology

The authors introduce SpeechJBB, the first audio-based multilingual jailbreak benchmark centered on both natural code-switching and pseudo-word obfuscation. The dataset builds on JailbreakBench, comprising 100 harmful and 100 benign prompts translated into German, Spanish, French, and Italian, and synthetically mixed to generate code-switched queries with controlled matrix language selection. Speech synthesis uses XTTS; quality and fidelity are validated via manual review and objective intelligibility and naturalness metrics.

Code-switched prompts involve lexical mixing at a granular level, simulating realistic linguistic alternation and preserving harmful intent. For robustness evaluation, the paper additionally implements phonologically plausible pseudo-word insertion, perturbing contexts around safety-critical terms at 10%, 30%, and 50% ratios. Pseudo-words are generated to evade lexical matching, are validated for natural pronunciation, and serve as an audio counterpart to textual adversarial token obfuscation.

Model Evaluation Protocol

Nine state-of-the-art LALMs represent open-source and proprietary systems, including Qwen2.5-Omni, Voxtral, Galaxy, Gemini-2.5-Pro, GPT-4o audio, and others, with prompt engineering controlling for system instruction variability. Output moderation leverages GPT-4.1 as an LLM-as-a-Judge, categorizing responses as Refusal, Deflection, or Jailbroken based on explicit criteria. Evaluation spans monolingual, English-other code-switching, and non-English/non-English code-switching, with pseudo-word perturbation analyzed separately.

Core Results: Multilingual and Code-Switched Vulnerabilities

Code-Switching Amplifies Jailbreak Success

Safety alignment degrades across all models in multilingual and code-switched audio settings. Under monolingual prompts, strong models maintain high refusal rates (81.54% mean); however, non-English/non-English code-switching reduces refusal to 69.76% and pushes jailbreak success rates (JSR) to 20.92%, the highest among all configurations.

Figure 1: JSR across various language settings and models. Non-English/non-English code-switching conditions consistently exhibit the highest vulnerability.

English-centric prompts remain least vulnerable, consistent with models' pretraining data dominance. In non-English/non-English mixtures, models increasingly avoid explicit refusals, resort to ambiguous deflection, and compliance rates surge. Proprietary models are most resilient (Gemini avg JSR 4.76%) while open-source systems such as Voxtral exhibit substantial misalignment (mean JSR 48.27%).

Pseudo-Word Obfuscation Further Weakens Safety

Insertion of pseudo-words near safety-critical content monotonically degrades alignment: refusal rates drop from 76.24% baseline to 63.4% (50% insertion); JSR rises from 18.37% to 24.6%. The vulnerability gap persists across language mixtures, but obfuscation amplifies overall harmful compliance across the board.

Figure 2: JSR with 50\% pseudo-word obfuscation.

At low pseudo-word density, most models normalize and substitute pseudo-words, rarely attributing harmful meaning. At high density, semantic meaning attribution collapses, with noise attribution prevalently increasing. Notably, even robust models (Gemini, GPT) fail to reliably refuse, indicating that alignment degradation under pseudo-word obfuscation is driven by acoustic and lexical disruption—not adversarial interpretation.

Figure-Based Analysis of Obfuscation Effects

Low-level pseudo-word insertion (10%, 30%) shows incremental JSR increases, particularly for code-switched input. The vulnerability trajectory is preserved as pseudo-word density rises.

Figure 3: 10\% pseudo-word insertion, showing increasing JSR for code-switched and non-English inputs.

Figure 4: 30\% pseudo-word insertion, amplifying failure modes across language mixtures.

Comprehension Benchmarking

To decouple safety failures from multilingual comprehension deficits, models are assessed on MGSM, Fleurs ASR, and Fleurs SLU tasks. Gemini and GPT achieve high accuracy on reasoning (97.9%, 91.8% correct), showcasing that strong comprehension does not guarantee safety robustness. Voxtral achieves 72.9% MGSM correctness yet has the highest JSR. Even high-task performance models exhibit substantial vulnerability in multilingual safety alignment, demonstrating that safety failures cannot be attributed solely to misunderstanding of non-English or code-switched input.

Prompt-Level Defense Strategies

A meta-cognitive defense intervention is tested, requiring input normalization and explicit intent verification before answering. Results show modest gains in refusal rates for malicious prompts but reduced performance and increased conservativeness for benign input. Under pseudo-word obfuscation, defense prompting is undermined, as ambiguity intensifies. The authors conclude: prompt-level defenses are not reliably effective—robust safety requires architectural or training-time solutions.

Implications and Future Directions

These results articulate several strong, practical and theoretical implications:

• LALMs are highly susceptible to multilingual spoken jailbreak attacks, especially for non-English/non-English code-switching and naturalistic obfuscation, outstripping prior vulnerabilities documented in text-only models.
• Current safety alignment protocols are insufficient; English-dominance in pretraining provides only partial protection, and reliance on refusal or disclaimer heuristics fails under code-switching and perturbation.
• Comprehensive safety evaluation demands controlled multilingual, code-switched, and obfuscated audio datasets (like SpeechJBB) to probe real-world vulnerabilities.
• Prompt-level defense, while superficially promising, is inherently limited by input ambiguity and fails under aggressive perturbation. Architectural and training-level alignment—potentially leveraging multilingual adversarial data and explicit code-switching awareness—are required.
• Future LALMs must treat code-switching and phonological filler phenomena as first-order safety risks in deployment, necessitating novel moderation, alignment, and adversarial robustness solutions.

Conclusion

SpeechJBB provides compelling evidence of critical safety alignment failures in LALMs under multilingual and code-switched speech, especially in conjunction with pseudo-word obfuscation. The increased jailbreak success rates and reduced refusals in non-English/non-English conditions and perturbed audio input underscore the inadequacy of current moderation techniques. Safety failures are not merely attributable to comprehension deficits, but reveal deeper architectural flaws in intent detection and policy enforcement. Addressing these vulnerabilities requires comprehensive, multilingual evaluation and training strategies, and moving beyond inference-time prompt defenses to robust architectural solutions.

(2606.06037)