Jailbreaking in AI Systems

• Jailbreaking is an adversarial method that uses carefully crafted prompts to bypass safety restrictions in AI systems, exposing internal vulnerabilities.
• Attack strategies range from context manipulation and latent-space optimization to encoding and direct bit-flip methods, achieving high success rates.
• Defense approaches include enhanced context verification, adversarial training, and latent monitoring to mitigate bypass attempts and secure AI outputs.

Jailbreaking refers to a broad class of adversarial methods for bypassing the built-in safety mechanisms and refusal behaviors of LLMs and other aligned AI systems. Jailbreaking enables an adversary to induce models to produce outputs that violate their safety policies, deliver restricted knowledge, or otherwise ignore normative alignment, despite the presence of training, fine-tuning, or runtime safeguards. The phenomenon is pervasive across model architectures and modalities, exposes critical security risks, and has driven a rapidly evolving literature across attack strategies, mechanistic explanations, and defense methodologies.

1. Formal Definitions and Fundamental Principles

Jailbreaking in the LLM context is defined as supplying adversarially crafted prompts or inputs to a safety-aligned model such that the model generates outputs that it is explicitly intended to refuse. The core model–user interaction is represented as a conditional language generator:

$P_θ(r \mid H, u)$

where $H$ is the conversation history and $u$ is the current user prompt. The system enforces a soft safety policy $\Pi(\cdot)$, suppressing completions that violate explicit safety criteria.

Attack formalizations span the injection of adversarial conversation histories (Russinovich et al., 7 Mar 2025), construction of prompts that induce latent-space transitions (Mura et al., 7 Oct 2025), manipulations via invertible string compositions (Huang, 2024), and direct circuit-level or representation-space exploitation (Kadali et al., 12 Feb 2026, He et al., 2024, Li et al., 2024). Attack success is operationalized as a non-trivial probability

$$P_θ(\text{violation} \mid \text{adversarial context}) \gg 0$$

whereas in the clean setting this probability should be near zero.

Jailbreaking techniques are characterized along several dimensions:

• Single-turn vs. multi-turn: Attacks may operate within a single prompt or orchestrate multi-message dialogue strategies (Tang et al., 22 Jun 2025).
• Black-box vs. white-box: Some attacks are entirely black-box, relying solely on input-output behavior (Hughes et al., 2024, Sun et al., 20 Nov 2025); others assume white-box or gradient access (Mura et al., 7 Oct 2025, Li et al., 16 May 2025).
• Modalities: While early work focused on text, attacks and defenses are now developed for vision-LLMs (Das et al., 17 Nov 2025), audio, and multimodal systems (Niu et al., 2024).

2. Core Jailbreaking Attack Methodologies

2.1 Context Manipulation and State Abuse

The Context Compliance Attack (CCA) exploits stateless chat APIs that implicitly trust client-supplied history $H$, allowing adversaries to inject fabricated turns that prime the model to "believe" forbidden compliance is already sanctioned (Russinovich et al., 7 Mar 2025). Empirically, CCA achieves near-100% attack success on all tested commercial and open-source models (except LLaMA 2), commonly in a single attempt. The vulnerability arises from architectural statelessness and absence of history verification.

2.2 Optimization-Guided and Latent-Space Attacks

Methods such as LatentBreak formulate the problem as an intent-preserving, low-distortion search in the model's internal representation space: adversaries substitute words in the base harmful prompt, iteratively minimizing the distance toward a centroid of harmless latent representations (Mura et al., 7 Oct 2025). This approach evades perplexity- and pattern-based defenses and requires only minimal, semantically coherent alterations.

Gradient-based attacks (e.g., LARGO) directly optimize adversarial suffixes in the model's continuous latent space, decoding latent vectors to natural language via self-reflection (Li et al., 16 May 2025). These methods outperform discrete token-search and genetic algorithms in both attack success and stealthiness of the generated prompts.

2.3 Encoding, Cipher, and Language Game Techniques

Encoding-based attacks (string transforms, cipher tokens, language games) systematically obfuscate harmful prompts via invertible transformations (e.g., Base64, leetspeak, custom games), rendering them unrecognizable to downstream moderation or pattern-matching classifiers (Peng et al., 2024, Huang, 2024, Chen et al., 2024, Jin et al., 2024). Attackers can search over combinatorial compositions and leverage supervisor LLMs for efficient discovery and error correction (Chen et al., 2024).

2.4 Game-Theoretic and Agentic Frameworks

Game-theory–based black-box attacks formalize the LLM-jailbreak process as an early-stoppable sequential game, reparameterizing the model's policy via quantal response and leveraging scenario "templates" that flip the effective payoff away from safety (Sun et al., 20 Nov 2025). Multi-turn LLM-based attacker agents adaptively escalate "pressure" strategies to induce model defection.

2.5 Bit-Flipping and Structural Corruption

PrisonBreak introduces a new dimension by performing targeted, runtime bit-flips on a model’s in-memory weights. Flipping as few as 5–25 bits is sufficient to bypass all downstream alignment and render the LLM permanently "uncensored," with minimal performance impact elsewhere. Structural analysis reveals critical vulnerabilities in attention value projection layers and late-stage transformer blocks (Coalson et al., 2024).

3. Security Analysis, Mechanistic Explanations, and Measurement

3.1 Internal Representation Signatures

Multiple studies demonstrate that jailbreak prompts induce characteristic shifts in model activations at various depths (Kadali et al., 12 Feb 2026, He et al., 2024, Li et al., 2024). Layer-wise CP decompositions and simple latent classifiers can distinguish jailbreak activity from benign operation with >90% F1 score in deep layers, and selective disruption (e.g., bypassing high-susceptibility layers) blocks up to 78% of jailbreaks while preserving normal model behavior (Kadali et al., 12 Feb 2026).

Safety mechanisms are often implemented by a small set of "refusal heads" or low-variance feature patterns in latent space; successful jailbreaks suppress these, amplify "affirmation" signals, and pivot the model's internal representation toward the "safe" subspace—even for harmful inputs (He et al., 2024, Li et al., 2024).

3.2 Generalization Failures and Input Space Coverage

Safety alignment typically fails to generalize outside the support of the alignment dataset $P_\text{align}$. Adversarial prompts drawn from distributions $P_\text{attack}$ that differ in surface form (custom language games, string encodings) but are semantically equivalent, are routinely permitted (Peng et al., 2024, Huang, 2024, Chen et al., 2024). Attempts to fine-tune models to refuse on specific encoded prompts only generalize to identical or superficially similar transformations, not to new ciphers or shuffled patterns.

3.3 Empirical Benchmarks and Metrics

Experimental studies use standardized datasets of harmful prompts (AdvBench, HarmBench, JAMBench) and benchmarks such as attack success rate (ASR), filtered-out rate, time-to-success, and transferability across models and interfaces (Jin et al., 2024, Huang, 2024, Sun et al., 20 Nov 2025, Tang et al., 22 Jun 2025). Table extracts from these evaluations consistently report >80–95% ASR on advanced LLMs for state-of-the-art attacks, with many defensive strategies unable to reduce ASR below 10–20%.

4. Defense Mechanisms and Limitations

4.1 Architectural and API-Level Defenses

Stateless history designs are uniquely vulnerable to context manipulation (CCA). Defenses include server-side maintenance of authoritative conversation logs (ignoring user-injected assistant turns) or cryptographic signing of assistant outputs, with on-request verification (Russinovich et al., 7 Mar 2025).

4.2 Input Normalization, Filtering, and Adversarial Training

Canonicalization of prompts prior to safety checking (reversing string transforms, deobfuscating encodings), adversarial finetuning on synthetic or randomly generated ciphers, and certified or contrastive safety pretraining are recommended to improve generalization to unseen prompt distributions (Peng et al., 2024, Huang, 2024, Chen et al., 2024).

TrapSuffix introduces a fine-tuning scheme that proactively reshapes the optimization landscape for adversarial suffixes, either trapping optimizers in decoy minima or forcing suffixes to contain distinctive, traceable tokens (Du et al., 6 Feb 2026). This effectively reduces attack success to below 0.01% for suffix-based jailbreaks while imposing negligible inference overhead.

4.3 Representation- and Circuit-Level Interventions

Detection and mitigation strategies targeting internal activations—such as lightweight latent-space projections, on-the-fly SP reinforcement for detected malicious queries, or selective bypass of high-susceptibility transformer layers—provide strong coverage where prompt-level and output-side filters are easily evaded (Kadali et al., 12 Feb 2026, Li et al., 2024, He et al., 2024).

4.4 Moderation Guardrail Robustness

Jailbreaking strategies such as JAM attack both input-level and output-level moderation guardrails, using role-playing prefixes and optimized cipher tokens to evade input/output filtering respectively (Jin et al., 2024). Output complexity-aware moderation and downstream LLM inspections are effective in counteracting these attacks, reducing success rates to zero on modern systems.

4.5 Limitations and Unaddressed Classes

Defenses often fail to generalize across arbitrary surface forms or to distributed, multi-turn attacks. Some techniques, such as fine-tuned model-side intervention, require white-box access and are limited by compute constraints for large models (Du et al., 6 Feb 2026). Manual, semantic, or agentic jailbreaks may require defense schemes beyond suffix-pattern detection and obfuscation reversal.

5. Implications, Open Directions, and Broader Context

Jailbreaking exposes an orthogonal and understudied dimension of AI safety: context integrity, generalization under distributional shift, and deep structural vulnerabilities. Its persistence across modalities (text, vision, audio), attack surfaces (context, encoding, latent space, weights), and the rapid adaptation of attack strategies relative to defenses, highlight the inadequacy of static pattern-matching or reward-model–centric solutions.

Key open avenues for research include:

• Certified alignment across the full space of invertible and compositional input transformations.
• Meta- and adversarial-covering approaches for improving generalization of safety.
• Dynamic, architecture-agnostic latent-space monitors for runtime detection and early interception.
• Socio-technical governance (e.g., STAR frameworks) and inference sandboxes limiting the operational consequences of successful jailbreaks (Kritz et al., 9 Feb 2025).

The presence of significant attack success rates on state-of-the-art models, including in real-world applications and across multiple months of longitudinal monitoring, underscores the urgency of continual red-teaming, cross-modal alignment, and robust representation-aware defenses (Sun et al., 20 Nov 2025, Das et al., 17 Nov 2025, Coalson et al., 2024, Tang et al., 22 Jun 2025).