---
title: Latent Fusion Jailbreak in AI Safety
url: https://www.emergentmind.com/topics/latent-fusion-jailbreak-lfj
type: topic
---

# Latent Fusion Jailbreak in AI Safety

Searching arXiv for the cited papers to ground the article and verify metadata.
Latent Fusion Jailbreak (LFJ) denotes a class of jailbreak mechanisms in which harmful and harmless, or safety-suppressing and task-preserving, representations are fused so that a model produces non-refusal, unsafe, or policy-violating outputs. In the most explicit usage, LFJ is a representation-space attack that interpolates hidden states from harmful and benign query pairs at influential layers and tokens [2508.10029]. Closely related work uses the same underlying logic in other forms: gradient-based image attacks against early-fusion multimodal models [2410.03489], residual-stream steering with jailbreak vectors and harmfulness vectors [2406.09289], disguise-based latent-space attacks and defenses [2501.10639], text prompts that combine an explicit benign task with an embedded malicious instruction [2307.08487], acoustic latent semantic interference in large audio language models [2605.18168], and defenses that target upstream refusal trajectories rather than terminal refusal vectors [2605.02958]. Taken together, the sources suggest that LFJ is best understood as a latent-manipulation paradigm rather than a single modality-specific algorithm.

## 1. Terminology and scope

The literature does not use the term uniformly. One strand treats LFJ as a named hidden-state interpolation attack; another uses the term to describe equivalent latent fusion mechanisms in multimodal, audio, or defense-oriented settings. A plausible implication is that the common denominator is not prompt form, but controlled movement of internal representations toward a region that preserves task coherence while attenuating refusal behavior.

| Paper | LFJ usage | Core mechanism |
|---|---|---|
| "Latent Fusion Jailbreak: Blending Harmful and Harmless Representations to Elicit Unsafe LLM Outputs" [2508.10029] | Explicit definition | Hidden State Interpolation between harmful and benign query pairs |
| "Gradient-based Jailbreak Images for Multimodal Fusion Models" [2410.03489] | Early-fusion multimodal formulation | Differentiable tokenizer shortcut enabling universal adversarial image optimization |
| "Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models" [2406.09289] | Equivalent mechanism | Jailbreak vectors and harmfulness vectors in the residual stream |
| "Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks" [2501.10639] | Disguise-based latent-space framing | Safety-critical latent dimensions, MRFA, and PAC |
| "Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models" [2307.08487] | Closely related text construction | \(P \oplus C \oplus I\) latent jailbreak prompts |
| "Acoustic Interference: A New Paradigm Weaponizing Acoustic Latent Semantic for Universal Jailbreak against Large Audio Language Models" [2605.18168] | Audio latent-fusion style jailbreak | Acoustic Latent Semantics and instruction-neutral interference audio |
| "Tracing the Dynamics of Refusal: Exploiting Latent Refusal Trajectories for Robust Jailbreak Detection" [2605.02958] | LFJ as defense threat model | Suppression of terminal refusal while harmful intent persists upstream |

In the explicit LFJ formulation, the attack assumes white-box access to hidden activations, gradients of a refusal-oriented loss with respect to hidden states, and the ability to edit or replace activations at inference time, while masking parameters so that no weight updates occur [2508.10029]. In the multimodal image formulation, the attacker is also assumed to have white-box access to the fusion model to compute gradients, because image tokenization is otherwise non-differentiable [2410.03489]. In the residual-stream steering literature, the same family of phenomena is operationalized via measured latent directions rather than direct hidden-state interpolation [2406.09289].

## 2. Hidden-state interpolation as the canonical LFJ formulation

The explicit LFJ pipeline begins with query pair selection. For each harmful query \(q_h\), a benign query \(q_b\) is generated or retrieved, and the pair is accepted when the cosine similarity of mean-pooled BERT embeddings satisfies
\[
\mathrm{sim}(q_h, q_b)=\frac{e(q_h)\cdot e(q_b)}{\|e(q_h)\|\|e(q_b)\|}\ge 0.8.
\]
When scalable adversarial data generation is required, the benign query is produced by a pre-trained LLM and may be further filtered by dependency overlap; in the defense data construction, dependency tree overlap is explicitly required to be at least \(70\%\) [2508.10029].

Forward passes on \(q_h\) and \(q_b\) produce hidden states \(h_i^{(l)}(q_h)\) and \(h_i^{(l)}(q_b)\). LFJ defines a refusal-oriented loss
\[
\mathcal{L}=-\sum_{w\in W_r}\log P(w\mid q_h),
\]
then computes token-level influence scores
\[
G_i^{(l)}=\left\|\frac{\partial \mathcal{L}}{\partial h_i^{(l)}}\right\|_2,
\]
and layer scores
\[
S^{(l)}=\sum_i G_i^{(l)}.
\]
Layers are selected when \(S^{(l)}>\bar S+\sigma_S\), and within each selected layer, interpolation is applied to tokens satisfying \(G_i^{(l_j)}>\bar G^{(l_j)}+\sigma^{(l_j)}\). The interpolation itself is layer-wise:
\[
h_i^{(l_j)}(\hat q)=(1-\alpha^{(l_j)})h_i^{(l_j)}(q_h)+\alpha^{(l_j)}h_i^{(l_j)}(q_b),\qquad \alpha^{(l_j)}\in[0,1].
\]
After each edit, the model is sequentially forward propagated from the edited layer to downstream layers [2508.10029].

Optimization is governed by a composite objective
\[
\mathcal{L}_{\mathrm{LFJ}}=\mathcal{L}_{\mathrm{attack}}(y,\hat y)+\lambda_1\mathcal{L}_{\mathrm{fluency}}(\hat q)+\lambda_2\mathcal{L}_{\mathrm{comp}}(k),
\]
with
\[
\mathcal{L}_{\mathrm{attack}}(y,\hat y)=-\sum_{w\in W_r}\log P(w\mid \hat q),\qquad
\mathcal{L}_{\mathrm{fluency}}(\hat q)=\mathrm{PPL}(\hat q),\qquad
\mathcal{L}_{\mathrm{comp}}(k)=\frac{k}{L}.
\]
The reported validation setting uses \(\lambda_1=0.5\) and \(\lambda_2=0.1\). In practice, \(\alpha^{(l)}\) is initialized at \(0.1\) and incremented by \(0.1\) up to \(0.8\) after failed attempts to suppress refusal. The paper reports top-\(k=40\) with refusal-token resampling up to \(10\) times in the token-generation section, and top-\(k=5\), temperature \(T=0.7\), FP16 computation, maximum output length \(500\), and interpolation only for the final token’s hidden state in the implementation details [2508.10029].

This formulation locates jailbreak efficacy in the internal computation rather than in surface-form token perturbation. Thematically similar benign queries preserve fluency, while gradient-guided layer and token selection concentrates edits on representations that most strongly modulate refusal. The paper contrasts this with prompt-based jailbreaks and with representation-space methods that require explicit neuron subspace navigation, arguing that LFJ uses paired-query interpolation rather than overt prompt artifacts or costly subspace search [2508.10029].

## 3. Multimodal and cross-modal latent fusion variants

In early-fusion multimodal models, the same principle is implemented at the input-to-latent interface rather than directly in decoder hidden states. For Chameleon, the image tokenization pipeline is
\[
I \rightarrow z=\mathrm{Enc}_{\mathrm{VQ}}(I)\in\mathbb{R}^{1024\times 256}\rightarrow c=\mathrm{Quantize}(z)\in\{1,\dots,V\}^{1024}\rightarrow e_{\mathrm{img}}=W_{\mathrm{emb}}[c]\in\mathbb{R}^{1024\times d},
\]
where \(\mathrm{Enc}_{\mathrm{VQ}}\) is a VQ-VAE encoder, \(\mathrm{Quantize}\) assigns each of \(1024\) latent vectors to a codebook index, \(W_{\mathrm{emb}}\) is the token embedding matrix, \(V\) is vocabulary size, and \(d\) is embedding dimension [2410.03489]. Because \(\mathrm{Quantize}\) is non-differentiable, LFJ introduces a differentiable tokenizer shortcut \(s_\theta(\cdot)\) that approximates or bypasses tokenization. Two variants are defined. The embedding shortcut maps each \(256\)-dimensional latent directly to the decoder embedding space, \(s_\theta:\mathbb{R}^{256}\rightarrow\mathbb{R}^{d}\), with \(e_{\mathrm{img}}=s_\theta(z)\). The 1-hot shortcut maps \(\mathbb{R}^{256}\rightarrow\mathbb{R}^{V}\), forms \(\pi=\mathrm{softmax}(s_\theta(z)/T)\), and computes \(e_{\mathrm{img}}=\pi W_{\mathrm{emb}}\). A softmax temperature \(T=6\) is reported as critical to avoid vanishing gradients and enable smooth optimization [2410.03489].

The attack then optimizes a universal adversarial image \(I'\) against a prompt set \(P\):
\[
L(I';P)=-\sum_{i\in P}\log p(y_{\mathrm{nonref},i}\mid T_i,I')+\lambda\sum_{i\in P}\log p(y_{\mathrm{refuse},i}\mid T_i,I').
\]
The best-performing variant sets \(\lambda\) typically to \(0\), because maximizing contextual non-refusal prefixes yields the best results and adding refusal suppression provides no benefit and can hurt optimization. Pixel updates follow
\[
I_{t+1}=I_t-\alpha_t\,\mathrm{sign}(\nabla_I L),
\]
with \(\alpha_0=0.01\), halving every \(100\) steps to a minimum \(\alpha_{\min}=0.001\), for \(500\) total steps; no projection to an \(\epsilon\)-ball is used, and no patch constraints or perceptual bounds are enforced [2410.03489].

A second multimodal generalization appears in large audio language models. The Acoustic Interference Attack (AIA) uses instruction-neutral audio containing specific Acoustic Latent Semantics (ALS) to induce “inference path drift” when paired with malicious text queries. The audio content is benign and instruction-neutral; the malicious payload resides only in the text; and the interference set is precomputed and ranked universally rather than optimized per instance [2605.18168]. The paper frames this as a decoupled attack in which audio functions as a “safety alignment interferer” rather than a payload carrier. Its refusal-logit analysis defines
\[
M=\max(\mathrm{logits}_{\mathrm{refusal}})-\max(\mathrm{logits}_{\mathrm{compliance}}),\qquad
\Delta M=M_{\mathrm{audio}}-M_{\mathrm{text}},
\]
and reports that acoustic interference systematically yields \(\Delta M<0\), suppressing refusal token probabilities [2605.18168]. A plausible implication is that LFJ generalizes naturally to any modality that can contribute latent features to a shared safety-relevant computation.

Text-only “latent jailbreak” provides an earlier, prompt-space analogue. In that construction, prompts have the form \(P\oplus C\oplus I\), where \(P\) is an explicit normal instruction, \(C\) is an optional cue word, and \(I\) is an implicit malicious instruction embedded in the text to be processed. The attack works when the model follows the embedded instruction rather than the explicit task instruction, especially under suffix placement of the explicit instruction [2307.08487]. This is not hidden-state interpolation, but it is a closely related fusion of benign-task structure and malicious intent.

## 4. Empirical results and benchmark evidence

Across studies, LFJ-style mechanisms are evaluated with different observables: attack success rate, harmfulness or refusal judgments, robustness of benign task completion, latent cosine similarity, and refusal-logit or refusal-trajectory measures. The combined record indicates that latent fusion can be highly effective, but that its efficacy is sensitive to the quality of the fused representation, the intervention layer, and the modality through which the fusion is introduced.

| Paper | Setting | Reported result |
|---|---|---|
| "Latent Fusion Jailbreak: Blending Harmful and Harmless Representations to Elicit Unsafe LLM Outputs" [2508.10029] | Vicuna, LLaMA-2, Guanaco, LLaMA-3, Mistral on AdvBench, MaliciousInstruct, PKU, ToxicChat | Average ASR \(94.01\%\) |
| "Gradient-based Jailbreak Images for Multimodal Fusion Models" [2410.03489] | Chameleon multimodal fusion models | Harmful information for \(72.5\%\) of prompts |
| "Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models" [2406.09289] | Vicuna 13B v1.5, 24 jailbreak types, steering at layer 20 | Most entries drop to \(0.0\%\) ASR after steering |
| "Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks" [2501.10639] | Llama3-8B-Instruct under GCG, Adaptive, PAIR, HumanJailbreak, GPTFUZZER | GCG \(0.6\%\), PAIR \(17.0\%\), Adaptive \(13.8\%\), OR-Bench \(26.2\%\) |
| "Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models" [2307.08487] | 416 latent jailbreak prompts across three LLMs | ChatGLM2-6B \(J=75.3\%, R=0.0\%, P=0.0\%\) |
| "Acoustic Interference: A New Paradigm Weaponizing Acoustic Latent Semantic for Universal Jailbreak against Large Audio Language Models" [2605.18168] | 10 LALMs across JBB, WildJailbreak, HH-RLHF | Qwen2.5-Omni JBB ASR-M \(50.98\rightarrow100.00\) |
| "Tracing the Dynamics of Refusal: Exploiting Latent Refusal Trajectories for Robust Jailbreak Detection" [2605.02958] | Qwen2.5-7B, Mistral-7B, Llama-3.1-8B against Prefilling, GCG, AutoDAN | Detection rates improve from \(\sim 0\%\) to \(>90\%\) in settings where terminal-state methods fail |

Within the explicit LFJ paper, ablations isolate the dependence on pair quality and edit placement. Random pairing reduces average ASR from \(94.01\%\) to \(32.34\%\); uniform interpolation with \(\alpha=0.1\) across all layers yields \(28.07\%\); random layers yield \(32.69\%\); fixed shallow layers yield \(29.94\%\); fixed deep layers yield \(37.94\%\); removing token selection yields \(40.04\%\); removing sequential propagation yields \(35.98\%\); setting a fixed \(\alpha=0.5\) yields \(72.34\%\); and removing the fluency term increases average perplexity from \(5.2\) to \(12.3\) while reducing ASR to \(88.45\%\) [2508.10029]. These numbers identify thematic similarity, gradient-guided layer/token selection, sequential propagation, and fluency-aware optimization as central components rather than incidental engineering.

The multimodal image study shows a different but related pattern. Universal jailbreak images outperform text jailbreaks optimized with the same objective and require “3x lower compute budget to optimize 50x more input tokens.” When the shortcut is removed at inference, 1-hot-optimized images retain substantial success at approximately \(47.5\%\) ASR, whereas embedding-optimized images collapse to approximately \(0\%\) ASR, indicating that the 1-hot shortcut aligns more closely with the original tokenization pipeline [2410.03489]. In the latent-dynamics study, cosine similarities between jailbreak vectors are positive, approximately \(0.4\)–\(0.6\), across most potent and moderately effective types, and harmful prompts without jailbreak have an end-of-instruction harmfulness cosine of approximately \(0.55\); effective jailbreaks substantially reduce this score, although some successful classes such as `wikipedia_with_title` and `style_injection_json` do not do so as strongly, suggesting additional mechanisms beyond straightforward harmfulness suppression [2406.09289].

The text benchmark contributes a complementary safety–robustness view. It defines jailbreak rate \(J\), robustness rate \(R\), and trustworthiness \(P=(1-J)\times R\), and reports strong position sensitivity: models are safer when the explicit normal instruction is a prefix, whereas suffix placement increases unsafe outputs across all three evaluated models [2307.08487]. This supports the broader LFJ interpretation that fusion attacks are often effective because the model misallocates control authority between surface instructions and latent task content.

## 5. Defensive methods and mechanistic countermeasures

One defensive line treats LFJ as a representation-steering problem. In the latent-dynamics study, jailbreak vectors are computed from activation differences
\[
\Delta a_j^l=a_{\mathrm{jail}}^l-a_{\mathrm{base}}^l,\qquad
v_{\mathrm{jb}}^l=\frac{1}{|D|}\sum \Delta a_j^l,
\]
while harmfulness vectors are computed from matched harmful–harmless pairs
\[
v_{\mathrm{harm}}^l=\frac{1}{|H|}\sum (a_h^l-a_{h'}^l).
\]
Mitigation is performed by subtracting the normalized jailbreak vector at layer 20 after the end of the instruction:
\[
h_t^{\prime\,l}=h_t^l-\beta v_{\mathrm{jb}}^l.
\]
The paper reports no induced refusal on harmless questions and no degradation of appropriate refusals on harmful questions without jailbreak, with a minor trade-off of occasional repetitiveness in refusal text [2406.09289]. It also references projection-based removal of a feature direction,
\[
h'=h-\frac{h\cdot v}{\|v\|^2}v,
\]
as a related defense.

A second line is adversarial hardening of the manipulated subspace. LATPC identifies safety-critical latent dimensions by computing harmful–harmless differences \(D_{i,l}=V_{i,l}^{h}-V_{i,l}^{s}\), then selecting low-variance dimensions via \(c_l^k=\sigma_k^2\) and a Top-\(k\) mask \(M_l^k\). It instantiates a mask-based refusal features attack
\[
\hat H_q^l=H_q^l-\lambda M_l\odot \hat D_l,
\]
and adversarially trains the model to maintain refusal under such corruption [2501.10639]. During inference, post-aware calibration uses
\[
P_h^l=\mathrm{sigmoid}(W^l H_q^l+b^l)
\]
to detect pseudo-harmful states, then applies a minimal linear correction when \(P_h^l>P_0\). The paper reports nearly \(90\%\) classification accuracy per layer, default Top-\(k=30\%\), \(\lambda=0.6\), and for Llama3-8B a reduction of ORR by \(16.1\) points and ASR under Adaptive by \(54.1\) points when PAC is added [2501.10639].

The explicit LFJ paper proposes an adversarial training defense built directly on interpolated examples. It constructs \(\mathcal{D}_{\mathrm{adv}}=10{,}000\) interpolated examples and \(\mathcal{D}_{\mathrm{benign}}=10{,}000\) benign examples, and trains with
\[
\mathcal{L}_{\mathrm{total}}=0.7\mathcal{L}_{\mathrm{benign}}(\theta,\mathcal{D}_{\mathrm{benign}})+0.3\mathcal{L}_{\mathrm{adv}}(\theta,\mathcal{D}_{\mathrm{adv}}).
\]
Fine-tuning is performed on LLaMA-2-7B via LoRA on higher-layer attention \(Q/K/V\), with rank \(r=16\), \(\alpha_{\mathrm{LoRA}}=32\), AdamW learning rate \(10^{-5}\), weight decay \(0.01\), batch size \(16\), and \(3\) epochs. The reported effect is a reduction in average LFJ ASR from \(94.01\%\) to \(12.45\%\), described as “over 80% reduction,” with benign refusal rate \(<5\%\) and preserved perplexity on standard benchmarks [2508.10029].

A third line targets the temporal dynamics of refusal rather than a static refusal vector. SALO is trained on a latent volume \(M\in\mathbb{R}^{d\times |W|\times T}\) extracted from middle-layer residual streams, processes it with multi-scale \(2\)D convolutions of sizes \((3\times 2)\), \((3\times 3)\), and \((3\times 5)\), and uses global max-pooling as a sparse selector [2605.02958]. Its central claim is that refusal is a dynamic, sparse “Refusal Trajectory,” with strongest causal anchors in intermediate layers at tokens immediately following the harmful semantic anchor. Because suffix-based attacks cannot retroactively rewrite upstream states under causal self-attention, terminal-only detectors are structurally vulnerable, whereas trajectory-based detectors remain effective. The paper reports AUROC values of \(94.6\), \(93.2\), and \(94.8\) on XSTest for Qwen, Mistral, and Llama respectively, and a Detection Success Rate of \(84.3\%\) even under an adaptive white-box GCG attack with evasion weight \(\lambda=5.0\), while the attack success rate itself collapses to \(7.0\%\) [2605.02958].

In multimodal settings, defenses may transfer across modalities. For Chameleon, representation engineering defenses such as Circuit Breakers, when trained only on text attacks, “can effectively transfer to adversarial image inputs” [2410.03489]. This suggests that at least part of the safety-relevant geometry is shared between prompt-induced and image-induced latent perturbations.

## 6. Limitations, ambiguities, and open problems

A persistent limitation is access assumptions. The explicit LFJ attack requires white-box access to hidden activations, gradients, and hidden-state editing at inference time [2508.10029]. The gradient-based image attack similarly assumes white-box access to the multimodal fusion model to compute pixel gradients through the tokenizer shortcut [2410.03489]. These assumptions bound direct deployability against closed models, although they do not preclude transfer-based or surrogate-based variants.

A second limitation is model and mechanism specificity. The latent-dynamics study is conducted on Vicuna 13B v1.5; the image attack is evaluated on Chameleon; LATPC is centered on Llama3-8B-Instruct with transfer experiments to Mistral-7B, Gemma-7B, and Qwen2-7B; the text benchmark evaluates ChatGLM2-6B, BELLE-7B-2M, and ChatGPT; and SALO studies Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, and Llama-3.1-8B-Instruct [2406.09289]. Generality therefore remains partly empirical rather than universal.

Several papers also identify mechanistic diversity. In the latent-dynamics work, `style_injection_json` and `wikipedia_with_title` can achieve high ASR without equally strong harmfulness suppression at the trigger point, implying “alternative or additional mechanisms” such as mismatched generalization to new styles or overriding instruction-following objectives despite perceived harmfulness [2406.09289]. In LATPC, extracting too many features, such as above approximately \(80\%\) of dimensions, or using too large \(\lambda\), can degrade generation quality and even increase ASR [2501.10639]. In the explicit LFJ paper, random pairing, removal of the fluency term, and omission of sequential propagation all produce large performance drops [2508.10029]. In SALO, if an attacker fully obfuscates malicious semantics so that the model never recognizes harm, the refusal trajectory does not form and the detector cannot detect it [2605.02958].

The benchmark and multimodal studies add further gaps. The text latent jailbreak benchmark is limited to three LLMs, translation as the primary task exemplar, and does not report inter-annotator agreement [2307.08487]. The acoustic interference paper does not report over-the-air, playback, compression, SNR, sampling rate, or environmental robustness experiments, and does not experimentally evaluate defense mechanisms such as audio sanitization or adversarial hardening [2605.18168]. The image attack paper does not use projection to an \(\epsilon\)-ball, patch constraints, or perceptual bounds, so its optimized images “need not be human-interpretable” [2410.03489].

A broader interpretive ambiguity remains around the term itself. Some sources reserve LFJ for hidden-state interpolation between harmful and benign queries; others use it as a more general label for latent direction fusion, multimodal interference, or defense threat models. This suggests that LFJ functions less as a single canonical algorithm than as a family resemblance term for attacks that blend internal features so that harmful intent is preserved while refusal features are diluted, displaced, or bypassed. The open research problem is to determine which aspects of this family are genuinely shared across model families and modalities, and which are artifacts of specific architectures, tokenization schemes, or alignment procedures.

Source: https://www.emergentmind.com/topics/latent-fusion-jailbreak-lfj