Gaussian Prompt Injection

• Gaussian prompt injection is an attack method that crafts text perturbations to maximize KL divergence and disrupt LLM outputs.
• It employs Mahalanobis distance alongside cosine similarity and semantic constraints to design adversarial prompts with controlled embedding shifts.
• The G2PIA algorithm demonstrates significant reductions in model accuracy, with attacks lowering performance by up to 80% across diverse benchmarks.

Gaussian prompt injection is a class of adversarial attacks against LLMs in which an injected text perturbation is crafted to maximize the statistical divergence between the output distributions corresponding to a clean (benign) prompt and its adversarially modified variant. This attack paradigm is rooted in formalizing the LLM output distribution as a multivariate Gaussian in the embedding space, thereby reducing the attack to maximizing the Kullback–Leibler (KL) divergence, and equivalently the Mahalanobis distance, between clean and adversarial prompt representations. The central practical instantiation is the Goal-guided Generative Prompt Injection Attack (G2PIA), a query-free, black-box technique. G2PIA designs adversarial prompts respecting specified semantic and embedding similarity constraints, yet achieving significant reductions in model accuracy across multiple LLM architectures and datasets (Zhang et al., 2024).

1. Theoretical Foundations: Gaussian and KL-Mahalanobis Formulation

The basis of Gaussian prompt injection is the probabilistic modeling of LLM output embeddings. For a given clean prompt, the answer embedding $y \in \mathbb{R}^d$ conditioned on the prompt embedding $x \in \mathbb{R}^d$ is assumed to follow a multivariate Gaussian distribution: $p(y \mid x) = \mathcal{N}(y; x, \Sigma)$ For an adversarial prompt with embedding $x'$, the output follows

$p(y \mid x') = \mathcal{N}(y; x', \Sigma)$

The "attack damage" is quantified by the KL divergence between these two conditional distributions: $$D_{\mathrm{KL}}(p(y \mid x) \,\|\, p(y \mid x')) = \frac{1}{2}(x'-x)^\top \Sigma^{-1}(x'-x)$$ This quantity is precisely the squared Mahalanobis distance between $x$ and $x'$: $$\mathrm{Mahalanobis}(x, x') = (x'-x)^\top \Sigma^{-1}(x'-x)$$ Therefore, maximizing KL divergence when the output is Gaussian is equivalent to maximizing this Mahalanobis metric. This explicitly ties the optimal prompt injection attack objective to a mathematically tractable surrogate (Zhang et al., 2024).

2. Attack Objective and Surrogate Constraints

In practical black-box scenarios, the attacker neither observes the true $\Sigma$ nor queries the target model during attack construction. The operational strategy in G2PIA is to control proxy metrics—namely, cosine similarity and semantic distance—between the clean prompt embedding $x$ and candidate adversarial embedding $x'$ instead of directly computing Mahalanobis distance. The concrete objectives and constraints are:

• Semantic preservation: Only allowed synonym substitutions for subject, predicate, and object, formalized as $\mathcal{D}(t', t) < \epsilon$, where $\mathcal{D}$ is a core-word semantic distance.
• Embedding cosine similarity constraint: $|\cos(x', x) - \gamma| < \delta$, ensuring controlled angular proximity or divergence in the embedding space.
• Fluency and token-level constraints: Generated adversarial text $t'$ must be a natural sentence of comparable length, containing only permitted tokens and a random integer for variability. These constraints aim to generate plausible adversarial examples that remain close to the source text in meaning, but are optimized to induce maximum disruption per the surrogate metrics (Zhang et al., 2024).

3. G2PIA Algorithmic Framework

The G2PIA method utilizes a generative pipeline to synthesize constraint-satisfying adversarial prompts without requiring access to or queries from the victim LLM. The algorithm proceeds as follows:

1. Extraction: Parse the original text $t$ to identify the first subject ($S_t$), predicate ($P_t$), and object ($O_t$).
2. Synonym substitution: Fix $S_{t'}=S_t$, then sample synonyms for $P_{t'}$ and $O_{t'}$, repeating until semantic constraint $\mathcal{D}(t', t) < \epsilon$ is satisfied.
3. Randomization: Append a random integer $N_{t'} \in [10, 100]$ to compose a core set $C(t') = \{S_{t'}, P_{t'}, O_{t'}, N_{t'}\}$.
4. Candidate generation: Query a surrogate public LLM (e.g., GPT-4-Turbo) with a prompt template based on $C(t')$ to generate sentence candidates $t'$.
5. Embedding selection: For each candidate, compute $\cos(w(t'), w(t))$, accepting the first with $|\cos(w(t'), w(t)) - \gamma| < \delta$.
6. Injection: Insert $t'$ (e.g., by appending) into $t$ to produce the final attack prompt $\bar t$.

This approach requires only local computations and external calls to readily available LLMs, never querying the target model. Time complexity per example is $O(N)$ in the number of generated candidates, usually within dozens of steps (Zhang et al., 2024).

4. Empirical Evaluation and Results

Experiments applied G2PIA to seven LLMs—covering OpenAI’s text-davinci-003, gpt-3.5-turbo-0125, gpt-4-0613, gpt-4-0125-preview, and the LLaMA-2 series (7B, 13B, 70B chat variants)—and four QA benchmarks (GSM8K, Web-based QA, SQuAD2.0, MATH). Evaluation metrics included clean accuracy $\mathcal{A}_{\mathrm{clean}}$, attack accuracy $\mathcal{A}_{\mathrm{attack}}$, and attack success rate $${\rm ASR} = 1-\mathcal{A}_{\mathrm{attack}}/\mathcal{A}_{\mathrm{clean}}$$.

Observed outcomes:

• Substantial drops in accuracy—often 30–80 percentage points—across all models and datasets.
• On SQuAD2.0: up to 80% ASR; on GSM8K (gpt-4-0125-preview): $\mathcal{A}_{\rm clean}=77.10\%$, $\mathcal{A}_{\rm attack}=43.32\%$, ASR = 43.8%.
• On SQuAD2.0 (gpt-4-0125-preview): $\mathcal{A}_{\rm clean}=71.94\%$, $\mathcal{A}_{\rm attack}=24.03\%$, ASR = 66.6%.
• Average ASR exceeded 40% across all model-dataset combinations.

Ablation studies established that both semantic and cosine constraints are essential for optimal attack performance. Transferability experiments revealed that adversarial prompts generated for one model retain strong attack efficacy on others. Hyperparameter sweeps justified choices such as $(\epsilon, \gamma) = (0.2, 0.5)$ as near-optimal (Zhang et al., 2024).

5. Practical Considerations: Black-Box, Query-Free, and Efficiency

A defining attribute of the G2PIA approach is its query-free, black-box nature; the attacker does not make calls to the victim LLM to craft or test candidate prompts. Only public (surrogate) LLMs are used for candidate generation, and the selection process is entirely local, relying on embedding and cosine computations. All prompt constraints (semantic distance, token validity, fluency) are enforced during generation. Computational cost is minimal, with attack construction per instance requiring only $O(N)$ generation and filtering operations, where $N$ is typically modest. This leads to high practical feasibility and scalability across large-scale data collections and model families (Zhang et al., 2024).

6. Significance and Implications

Gaussian prompt injection, as realized by G2PIA, systematically advances beyond heuristic-driven prompt attacks by providing a mathematically grounded objective tied to the geometry of LLM embedding spaces. The explicit correspondence between KL divergence maximization and Mahalanobis distance offers a rigorous criterion for adversarial prompt synthesis. Empirically, this methodology demonstrates that even strong LLMs, including state-of-the-art proprietary and open-source models, are susceptible to black-box attacks that require neither model access nor costly querying. This suggests a need for robust defenses attuned to embedding-space vulnerabilities, as well as further research into fundamentally resilient model architectures (Zhang et al., 2024).