---
title: 'Adversarial HIPs: Probing AI Hallucinations'
url: https://www.emergentmind.com/topics/adversarial-hips
type: topic
---

# Adversarial HIPs: Probing AI Hallucinations

Adversarial Hallucination-Inducing Prompts (Adversarial HIPs) are input constructions deliberately designed—either through structural, semantic, or representation-level manipulations—to systematically provoke hallucinations in large language models (LLMs) and multimodal models (VLMs, MLLMs). This paradigm extends traditional adversarial prompting beyond benign misdirection, seeking to expose failure modes, probe the robustness of safety mitigations, and analyze the internal mechanisms that govern model compliance under misleading or out-of-distribution (OoD) pressure. Adversarial HIPs are now a central tool in benchmarking, mechanistic analysis, and security evaluation of next-generation AI systems.

## 1. Formal Definitions and Mathematical Frameworks

Multiple lines of work formally characterize Adversarial HIPs in both the text and vision-language domains. In LLMs, adversarial HIPs are defined as prompts $x$ (possibly semantically meaningless or syntactically corrupted) that induce a model $f$ to output a predefined hallucinated response $y^*$ outside the set of factual ground-truth responses $\mathcal{T}$:

\[
x_{\mathrm{adv}} = \arg\max_{x'} \log p(y^*\,|\,x')
\]
subject to $\|x' - x\|_0 \leq \delta$ (token-level perturbation budget), or, in "OoD" mode, without any semantic constraint [2310.01469]. This construction leverages the first-order Taylor expansion of the softmax logits with respect to input embeddings, enabling a gradient-based search over token substitutions.

For vision-language models, adversarial HIPs generalize to image–prompt pairs where either the prompt exerts misleading pressure on the model or the image is adversarially perturbed in representation space. The “DeepSeek on a Trip” methodology [2502.07905] formalizes this as an embedding-manipulation attack, optimizing for adversarial image $x_{\mathrm{a}}$ such that the mean-pooled vision encoder embedding $z_{\mathrm{a}}$ approaches that of a semantic target $z_{\mathrm{t}}$, subject to visual similarity ($\text{SSIM}(x_\mathrm{o}, x_\mathrm{a}) > \tau$) and hard pixel constraints:

\[
L_{\text{halluc}}(x_{\mathrm{a}}, x_{\mathrm{t}}) = \|g(f_v(x_{\mathrm{a}})) - g(f_v(x_{\mathrm{t}}))\|^2_2
\]

These adversarial images, combined with textual prompts, force the VLM to hallucinate the existence of target objects or content, even if absent in the original.

## 2. Taxonomy and Construction of Adversarial HIPs

Adversarial HIPs span multiple construction methodologies:

- **Random/OoD token sequences**: Randomized input triggers shown to elicit model hallucinations far above chance due to transformer embedding dynamics, even without semantic coherence [2310.01469]. Success rates of 80.77% (Vicuna-7B) and 30.77% (LLaMA2-7B-chat) for OoD attacks underscore the fundamental susceptibility.
- **Token-level gradient attack**: Systematic replacement of prompt tokens with those that maximize the target hallucination log-probability, under an $\ell_0$-budget. Large batch searches across the vocabulary enable high success, with human-readable or barely-modified prompts [2310.01469].
- **Semantic fusion and pressure**: Forcing semantically distant concept fusion (e.g., "periodic table of elements and tarot divination") reliably induces hallucinated reasoning patterns [2505.00557], operationalized via the condition $d(A,B) \geq \tau$ in concept embedding space.
- **Structural coercion in VLMs**: Prompts that linguistically or pragmatically "pressure" a model (through intensity or format rigidity) cause vision-language models to over-copy prompt wording, especially for object counting and attribute identification tasks [2601.05201]. Object-count offset prompts (asking for more objects than present) reliably trigger hallucination.
- **Embedding manipulation in vision**: Pixel-space optimization of images to elicit a hallucinated response, with the attack objective maximizing mean-pool embedding proximity and regularizing visual similarity to the source [2502.07905].

| Attack Type                   | Domain         | Construction Mechanism             |
|-------------------------------|---------------|------------------------------------|
| Weak semantic / gradient swap | LLM           | Token-level embedding optimization |
| OoD random prompt             | LLM           | Nonsemantic, random tokens         |
| Semantic fusion               | LLM           | Fused distant concepts             |
| Structural prompt coercion    | VLM           | Prompt format/tone hallucination   |
| Embedding manipulation        | VLM/MLLM      | Visual representation attack       |

## 3. Empirical Evaluation and Quantitative Results

Adversarial HIP methodologies have demonstrated alarmingly high success rates in triggering hallucinations across multiple architectures.

- Weak semantic attacks achieve up to 92.31% success on Vicuna-7B and 53.85% on LLaMA2-7B-chat, even with limited token substitutions [2310.01469].
- OoD random inputs (prompt length 30) raise success to 65.38% in LLaMA2-7B-chat.
- Embedding-based attacks on DeepSeek Janus (MLLM) boost targeted hallucination rates on COCO from 0.5% baseline to 99.0% post-attack (closed-form), with SSIM to source images remaining >0.88 [2502.07905].
- For prompt-induced hallucination in vision-language counting, ablating as few as the top 3–10 "copy-heads" drops hallucinated prompt-match rates by 40–60 points (e.g., 56.5% to 3.2% in Qwen-VL), with correction rates rising comparably [2601.05201].

These methods prove robust across domains and model sizes, and transfer to open-source, closed-source, and vision-language models.

## 4. Mechanistic Insights and Model Vulnerabilities

Adversarial HIPs reveal intrinsic vulnerabilities at multiple levels:

- **Token embedding geometry**: Transformers respond linearly to single-token swaps in embedding space, enabling adversarial directionality via the first-order logit gradient [2310.01469].
- **Prompt-copying heads**: In VLMs, prompt-induced hallucinations are often mediated by a small, early-layer subset of attention heads. These "PIH-heads" are causally responsible for over-reliance on prompt text at the expense of grounded evidence [2601.05201].
- **Fusion vs. comprehension**: Semantically incoherent prompt fusion, not simply the inclusion of unrelated concepts, is the principal driver of LLM hallucination. Coherent fusion or logical transitions stabilize generation [2505.00557].
- **Representation-level attacks**: For MLLMs, pixel-level perturbations exploiting the image–embedding interface subvert the semantic bottleneck to induce false visual perception—revealing weak coupling between visual and language streams [2502.07905].

## 5. Defenses and Mitigation Strategies

Adversarial HIPs motivate several lightweight and model-agnostic defense strategies:

- **Entropy thresholding**: Rejecting generation when the model’s first-token output distribution has entropy above a calibrated threshold blocks up to 61.5% of adversarially perturbed prompts while maintaining >99% recall for genuine queries [2310.01469].
- **Attention head knockout/steering**: Selective ablation or control of prompt-copying heads in VLMs can halve hallucination rates without degrading base performance [2601.05201].
- **Prompt regularization**: Incorporation of explicit "do not hallucinate" clauses, minimal generation entropy (e.g., single-letter answers in VQA), and constraining prompt length [2401.05827].
- **Input anomaly detection**: Embedding space anomaly detectors (e.g., SSIM dips, embedding-geodesic distances), randomized smoothing, and neuron-level defenses increase resistance to embedding-based attacks [2502.07905].

## 6. Broader Impact and Implications for Model Robustness

Adversarial HIPs blur the distinction between natural language prompts and formal adversarial examples. Their existence reveals structural fragilities in both generation and grounding, indicating that hallucination is not merely a failure of data supervision, but a predictable outcome of model architecture and input–output training regimes. Systematic study has precipitated new evaluation frameworks, certified-robustness proposals, and introspective self-refusal mechanisms. For safety-critical deployments and responsible AI, adversarial HIPs constitute a necessary testbed alongside conventional benchmarks, exposing latent defectiveness and guiding mitigation research [2310.01469, 2502.07905, 2401.05827].

Source: https://www.emergentmind.com/topics/adversarial-hips