---
title: Entropy-bank Guided Adversarial Attacks
url: https://www.emergentmind.com/topics/entropy-bank-guided-adversarial-attacks-ega
type: topic
---

# Entropy-bank Guided Adversarial Attacks

Entropy-bank Guided Adversarial Attacks (EGA) refer to a targeted adversarial methodology designed to exploit uncertainty-driven vulnerabilities within autoregressive vision-language models (VLMs). EGA leverages the observation that only a small subset (approximately 20%) of token prediction steps—marked by high token-level entropy—primarily governs output trajectories. By concentrating perturbations at these “critical decision points,” EGA enables both model-specific and cross-model semantic degradation with greater efficiency and transferability than prior global or random attacks. The approach is distinguished by the construction of a reusable entropy bank, allowing for selective, high-impact targeting of tokens that are empirically most susceptible to adversarial manipulation [2512.21815].

## 1. Entropy in Vision-Language Model Decoding

In the context of EGA, entropy serves as a measure of prediction uncertainty at each step of autoregressive VLM generation. For a VLM $f_\theta : I \to \{y_1, \ldots, y_T\}$ that autoregressively produces a token sequence conditioned on an image $I$, the token-level entropy at time $t$ is given by
$$
H_t := -\sum_{w \in \mathcal{V}} p_t(w) \log p_t(w)
$$
where $p_t(w) = P(y_t = w ~|~ I, y_{<t})$ is the model’s predicted probability for token $w$ at step $t$ and $\mathcal{V}$ is the vocabulary. A single teacher-forced pass on clean images yields an entropy profile $\{H_1, ..., H_T\}$, quantifying generative instability at each position.

## 2. Selection and Characterization of High-Entropy Tokens

EGA identifies the tokens responsible for the majority of generation “forks” by selecting the top quantile of positions with the highest entropies. Specifically, entropies are ranked in descending order, and a set $S_q = \{\sigma(1), ..., \sigma(k)\}$ is formed, where $k = \lceil q \cdot T \rceil$ and $q$ is the selection ratio (default $q=0.20$). Alternatively, a threshold $\tau$ can define the set $S = \{ t \,|\, H_t \geq \tau \}$ with $|S|/T \approx 0.20$. Empirical findings establish that this ≈20% subset governs the bulk of generative trajectory changes, thus presenting an optimal attack surface.

## 3. Construction and Integration of the Entropy Bank

EGA constructs a reusable entropy bank $B$ to generalize across images and models. This bank comprises the top $K$ tokens from the vocabulary $\mathcal{V}$ with the highest empirical “flip-rate,” calculated as:
$$
\mathrm{FlipRate}(w) = \frac{\#\,\text{images where token } w \text{ is replaced in a high-entropy position}}{\text{total images}}
$$
Tokens are ranked by flip-rate, and the bank $B$ consists of the top-$K$ most “flippable.” At attack time, per-image high-entropy masks $S_q$ are augmented with token positions in the bank: $S_\text{bank} = \{ t ~|~ \hat{y}_t \in B \}$, yielding the mask $S_\text{tr} = S_q \cup S_\text{bank}$. This augmentation increases attack effectiveness and enables transferability across VLM architectures.

## 4. EGA Attack Optimization Procedure

EGA optimizes adversarial perturbations by maximizing the average entropy over the selected positions $S_\text{tr}$, under an $L_\infty$ norm constraint on the pixel-level perturbation:
$$
\max_\delta \frac{1}{|S_\text{tr}|} \sum_{t \in S_\text{tr}} H_t (f_\theta (v_0 + \delta, \hat{y}_{<t}) ) \qquad \text{subject to} \quad \|\delta\|_\infty \leq \varepsilon
$$
where $v_0 = \psi(I)$ is the normalized pixel input and $\varepsilon=8/255$ is the standard perturbation budget. Projected gradient ascent with momentum (or Adam optimizer) is performed over 300 steps, with periodic mask refresh (every $R=50$ steps). Exact pseudocode is provided in the source [2512.21815], specifying key steps: initial greedy decoding, entropy computation, mask formation, iterative gradient updates, and final adversarial decoding.

## 5. Empirical Results and Baseline Comparisons

Quantitative evaluation on standard VLMs (Qwen2.5-VL-7B-Instruct, InternVL3.5-4B, LLaVA-1.5-7B) demonstrates that EGA achieves attack success rates (ASR) of $93.1\%$–$94.8\%$ with $\Delta$CIDEr scores near 0.85–0.88. Crucially, EGA induces harmful content (violence, self-harm, hate, etc.) in $37.3\%$–$47.1\%$ of outputs—more than doubling the harmful rate achieved by global-entropy (MIE) baselines ($14\%$–$23\%$) and greatly exceeding standard methods such as PGD, VLA, and COA (all $<2\%$ harmful conversion). In visual question answering, similar patterns are observed, with EGA producing $25\%$–$29\%$ harmful conversions and retaining $>80\%$ ASR. Transferability metrics show EGA achieves $17\%$–$26\%$ harmful rates on unseen target models, compared with $<12\%$ for transferred XTA or MIE attacks.

| Model                  | ASR (%) | ΔCIDEr | Harmful Rate (%) |
|------------------------|---------|--------|------------------|
| Qwen2.5-VL-7B-Instruct | 94.8    | 0.88   | 42.5             |
| InternVL3.5-4B         | 93.8    | 0.86   | 37.3             |
| LLaVA-1.5-7B           | 93.1    | 0.85   | 47.1             |

## 6. Key Insights and Implications

Empirical findings reveal that approximately 20% of tokens with highest entropy govern the majority of generative variability. Restricting adversarial perturbations to these “decision tokens” not only preserves attack efficacy but leads to disproportionate semantic drift—enabling conversion of $35\%$–$49\%$ of benign outputs to harmful content, a vulnerability underreported by prior global-entropy attacks. High-entropy tokens show recurrence across different VLM architectures, enabling practical cross-model transfer of attacks.

A salient implication is that robustness interventions should focus on stabilizing next-token distributions specifically at high-entropy forks, for example, via uncertainty regularization or dynamic token masking, instead of merely increasing average-case robustness. EGA thus exposes a concentrated vulnerability in modern VLM decoding and motivates reevaluation of multimodal safety mechanisms [2512.21815].

## 7. Practical Considerations and Implementation

EGA is implemented under a standardized $L_\infty$-norm constraint on normalized pixels ($\varepsilon=8/255$ as default), with optimizer options including Adam ($\beta_1=0.9$, $\beta_2=0.999$) or momentum-based PGD ($\mu=0.6$). Perturbations are iteratively updated, and the selected entropy mask may be optionally refreshed to account for evolving generative trajectories. Parameters such as token budget ($q=0.20$), entropy bank size ($K=100$), and mask refresh interval ($R=50$) are configurable, with efficacy robust across reasonable variations. Decoding is performed greedily with typical sequence length restrictions.

A plausible implication is that the EGA framework and entropy bank concept may be adapted to broader autoregressive, multimodal, or sequence-generation settings where token-level uncertainty can be exploited for selective adversarial control.

Source: https://www.emergentmind.com/topics/entropy-bank-guided-adversarial-attacks-ega