Hierarchical Refinement Attack (HRA)

• Hierarchical Refinement Attack is a multi-level adversarial strategy that iteratively refines perturbations to enhance attack success and transferability.
• It leverages dual-stage refinement at sample and optimization levels, incorporating historical and future gradient feedback to overcome local optima.
• Its applications include jailbreaking large language models and executing universal attacks on vision-language systems with significantly improved performance.

The Hierarchical Refinement Attack (HRA) is a class of adversarial attack strategies that leverage multi-level, feedback-driven refinement and optimization to maximize the success, diversity, and transferability of attacks on machine learning models. HRA has been instantiated in both the vision-language and LLM domains to overcome limitations of single-level or flat adversarial approaches. Architecturally, HRA unifies hierarchical semantic expansion, iterative refinement based on structured feedback, and adaptive execution, producing robust and efficient adversarial perturbations or multi-turn queries. HRA yields notably improved attack success rates and compositional generalization relative to prior universal and targeted attack paradigms (Narula et al., 21 Oct 2025, Zhang et al., 15 Jan 2026).

1. Motivation and Key Conceptual Advances

Hierarchical Refinement Attack is motivated by two central challenges faced by adversarial attack frameworks: (1) the tendency for adversarial perturbations or prompts to overfit to surrogates or local optima, resulting in poor transferability, and (2) the inefficiency and narrow coverage of attacks constructed in a sample-specific or single-stage manner (Zhang et al., 15 Jan 2026). HRA introduces hierarchical refinement at two interdependent levels:

• Sample Level: Independently optimizes the clean input (e.g., image, prompt) and its adversarial perturbation, leveraging targeted augmentations to increase diversity and generalization.
• Optimization Level: Controls the optimization trajectory through incorporation of historical and estimated future gradients, avoiding stagnation in poor optima and improving cross-model effectiveness.

This dual-level refinement is operationalized in frameworks such as HarmNet for LLMs and universal multimodal attacks for VLP architectures (Narula et al., 21 Oct 2025, Zhang et al., 15 Jan 2026).

2. Architectural Frameworks

A. Vision-LLMs: Multimodal HRA

In the context of universal attacks on vision-language pre-trained (VLP) models, HRA refines both image and text-based adversarial perturbations through multi-stage procedures (Zhang et al., 15 Jan 2026):

• Image Domain: Perturbations $\Delta_I$ are disentangled from clean images $x$ and refined via:
  • ScMix: A two-stage augmentation (self-mix and cross-mix) designed to expose global and local features to attack, defined as

  $$\hat x_i = \eta x_i^1 + (1-\eta) x_i^2, \quad \tilde x_i = \beta_1 \hat x_i + \beta_2 x_j$$

  where $$\eta \sim \max(\text{Beta}(\alpha, \alpha), 1-\text{Beta}(\alpha, \alpha))$$ and $\beta_1 > \beta_2$. - Local Utility: Random local crops $\mathcal{A}_s(\Delta_I)$ are optimized to disrupt patch-level feature alignment.

• Optimization-Level Refinement: Gradient updates for $\Delta_I$ aggregate the current, past, and estimated future gradients:

$$\tilde g^t = g^t + \gamma_1 g_p^{t-1} + \gamma_2 g_f^t$$

Where $g_f^t$ is the mean of next $d$-step gradients, stabilizing universal perturbation learning.

• Text Domain: Universal trigger extraction selects globally influential tokens based on intra/inter-sentence importance using masking strategies and KL-divergence in shared feature space. The score for a token $w$ is

$$S(w) = \ell(f_I(\hat y), f_T(y)) + \ell(f_I(\hat y), f_I(x))$$

where $\hat y$ is $y$ with $w$ masked.

B. LLMs: Hierarchical Multi-Turn Jailbreak

In HarmNet, HRA is instantiated as a multi-module framework for adversarially extracting harmful completions from LLMs (Narula et al., 21 Oct 2025):

• ThoughtNet (Hierarchical Semantic Network): Generates a forest structure from the high-level user prompt, expanding through topics $z_i$, sentences $s_{ij}$, and entities $e_{ijk}$ subject to cosine-similarity thresholds (e.g., $\cos(\mathbf{v}_{z_i},\mathbf{v}_g)\ge\tau_z$). Each semantic path leads to a multi-turn candidate query chain $\mathcal{C}_{ijk}$.
• Simulator: Iteratively interacts with the victim model, refining each prompt $c_t$ based on temporal reward signals:

$$\mathcal{L}_t(c_t) = -[\alpha \Delta H_t + (1-\alpha) \Delta S_t]$$

where $H_t$ is a judge-derived harmfulness score and $S_t$ measures semantic alignment.

• Network Traverser: Executes and adaptively refines the highest-scoring query chain in real time, using early stopping heuristics if maximal harm is attained ($H_t=5$).

3. Algorithmic Realizations

The implementation of HRA is characterized by explicit multi-level loops and feedback incorporation. For VLP models (Zhang et al., 15 Jan 2026), algorithmic steps involve:

• Iterative refinement of $\Delta_I$ using augmented data and temporally smoothed gradients (as above).
• Post-hoc extraction of a global textual trigger $\Delta_T$ by ranking word importance scores $S(w)$, considering both intra- and inter-sentence effects.

For LLM jailbreak, the hierarchical refinement proceeds as:

• Expansion of the ThoughtNet hierarchy with selection and embedding similarity constraints.
• Simulation-driven pruning and feedback-based local refinements within multi-turn query chains, with explicit convergence and diversity criteria.

Pseudo-code and detailed decision criteria are given in Algorithm 1 and Section 2 of (Zhang et al., 15 Jan 2026) and Section 2 of (Narula et al., 21 Oct 2025), respectively.

4. Mathematical Formulation

Mathematically, HRA is defined by composite objectives and refinement equations:

Vision-LLM Universal Attack

• Primary Image Objective:

$$\underset{\Delta_I}{\arg\max}\; L_1(x, y, \Delta_I) = \sum_{i=1}^n \Bigl[\ell(f_I(x_i + \Delta_I), f_I(x_i)) + \ell(f_I(x_i + \Delta_I), f_T(y_i))\Bigr]$$

• Local Utility Loss:

$$L_2(x, y, \Delta_I) = \sum_{i=1}^{n_t} \left\{ \cdots \right\}$$

• Total Objective:

$$L(x, y, \Delta_I) = L_1(x, y, \Delta_I) + L_2(x, y, \Delta_I)$$

LLM Multi-Turn Jailbreak

• Expansion Criteria:

$$\cos(\mathbf{v}_{z_i}, \mathbf{v}_g) \ge \tau_z,\;\; \cos(\mathbf{v}_{s_{ij}}, \mathbf{v}_g) \ge \tau_s,\;\; \cos(\mathbf{v}_{s_{ij}}, \mathbf{v}_{s_{ik}}) < \tau_d$$

• Turn-level Loss:

$$\mathcal{L}_t(c_t) = -\bigl[\alpha\,\Delta H_t + (1-\alpha)\,\Delta S_t\bigr]$$

• Chain-level Score:

$$\mathcal{R}(\mathcal{C}) = \sum_{t=1}^T[\alpha H_t + (1-\alpha)S_t]$$

5. Experimental Validation and Comparative Performance

HRA demonstrates strong empirical results across both vision-language and language modeling tasks. In universal VLP attacks, HRA attains significantly higher recall-at-1 (R@1) and transferability across CLIP architectures relative to ETU and FD-UAP (e.g., HRA $\sim$90% white-box I2T on Flickr30K vs. 87–88% baselines; HRA 79.8% black-box I2T vs. 56.8–68.5% for alternatives) (Zhang et al., 15 Jan 2026). In LLM jailbreak settings, HarmNet achieves attack success rates (ASR) of 99.4% on Mistral-7B and 98.4% on LLaMA 3-8B, exceeding the best prior methods by 13.9 and 19.4 percentage points, respectively. HarmNet also reduces the required number of turns (3.2 vs. 4.5–5.8) and achieves shorter adversarial prompts per turn (Narula et al., 21 Oct 2025).

Ablation analysis confirms the significant contribution of hierarchical structure: removing it causes a drop of 12% (LLM) or 2–5% (VLP) in ASR, and omitting iterative Simulator feedback reduces diversity and coverage metrics by up to 20 points. Each HRA component—ScMix, local utility, future-aware momentum, and text trigger—independently increases attack strength (Narula et al., 21 Oct 2025, Zhang et al., 15 Jan 2026).

Table: Attack Success Rate (ASR, %) across Major LLMs ((Narula et al., 21 Oct 2025), Table, abridged)

Method	GPT-3.5	GPT-4o	LLaMA 3 8B	Mistral 7B
GCG	55.8	12.5	34.5	27.2
RACE	80.0	82.8	75.5	78.0
ActorAttack	86.5	84.5	79.0	85.5
HarmNet (HRA)	91.5	94.8	98.4	99.4

6. Constraints, Insights, and Future Research

Although HRA establishes strong performance benchmarks, several constraints and open research directions are identified (Narula et al., 21 Oct 2025, Zhang et al., 15 Jan 2026):

• Threshold Sensitivity: HRA parameters (e.g., $$\tau_z, \tau_s, \tau_d, \mu, \nu, \gamma_1, \gamma_2$$) require non-trivial tuning for optimal performance.
• Evaluator Dependence: Hierarchical refinement for LLMs depends on access to high-quality judge models; weaker judges degrade attack outcomes.
• Scalability: The computational overhead of simulating and pruning candidate chains (in LLMs) or evaluating hundreds of sample-level augmentations (in VLP models) is non-negligible.

Future directions include automated threshold optimization (e.g., Bayesian methods), integration of full Monte Carlo Tree Search in hierarchical graph traversal (LLM attacks), and constraint-aware refinement subject to token or computation budgets. The temporal hierarchy for optimization in VLP models may be further extended to other sequential or structured adversarial settings.

7. Applications and Broader Implications

HRA has demonstrated utility in jailbreaking state-of-the-art LLMs and generating transferable, universal perturbations against vision-language systems, including black-box architectures and across diverse end-tasks such as retrieval, captioning, and grounding (Narula et al., 21 Oct 2025, Zhang et al., 15 Jan 2026). In both modalities, HRA yields increased attack diversity, higher adversarial coverage, and robustness to overfitting relative to flat or single-pass attack strategies. This suggests HRA will remain a central analytic and practical tool for benchmarking and improving adversarial robustness in multimodal and language-based AI systems.

---

References:

• "HarmNet: A Framework for Adaptive Multi-Turn Jailbreak Attacks on LLMs" (Narula et al., 21 Oct 2025)
• "Hierarchical Refinement of Universal Multimodal Attacks on Vision-LLMs" (Zhang et al., 15 Jan 2026)