Proxy Targeted Attack (PTA)

• PTA is a design pattern that uses tractable surrogates—such as proxy models, anchor codes, or guided exemplars—to approximate hard-to-access targets.
• It optimizes targeted adversarial objectives by aligning against proxy sets and leveraging multimodal, distributional, or discrete token proxies across various applications.
• PTA demonstrates improved transferability and stealth in areas like deep hashing retrieval, object detection, diffusion customization, and black-box LLM attacks while facing proxy quality challenges.

Searching arXiv for papers on Proxy Targeted Attack and closely related proxy-based targeted attack methods. arXiv search query: "Proxy Targeted Attack" Proxy Targeted Attack (PTA) denotes a family of targeted adversarial attacks in which optimization is redirected from the exact victim target to a proxy representation, proxy model, proxy subset, or proxy delivery channel. In the explicit multimodal formulation, PTA “leverages multiple source-modal and target-modal proxies to optimize targeted AEs, ensuring they remain evasive to defenses while aligning with multiple potential targets” (Zhang et al., 24 Sep 2025). Closely related work realizes the same pattern in deep hashing retrieval, diffusion customization, object detection, transferable image classification, black-box LLM jailbreaking, and agentic oversight, even when the term itself is not used (Bai et al., 2020).

1. Conceptual basis

The core PTA move is to replace a hard or inaccessible target with a tractable surrogate that preserves the attack objective. In deep hashing retrieval, the proper target is a set of hash codes $F(X^{(t)})$, but the attack is made tractable by constructing a single anchor code $\mathbf{h}_a$ that represents that set (Bai et al., 2020). In multimodal encoders, the inaccessible target is a distribution of possible prompts, queries, or cross-modal targets, and the attack instead optimizes against multiple target-modal proxies while remaining close to source-modal proxies (Zhang et al., 24 Sep 2025). In black-box LLM jailbreaks, the inaccessible victim gradients are replaced by a white-box surrogate model $f_\phi$ that guides discrete token optimization (Sitawarin et al., 2024). In agentic monitoring, the proxy is not merely a model but an entire component: the agent itself acts as a delivery mechanism that places optimized strings into the monitor’s input stream (Isbarov et al., 4 Feb 2026).

This suggests that PTA is better understood as a design pattern than as a single algorithm. The proxy can be a representative code, a centroid-like object, a surrogate model, a guided exemplar, an easy-to-attack subset, or a relay component. What remains invariant is the targeted nature of the objective: the attack is not satisfied with arbitrary failure, but seeks a specified downstream behavior, label, retrieval set, harmful string, or safe verdict.

A recurrent distinction in the literature is between direct point-to-point targeting and proxy-based targeting. The deep hashing formulation makes this explicit: point-to-set targeting is reduced to point-to-proxy targeting, and the paper states that “the attack never directly optimizes against the full set; it optimizes against $\mathbf{h}_a$ as a summary of the set” (Bai et al., 2020). The multimodal PTA paper sharpens the same distinction in probabilistic terms by introducing estimated conditional target distributions and proxy sets drawn from them (Zhang et al., 24 Sep 2025).

2. Formal structures and objective functions

A canonical PTA construction appears in deep hashing retrieval. Let $F(X^{(t)})$ be the set of hash codes for the target label. The original objective is a point-to-set problem,

$\min_{x'}\ d\big(F(x'),\, F(X^{(t)})\big),$

which is instantiated with an average-case Hamming metric (Bai et al., 2020). DHTA replaces the full target set by an anchor code $\mathbf{h}_a$ built by component voting, with the theorem

$$\mathbf{h}_a = \arg\min_{\mathbf{h}\in\{-1,+1\}^K} \sum_{i=1}^{n_t} d_H(\mathbf{h},\mathbf{h}_i).$$

The attack then becomes

$$\min_{x'}\ -\frac{1}{K}\,\mathbf{h}_a^\top \tanh\big(\alpha f_\theta(x')\big) \quad \text{s.t.}\quad \|x'-x\|_\infty \le \epsilon,$$

which is a direct point-to-proxy optimization (Bai et al., 2020).

The explicit multimodal PTA formalizes proxy use at the distributional level. With source-modal proxies $\{\hat{\mathbf{x}}_i\}_{i=1}^{N_s}$ and target-modal proxies $\mathbf{h}_a$0, it defines

$\mathbf{h}_a$1

and

$\mathbf{h}_a$2

The combined optimization is

$\mathbf{h}_a$3

so the attack jointly optimizes for targeted alignment and in-manifold stealth (Zhang et al., 24 Sep 2025).

In black-box LLM jailbreaking, PAL uses a surrogate model as the proxy. Given prompt $\mathbf{h}_a$4, adversarial suffix $\mathbf{h}_a$5, and target harmful string $\mathbf{h}_a$6, the targeted objective is

$\mathbf{h}_a$7

with

$\mathbf{h}_a$8

Because the victim is query-only, PAL substitutes proxy gradients $\mathbf{h}_a$9 from a white-box surrogate $f_\phi$0 and then validates candidates against the black-box target (Sitawarin et al., 2024).

In agentic monitoring, the proxy structure is compositional. If an optimized string $f_\phi$1 is inserted at $f_\phi$2 trace locations, the synthetic trace template is

$f_\phi$3

and the attack minimizes a target-monitor loss such as

$f_\phi$4

where $f_\phi$5 is a safe monitor output like "conclusion": false or <score\>0</score> (Isbarov et al., 4 Feb 2026). Here the proxy is the agent trajectory itself.

3. Proxy representations in retrieval, detection, and transferable classification

In retrieval, PTA appears as a response to the mismatch between the semantic target and the operational target. Deep hashing retrieval returns database items by Hamming proximity, so targeted manipulation cannot reliably be reduced to a single target image. DHTA therefore introduces the anchor code $f_\phi$6 as a proxy that is “computed from multiple target-class samples via voting,” is “guaranteed to minimize the average Hamming distance,” and is “used as the sole target in the attack loss” (Bai et al., 2020). The paper reports that DHTA outperforms point-to-point attacks on both image and video retrieval, with ImageNet t-MAP rising from point-to-point values of about $f_\phi$7–$f_\phi$8 to about $f_\phi$9–$\mathbf{h}_a$0, and UCF-101 t-MAP rising from about $\mathbf{h}_a$1–$\mathbf{h}_a$2 to about $\mathbf{h}_a$3–$\mathbf{h}_a$4 (Bai et al., 2020).

In object detection, the proxy is an exemplar rather than a code. The object-fabrication targeted attack introduces a guided image $\mathbf{h}_a$5 that contains the target object and uses dual attention to extract “the critical features of the target objects efficiently” (Zhang et al., 2022). The loss

$\mathbf{h}_a$6

aligns victim-image features to guided-image features, so the guided image functions as a proxy target in detector feature space (Zhang et al., 2022). This attack fabricates designated objects “regardless of whether the victim image contains objects or not” and substantially exceeds TOG and CAA in image-specific, universal, and generalization settings; on MS COCO with YOLOv5, image-specific success rate reaches $\mathbf{h}_a$7 compared with $\mathbf{h}_a$8 for CAA (Zhang et al., 2022).

Transferable image classification introduces a different proxy logic. “Everywhere Attack” assumes the victim is accessible only through surrogate models and attacks both the global image and multiple local blocks, thereby optimizing “an army of targets” rather than “a high-confidence target in the image” (Zeng et al., 1 Jan 2025). The surrogate is the proxy model, and the local blocks act as spatial target proxies that mitigate “transfer failures caused by attention inconsistency between surrogate and victim models” (Zeng et al., 1 Jan 2025). The paper states that the transferability of the widely adopted Logit attack can be improved by $\mathbf{h}_a$9-$F(X^{(t)})$0 (Zeng et al., 1 Jan 2025).

A more concessive interpretation appears in BAST. That paper does not use the term PTA explicitly; the provided interpretation treats non-targeted success on robust models as a proxy outcome for full targeted success on the ensemble (He et al., 2019). BAST divides models into easy-to-attack and robust groups, applies targeted attacks to the former and non-targeted attacks to the latter, and evaluates success with

$F(X^{(t)})$1

This suggests a broadened PTA notion in which proxying may occur at the level of system objective rather than target representation (He et al., 2019).

4. Generative and multimodal PTAs

In diffusion customization, the proxy takes the form of both a surrogate model and a designed target in score space. ACE defines a targeted objective

$F(X^{(t)})$2

and ACE+ adds a latent-space term

$F(X^{(t)})$3

The target $F(X^{(t)})$4 is a hand-picked latent corresponding to a dense, high-contrast texture, while the attack itself is crafted on a proxy Stable Diffusion backbone and transferred to victim customization pipelines (Zheng et al., 2023). The paper reports transfer across SD $F(X^{(t)})$5, $F(X^{(t)})$6, and $F(X^{(t)})$7, and across LoRA and SDEdit. In one cited cross-model result, ACE+ crafted on SD $F(X^{(t)})$8 and evaluated on SD $F(X^{(t)})$9 for LoRA changes CLIP-IQA from about $\min_{x'}\ d\big(F(x'),\, F(X^{(t)})\big),$0 under no attack to about $\min_{x'}\ d\big(F(x'),\, F(X^{(t)})\big),$1, indicating severe quality degradation (Zheng et al., 2023).

The multimodal PTA paper provides the most explicit theoretical treatment of the proxy idea. It argues that existing targeted attacks suffer from “limited generalizability” and “limited undetectability,” and addresses both by optimizing against source-modal and target-modal proxy sets (Zhang et al., 24 Sep 2025). Its theory rewrites the constrained optimization in terms of the source and target means $\min_{x'}\ d\big(F(x'),\, F(X^{(t)})\big),$2 and $\min_{x'}\ d\big(F(x'),\, F(X^{(t)})\big),$3, the modality gap $\min_{x'}\ d\big(F(x'),\, F(X^{(t)})\big),$4, and a detector threshold $\min_{x'}\ d\big(F(x'),\, F(X^{(t)})\big),$5, yielding

$\min_{x'}\ d\big(F(x'),\, F(X^{(t)})\big),$6

This formalizes the trade-off between generalizability and undetectability (Zhang et al., 24 Sep 2025). Empirically, the paper reports average classification ASR of about $\min_{x'}\ d\big(F(x'),\, F(X^{(t)})\big),$7 without detection and average classification ASRD of about $\min_{x'}\ d\big(F(x'),\, F(X^{(t)})\big),$8 with anomaly detection, versus about $\min_{x'}\ d\big(F(x'),\, F(X^{(t)})\big),$9 and about $\mathbf{h}_a$0 for Illusion Attack (Zhang et al., 24 Sep 2025). In retrieval, PTA raises average $\mathbf{h}_a$1 ASR from about $\mathbf{h}_a$2 to about $\mathbf{h}_a$3, and average $\mathbf{h}_a$4 ASRD from about $\mathbf{h}_a$5 to about $\mathbf{h}_a$6 (Zhang et al., 24 Sep 2025).

Across these generative and multimodal settings, the proxy is not merely a computational convenience. It is engineered to impose structure on the attack: ACE uses a common score-space pattern $\mathbf{h}_a$7 so all poisoned examples induce a consistent reverse bias, while multimodal PTA uses multiple proxies so the adversarial example remains close to a benign source manifold while covering a target distribution (Zheng et al., 2023).

5. LLMs, proxy models, and agentic oversight

In language-model settings, PTA becomes a black-box optimization problem over discrete tokens. PAL is described as “the first optimization-based attack on LLMs in a black-box query-only setting” and “relies on a surrogate model to guide the optimization and a sophisticated loss designed for real-world LLM APIs” (Sitawarin et al., 2024). The proxy model supplies gradients, filters candidates, and can be fine-tuned on collected victim outputs, functioning as a local model-extraction mechanism. PAL achieves $\mathbf{h}_a$8 attack success rate on GPT-3.5-Turbo and $\mathbf{h}_a$9 on Llama-2-7B, compared to $$\mathbf{h}_a = \arg\min_{\mathbf{h}\in\{-1,+1\}^K} \sum_{i=1}^{n_t} d_H(\mathbf{h},\mathbf{h}_i).$$0 for the current state of the art (Sitawarin et al., 2024).

PAL’s loss design is explicitly targeted. For token $$\mathbf{h}_a = \arg\min_{\mathbf{h}\in\{-1,+1\}^K} \sum_{i=1}^{n_t} d_H(\mathbf{h},\mathbf{h}_i).$$1, it uses a CW-style objective

$$\mathbf{h}_a = \arg\min_{\mathbf{h}\in\{-1,+1\}^K} \sum_{i=1}^{n_t} d_H(\mathbf{h},\mathbf{h}_i).$$2

summed over the target string, and introduces the logit-bias trick

$$\mathbf{h}_a = \arg\min_{\mathbf{h}\in\{-1,+1\}^K} \sum_{i=1}^{n_t} d_H(\mathbf{h},\mathbf{h}_i).$$3

to recover otherwise hidden token log-probabilities from real APIs (Sitawarin et al., 2024). The attack remains targeted at the sequence level even when its harmful success is evaluated separately from exact target-string reproduction.

Agent-as-a-Proxy attacks extend PTA from model inference to control protocols. Here the proxy is the agent itself, which repeats optimized substrings in chain-of-thought and tool parameters so that the monitor misclassifies a malicious trajectory as safe (Isbarov et al., 4 Feb 2026). The attack is optimized against monitors such as AlignmentCheck, PromptGuard 2 + AlignmentCheck, and Extract-and-Evaluate, and is evaluated on AgentDojo banking, Slack, travel, and workspace tasks. On Qwen2.5-72B as the monitor, ASR@5 reaches $$\mathbf{h}_a = \arg\min_{\mathbf{h}\in\{-1,+1\}^K} \sum_{i=1}^{n_t} d_H(\mathbf{h},\mathbf{h}_i).$$4 with GPT-4o mini and $$\mathbf{h}_a = \arg\min_{\mathbf{h}\in\{-1,+1\}^K} \sum_{i=1}^{n_t} d_H(\mathbf{h},\mathbf{h}_i).$$5 with Llama-3.1-70B (Isbarov et al., 4 Feb 2026).

A central controversy concerns the value of more monitor visibility. The paper reports that static attacks are better handled by hybrid monitoring than by CoT-only monitoring, but adaptive Agent-as-a-Proxy attacks can make hybrid monitoring more vulnerable because tool outputs become an attack surface (Isbarov et al., 4 Feb 2026). This directly challenges the assumption that larger or more comprehensive monitors are sufficient for scalable oversight.

6. Limitations, controversies, and proxy-free responses

PTA inherits the strengths and weaknesses of its proxies. If the proxy is misaligned with the actual victim objective, performance can degrade. The backdoor-poisoning paper on Proxy attack-Free Strategy states this criticism explicitly: proxy attack-based sample selection “can lead to performance degradation if the proxy attack settings differ from those of the actual victims, due to the shortcut nature of backdoor learning,” and such methods are “extremely time-consuming” because they require “numerous complete backdoor injection processes for sample selection” (Li et al., 2023). PFS is proposed precisely as a “Proxy attack-Free Strategy,” which is a direct counterpoint to proxy-based attack design (Li et al., 2023).

Proxy quality is also a recurring bottleneck. In multimodal PTA, theoretical guarantees depend on the source and target proxies adequately covering the target distributions; the paper states that if the true target lies in the convex polytope spanned by proxies, lower bounds on alignment follow (Zhang et al., 24 Sep 2025). In object detection, guided-image quality affects performance, and the method remains white-box because class attention requires gradients of the detector (Zhang et al., 2022). In PAL, effectiveness depends on API features such as logprobs and logit bias, and the paper notes that models without such features require alternative scoring schemes (Sitawarin et al., 2024).

A second controversy is whether proxying reduces or increases robustness. In some contexts proxying improves transfer and stability. DHTA replaces a single target image with an anchor code computed from multiple target samples; ACE uses a consistent target tensor to induce a shared corruption pattern; multimodal PTA uses multiple source and target proxies to jointly optimize generalizability and stealth (Bai et al., 2020). In other contexts proxying is the vulnerability itself. Agent-as-a-Proxy attacks show that when a defense must ingest agent traces, the proxy channel becomes a mechanism for directly attacking the monitor (Isbarov et al., 4 Feb 2026).

This suggests that PTA should not be treated as uniformly beneficial or uniformly harmful. It is a structural technique for reparameterizing targeted attack objectives under partial knowledge, optimization difficulty, or architectural separation. Its practical efficacy depends on whether the proxy preserves the relevant target geometry while avoiding detectable mismatch. The strongest recent work therefore treats proxy design as the central problem: choosing representative anchors, multiple proxy sets, surrogate models, or delivery channels that are sufficiently faithful to the true target while remaining computationally exploitable (Zhang et al., 24 Sep 2025).