Heuristic Adversarial Skill Generation

• Heuristic adversarial skill generation is a unified framework that uses structured, closed-loop heuristics to propose, evaluate, and refine reusable skills across diverse domains.
• Techniques include prompt-only exploit refinement, documentation-driven payload embedding, memory-based attack evolution, and regret-aware task discovery in reinforcement learning.
• Empirical results show iterative, feedback-driven approaches significantly enhance adversarial success rates and system robustness in settings from autonomous driving to embodied control.

Searching arXiv for the cited papers and related work on heuristic adversarial skill generation. Heuristic adversarial skill generation denotes a family of methods in which adversarially useful skills are proposed, selected, or refined through heuristic procedures rather than exhaustive search. The literature uses the phrase across several technical settings: prompt-only exploitation of agent skills in open registries, documentation-driven poisoning of coding-agent ecosystems, memory-based jailbreak strategy evolution, regret-guided skill discovery in reinforcement learning, adversarially regularized motor-skill composition, realistic scenario perturbation in autonomous driving, and even robust decoding viewed as a minimax game over heuristic generation policies (Duan et al., 5 Apr 2026, Qu et al., 3 Apr 2026, Zhang et al., 28 May 2026, Zhang et al., 26 Jun 2025, Lee et al., 2021, Stoler et al., 2024, Chen et al., 2024). Across these usages, the common pattern is an adversarial objective coupled to a structured heuristic controller that proposes candidate skills, observes outcomes, and updates future proposals using explicit feedback.

1. Terminology and scope

The term “skill” is not uniform across this literature. In open agent ecosystems, a skill is a reusable module that bundles executable code, domain knowledge, and natural-language instructions; in SkillAttack, a skill is formalized as $s = (s_{\text{inst}}, s_{\text{impl}})$, where $s_{\text{inst}}$ is a natural-language instruction file and $s_{\text{impl}}$ is executable implementation (Duan et al., 5 Apr 2026). In coding-agent supply chains, a skill is likewise an executable unit, but the critical surface is often its metadata or documentation; the paper on supply-chain poisoning formalizes a skill as $s = (m, e)$, where $m$ contains metadata and inline examples and $e$ is the execution body (Qu et al., 3 Apr 2026). In jailbreak work, “skill” can mean an abstract reusable attack mechanism stored with templates and evidence, as in MemoAttack’s memory unit $m = \{s, \tau, E, \ell\}$ (Zhang et al., 28 May 2026). In reinforcement learning and control, skills are latent goals, option-like policies, or low-level motor behaviors, including skill-conditioned policies $\pi_\theta(a \mid s, z)$, teacher-generated task distributions $p_\phi(T \mid h)$, or terminal-state-compatible subtasks (Zhang et al., 26 Jun 2025, Xu et al., 21 Oct 2025, Lee et al., 2021).

Domain	Object called a skill	Adversarial mechanism
Open agent skills	Module $s = (s_{\text{inst}}, s_{\text{impl}})$	Prompt-only exploit search over latent vulnerabilities
Coding-agent supply chain	Skill $s_{\text{inst}}$0 with executable docs	DDIPE through code examples and config templates
Jailbreaking	Memory unit $s_{\text{inst}}$1	Reuse, mutation, invention, and lifecycle-managed search
RL and ACL	Latent skill $s_{\text{inst}}$2 or task $s_{\text{inst}}$3	Min-max generator–learner or teacher–student dynamics
Embodied control	Motor prior, subtask policy, or disturbance policy	Adversarial regularization, imitation, or disturbance learning
Decoding	Generation policy	Robust minimax response to adversarial distribution shift

A plausible unifying abstraction is that heuristic adversarial skill generation decomposes into four elements: a skill space, an adversarial objective, a heuristic proposal rule, and an evidence channel. The proposal rule may be path refinement, template mutation, Thompson sampling, regret-aware sampling, entropy-regularized teacher updates, or discriminator-shaped policy learning. The evidence channel may be execution traces, evaluator scores, value-function deltas, discriminator outputs, or realism metrics.

2. Prompt-only exploit generation for open agent skills

The most direct use of the term in agent security is SkillAttack, which studies prompt-only exploitation of latent vulnerabilities in non-malicious agent skills (Duan et al., 5 Apr 2026). The threat model is explicit: the attacker can craft arbitrary user prompts $s_{\text{inst}}$4 but cannot modify the skill, the agent’s system prompt, or the runtime. Given $s_{\text{inst}}$5, the agent executes a trajectory $s_{\text{inst}}$6, and the goal is to find an $s_{\text{inst}}$7 that elicits at least one unsafe behavior under an eight-class taxonomy, evidenced by the trajectory, artifacts $s_{\text{inst}}$8, or final response $s_{\text{inst}}$9.

SkillAttack organizes exploit generation as a closed-loop heuristic search with three components. First, Skill Vulnerability Analysis uses an LLM auditor to extract vulnerability candidates $s_{\text{impl}}$0, where $s_{\text{impl}}$1 is a taxonomy type, $s_{\text{impl}}$2 attacker-controllable inputs, $s_{\text{impl}}$3 sensitive operations, and $s_{\text{impl}}$4 triggering conditions. Second, Surface-Parallel Attack Generation constructs, for each surface, an attack path $s_{\text{impl}}$5 that links user input to a vulnerable operation $s_{\text{impl}}$6 and an unsafe behavior $s_{\text{impl}}$7. Third, Feedback-Driven Exploit Refinement executes the prompt, judges success, diagnoses deviation or failure cause, and refines both path and prompt up to budget $s_{\text{impl}}$8 rounds. The formal objective is to maximize the binary judge $s_{\text{impl}}$9 subject to no modification of $s = (m, e)$0 or $s = (m, e)$1 and $s = (m, e)$2 (Duan et al., 5 Apr 2026).

The heuristics are trace-based rather than search-theoretic in the bandit or beam-search sense. The framework explicitly analyzes the earliest path deviation, categorizes failures such as refusal, sanitization, or irrelevant branch selection, and keeps prompts contextually plausible so that the agent is steered toward necessary tool use without triggering blanket refusal. This is operationally important because the paper’s case study shows two common false positives for naive red teaming: no tool invocation, and hallucinated credentials with empty traces. The successful exploit emerged only after refinement promoted file inspection to the primary instruction.

The evaluation establishes the empirical stakes. Across 10 backbones, 71 adversarial skills from Skill-Inject, and 100 real-world skills from ClawHub Hot100, SkillAttack reports ASR $s = (m, e)$3–$s = (m, e)$4 on Obvious adversarial skills, $s = (m, e)$5–$s = (m, e)$6 on Contextual adversarial skills, and up to $s = (m, e)$7 on Hot100 real-world skills; Direct reaches at most $s = (m, e)$8 on Hot100, and on GPT-5.4 the comparison is Obvious $s = (m, e)$9 vs. Direct $m$0 and SI $m$1, Contextual $m$2 vs. Direct $m$3 and SI $m$4 (Duan et al., 5 Apr 2026). First success appears predominantly in rounds 3–4, accounting for about 65% of successes, which directly supports the paper’s claim that early-turn safety is deceptive. The larger implication is that open skill registries cannot be adequately characterized by static auditing or one-shot prompting alone.

3. Documentation-driven poisoning and memory-structured attack skills

A second branch of the literature moves from exploiting existing skills to generating adversarial ones. “Supply-Chain Poisoning Attacks Against LLM Coding Agent Skill Ecosystems” introduces Document-Driven Implicit Payload Execution, or DDIPE, whose core claim is that coding agents treat skill documentation as operational guidance and may copy code examples or configuration templates into executable outputs (Qu et al., 3 Apr 2026). The retrieve–load–execute flow is written as

$m$5

with the vulnerability arising because retrieved skill metadata is injected into context without content-level integrity checks.

DDIPE embeds malicious logic $m$6 into documentation rather than imperative instructions. The two principal embedding strategies are Code Example Poisoning and Configuration Template Poisoning. The generation pipeline scales from 81 expert-crafted seed skills to 1,070 valid adversarial skills across 15 MITRE ATT&CK-derived categories, using template selection, camouflage operators such as ContextShift, AuthorityInject, EncodingWrap, TriggerChain, DomainMigrate, SemanticDisguise, and validation steps including Markdown parsing, syntax linting, and similarity filtering at $m$7 (Qu et al., 3 Apr 2026). The reported bypass rates are $m$8–$m$9 across four frameworks and five models. Under the strongest defended setup, explicit instruction injection achieves $e$0, while DDIPE still reaches DER $e$1 and BR $e$2 on Claude Code + Sonnet 4.6. Static detection achieves $e$3 and $e$4, yet $e$5 of samples evade both static detection and model alignment and are executed while undetected (Qu et al., 3 Apr 2026).

A complementary development appears in MemoAttack, which treats heuristic adversarial skill generation as a memory-management problem rather than a per-sample rewriting problem (Zhang et al., 28 May 2026). Its unit of abstraction is the skill-structured memory item $e$6, containing an attack skill, template guidance, evidence, and lifecycle state. Evidence is modeled with dual Beta–Bernoulli posteriors for partial progress and final success, while selection is driven by contextual Thompson Sampling,

$e$7

with $e$8 and $e$9, followed by a contextual bonus and a retired-memory penalty. Lifecycle transitions—candidate, active, retired, eliminated—are updated from evidence through probation, promotion, retirement, reactivation, elimination, and storage cleanup.

The empirical result is unusually strong: on AdvBench, MemoAttack reports average ASR $m = \{s, \tau, E, \ell\}$0, outperforming the strongest baseline by $m = \{s, \tau, E, \ell\}$1 percentage points while reducing request count by $m = \{s, \tau, E, \ell\}$2; per-target ASR is $m = \{s, \tau, E, \ell\}$3 on kimi-k2.5, $m = \{s, \tau, E, \ell\}$4 on minimax-M2.5, and $m = \{s, \tau, E, \ell\}$5 on qwen3.5-397B-A17B (Zhang et al., 28 May 2026). The deeper significance is methodological. SkillAttack emphasizes path-aware online refinement against a single target skill; DDIPE emphasizes corpus-scale synthesis of adversarial skills; MemoAttack emphasizes persistent attack memory and evidence accumulation across samples. Together they define a security-oriented strand of heuristic adversarial skill generation in which attack behavior is treated as a structured, reusable asset.

4. Adversarial curricula, task generators, and regret-aware skill discovery

In reinforcement learning, heuristic adversarial skill generation often means proposing tasks or latent goals that are neither random nor fixed, but targeted at the current frontier of learner competence. “Efficient Skill Discovery via Regret-Aware Optimization” formulates this as a min-max game between a skill-conditioned policy and a learnable skill generator over a latent skill space $m = \{s, \tau, E, \ell\}$6 (Zhang et al., 26 Jun 2025). The policy receives an intrinsic reward

$m = \{s, \tau, E, \ell\}$7

while the generator is trained against a regret signal

$m = \{s, \tau, E, \ell\}$8

High-regret skills are interpreted as under-mastered; low-regret skills as converged. The generator objective combines expected regret with a KL term away from the current population and a proximity term toward the seen frontier.

The reported behavior is a curriculum that expands along directions of remaining policy improvement. On Maze2d-large, RSD reports FD $m = \{s, \tau, E, \ell\}$9 and AR $\pi_\theta(a \mid s, z)$0 versus METRA-d FD $\pi_\theta(a \mid s, z)$1 and AR $\pi_\theta(a \mid s, z)$2; on Antmaze-large, FD is $\pi_\theta(a \mid s, z)$3 versus $\pi_\theta(a \mid s, z)$4, and AR is $\pi_\theta(a \mid s, z)$5 versus $\pi_\theta(a \mid s, z)$6, corresponding to up to about $\pi_\theta(a \mid s, z)$7–$\pi_\theta(a \mid s, z)$8 relative AR improvement in high-dimensional settings. In Kitchen, at 400k steps, RSD reports $\pi_\theta(a \mid s, z)$9 versus $p_\phi(T \mid h)$0 for skill dimension 24 and $p_\phi(T \mid h)$1 versus $p_\phi(T \mid h)$2 for skill dimension 32 (Zhang et al., 26 Jun 2025).

Heterogeneous Adversarial Play generalizes the same logic from latent skills to discrete task curricula (Xu et al., 21 Oct 2025). A teacher network defines

$p_\phi(T \mid h)$3

and the teacher–student game is written as

$p_\phi(T \mid h)$4

Entropy regularization, warm-up, history windows, and minimum task-probability bounds stabilize the curriculum so that the teacher does not collapse onto unsolved or already mastered tasks.

The quantitative results show that this adversarial curriculum is not merely a theoretical analogy. HAP reports general scores of $p_\phi(T \mid h)$5 on Minigrid versus PPO $p_\phi(T \mid h)$6, SAC $p_\phi(T \mid h)$7, DreamerV3 $p_\phi(T \mid h)$8, and EXP3 $p_\phi(T \mid h)$9; $s = (s_{\text{inst}}, s_{\text{impl}})$0 on CRAFT versus DreamerV3 $s = (s_{\text{inst}}, s_{\text{impl}})$1 and PPO $s = (s_{\text{inst}}, s_{\text{impl}})$2; and $s = (s_{\text{inst}}, s_{\text{impl}})$3 on Crafter versus DreamerV3 $s = (s_{\text{inst}}, s_{\text{impl}})$4 and PPO (ResNet) $s = (s_{\text{inst}}, s_{\text{impl}})$5 (Xu et al., 21 Oct 2025). The paper also reports a human-subject experiment in Minigrid in which the HAP-generated adaptive tutorial significantly accelerates learning relative to no tutorial and is comparable to expert step-by-step instruction. The broad implication is that heuristic adversarial skill generation can function as automatic curriculum learning when the generated object is a task distribution rather than an exploit prompt.

5. Embodied control, skill chaining, and realistic adversaries

Embodied control introduces a different problem: generated skills must remain dynamically feasible and often human-like. In long-horizon manipulation, “Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization” addresses the classical failure mode that independently trained skills terminate outside the next skill’s initiation set (Lee et al., 2021). The method trains an initiation-set discriminator

$s = (s_{\text{inst}}, s_{\text{impl}})$6

and rewards each skill for producing terminal states that score highly under the next skill’s discriminator through

$s = (s_{\text{inst}}, s_{\text{impl}})$7

On the furniture-assembly tasks table_lack and chair_ingolf, T-STAR reports average task progress $s = (s_{\text{inst}}, s_{\text{impl}})$8 and $s = (s_{\text{inst}}, s_{\text{impl}})$9, versus Policy Sequencing at $s_{\text{inst}}$00 and $s_{\text{inst}}$01; success on chair_ingolf improves from $s_{\text{inst}}$02 to $s_{\text{inst}}$03, and on table_lack from $s_{\text{inst}}$04 to $s_{\text{inst}}$05 (Lee et al., 2021).

Adversarial skill learning also appears in low-level robustness training. In “Adversarial Skill Learning for Robust Manipulation,” the disturbance policy is learned as the adversary in a zero-sum Markov game, either by additive internal disturbance $s_{\text{inst}}$06 or actuator concatenation for an external robot (Jian et al., 2020). On DClawTurnFixed, the success-rate table reports Ours + Rand. at $s_{\text{inst}}$07, $s_{\text{inst}}$08, and $s_{\text{inst}}$09 for amplitudes $s_{\text{inst}}$10, $s_{\text{inst}}$11, and $s_{\text{inst}}$12, versus SAC + Rand. at $s_{\text{inst}}$13, $s_{\text{inst}}$14, and $s_{\text{inst}}$15; under learned adversarial attack, Ours achieves $s_{\text{inst}}$16, $s_{\text{inst}}$17, and $s_{\text{inst}}$18, while SAC collapses to $s_{\text{inst}}$19, $s_{\text{inst}}$20, and $s_{\text{inst}}$21 (Jian et al., 2020). In DoubleArmPick, the learned adversary causes about a $s_{\text{inst}}$22 larger drop in success than random actions on vanilla SAC. Here the “generated skills” are disturbance strategies such as “stealing the object” and “hindering the protagonist,” discovered by min–max training rather than manually scripted.

A more representation-centric view appears in ASE and ASN. ASE learns a latent skill embedding for physically simulated characters by combining adversarial imitation, a skill posterior $s_{\text{inst}}$23, and a diversity regularizer, then transfers the resulting motor prior to downstream tasks using simple heuristic rewards such as Reach, Speed, Steering, Location, and Strike (Peng et al., 2022). The pre-training scale exceeds 10 billion samples, uses 4096 parallel Isaac Gym environments, and yields high transfer returns such as Reach $s_{\text{inst}}$24 and Strike $s_{\text{inst}}$25, with policies trained from scratch often obtaining higher nominal returns but behaving unrealistically (Peng et al., 2022). ASN likewise learns a transferable skill embedding from unlabeled synchronized multi-view video by combining metric learning with an entropy-regularized adversarial skill-transfer loss; in simulation, it reports alignment loss $s_{\text{inst}}$26 on A,B,C$s_{\text{inst}}$27A,B,C and $s_{\text{inst}}$28 on A,B$s_{\text{inst}}$29C, and in real-world transfer on A,B,D$s_{\text{inst}}$30C it reports $s_{\text{inst}}$31 versus TCN-lifted at $s_{\text{inst}}$32 (Mees et al., 2019).

In autonomous driving, SEAL replaces rule-based stress scenarios with learned criticality scoring and skill-enabled adversaries (Stoler et al., 2024). Candidate trajectories from DenseTNT are ranked by a learned predictor of collision closeness and ego deviation, then executed by a hierarchical adversarial skill policy with an adversarial prior trained on collision and near-miss demonstrations. Across five evaluation settings, SEAL-trained policies improve Success rates by an average $s_{\text{inst}}$33 relative to the best baseline, while scenario realism on WOMD-Normal perturbed scenes reaches Realism WD $s_{\text{inst}}$34 versus CAT-Gen $s_{\text{inst}}$35 and GOOSE-Gen $s_{\text{inst}}$36 (Stoler et al., 2024). This work is notable because it explicitly positions heuristic adversarial generation as insufficient when it produces overly aggressive, non-reactive behavior.

6. Shared design patterns, theoretical perspectives, and open issues

Despite the heterogeneity of domains, the literature converges on several recurring design patterns. First, heuristic adversarial skill generation is usually closed-loop. SkillAttack refines prompts from execution traces; MemoAttack updates posteriors and lifecycle states from progress and success; RSD updates the generator from regret; HAP updates the teacher from student returns; T-STAR updates each policy from discriminator-scored terminal compatibility (Duan et al., 5 Apr 2026, Zhang et al., 28 May 2026, Zhang et al., 26 Jun 2025, Xu et al., 21 Oct 2025, Lee et al., 2021). Second, the heuristics are almost always structured rather than free-form. They operate over attack surfaces, templates, memory units, latent skills, task distributions, or scenario candidates. Third, realism or plausibility is a central constraint in the stronger systems: SkillAttack keeps prompts contextually plausible, DDIPE disguises payloads inside documentation-native artifacts, SEAL constrains adversaries through demonstration-grounded priors, and ASE uses an adversarial motion prior to keep heuristic downstream rewards from producing unnatural behavior (Duan et al., 5 Apr 2026, Qu et al., 3 Apr 2026, Stoler et al., 2024, Peng et al., 2022).

A recurring misconception is that heuristic methods are necessarily ad hoc and theoretically ungrounded. The Decoding Game offers a counterexample at the level of generation policy: it casts next-token generation as a two-player zero-sum game between a Strategist and adversarial Nature, shows that the one-step optimum is a regularized truncation problem, and derives Top-$s_{\text{inst}}$37, Top-$s_{\text{inst}}$38, temperature scaling, and hybrids as first-order approximations to the minimax-optimal strategy under total-variation uncertainty (Chen et al., 2024). In that formulation, Nature induces an implicit $s_{\text{inst}}$39-type regularization and a hard support truncation, which explains why heuristic truncation-normalization methods can outperform brittle MAP-style decoding in practice. This suggests, though does not prove, that some heuristic adversarial skill generators are better understood as approximate robust optimizers rather than merely engineering expedients.

Another misconception is that static auditing, one-shot evaluation, or simple heuristic perturbations are sufficient. SkillAttack shows that prompt-only exploitation of latent vulnerabilities often emerges only after iterative path refinement, with most successes appearing in rounds 3–4 (Duan et al., 5 Apr 2026). DDIPE shows that documentation-driven payloads can bypass both model alignment and framework guardrails even when static analysis detects most samples (Qu et al., 3 Apr 2026). SEAL shows that contact-centric stress heuristics can be effective yet unrealistic, and that realism-aware adversaries can be simultaneously more plausible and more useful for closed-loop training (Stoler et al., 2024).

The major limitations are also recurrent. Security systems often depend on a judge model or evaluator, as in SkillAttack’s use of Gemini 3.0 Pro Preview and MemoAttack’s LLM-based evaluation pipeline (Duan et al., 5 Apr 2026, Zhang et al., 28 May 2026). Supply-chain attacks assume retrieval success and can be sensitive to defense changes, which is why provenance, signing, allowlists, per-skill permission manifests, documentation sanitization, and dynamic sandbox testing recur as recommended mitigations (Qu et al., 3 Apr 2026). Curriculum and RL systems frequently lack formal convergence guarantees; HAP explicitly reports practical stabilization rather than equilibrium proofs, and RSD notes that regret is a heuristic surrogate whose estimates can be biased for unseen skills or nonstationary buffers (Xu et al., 21 Oct 2025, Zhang et al., 26 Jun 2025). Embodied systems remain sensitive to hyperparameters, discriminator drift, or demonstration quality, as emphasized in T-STAR, SEAL, and ASE (Lee et al., 2021, Stoler et al., 2024, Peng et al., 2022).

The field therefore supports a restrained generalization. Heuristic adversarial skill generation is not a single algorithmic family, but a recurring design principle: adversarial objectives are made tractable by generating structured skill candidates through heuristics that are informed by traces, priors, posteriors, discriminators, or performance histories. In agent security, this principle exposes latent exploitability in nominally benign skills. In reinforcement learning and control, it produces adaptive curricula, more chainable subtasks, more robust manipulation policies, and more realistic stress scenarios. In generation theory, it can even recover familiar sampling rules as robust responses to adversarial uncertainty.