--- title: Adversarial Skill Compositional Training url: https://www.emergentmind.com/topics/adversarial-skill-compositional-training-ascot type: topic --- # Adversarial Skill Compositional Training Adversarial Skill Compositional Training (ASCoT) designates a class of frameworks and algorithms for achieving robust compositional generalization in both reinforcement learning (RL) and large language model (LLM) domains. The central principle is to train policies or models not merely to master isolated skills or attack-defense instances, but to perform well (in the robustness or alignment sense) under adversarial—often combinatorially complex—compositions of skills or input primitives. ASCoT frameworks unify adversarially-motivated min–max training, skill/task compositionality, and scalable implementation mechanics grounded in recent theoretical and empirical advances [2111.07999], [2510.21910], [2302.02984]. ## 1. Core Theoretical Frameworks Three primary instantiations of ASCoT have been developed in the literature, each addressing distinct challenges: - **Adversarial Skill Chaining for RL**: In long-horizon manipulation, the challenge is to sequence $K$ pre-trained or learned subskills $\pi^1,\dotsc,\pi^K$ such that the terminal-state distribution $\beta_{i-1}$ of skill $i-1$ lies within the initiation set $\mathcal{I}_i$ of $i$ for all $i=2,\dotsc,K$. Naïve methods expand initiation sets, leading to compounding difficulty and explosion of reachable states. ASCoT (notably the T-STAR algorithm) regularizes the terminal states to ensure $\beta_{i} \approx \mathcal{I}_{i+1}$ via adversarial losses, enforcing a compositional interface between skills [2111.07999]. - **Robust Subtask Learning (Adversarial Subtask Games)**: Here, a collection of subtask policies $\Pi = \{\pi_\sigma : \sigma\in \Sigma\}$ is trained in a multi-task MDP, facing an adversary that selects the worst-case orderings of subtasks. The design objective is to maximize $\inf_{\tau} J(\Pi; \tau)$, with $\tau$ any subtask sequence, reducing to a two-player zero-sum Markov game [2302.02984]. - **Compositional Adversarial Robustness in LLM Alignment**: In the context of language model jailbreak defense, most new adversarial attacks are compositional recombinations of a finite “dictionary” of skill primitives. ASCoT here refers to fine-tuning on adversarial examples synthesized by composing several detectable primitives, thereby ensuring robustness to unseen skill recombinations [2510.21910]. ## 2. Formal Objectives and Min–Max Training ASCoT instantiates compositional training via explicit min–max or adversarial optimization objectives tailored to the application domain. - **RL/Skill Chaining**: The core reward for each skill combines environment rewards $R^i_{\mathrm{ENV}}$, a GAIL-style imitation loss for subtask mastery, and a terminal-state regularization bonus: $$ R^i(s,a,s';\phi,\omega) = \lambda_1 R^i_{\mathrm{ENV}}(s,a,s') + \lambda_2 R^i_{\mathrm{GAIL}}(s,a;\phi) + \lambda_3 R^i_{\mathrm{TSR}}(s';\omega) $$ GAIL adversarial imitation loss is enforced by a discriminator $f_\phi^i$; terminal-state regularization uses a discriminator $D^i_\omega$ trained to distinguish successful starts from previous skill terminations, directly incentivizing terminal distributions to match initiation sets [2111.07999]. - **Worst-case Subtask Sequencing**: The value function is optimized for adversarial sequences: $$ \max_{\Pi} \min_{\tau} \mathbb{E}_{\pi_\Pi} \left[ \sum_{t=0}^\infty \gamma^t r(s_t, a_t) \right] $$ Each subtask-nexus transition is adversarially chosen, and Bellman-adversarial equations capture worst-case reasoning [2302.02984]. - **Skill-Space Adversarial Training (LLMs)**: Given a learned dictionary $D_{final} = \{d_1, ..., d_k\}$ of primitives, adversarial prompts are generated by composing random subsets with harmful instructions. The loss is ordinary negative log-likelihood on these synthetic compositions, so the LLM must learn to refuse combinations not seen in training but systematically covered via the skills combinatorial space [2510.21910]: $$ \mathcal{L}_{\mathrm{ASCoT}}(\theta) = \mathbb{E}_{(q',r)\in S} [ -\log p_\theta(r \mid q') ] $$ ## 3. Algorithms and Training Procedures ASCoT implementations leverage adversarial pipeline design, RL, and dictionary learning. - **T-STAR (RL, Long-Horizon Manipulation)**: - Initial pre-training of skill policies via GAIL and RL in isolation. - Iterative alternate training: - For subtask $i$, rollout from mixture of environment and prior-skill terminal states. - Update GAIL discriminators, initiation-set discriminators. - RL update for policy incorporating all reward terms. - Terminal state buffers maintain compactness of $\beta_i$ [2111.07999]. - **Rosac/Arosac (Compositional RL)**: - Data collection alternates skill execution. - At each subtask termination, select next subtask adversarially (greedy/minimum-value or with MCTS). - Q-function and policy networks updated via Soft Actor-Critic; Asynchronous variant trains multiple subtasks in parallel with periodic cross-subtask value estimation. - Pseudocode specifies buffer management, value backups, and adversarial sampling [2302.02984]. - **ASCoT (LLM Alignment, Jailbreak Robustness)**: - Primitives extracted via LLM-aided annotation and dictionary learning with K-SVD and redundant skill pruning. - Harmful initial queries are composed (at depth 1–5) with sampled skill primitives via LLM prompt engineering. - Training batches interleave compositional adversarial, benign, and over-refusal calibration examples. - Model fine-tuned with LoRA adapters on the merged dataset, using cross-entropy loss [2510.21910]. ## 4. Empirical Results and Comparative Performance ASCoT variants demonstrate empirical state-of-the-art performance relative to baselines in their domains. - **Task and Environment Benchmarks**: - **RL/Robotics**: IKEA furniture assembly, Rooms navigation, and F1/10 car racing benchmarks. - **LLM Defense**: StrongReject suite (LLM jailbreaks), with direct requests, unseen attack transfer (AutoDAN-Turbo), and implicit reference settings. - **Quantitative Results**: | Method | table_lack | chair_ingolf | |-----------------------|-------------------|-------------------| | BC | 0.03 ± 0.00 | 0.04 ± 0.01 | | PPO | 0.09 ± 0.11 | 0.14 ± 0.03 | | GAIL | 0.00 ± 0.00 | 0.00 ± 0.00 | | GAIL + PPO | 0.21 ± 0.11 | 0.22 ± 0.08 | | SPiRL | 0.05 ± 0.00 | 0.03 ± 0.00 | | Policy Sequencing | 0.63 ± 0.28 | 0.77 ± 0.12 | | T-STAR (ASCoT) | **0.90 ± 0.07** | **0.89 ± 0.04** | - **Skill Chaining**: T-STAR achieves the first model-free RL solution to long-horizon assembly tasks, with ≥87% success rates and 0.90 progress, outperforming prior skill-chaining and imitation learning methods [2111.07999]. - **Compositional RL**: ASCoT (Rosac/Arosac) matches/exceeds alternate multi-agent and unsupervised adversarial curriculum baselines, demonstrating superior robustness and data efficiency as measured by subtask-completion under both random and MCTS adversaries [2302.02984]. - **LLM Robustness**: ASCoT consistently lowers harmfulness on new unseen adversarial attacks (harmfulness ≈ 0.10–0.12 vs. 0.14–0.43 for other methods), and maintains low over-refusal rates (ORR ≈ 0.05), competitive with or surpassing leading closed-source and open-source models [2510.21910]. ## 5. Theoretical Properties and Insights - **Chaining Condition via Distributional Regularization**: In the limit of discriminator capacity, the ASCoT terminal-state regularizer enforces $$ D^i_\omega(s) = \frac{p_{\mathcal{I}_i}(s)}{p_{\mathcal{I}_i}(s) + p_{\beta_{i-1}}(s)} $$ and maximizing the terminal-state regularization encourages $p_{\beta_i} \rightarrow p_{\mathcal{I}_{i+1}}$ (Jensen–Shannon divergence minimization), formalizing modular compositionality [2111.07999]. - **Skill-Space Coverage vs. Data Scale**: In LLM settings, robustness is shown empirically to depend not just on the amount of adversarial data, but the breadth of primitive skill combinations covered. Monotonic improvements are observed with increased skill primitives included in training (coverage dividend), but diminishing returns occur as dictionary completeness is approached [2510.21910]. - **Game-Theoretic Guarantees**: The reduction of adversarial subtask sequencing to a two-player stagewise Markov game provides a foundation for convergence analysis—both for policy/value iteration in tabular settings and in continuous spaces via approximate RL [2302.02984]. ## 6. Limitations and Prospective Extensions - **Current Constraints**: - Fixed set of subtasks/skills or dictionary primitives is assumed; automatic discovery remains open. - Adversarial training can be unstable, especially with high weighting of regularization terms (e.g., $\lambda_3$ in T-STAR). - Most instantiations require access to per-skill success indicators, expert demonstrations, or explicit harmful prompts. - Simulated environments assume automatic resets and precise subgoal definitions. - **Potential Extensions**: - Automated, unsupervised option or primitive discovery—e.g., via hierarchical or information-theoretic methods. - Online or continual augmentation of skill dictionaries to address drift in adversarial tactics (for LLMs). - Higher-order compositionality: joint training of high-level planners atop ASCoT-trained skills or primitives; non-linear skill interactions. - Transferring methods to real-world/multi-modal domains, vision-based observation models, or interactive/multi-turn dialogue settings. - Reducing dependence on demonstrations or labeled data via large-scale offline RL or unsupervised exploration [2510.21910], [2302.02984], [2111.07999]. ## 7. Relevance and Impact ASCoT frameworks provide a scalable and general paradigm for robust skill composition across agentic and language domains. By framing robustness, adaptability, and alignment as problems of adversarial compositional generalization, these methods overcome brittleness in prior imitation or task decomposition approaches. Empirical and theoretical advances confirm that adversarial skill compositionality is critical for high performance under distributional shift, novel task sequencing, and adversarial attack generalization. ASCoT has established new performance frontiers in robotic manipulation, continuous-control RL, and LLM jailbreak defense, and continues to motivate active research in compositionality, min–max optimization, and automated skill discovery [2111.07999], [2510.21910], [2302.02984]. Source: https://www.emergentmind.com/topics/adversarial-skill-compositional-training-ascot