Closing the Distribution Gap in Adversarial Training for Large Language Models
This presentation explores a critical vulnerability in current adversarial training methods for large language models: they fail to cover the full data distribution, leaving models susceptible to simple exploits like prompt translation or tense changes. The talk introduces Distributional Adversarial Training, a novel framework that combines diffusion models with continuous adversarial training to close this gap and achieve more robust language models that can withstand a broader range of natural language variations.Script
A language model survives sophisticated attacks but crumbles when you simply translate the prompt to French. This paper reveals why adversarial training fails at such basic exploits and introduces a solution that closes the distribution gap.
The authors identify a fundamental flaw: existing adversarial training augments with empirical examples, creating robustness in a narrow neighborhood while leaving vast portions of the input distribution unprotected. A model might withstand carefully crafted adversarial tokens yet collapse when faced with everyday linguistic variations it never trained on.
The key insight is to treat adversarial training as a distribution problem, not just a data problem.
Distributional Adversarial Training, or DAT, leverages diffusion models to sample from the actual distribution of language rather than relying solely on observed examples. This generative approach produces adversarial examples that span the natural variation in real-world prompts.
Where traditional methods fortify individual data points, DAT fortifies the entire distribution. The diffusion model acts as a surrogate, generating high-likelihood samples that traditional training would never encounter, then continuous adversarial training ensures the model responds robustly to worst-case perturbations across this expanded space.
The method operates in two stages. First, diffusion models generate diverse adversarial samples that reflect the true variety of possible inputs. Second, continuous adversarial training updates model parameters against worst-case perturbations within this richer sample space, building robustness that generalizes beyond the training set.
The results demonstrate that DAT achieves substantially higher adversarial robustness than existing methods. By closing the distribution gap, models become resilient not just to crafted attacks but to the full spectrum of natural language variation, a critical step toward deployment-ready safety.
The authors acknowledge that DAT depends on the quality of the diffusion surrogate. While current diffusion models provide strong approximations, exploring other generative architectures could yield even tighter coverage of the true data distribution and further enhance robustness guarantees.
This work matters because it reframes adversarial training as a problem of coverage, not just defense. A truly robust language model must withstand the full richness of human language, and DAT offers the first principled method to train for that reality, not just against known attacks.
Distributional Adversarial Training closes the gap between what models train on and what they encounter in the wild. Visit EmergentMind.com to learn more and create your own videos exploring the latest research.
