From Game Theory to Foundation Models: The Evolution of Reinforcement Learning
This presentation traces the remarkable evolution of reinforcement learning from classical algorithmic foundations through competitive multi-agent game theory to the integration of generative foundation models for world simulation and control. We examine how principled equilibrium-seeking algorithms achieve robust, non-exploitable policies in competitive video games, and how diffusion-based world models overcome the fundamental limitations of traditional approaches to enable long-horizon planning and generalization in high-dimensional domains.Script
Most reinforcement learning algorithms struggle in competitive games because they optimize for winning, not for being unexploitable. This dissertation reveals how game-theoretic foundations and generative world models converge to solve that problem.
The authors introduce Nash-DQN, an algorithm that directly computes Nash equilibria during training rather than simply finding best responses. In competitive fighting games, this approach produces policies with order-of-magnitude lower exploitability than standard population-based methods.
Traditional world models predict one step at a time, and errors compound catastrophically over long horizons. Diffusion world models generate entire trajectories jointly, which this work shows reduces compounding errors and achieves 44 percent improvement over conventional approaches in offline reinforcement learning benchmarks.
When tested in the FightLadder benchmark across multiple fighting games, league training consistently outperforms independent learning and meta-solver frameworks. The Elo distributions reveal that population diversity alone is insufficient without explicit equilibrium computation to achieve robust policies.
The thesis identifies a fundamental limitation in autoregressive video world models: they lack explicit memory and suffer from compounding visual drift. By introducing global state conditioning and video retrieval augmented generation, the authors achieve temporally consistent simulation over long horizons, making these models viable for actual planning and control.
This work reveals a clear trajectory: reinforcement learning is evolving from isolated algorithmic tricks into a unified framework where game-theoretic robustness, generative world modeling, and foundation model priors combine to enable decision-making in domains classical methods cannot reach. To explore these ideas further and create your own explanations, visit EmergentMind.com.
