Reinforcement Learning: From Algorithms To Foundation Models

Abstract: Reinforcement learning (RL) provides a framework for sequential decision making under explicit objectives. In its classical form, RL studies how an agent should act to maximise long-term reward in a dynamic environment. In richer settings, the problem extends beyond a single agent and fixed environment: intelligent behavior may require strategic interaction, adaptation to uncertainty, and reasoning over high-dimensional worlds. This thesis studies RL from two perspectives: algorithms in games and RL in the era of foundation models. The first part focuses on multi-agent RL in games. It examines how incentives, policies, and equilibrium concepts interact in competitive and general-sum environments, spanning two-player zero-sum games, large-scale video games, and multi-player settings with general structure. These works investigate learning in multi-agent systems and the behavior of RL methods in interactive environments. The second part studies RL with generative and foundation models, motivated by the idea that prior knowledge can enrich sequential decision making. Pretrained generative models and learned world models serve as representation tools and structured priors for planning, control, and policy optimization. The thesis develops diffusion-based world models, investigates RL for efficient video generation, explores generative models as policy classes, and studies interactive video world models in which actions shape future observations. It also addresses long-horizon modeling through architectures with memory. Together, these contributions present a unified view of RL as objective-driven adaptation in complex sequential domains. From strategic games to generative world models, the thesis highlights how RL connects decision making, environment modeling, and emerging foundation-model capabilities, offering a broader perspective on the principles underlying intelligent behavior.

• The paper establishes rigorous reinforcement learning theory using MDPs, Q-learning, and policy gradients to ensure convergent and sample-efficient deep RL methods.
• The paper introduces Nash-DQN algorithms for multi-agent environments, demonstrating order-of-magnitude improvements in robustness and non-exploitability in competitive settings.
• The paper integrates diffusion-based world models with RL, achieving significant performance gains in long-horizon trajectory prediction and scalable generative policy learning.

Reinforcement Learning: Algorithmic Foundations and the Transition to Foundation Models

Introduction

This comprehensive dissertation presents a cohesive narrative on reinforcement learning (RL), systematically examining its algorithmic underpinnings, progression to multi-agent settings, integration with generative world modeling, and the paradigm shift induced by the advent of large-scale foundation models. The work is organized into two main themes: (1) the study of RL and multi-agent systems from classical algorithmic perspectives, and (2) the incorporation of sophisticated representation learning and generative modeling—specifically through diffusion and foundation models—for advancing RL efficacy in high-dimensional, dynamic, and interactive domains.

---

Classical RL Foundations: Theory and Algorithms

The thesis begins by establishing the mathematical and computational foundations of RL, primarily within the Markov Decision Process (MDP) formalism. It provides rigorous treatments of value-based and policy-gradient methods:

• Value Iteration and Q-learning: The convergence proofs for both value and Q-iteration are formally derived using contraction mapping arguments and Banach’s fixed-point theorem. This establishes the theoretical basis for dynamic programming and sample-based learning in tabular and large-scale environments.
• Policy Gradient Methods: The analysis encompasses both stochastic and deterministic policy gradients, including the foundational PG theorem, reparameterization trick for continuous actions, and entropy-regularized optimization. The material includes new theoretical results on gradient estimators for reparameterizable stochastic policies, generalizing earlier results and connecting to modern actor-critic variants.

These foundational results underpin much of contemporary deep RL and are prerequisites for further exploration into multi-agent systems.

---

Multi-Agent Reinforcement Learning and Game Theory

A core section of the work addresses RL in competitive and collaborative multi-agent environments, leveraging game-theoretic constructs:

• Zero-Sum Markov Games: The extension from single-agent MDPs to multi-agent Markov games is rigorously presented, including the minimax theorem, value function definitions for stochastic policies, and Nash equilibria conditions. The discussion navigates the interplay between best-response dynamics and equilibrium computation.
• Algorithmic Innovations: Notably, the thesis proposes Nash-DQN and Nash-DQN-Exploiter—practical deep algorithms for two-player zero-sum Markov games. These variants hybridize Q-learning with Nash equilibrium computation, moving beyond reductions to normal-form games and embodying tabular RL convergence guarantees in deep function approximation settings.
• Empirical Validation: Extensive experiments on tabular games and a suite of competitive two-player video games (see Figure 1 and Figure 2) illustrate the superiority of the proposed Nash-based algorithms over standard best-response population baselines (NFSP, PSRO). Key results include order-of-magnitude lower exploitability and demonstrable empirical robustness against high-capacity exploiters, substantiating claims of non-exploitability.

  Figure 1: Screenshots from the two-player video games used in the evaluation of deep MARL algorithms.

  

  Figure 2: Learning curves for exploiters trained against different algorithms, showing that Nash-DQN produces policies that are difficult to exploit.

• Markov Potential Games: The thesis also covers the characterization of general-sum Markov games that admit potential functions, enabling the reformulation of equilibrium finding as global optimization over these potentials. Sufficient structural conditions for Markov potential games are proven, and distributed load balancing is presented as an application.

---

Competitive RL at Scale: Benchmarking and Population Methods

Building on the algorithmic advances, the thesis introduces "FightLadder"—a scalable, high-fidelity environment for benchmarking multi-agent reinforcement learning in competitive video games (see Figure 3 and Figure 4):

Figure 3: FightLadder supports multiple cross-platform fighting games for structured multi-agent RL benchmarks.

Figure 4: The motion and attack action spaces in fighting games highlighting the complexity of the action representation.

Key experimental modules include:

• Evaluation with Population-Based Methods: Comparison of independent learning (IPPO, 2Timescale), classical meta-solver-based frameworks (FSP, PSRO), and league training (à la AlphaStar). League training outperforms other baselines in terms of Elo ratings and resistance to exploitation.
• Scalable Evaluation Metrics: Introduction and use of win-rate curves, real-time Elo computation (Figure 5), and direct exploitability measurement via adversarial learning.
• Implications:
  • Standard deep RL algorithms (DQN, PPO) and population variants fail to achieve non-exploitable equilibria in high-dimensional, adversarial environments.
  • Nash-DQN and its variants are explicitly constructed to minimize exploitability, providing a principled approach to robust multi-agent policy learning.
  • 

  • Figure 5: Distributions of Elo ratings for the top agents produced by different baseline algorithms.

---

Distributed RL in General-Sum Settings: Load Balancing as Markov Potential Game

In the context of distributed networking, the thesis develops a framework for RL-driven load balancing, where fairness-based rewards allow for potential game structure:

• Markov Potential Game Formulation: The reward structure is engineered such that each agent’s best-response dynamics align with optimizing a global fairness objective. The work demonstrates how independent natural policy gradients converge to Nash equilibria under this structure.
• Empirical Performance: Experiments in both simulation and real datacenter networks demonstrate that the proposed distributed MARL approach can outperform classic algorithmic and heuristic baselines, providing both fairness and throughput improvements.

Figure 6: Accurate estimation of server processing speed is challenging and has direct implications for load balancing performance.

Figure 7: The distributed MARL framework for network load balancing, enabling scalable decentralized optimization.

---

RL Meets Foundation Models: World Modeling and Generative Policies

The second major part of the thesis focuses on the integration of RL with generative modeling, foundation architectures, and latent world simulation. Several core advances are documented:

• Diffusion-Based World Models for RL: The thesis presents diffusion world models (DWM) for multi-step dynamics prediction in offline RL settings. Unlike traditional one-step models, DWM can jointly generate long trajectories, dramatically reducing compounding errors in model rollout (see Figure 8).

  Figure 8: Performance of DWM-based RL as a function of planning horizon, demonstrating stable returns even for long rollouts.

• Algorithmic Implications: The DWM framework is plugged into various offline RL pipelines as a model-based value expansion method, which is shown both theoretically and empirically to regularize value estimation and stabilize policy optimization even under severe distribution shift.
• Numerical Results: DWM-based MBRL achieves up to 44% improvement over conventional one-step models and outperforms strong model-free baselines in D4RL benchmarks. The work also demonstrates the intrinsic limitations of transformer-based world models for long-horizon rollout in the absence of non-autoregressive joint sequence modeling (see Figure 9).

Figure 9: Average prediction errors for DWM as a function of diffusion steps and inference configurations, highlighting superior long-horizon predictive fidelity.

• Expressive and Efficient Policy Classes: The thesis advances the use of consistency models—fast, few-step generative models—for policy parameterization in both offline and online RL. This addresses the trade-off between multi-modality in policy distributions (critical for offline RL/IL) and fast policy inference required for interaction with complex simulators or robotics.

---

Video Generation, Interactive Simulators, and Memory-Augmented World Modeling

Progressing from single-step generative world modeling to structured, interactive video generation, the work examines issues arising in long-horizon, high-dimensional video simulation:

• Distillation, Consistency, and Latent Reward Optimization: A pipeline is introduced for distilling high-quality, few-step video diffusion models from teacher models using a combination of distillation, self-consistency, and reward-based tuning in latent space. This permits order-of-magnitude speedups in video synthesis without quality loss, making reinforcement learning feasible in complex environments.
• Interactive Video World Models: The work identifies the compounding error and insufficient memory bottlenecks present in contemporary autoregressive, diffusion-based video world models. The introduction of explicit global state conditioning and video retrieval augmented generation (VRAG) substantially improves both interactive fidelity and long-term consistency.
• Global Memory and Local Attention: Fundamentally, the thesis proposes a general recurrent-autoregressive diffusion framework using global memory and local attention (combining LSTMs/Mamba/TTT with DiT backbones). This achieves efficient, scalable, and temporally-consistent long video world modeling with provable improvements in simulation quality.

---

Implications, Theoretical Insights, and Future Directions

The body of work demonstrates several notable theoretical and empirical claims:

• Principled Exploitability Minimization: Direct algorithmic alignment with game-theoretic equilibrium concepts yields highly robust, non-exploitable policies in multi-agent settings.
• Joint Generative-Policy Modeling: Expressive policy classes—diffusion and consistency models—not only improve data-driven imitation and offline RL, but also enable the unification of generative modeling and RL optimization objectives.
• Latent and World Modeling as a New RL Substrate: Incorporating pretrained generative or foundation world models as queryable simulators leverages broad priors and inductive biases, making RL feasible in domains far beyond the reach of classical methods.
• Memory and Explicit State Tracking: For world models to truly support long-term planning and generalization, explicit memory and global state conditioning are necessary, as implicit context propagation is fundamentally insufficient—the analysis of compounding errors and memory faithfulness is decoupled and addressed.

---

Conclusion

This dissertation encapsulates a unified vision of reinforcement learning: from its classical algorithmic basis, through robust equilibrium-seeking in competitive multi-agent environments, to the integration of deep generative and foundation models for complex world simulation and control. By bridging theory and scalable algorithmic practice, the work offers a pathway toward RL agents capable of robust adaptation, efficient generalization, and principled interaction in high-dimensional, dynamic domains. The progression from sample-efficient tabular algorithms to scalable, foundation model-augmented RL foreshadows the ongoing convergence of decision making, generative modeling, and general intelligence.