Learning Graphon Mean Field Games and Approximate Nash Equilibria

Abstract: Recent advances at the intersection of dense large graph limits and mean field games have begun to enable the scalable analysis of a broad class of dynamical sequential games with large numbers of agents. So far, results have been largely limited to graphon mean field systems with continuous-time diffusive or jump dynamics, typically without control and with little focus on computational methods. We propose a novel discrete-time formulation for graphon mean field games as the limit of non-linear dense graph Markov games with weak interaction. On the theoretical side, we give extensive and rigorous existence and approximation properties of the graphon mean field solution in sufficiently large systems. On the practical side, we provide general learning schemes for graphon mean field equilibria by either introducing agent equivalence classes or reformulating the graphon mean field system as a classical mean field system. By repeatedly finding a regularized optimal control solution and its generated mean field, we successfully obtain plausible approximate Nash equilibria in otherwise infeasible large dense graph games with many agents. Empirically, we are able to demonstrate on a number of examples that the finite-agent behavior comes increasingly close to the mean field behavior for our computed equilibria as the graph or system size grows, verifying our theory. More generally, we successfully apply policy gradient reinforcement learning in conjunction with sequential Monte Carlo methods.

• The paper introduces a novel formulation of graphon mean field games that converges to approximate Nash equilibria for large-scale systems.
• It leverages graph limit theory and deterministic mean field approximations to model complex interactions among infinitely many agents.
• Empirical validations on SIS-Graphon and Investment-Graphon problems demonstrate the methods' accuracy, convergence rates, and practical applicability.

Learning Graphon Mean Field Games and Approximate Nash Equilibria

Introduction

The paper "Learning Graphon Mean Field Games and Approximate Nash Equilibria" explores the intersection of graph limit theory and mean field games (MFGs) to provide scalable approaches for analyzing dynamical systems involving a large number of agents, specifically focusing on discrete-time graphon mean field games (GMFGs). The authors propose novel methods for formulating these games by extending traditional mean field and graph-based models. They leverage equivalence classes of agents and reformulations into classical mean field systems to derive approximate Nash equilibria efficiently.

Dense Graph Mean Field Games

The methodology underlying GMFGs involves modeling the interactions of infinitely many agents on a dense graph structure. Here, the concept of a graphon—a limit object representing large graphs—is instrumental (Figure 1). The authors define the limiting behavior of agent dynamics as the system size $N$ tends to infinity, enabling the usage of deterministic mean field approximations. The graphon framework permits capturing complex interaction patterns across agents.

Figure 1: Graphical model visualization. (a): A graph with 5 nodes; (b): The associated step graphon of the graph in (a) as a continuous domain version of its adjacency matrix; (c): A visualization of the dynamics, i.e. the center agent is affected only by its neighbors (grey).

The finite-agent game models are adapted to interact weakly, each agent's influence diminishing as the system scales. This assumption underlies the transition from individual agent objectives to approximations by mean field terms, permitting tractable analysis for large systems.

Theoretical Analysis

Central to the analysis is the existence and properties of GMFG equilibria, rigorously derived under assumptions pertinent to graph convergence and the Lipschitz continuity of dynamics and rewards. Theoretical results establish that GMFG solutions furnish $(\varepsilon, p)$-Markov Nash equilibria in finite systems as $N \to \infty$:

• Existence of GMFE: Employs classical MFG reformulations to ensure a graphon mean field equilibrium exists.
• Approximation Quality: Demonstrates convergence rates and error bounds for state distributions and agent dynamics approximated by mean plateaus as $N$ scales.

The results indicate a significant advance in resolving complex interactions in dense graph-based agent systems, particularly in assuring near Nash-optimality via computationally feasible means (Figure 2).

Figure 2: Decreasing maximum deviation between average $N$-agent objective and mean field objective over all agents for the GMFE policy and 5 $W$-random graph sequences. (a): Uniform attachment graphon; (b): Ranked attachment graphon; (c): ER graphon.

Empirical Validation

The empirical demonstrations focus on two main experiments utilizing graphons to model distinct problem settings:

• SIS-Graphon Problem: Models epidemic spread with agents choosing precautions. Different graphon topologies (uniform, ranked, and Erdős–Rényi) lead to distinct precautionary behavior based on agent connectivity (Figure 3).

  Figure 3: Achieved equilibrium via $M=100$ approximate equivalence classes in SIS-Graphon, plotted for each agent $\alpha \in \mathcal I$. Top: Probability of taking precautions when healthy. Bottom: Probability of being infected. It can be observed that agents with less connections (higher $\alpha$) will take less precautions. (a): Uniform attachment graphon; (b): Ranked attachment graphon; (c): ER graphon.

• Investment-Graphon Problem: Captures competitive investments among firms where profit depends on relative quality within a neighborhood. Firms with fewer competition invest to higher quality thresholds (Figure 4).

  Figure 4: The $M=100$ approximate equivalence classes solution of Investment-Graphon. We plot the probability of investing at state $x=0$ (top) together with the evolution of average quality (bottom). (a): Uniform attachment graphon; (b): Ranked attachment graphon; (c): ER graphon.

Performance metrics show strong alignment with theoretical predictions, indicating the approximations' accuracy and the methods' efficacy in obtaining solutions that align well with actual finite-agent dynamics even for modest $N$.

Conclusion

The research broadens the potential for deploying mean field approximations in graph-based multi-agent settings, offering theoretically sound methods for high-dimensional, large-scale games. Future work could expand into sparse graph formulations and enhance real-world applicability via robust graphon estimation techniques. Potential developments include addressing partial observability and noise, further enriching the model's scope to tackle more nuanced application domains.