$α$-Rank: Multi-Agent Evaluation by Evolution (1903.01373v4)

Abstract: We introduce $\alpha$-Rank, a principled evolutionary dynamics methodology for the evaluation and ranking of agents in large-scale multi-agent interactions, grounded in a novel dynamical game-theoretic solution concept called Markov-Conley chains (MCCs). The approach leverages continuous- and discrete-time evolutionary dynamical systems applied to empirical games, and scales tractably in the number of agents, the type of interactions, and the type of empirical games (symmetric and asymmetric). Current models are fundamentally limited in one or more of these dimensions and are not guaranteed to converge to the desired game-theoretic solution concept (typically the Nash equilibrium). $\alpha$-Rank provides a ranking over the set of agents under evaluation and provides insights into their strengths, weaknesses, and long-term dynamics. This is a consequence of the links we establish to the MCC solution concept when the underlying evolutionary model's ranking-intensity parameter, $\alpha$, is chosen to be large, which exactly forms the basis of $\alpha$-Rank. In contrast to the Nash equilibrium, which is a static concept based on fixed points, MCCs are a dynamical solution concept based on the Markov chain formalism, Conley's Fundamental Theorem of Dynamical Systems, and the core ingredients of dynamical systems: fixed points, recurrent sets, periodic orbits, and limit cycles. $\alpha$-Rank runs in polynomial time with respect to the total number of pure strategy profiles, whereas computing a Nash equilibrium for a general-sum game is known to be intractable. We introduce proofs that not only provide a unifying perspective of existing continuous- and discrete-time evolutionary evaluation models, but also reveal the formal underpinnings of the $\alpha$-Rank methodology. We empirically validate the method in several domains including AlphaGo, AlphaZero, MuJoCo Soccer, and Poker.

• The paper introduces α-Rank, an evolutionary framework that leverages Markov-Conley chains to overcome Nash equilibrium limitations in multi-agent games.
• It employs a generalized discrete-time evolutionary model with a ranking-intensity parameter to simplify transitions and efficiently compute agent rankings.
• Empirical validations on domains like AlphaGo Chess and MuJoCo Soccer demonstrate its scalability and ability to capture complex, cyclic agent interactions.

This paper introduces $\alpha$-Rank, a principled methodology for evaluating and ranking agents in complex multi-agent interactions. The core problem addressed is the difficulty of assessing agent strength in games that involve more than two players, exhibit asymmetric payoffs, or have large strategy spaces, especially when standard game-theoretic concepts like Nash equilibrium are computationally intractable or do not align with the observed dynamics.

The authors argue that traditional methods, often rooted in finding static solutions like Nash equilibria, are fundamentally incompatible with the inherently dynamic nature of multi-agent learning and interaction processes. Computing a Nash equilibrium for general-sum games is computationally hard (PPAD-complete), and even if computed, selecting among multiple equilibria remains a challenge. Furthermore, evolutionary dynamics, which model how populations of agents might adapt over time, are not guaranteed to converge to a Nash equilibrium; they often exhibit complex behaviors like cycles.

Instead of relying on static equilibrium concepts, $\alpha$-Rank is grounded in a novel dynamical game-theoretic solution concept called Markov-Conley chains (MCCs). MCCs are inspired by Conley's Fundamental Theorem of Dynamical Systems, which posits that the long-term behavior of any dynamical system can be understood in terms of its recurrent sets (specifically, chain components) and transient behavior. Chain components capture irreducible behaviors that are robust to small perturbations, analogous to cycles or equilibria but applicable in higher dimensions.

The paper formally defines MCCs as irreducible Markov chains whose state space corresponds to the sink strongly connected components of the game's response graph. The response graph has pure strategy profiles as vertices and edges representing weakly better responses. Sink strongly connected components are sets of pure profiles that are reachable from each other via improving responses and have no outgoing improving responses to states outside the set. MCCs represent the "final" recurrent regions in the discrete space of pure strategies. The paper proves a correspondence between MCCs and the asymptotically stable sink chain components of the continuous-time replicator dynamics [(1903.01373), Theorem 3.2], showing that MCCs capture the essence of the dynamical system's long-term behavior.

To make this concept practical for ranking, $\alpha$-Rank leverages a generalized discrete-time finite-population evolutionary model. This model describes transitions between multi-population pure strategy profiles based on the probability that a rare mutant strategy takes over a population. The probability of a strategy copying another is determined by a selection function (like the Fermi distribution) influenced by payoffs and a "ranking-intensity" parameter, $\alpha$. Under a small mutation rate assumption, only one population changes strategy at a time, simplifying the transitions.

The crucial link established is that in the limit of infinite ranking-intensity ($\alpha \rightarrow \infty$), the Markov chain defined by this discrete-time model coincides with the MCCs [(1903.01373), Theorem 3.3]. For practical purposes, a sufficiently large $\alpha$ creates an irreducible Markov chain over all pure strategy profiles (not just MCCs), which guarantees a unique stationary distribution. This stationary distribution quantifies the proportion of time the system spends in each pure strategy profile over the long run.

The $\alpha$-Rank methodology is summarized as follows:

1. Construct the meta-game: Obtain empirical payoffs for all pairs of agent interactions in the multi-agent system under evaluation. This defines a multi-player normal form game where agent variants are strategies and players are agent roles/populations.
2. Compute the transition matrix: Build the transition matrix for the generalized discrete-time evolutionary model over all pure strategy profiles (combinations of agents for each player/population). Use a sufficiently large ranking-intensity $\alpha$ (found via a sweep) to approximate the MCC limit.
3. Compute the stationary distribution: Solve for the unique stationary distribution $\pi$ of this Markov chain.
4. Derive rankings and scores: The stationary distribution assigns a probability mass to each pure strategy profile. The $\alpha$-Rank score for an individual agent is its total mass across all profiles in which it is played. Agents are ranked based on these scores.

Key practical advantages highlighted:

• Generality: $\alpha$-Rank can be applied to games with any number of players (K-player), asymmetric payoffs, and large numbers of agents/strategies.
• Tractability: Computing the stationary distribution of the sparse transition matrix is polynomial in the number of pure strategy profiles [(1903.01373), Property 3.4], which is significantly more tractable than computing Nash for large general-sum games.
• Unique Ranking: The use of a perturbed Markov chain guarantees a unique stationary distribution, avoiding the equilibrium selection issues of Nash.
• Dynamical Insights: The ranking is based on a dynamical process, naturally capturing intransitive relationships (cycles) and identifying evolutionarily robust strategies (those with significant mass in the stationary distribution) versus transient ones (those with zero mass).

The paper empirically validates $\alpha$-Rank on diverse domains:

• Canonical games (RPS, Biased RPS, Battle of the Sexes): Illustrates how $\alpha$-Rank captures cyclic dynamics and handles asymmetry, contrasting its output with Nash in the biased RPS case.
• AlphaGo and AlphaZero Chess: Demonstrates evaluation on large pools of agents derived from snapshots during training. $\alpha$-Rank successfully ranks agents and reveals which ones are evolutionarily stable, highlighting its scalability (56 agents in AlphaZero Chess leads to a Markov chain over $56^1=56$ states in the symmetric single-population model, easily handled).
• MuJoCo Soccer: Shows application to a continuous-action physics-based domain, identifying robust agents and uncovering cyclic relationships.
• Kuhn and Leduc Poker: Validates the approach on K-player (3- and 4-player Kuhn) and asymmetric (Leduc) poker variants, domains where traditional pairwise analysis or Nash computation is challenging. For 3-player Kuhn with 4 strategies per player, the state space is $4^3=64$ profiles; for 4-player, $4^4=256$ profiles. $\alpha$-Rank handles these state spaces efficiently.

The authors propose $\alpha$-Rank as a descriptive evaluation methodology, providing insights into which strategies are evolutionarily favored or resistant to invasion in the long term, rather than prescribing rational behavior. Its computational feasibility and ability to handle complex game structures make it a strong candidate for use in AI agent leaderboards and potentially for integration into multi-agent training pipelines.