---
title: Two-Agent Adversarial Flow Networks
url: https://www.emergentmind.com/topics/two-agent-adversarial-flow-networks-aflownets
type: topic
---

# Two-Agent Adversarial Flow Networks

Two-Agent Adversarial Flow Networks (AFlowNets) are a variant of generative flow networks (GFlowNets) designed for modeling and solving two-player zero-sum games within a flow-based formalism. By extending expected flow network (EFlowNet) principles to adversarial, turn-based environments, AFlowNets provide a theoretically grounded, off-policy self-play algorithm that seeks a unique Nash equilibrium policy pair through the minimization of trajectory-balance loss. Empirical evaluations demonstrate that AFlowNets achieve high optimal move rates and outperform AlphaZero in the classical game of Connect-4 under comparable computational constraints [2310.02779].

## 1. Background and Theoretical Foundation

GFlowNets are sequential generative models that sample trajectories on directed acyclic trees, producing objects (terminal states) with probability proportional to a predefined reward function. Each state $s$ is associated with a non-negative flow $F(s)$, and policy $P_F(s'|s) = F(s')/F(s)$ governs transitions along the tree. The flow-matching constraints require
\[
F(s) = \sum_{s' \in \Ch(s)} F(s'),\quad \forall s \notin \mathcal X, \qquad F(x) = R(x),\quad \forall x \in \mathcal X
\]
where $\Ch(s)$ denotes children of $s$ and $R$ is the reward function on terminal states $\mathcal X$.

EFlowNets extend this formalism to stochastic environments, partitioning nonterminal states into agent states and environment states. For environment states $s \in \mathcal V_{\rm env}$, transitions follow a fixed stochastic kernel $P_{\rm env}$. The EFlowNets introduce expected detailed balance (EDB) constraints:
- **Agent step:** $F(s)P_{\rm agent}(s'|s) = F(s')$ for $s \in \mathcal V_{\rm agent}$.
- **Environment step:** $F(s) = \mathbb{E}_{s'\sim P_{\rm env}(\cdot|s)}[F(s')]$ for $s \in \mathcal V_{\rm env}$.
- **Terminal step:** $F(x) = R(x)$ for $x \in \mathcal X$.

Flows and agent policies are uniquely determined under mild conditions; when no environment states are present, this reduces to standard (deterministic) GFlowNets [2310.02779].

## 2. Adversarial Flow Networks for Two-Player Zero-Sum Games

In the two-agent adversarial context, the game tree $G=(\mathcal V, \mathcal E)$ is partitioned into disjoint player states $(\mathcal V_1, \mathcal V_2)$ and terminal states $\mathcal X$. Player $i$ acts at states $s \in \mathcal V_i$, while the other player’s actions are modeled as part of a stochastic environment from $i$’s perspective.

Each player maintains:
- A flow function $F_i:\mathcal V \to \mathbb R_{>0}$,
- A policy $P_i(s'|s) = F_i(s')/F_i(s)$ for $s \in \mathcal V_i$.

The adversarial EDB constraints for player $i$ are:
- **Own move:** $F_i(s)P_i(s'|s) = F_i(s')$ for $s \in \mathcal V_i$.
- **Opponent move:** $F_i(s) = \mathbb E_{s'\sim P_j(\cdot|s)}[F_i(s')]$ for $s \in \mathcal V_j$ ($j \ne i$).
- **Terminal:** $F_i(x) = R_i(x)$ where $R_1(x) R_2(x) = 1$ to satisfy a zero-sum relationship (in log-domain).

To bias toward shorter wins, a branch-adjustment is introduced:
\[
R_i(x) = \frac{R_i^\circ(x)}{B_i(x)},\quad B_i(x) = \prod_{k: s_k \in \mathcal V_i} |\Ch(s_k)|
\]
with $R_i^\circ(x) = e^{+\lambda}$ for win, $1$ for draw, $e^{-\lambda}$ for loss.

## 3. Trajectory-Balance Constraints and Training Objective

The global trajectory-balance (TB) constraint offers an alternative to local EDB constraints. For any trajectory $\tau=(s_0, \dots, s_n=x)$:
\[
Z\prod_{i: s_i \in \mathcal V_1} P_1(s_{i+1}|s_i) = R_1(x) B_2(x) \prod_{i: s_i \in \mathcal V_2} P_2(s_{i+1}|s_i)
\]
where $Z>0$ is unique.

The trajectory-balance loss for a trajectory $\tau$ is
\[
\mathcal L_{\rm TB}(\tau; \theta_1, \theta_2, Z) = \left[ \log Z + \sum_{s_i \in \mathcal V_1} \log P_1^\theta(s_{i+1}|s_i) - \log R_1(x) - \log B_2(x) - \sum_{s_i \in \mathcal V_2} \log P_2^\theta(s_{i+1}|s_i) \right]^2.
\]
Parameters $(\theta_1, \theta_2, \log Z)$ are updated by minimizing the expected TB loss over sampled self-play trajectories. The resulting Nash equilibrium is unique [2310.02779].

## 4. Training Algorithm and Implementation

Training proceeds via on-policy self-play, coupled with experience replay. The algorithm alternates between two phases:
1. **Self-play Episode Generation:** $N_{\rm new}$ complete episodes are played by alternating $P_1$ and $P_2$, with all trajectories stored in a buffer $\mathcal B$.
2. **Parameter Updates:** For $L$ steps, batches of trajectories $\{\tau_j\}$ are sampled from $\mathcal B$; gradients $\nabla_{\theta_1}$, $\nabla_{\theta_2}$, and $\nabla_{\log Z}$ of the summed $\mathcal L_{\rm TB}$ are computed and used to perform gradient descent steps.

This approach enables off-policy updates, facilitating sample efficiency and stabilizing learning due to the existence of a unique minimax equilibrium.

## 5. Empirical Evaluation in Connect-4

AFlowNets have been evaluated on the Connect-4 environment, with states defined as board configurations plus the current player. Legal actions are drop-column moves, and the tree structure is memory-augmented to prevent transpositions.

Metrics include:
- **Elo rating** (BayesElo) from matches against uniform random agents and AlphaZero (with and without MCTS during test time).
- **Optimal move rate**: The fraction of positions in which the chosen move matches a minimax solver.

Key reported results after 3 hours on a single RTX 8000 GPU include:
- **AFlowNet$_{15}$** selects the minimax move in over 80% of positions.
- **Tournament Elo (mean ± std, 3 seeds):**

  | Agent             | Elo                |
  |-------------------|--------------------|
  | AFlowNet$_2$      | 1190.8 ± 64.2      |
  | AFlowNet$_{10}$   | 1700.1 ± 60.0      |
  | AFlowNet$_{15}$   | 1835.3 ± 154.9     |
  | AlphaZero (no MCTS) | ~700             |
  | AlphaZero + MCTS  | ~900               |

- **Win–draw–loss scores (first-player view, out of 50 games) against AlphaZero + MCTS:**

  | Agent         | vs AFlowNet$_2$ | vs AFlowNet$_{10}$ | vs AFlowNet$_{15}$ |
  |---------------|-----------------|--------------------|--------------------|
  | AFlowNet$_2$  | -               | 0–0–50             | 5–0–45             |
  | AFlowNet$_{10}$ | 50–0–0         | -                  | 20–0–30            |
  | AFlowNet$_{15}$ | 50–0–0         | 35–0–15            | -                  |
  | AlphaZero+MCTS | 25–0–25         | 53–0–47            | 50–0–50            |

This demonstrates that AFlowNets substantially outperform AlphaZero baselines, especially when acting without MCTS for inference [2310.02779].

## 6. Advantages, Limitations, and Extensions

AFlowNets confer several benefits:
- Single-pass policy rollout—eliminating the need for Monte Carlo tree search during training and inference.
- Off-policy self-play with replay buffers, facilitating convergence to a unique equilibrium without cyclic instability.
- Adaptability to both stochastic transitions (via EFlowNet formalism) and adversarial, turn-based interactions.

Limitations include:
- High variance of trajectory-balance objectives for long games and the requirement to store entire episodes.
- Necessity to tune the branch-adjustment factor $B_i(x)$ and the hyperparameter $\lambda$ to balance the diversity of exploration and exploitation.
- Scalability to very large games, such as chess or Go, may require future advances, e.g., subtrajectory-based trajectory-balance formulations.

Potential extensions—motivated by the generality of the formalism—include multi-player and general-sum games, latent-variable modeling for incomplete-information games, adaptation to continuous action spaces, and hybridization of AFlowNet policies with limited-depth look-ahead or search.

AFlowNets thus import the diversity-seeking, off-policy sampling virtues of GFlowNets into adversarial domains, yielding a principled and effective self-play learning method [2310.02779].

Source: https://www.emergentmind.com/topics/two-agent-adversarial-flow-networks-aflownets