Multi-Agent Generative Flow Networks

• MA-GFlowNets are multi-agent generative flow networks that sample terminal objects proportionally to rewards by enforcing a flow-matching constraint.
• They integrate global state flows with local agent observations to enable centralized training coupled with decentralized execution.
• Algorithmic variants (CFN, IFN, JFN, CJFN, and MACFN) illustrate practical trade-offs between modeling accuracy and computational efficiency in diverse environments.

Searching arXiv for the cited MA-GFlowNet papers to ground the article in the relevant literature. Multi-Agent Generative Flow Networks (MA-GFlowNets) are a multi-agent generalization of generative flow networks in which multiple agents generate objects collaboratively through a series of joint actions, with the objective that the probability of generating a terminal object is proportional to a given reward. In the measurable formulation introduced for MA-GFlowNets, global flows live on a joint state space, local flows live on agent-specific observation spaces, and centralized or decentralized training protocols are organized around a flow-matching constraint that extends single-agent GFlowNet theory to multi-agent settings (Brunswic et al., 24 Sep 2025). Related work extends GFlowNet ideas to stochastic environments and adversarial multi-agent games through expected flow networks and adversarial flow networks, and to continuous cooperative control through Multi-Agent generative Continuous Flow Networks (MACFN), situating MA-GFlowNets at the intersection of flow-based generative modeling, centralized training with decentralized execution, and multi-agent exploration (Jiralerspong et al., 2023, Luo et al., 2024).

1. Foundational formulation

Single-agent GFlowNets provide the formal substrate for MA-GFlowNets. In the measurable setting, states $s$ live in a measurable state space $S$ with special source $s_0$ and sink $s_f$, terminal objects are the last pre-sink states $s_\tau$ reached when a STOP action is taken, and a trajectory has the form $$(s_0, a_0, s_1, a_1, \ldots, s_\tau, \mathrm{STOP}, s_f)$$. The reward is a nonnegative measure $R$ on $S^* := S \setminus \{s_0,s_f\}$, and the generative target is to sample objects proportionally to reward, $P_\theta(x) \propto R(x)$, by enforcing a measurable flow-matching constraint (Brunswic et al., 24 Sep 2025).

In that formulation, a forward policy $\pi$ is a Markov kernel $S$0 with $S$1, and the key flow objects are $S$2, $S$3, and $S$4. The measurable flow-matching constraint is

$S$5

as measures on $S$6, with termination mass

$S$7

The STOP probability at state $S$8 is given by the Radon–Nikodym ratio

$S$9

or, when densities exist,

$s_0$0

If $s_0$1 and flow matching holds, then $s_0$2 is almost surely finite, $s_0$3, and the terminal state satisfies $s_0$4 (Brunswic et al., 24 Sep 2025).

MA-GFlowNets retain this reward-proportional sampling objective while distributing control across agents. In the multi-agent formalization, there are $s_0$5 agents indexed by $s_0$6, each with an observation space $s_0$7 and projection $s_0$8. The global state is written as $s_0$9. Each agent has its own action bundle $s_f$0 over $s_f$1 with special action $s_f$2, and the environment couples actions through the global transition $s_f$3. At time $s_f$4, agents choose local actions $s_f$5 from local observations $s_f$6, yielding joint action $s_f$7 (Brunswic et al., 24 Sep 2025).

A decentralized-execution parameterization factors the joint forward policy as

$s_f$8

Global flows live on $s_f$9, local flows live on $s_\tau$0, and asynchronous stopping requires an explicit convention: when an agent chooses $s_\tau$1, it is put on hold until all agents STOP, and the global episode ends with reward $s_\tau$2 awarded on the last global state before STOP (Brunswic et al., 24 Sep 2025).

2. Local–global principle and structural assumptions

The central theoretical contribution of MA-GFlowNets is a local–global principle establishing when a collection of local GFNs can be trained as a unique global GFN. The key theorem states that, given local GFNs $s_\tau$3, there exists a global joint GFN $s_\tau$4 on the product observation space that is consistent with the local ones and factorizes as

$s_\tau$5

Moreover, if the joint GFN satisfies the flow-matching constraint for reward $s_\tau$6 and local virtual rewards $s_\tau$7, then

$s_\tau$8

This factorized construction is the basis for tractable multi-agent flow matching and for centralized training with decentralized execution (Brunswic et al., 24 Sep 2025).

The proof sketch given for the theorem constructs the joint policy as the product of local policies, constructs a joint transition $s_\tau$9 from local transitions by marginalizing over other agents’ actions and states, defines joint outflows as products of local outflows, and shows that compatibility of local flow-matching constraints on the appropriate “live” domains induces a valid global flow-matching relation with reward splitting $$(s_0, a_0, s_1, a_1, \ldots, s_\tau, \mathrm{STOP}, s_f)$$0. The paper explicitly notes that, in general, $$(s_0, a_0, s_1, a_1, \ldots, s_\tau, \mathrm{STOP}, s_f)$$1 unless agents can perfectly synchronize (Brunswic et al., 24 Sep 2025).

The theory depends on start and STOP conventions. For synchronous start,

$$(s_0, a_0, s_1, a_1, \ldots, s_\tau, \mathrm{STOP}, s_f)$$2

whereas for asynchronous start,

$$(s_0, a_0, s_1, a_1, \ldots, s_\tau, \mathrm{STOP}, s_f)$$3

STOP can be handled by unilateral STOP, where any $$(s_0, a_0, s_1, a_1, \ldots, s_\tau, \mathrm{STOP}, s_f)$$4 triggers global STOP, or by asynchronous unanimous STOP, in which agents entering “purgatory” self-loop until the last agent STOPs; in the latter case, reward is supported on purgatory states and becomes nonzero only when all have STOPped (Brunswic et al., 24 Sep 2025).

The significance of the local–global principle lies in its identification of the conditions under which independent local GFNs induce a valid joint GFN. The paper states these conditions as factorization and compatibility conditions, including independence of local policies given global state, product reward structure, and a consistent virtual transition $$(s_0, a_0, s_1, a_1, \ldots, s_\tau, \mathrm{STOP}, s_f)$$5. Under those assumptions, decentralized policies generate samples with probability proportional to the reward function (Brunswic et al., 24 Sep 2025). This suggests that MA-GFlowNets are not merely a heuristic decomposition of a central GFlowNet, but a theory of when decentralized policies can inherit the reward-proportional sampling guarantees of a single global flow network.

A related but distinct line of work formulates stochastic and adversarial multi-agent extensions by treating other agents or environmental randomness as part of the environment. In expected flow networks, environment states satisfy expected detailed balance constraints, and in adversarial flow networks each player’s environment is the other player’s policy, yielding a joint EDB fixed point in two-player zero-sum games (Jiralerspong et al., 2023). A plausible implication is that MA-GFlowNets and EFlowNet/AFlowNet frameworks address complementary multi-agent regimes: the former centers cooperative joint generation with factorized flows, while the latter centers stochasticity and adversarial interaction.

3. Algorithmic families

The MA-GFlowNet theory introduces four algorithms: Centralized Flow Network (CFN), Independent Flow Network (IFN), Joint Flow Network (JFN), and Conditioned Joint Flow Network (CJFN). All are expressed in terms of a GFN $$(s_0, a_0, s_1, a_1, \ldots, s_\tau, \mathrm{STOP}, s_f)$$6 with inflow

$$(s_0, a_0, s_1, a_1, \ldots, s_\tau, \mathrm{STOP}, s_f)$$7

and stable flow-matching loss

$$(s_0, a_0, s_1, a_1, \ldots, s_\tau, \mathrm{STOP}, s_f)$$8

where $$(s_0, a_0, s_1, a_1, \ldots, s_\tau, \mathrm{STOP}, s_f)$$9 is positive, increasing on $R$0, and decreasing on $R$1 (Brunswic et al., 24 Sep 2025).

CFN trains a single global GFN on the joint state–action space using the true global transition $R$2 and reward $R$3. The objective is

$R$4

with a joint policy over the full action space. The paper states that its joint action complexity scales as

$R$5

which is exponential in the number of agents. CFN is therefore accurate but potentially slow, and fully centralized (Brunswic et al., 24 Sep 2025).

IFN trains each local GFN independently on its own observation/action space by minimizing

$R$6

with local inflow

$R$7

Its per-agent action complexity is $R$8, but local termination flows are hard to estimate. The paper explicitly notes that this can cause spurious rewards and nonstationarity, and that IFN can lead to mode collapse (Brunswic et al., 24 Sep 2025).

JFN realizes centralized training with decentralized execution. It defines joint flows from locals by

$R$9

and minimizes a global stable FM loss

$S^* := S \setminus \{s_0,s_f\}$0

Its decentralized execution policy is

$S^* := S \setminus \{s_0,s_f\}$1

modulo STOP quotienting. The paper emphasizes that the loss only involves the global reward $S^* := S \setminus \{s_0,s_f\}$2, while coupling is achieved through the product structure of flows (Brunswic et al., 24 Sep 2025).

CJFN addresses the case where $S^* := S \setminus \{s_0,s_f\}$3 does not factor as $S^* := S \setminus \{s_0,s_f\}$4 or where $S^* := S \setminus \{s_0,s_f\}$5. It augments the state and observation spaces with a shared episode-level condition $S^* := S \setminus \{s_0,s_f\}$6, described as a latent cooperative strategy. With $S^* := S \setminus \{s_0,s_f\}$7 and $S^* := S \setminus \{s_0,s_f\}$8, each agent conditions on $S^* := S \setminus \{s_0,s_f\}$9, and the training objective becomes

$P_\theta(x) \propto R(x)$0

The paper states that CJFN preserves the tractability benefits of JFN but yields more expressive joint models, because $P_\theta(x) \propto R(x)$1 can encode coordination strategies that make $P_\theta(x) \propto R(x)$2 closer to $P_\theta(x) \propto R(x)$3 and relax strict product-reward limitations (Brunswic et al., 24 Sep 2025).

The relation to prior extensions is informative. EFlowNets extend GFlowNets to stochastic environments by enforcing expected detailed balance at environment states, while AFlowNets define per-player flows and policies in two-player zero-sum games and derive a trajectory-balance-style objective for self-play (Jiralerspong et al., 2023). MACFN, by contrast, addresses cooperative continuous control and trains decentralized individual-flow-based policies in a centralized global-flow-based matching fashion (Luo et al., 2024). This suggests a broader taxonomy in which MA-GFlowNets denote a family of multi-agent flow-based samplers rather than a single algorithmic template.

4. Continuous-control instantiation: MACFN

MACFN formulates cooperative multi-agent continuous control as a Dec-POMDP

$P_\theta(x) \propto R(x)$4

with continuous joint action space $P_\theta(x) \propto R(x)$5, deterministic transition $P_\theta(x) \propto R(x)$6, and a global reward shared by all agents. The paper argues that the individual-flow matching constraint used in vanilla GFlowNets is insufficient for multi-agent systems with continuous joint actions because of curse of dimensionality in joint actions, partial observability leading to non-stationarity when training independent per-agent flows without global information, and reward sparsity and multimodality making it hard for agents to deduce their contributions from global terminal rewards only (Luo et al., 2024).

The central structural assumption is that, for any state pair $P_\theta(x) \propto R(x)$7, there is a unique joint action $P_\theta(x) \propto R(x)$8 such that $P_\theta(x) \propto R(x)$9, and actions are translation actions, $\pi$0. Under this assumption, MACFN introduces an inverse transition network $\pi$1 to estimate parent observations from child observations and actions: $\pi$2 The global flow decomposition is then defined as

$\pi$3

where $\pi$4 is the individual edge flow of agent $\pi$5. The corresponding policy factorization result states that, if

$\pi$6

then under the decomposition

$\pi$7

Thus decentralized local policies reconstruct the joint policy when the global flow factorizes (Luo et al., 2024).

MACFN further derives decompositions of continuous inflows and outflows into per-agent integrals. For outflows,

$\pi$8

For inflows,

$\pi$9

On that basis, the training loss enforces inflow $S$00 outflow $S$01 reward along a trajectory: $S$02 Because the actions are continuous, integrals are approximated by Monte Carlo sampling, and the paper gives expectation equalities and tail bounds controlling the deviation of sampled inflows and outflows from the true integrals (Luo et al., 2024).

For numerical stability, MACFN employs a log-scale regularized loss

$S$03

with inflows and outflows formed by products of per-agent sums of exp-transformed log-flows. During centralized training, gradients from global inflow/outflow matching backpropagate to the individual flow networks $S$04 and inverse transition network $S$05. During decentralized execution, the decomposition machinery is removed and each agent acts using its own individual flow function over local observations (Luo et al., 2024).

The paper describes this as training decentralized individual-flow-based policies in a centralized global-flow-based matching fashion, and states that the method enables agents to deliver actions solely based on their assigned local flow in a decentralized way, forming a joint policy distribution proportional to the rewards (Luo et al., 2024). This is closely aligned with the CTDE interpretation of JFN in the discrete measurable MA-GFlowNet framework (Brunswic et al., 24 Sep 2025).

5. Guarantees, complexity, and implementation

The measurable MA-GFlowNet theory inherits the sampling theorem from single-agent measurable GFNs: under the joint FM constraint, terminal states are sampled proportionally to $S$06, with finite expected stopping time (Brunswic et al., 24 Sep 2025). The local–global consistency results further state that there exists a joint GFN with factorized flows

$S$07

and that, if the joint FM holds on the “live” subdomains and local virtual rewards $S$08, then $S$09. Conversely, if local FM holds on “live” subdomains under asynchronous unanimous STOP, then on purgatory $S$10, achieved as $S$11 in the purgatory self-loop construction (Brunswic et al., 24 Sep 2025).

The principal computational distinction among the algorithms is the trade-off between exact joint modeling and factorized tractability. CFN is accurate but scales exponentially in the joint action space. IFN avoids global combinatorial explosion by using per-agent action complexity $S$12, but its local inflow estimation depends on $S$13, itself dependent on other agents’ states and actions, and its local rewards are generally intractable; the paper links this to nonstationarity and mode collapse. JFN and CJFN keep action complexity linear in the number of agents during training because joint measures factorize, so the loss reduces to products of local FM quantities (Brunswic et al., 24 Sep 2025).

The implementation details given for the measurable framework are specific. Policies and flow networks use MLPs, for example two hidden layers of 256 units. Training uses PyTorch, Adam optimizer, learning rate $S$14, and 16 trajectories per step. The loss function $S$15 can be $S$16. Termination is derived from estimated $S$17 via $S$18, and in practice the maximum flow action is used at test time for evaluation (Brunswic et al., 24 Sep 2025).

MACFN provides a separate implementation profile for continuous control. Optimization uses Adam at learning rate $S$19; training runs for 1M timesteps on MPE scenarios and 2M timesteps on MAMuJoCo scenarios; evaluation reports average test return under greedy action selection by maximum flow and number of distinctive trajectories under sampling by flow distributions over 5 seeds; the framework is PyMARL; and trajectory collection uses 10,000 trajectories with distinctiveness counted by filtering near-duplicate trajectories via a distance threshold (Luo et al., 2024).

Expected flow networks and adversarial flow networks contribute a different set of guarantees. On trees, EDB constraints admit a unique global optimum and are satisfiable simultaneously at all states, unlike prior stochastic GFlowNet treatments that fold environment stochasticity into immutable actions. In AFlowNets, there exist unique flows and policies jointly satisfying each player’s EDB constraints, and product flows $S$20 define a valid GFlowNet flow for reward $S$21 in the two-player setting (Jiralerspong et al., 2023). This suggests that fixed-point and product-flow arguments play an analogous role across cooperative, stochastic, and adversarial variants, even though the formal constraints differ.

6. Empirical findings, positioning, and open issues

The empirical evaluation in the MA-GFlowNet theory paper centers on hyper-grid environments, StarCraft II tasks, and Sparse Simple-Spread. In the $S$22-dimensional hypercubic grids, agents start at $S$23, can increment coordinates or STOP, and the reward is

$S$24

with $S$25, producing multiple modes. The metrics are Mode Found and normalized $S$26 error between the learned distribution and the normalized reward distribution (Brunswic et al., 24 Sep 2025).

In those hyper-grid experiments, CFN excels when the joint action space is small; as complexity grows, CFN degrades while JFN and CJFN remain accurate and converge faster. IFN suffers from nonstationarity and mode collapse due to spurious local rewards. Compared to RL methods, specifically MAPPO and MASAC, and modified MCMC, MA-GFlowNets better match reward distributions, achieving lower $S$27 error and recovering diverse modes (Brunswic et al., 24 Sep 2025).

On StarCraft II tasks 3m and 2s3z, the paper reports win-rate comparisons and qualitative trajectories, stating that MA-GFlowNets produce diverse, high-reward trajectories and, while not dominating MAPPO or QMIX in average return, demonstrate the ability to sample from richer reward distributions and maintain competitive performance in large-scale decision-making. In Sparse Simple-Spread, JFN reaches comparable average returns to MAPPO and QMIX but finds significantly more distinct trajectories, approximately $S$28, which the paper presents as evidence of diversity and exploration advantages (Brunswic et al., 24 Sep 2025).

MACFN evaluates cooperative continuous control in MPE and MAMuJoCo against IDDPG, MADDPG, COVDN, COMIX, and FACMAC. In MPE, the environments are Robot-Navigation-Sparse, Food-Collection-Sparse, and Predator-Prey-Sparse; in MAMuJoCo, they are 2-Agent-Reacher-Sparse, 2-Agent-Swimmer-Sparse, and 3-Agent-Hopper-Sparse. The reported metrics are average test return and number of distinctive trajectories (Luo et al., 2024).

The detailed results include the following values. In Food-Collection-Sparse, MACFN reports $S$29 for $S$30, $S$31 for $S$32, and $S$33 for $S$34; corresponding MADDPG values are $S$35, $S$36, and $S$37, with IDDPG and others lower. In Predator-Prey-Sparse, MACFN reports $S$38 for $S$39, $S$40 for $S$41, and $S$42 for $S$43; COVDN reaches $S$44 at $S$45, MADDPG reports $S$46 at $S$47 and $S$48 at $S$49, with others lower (Luo et al., 2024). The paper states that MACFN explores substantially more valid-distinctive trajectories than the baselines and highlights improved cooperative exploration under sparsity (Luo et al., 2024).

The stochastic and adversarial branch reports distinct empirical results. EFlowNets outperform other GFlowNet formulations in stochastic tasks such as protein design, and in TFBind they consistently find more diverse, high-reward sequences, especially when the reward is peaky and environment stochasticity is high. AFlowNets learn to find above 80% of optimal moves in Connect-4 via self-play, outperform AlphaZero in tournaments, and select $S$50 optimal moves after approximately 3 hours on a single RTX8000 GPU (Jiralerspong et al., 2023). These findings are not cooperative MA-GFlowNets in the narrow sense of (Brunswic et al., 24 Sep 2025), but they show that multi-agent and stochastic flow-network formulations can compete with established RL and search-based baselines in regimes where diversity and amortized sampling are important.

The principal limitations are stated explicitly. JFN requires product-form flows and often product-form rewards; strong coupling in $S$51 or non-factorized $S$52 can hinder convergence. IFN suffers from the difficulty of defining and estimating local rewards, which yields spurious rewards, nonstationarity, and mode collapse. The current theoretical guarantees focus on cooperative agents, particular STOP and start conventions, and measurable flow matching; noncooperative settings and richer termination and initialization rules remain open (Brunswic et al., 24 Sep 2025). MACFN adds assumptions of deterministic transitions with unique joint actions and translation dynamics, and notes limitations in scalability, parent estimation accuracy under partial observability, and stability under highly complex dynamics (Luo et al., 2024). EFlowNets and AFlowNets identify additional practical constraints, including TB variance and memory on long horizons and the possibility that, in very deep trees, lack of search may underexploit tactical lines compared to MCTS unless supplemented by curriculum or hybrid training (Jiralerspong et al., 2023).

Taken together, these works define MA-GFlowNets as a family of flow-based multi-agent samplers whose central invariant is reward-proportional terminal generation under suitable flow-matching constraints. In the cooperative measurable setting, this invariant is realized through the local–global principle and the CFN, IFN, JFN, and CJFN algorithms (Brunswic et al., 24 Sep 2025). In continuous control, it appears as multiplicative continuous flow decomposition with CTDE (Luo et al., 2024). In stochastic and adversarial settings, it appears through expected detailed balance and joint fixed-point constructions (Jiralerspong et al., 2023). A plausible implication is that future extensions will continue to revolve around the same structural problem: how to preserve global flow consistency while distributing decision-making, credit assignment, and termination control across agents and environments that are only partially aligned with product-form assumptions.