Efficient expressive generative policies for online multi-agent reinforcement learning

Develop expressive generative policies for online multi-agent reinforcement learning that retain strong multi-agent coordination performance while enabling efficient one-step action generation.

Background

The paper identifies a tension in online multi-agent reinforcement learning between policy expressiveness and computational efficiency. Diffusion- and flow-based generative policies can represent complex, multimodal, and coordinated action distributions, but iterative generation is costly, particularly as the number of agents and the dimensionality of the joint state-action space increase.

One-step flow policies reduce this computational burden, but aggressive step reduction can introduce transport errors and degrade action quality. The unresolved challenge is therefore to develop a generative policy mechanism that preserves expressive coordination behavior and achieves effective online learning with only one action-generation step. The proposed OMAF framework is presented as an approach to this challenge, but the challenge itself is explicitly characterized as open.