Comparative performance of fully asynchronous versus backward-synchronized policy gradient methods

Determine whether fully asynchronous policy gradient algorithms with decoupled training and data collection outperform backward-pass-synchronized methods under a fixed compute budget, accounting for policy lag effects.

Background

The paper discusses that fully asynchronous policy gradient methods can train faster but often omit analysis of policy lag, which introduces off-policy effects when training and data collection are decoupled. This raises uncertainty about the true performance benefits compared to synchronized approaches.

The authors’ AC-PPO keeps a synchronized backward pass to avoid policy lag, and they explicitly note that it remains unclear whether fully asynchronous methods are better under equal compute constraints.