Dice Question Streamline Icon: https://streamlinehq.com

Fundamental open problem in opponent shaping: arrogant behavior persists under consistency

Investigate the fundamental issue in opponent shaping algorithms for differentiable games whereby agents exhibit "arrogant" behavior even when their update functions are consistent under mutual opponent shaping; characterize this phenomenon and determine what, if any, principled modifications resolve it.

Information Square Streamline Icon: https://streamlinehq.com

Background

Opponent shaping methods such as Learning with Opponent-Learning Awareness (LOLA) aim to influence anticipated opponent learning steps, but the original LOLA is inconsistent when both agents shape each other. Higher-order LOLA (HOLA) and the limiting iLOLA address consistency under convergence, and Consistent LOLA (COLA) directly learns update functions that satisfy mutual consistency equations.

Despite achieving consistency, the paper proves that consistent update functions can still fail to preserve stable fixed points (SFPs) and observes that opponent shaping continues to display so-called "arrogant" behavior. This persistence suggests a deeper, unresolved challenge in the design of opponent-shaping updates.

References

It was believed that inconsistency leads to arrogant behaviour and lack of preservation of SFPs. We showed that even with consistency, opponent shaping behaves arrogantly, pointing towards a fundamental open problem for the method.

— COLA: Consistent Learning with Opponent-Learning Awareness (2203.04098 - Willi et al., 2022) in Conclusion and Future Work (Section 7)