Proximal Learning With Opponent-Learning Awareness

• The paper introduces POLA, a multi-agent reinforcement learning algorithm that reformulates LOLA updates as proximal steps to achieve parameterization invariance.
• POLA uses divergence-based penalties in policy space to eliminate inconsistencies from neural network parameterizations, ensuring reliable reciprocal strategies.
• Empirical evaluations show that POLA outperforms LOLA across diverse environments by consistently inducing cooperation and mitigating defection behaviors.

Proximal Learning With Opponent-Learning Awareness (POLA) is a multi-agent reinforcement learning algorithm designed to achieve reciprocity-based cooperation in partially competitive environments while ensuring invariance to policy parameterization. POLA addresses known instabilities and specification issues in Learning With Opponent-Learning Awareness (LOLA) when operating over complex or neural network–parameterized policy spaces by reformulating opponent-aware learning as a proximal point problem in policy space. This conception guarantees that behaviourally equivalent policies will always induce the same update, eliminating a significant class of failure modes observed in prior approaches (Zhao et al., 2022).

1. Background and Motivation

Learning With Opponent-Learning Awareness (LOLA) augments the standard agent policy-gradient update by explicitly differentiating through an opponent’s learning step. Specifically, for agent 2’s update $$\Delta\theta^2(\theta^1) = \eta \nabla_{\theta^2} L^2(\pi_{\theta^1}, \pi_{\theta^2})$$, LOLA computes agent 1's new parameters as $$\theta^{1\prime} = \theta^1 - \alpha \nabla_{\theta^1} L^1(\pi_{\theta^1}, \pi_{\theta^2-\Delta\theta^2(\theta^1)})$$, thus shaping agent 2's objective as perceived by agent 1.

While LOLA reliably induces reciprocity-based strategies like Tit-for-Tat in tabular (small, discrete) policy spaces, its efficacy is severely degraded when using neural policies or when an agent must learn an opponent model. This arises because LOLA’s parameter-space updates depend on the specific parameterization: for two different vectors $\theta$ yielding exactly the same policy behavior $\pi_{\theta}$, the Euclidean gradient $\nabla_{\theta}$ may differ arbitrarily. Consequently, behaviourally equivalent policies can lead LOLA to divergent learning dynamics and pathological outcomes. This sensitivity fundamentally limits LOLA's applicability in modern deep reinforcement learning contexts.

2. Proximal Reformulation of Opponent-Shaping

The POLA methodology emerges by reinterpreting LOLA updates as approximate proximal steps. The classical proximal operator for a function $f$ is $$\operatorname{prox}_{\eta f}(v) = \arg\min_{x} [f(x) + \frac{1}{2\eta}\|x-v\|^2]$$, with the gradient step recovered by linearizing $f$ near $v$. Under this view, LOLA’s update is a specific gradient-based approximation of such a proximal update on an agent’s learning-aware loss.

POLA generalizes this construction: all objectives and penalties are defined in policy space, replacing the classical Euclidean penalty with a divergence $D$ over policies. The ideal two-player POLA update is specified as:

• Inner update for agent 2:

$$\theta^{1\prime} = \theta^1 - \alpha \nabla_{\theta^1} L^1(\pi_{\theta^1}, \pi_{\theta^2-\Delta\theta^2(\theta^1)})$$0

• Outer update for agent 1:

$$\theta^{1\prime} = \theta^1 - \alpha \nabla_{\theta^1} L^1(\pi_{\theta^1}, \pi_{\theta^2-\Delta\theta^2(\theta^1)})$$1

With all losses and regularizers defined over policies, not parameters, POLA eliminates inconsistencies caused by parameterization (Zhao et al., 2022).

3. Parameterization Invariance

An update $$\theta^{1\prime} = \theta^1 - \alpha \nabla_{\theta^1} L^1(\pi_{\theta^1}, \pi_{\theta^2-\Delta\theta^2(\theta^1)})$$2 is parameterization-invariant if $$\theta^{1\prime} = \theta^1 - \alpha \nabla_{\theta^1} L^1(\pi_{\theta^1}, \pi_{\theta^2-\Delta\theta^2(\theta^1)})$$3 and $$\theta^{1\prime} = \theta^1 - \alpha \nabla_{\theta^1} L^1(\pi_{\theta^1}, \pi_{\theta^2-\Delta\theta^2(\theta^1)})$$4 imply $$\theta^{1\prime} = \theta^1 - \alpha \nabla_{\theta^1} L^1(\pi_{\theta^1}, \pi_{\theta^2-\Delta\theta^2(\theta^1)})$$5.

Ideal POLA achieves parameterization invariance by construction: each loss and penalty in the subproblems depends only on the realized policies. If the minimizer is unique in policy space, the result does not depend on how policies are parameterized or implemented (as proved in Appendix A.2 of (Zhao et al., 2022)). In contrast, LOLA’s Euclidean penalties break this invariance: empirically, two distinct neural network parameterizations yielding the same input-output mapping lead to widely different learning updates under LOLA, while POLA produces consistent updates in policy space. This property is critical for scalable multi-agent reinforcement learning with deep function approximators.

4. Practical Algorithms and Approximations

Solving the ideal POLA subproblems exactly is tractable only for low-dimensional or tabular policies. The following approximations are proposed for practical deployment:

4.1 Outer POLA (Tabular or Small Networks):

• Agent 2’s policy is updated with a standard gradient step: $$\theta^{1\prime} = \theta^1 - \alpha \nabla_{\theta^1} L^1(\pi_{\theta^1}, \pi_{\theta^2-\Delta\theta^2(\theta^1)})$$6.
• Agent 1’s parameters $$\theta^{1\prime} = \theta^1 - \alpha \nabla_{\theta^1} L^1(\pi_{\theta^1}, \pi_{\theta^2-\Delta\theta^2(\theta^1)})$$7 are then updated by repeated gradient steps on $$\theta^{1\prime} = \theta^1 - \alpha \nabla_{\theta^1} L^1(\pi_{\theta^1}, \pi_{\theta^2-\Delta\theta^2(\theta^1)})$$8 until convergence.

Pseudocode (Algorithm 1 from (Zhao et al., 2022)): $\pi_{\theta}$0

4.2 POLA-DiCE (Sample-Based, Deep Networks, Opponent Modeling):

• Uses DiCE-style objectives $$\theta^{1\prime} = \theta^1 - \alpha \nabla_{\theta^1} L^1(\pi_{\theta^1}, \pi_{\theta^2-\Delta\theta^2(\theta^1)})$$9 for unbiased higher-order gradients.
• Inner loop: Agent 2 is updated for $\theta$0 steps, penalized by $\theta$1.
• Agent 1 is then updated on the outer objective, with $\theta$2 controlling proximity.
• If the opponent’s policy is unknown, a behaviour-cloned model $\theta$3 is used in the inner loop.

Algorithm 2 in the reference details this structure. These approximations maintain parameterization invariance to the extent that subproblems are solved to convergence.

5. Empirical Evaluation

POLA was systematically evaluated in several domains:

5.1 One-Step-Memory IPD:

• Tabular parameters, neural nets, and pre-conditioned tabular representations were tested.
• Measure: % runs converging to Tit-for-Tat (TFT).
• Only Outer POLA found TFT policies reliably across all parameterizations and for all $\theta$4. LOLA performed well only for tabular $\theta$5, failing under neural or pre-conditioned settings. Naïve gradient-based agents never found TFT.

5.2 Full-History IPD with Rollouts:

• Both LOLA-DiCE and POLA-DiCE were used with GRU policy parameterizations.
• Measure: average episodic return, probability of cooperation against unconditional defection.
• POLA-DiCE almost always discovered reciprocity-based cooperation, defecting when facing an uncooperative agent. LOLA-DiCE was unstable and often fell into defection modes.

5.3 Coin Game (Spatial Social Dilemma):

• Agents navigated a $\theta$6 grid, collecting coins for themselves and/or penalizing others.
• Metric: Proportion of own-colour coins collected, self-play returns, and score against defectors.
• POLA-DiCE agents achieved high cooperation metrics ($\theta$7 of own-colour coins), near-optimal self-play returns, and defensible strategies. LOLA-DiCE performed considerably worse, defaulting to defection.

Summary of experimental findings:

Environment	LOLA Outcome	POLA Outcome
Tabular IPD	Cooperation in some settings	Robust cooperation everywhere
NN IPD / Precond.	Fails	Robust cooperation
Full-history IPD	Unstable, defection common	Consistent cooperation
Coin Game	Defection dominant	Near-optimal cooperation

6. Limitations and Directions for Future Research

POLA’s primary limitations are increased sample complexity due to inner/outer optimization loops and the introduction of additional hyperparameters ($\theta$8). Parameterization invariance is only approximate when using finite-step inner/outer loops instead of global optima. Practical deployment in very high-dimensional settings or with large opponent populations remains computationally demanding.

Open research directions include:

• Improving sample efficiency of proximal updates (e.g., trust-region or clipped-proximal variants).
• Extending parameterization-invariant techniques to other opponent-shaping algorithms (such as COLA, SOS).
• Generalizing the approach to $\theta$9-agent settings.
• Developing adaptive penalty schedules and establishing connections to mirror descent and extra-gradient methods.

Reference implementation and all experimental details, including hyperparameters and figure replication, are available at https://github.com/Silent-Zebra/POLA (Zhao et al., 2022).