Regret Minimization with Adaptive Opponents in Repeated Games

Abstract: In this paper, we study regret minimization in repeated games with \emph{adaptive} opponents who can respond based on histories of play. The standard metric of \emph{external regret} in online learning is known to fail to capture such adaptivity. To account for players' counterfactual reasoning, we introduce {\tt Repeated Policy Regret (RP-Regret)}, a game-theoretic metric that measures the difference between the \emph{realized} and the \emph{best-in-hindsight} accumulated utility when all players can \emph{respond} to the history of play. Compared to existing regret notions in this setting, ours is native to repeated game playing, enabling stronger comparators and opponents with fewer constraints, while maintaining the possibility of finding better equilibria when all players minimize it. We first identify necessary conditions for obtaining {\tt RP-Regret} sublinear in time, on the variation of the player's comparator strategies in the regret definition and on the memories of both the comparator and opponents' strategies. We then study additional conditions and provable algorithms to minimize {\tt RP-Regret}, which is by definition \emph{non-convex} in the strategy space. To address this challenge, we propose three algorithms: (i) one based on an optimization oracle, as assumed in some prior work in online non-convex learning; (ii) one that minimizes a convex and \emph{linearized} surrogate of {\tt RP-Regret} at each iteration; (iii) one that directly minimizes {\tt RP-Regret} when opponents change strategies slowly. Furthermore, when all players can run algorithms to minimize the {\tt RP-Regret} (or its linearized variant), certain subgame perfect equilibria of the repeated game can be learned. We also provide experiments showing that minimizing our regret notions can lead to more cooperative solutions with higher utility in games such as Stag-Hunt.

• The paper introduces a novel regret notion, RP-Regret, to assess performance in repeated games with adaptive, history-dependent strategies.
• It develops practical algorithms using projected gradient descent and convex relaxations to achieve sublinear regret bounds under adaptive opponent play.
• Empirical experiments, including in games like Stag Hunt, demonstrate that minimizing LRP-Regret leads to higher cooperation and efficient equilibrium convergence.

Regret Minimization Against Adaptive Opponents in Repeated Games

Introduction and Motivation

The paper "Regret Minimization with Adaptive Opponents in Repeated Games" (2606.06486) advances the theory of regret minimization in repeated games where all players are adaptive—they select strategies based on the observed history of play. Traditional external regret, a core online learning notion, fails to capture the counterfactual impact of a player's deviations when opponents are fully responsive to every action sequence. This limitation manifests acutely in classic settings such as the Iterated Prisoner's Dilemma, where players minimizing external regret are trapped in low-utility, defect-defect equilibria, even though more cooperative strategies (e.g., Tit-for-Tat) can achieve higher collective utilities but do not minimize external regret under adaptive play.

The paper introduces Repeated Policy Regret (RP-Regret), a new game-theoretic regret notion suitable for multi-agent learning with counterfactually adaptive opponents, and develops tractable algorithms and theoretical results for its minimization. The RP-Regret framework permits both the comparator and opponents to select time-varying, history-dependent ("policy") strategies, significantly generalizing earlier regret notions that require oblivious, memory-bounded, or non-adaptive environments.

Regret Notions: RP-Regret and LRP-Regret

Definition and Comparison

RP-Regret quantifies the difference between the realized accumulated utility and that of the best-in-hindsight policy the player could have adopted if allowed to react to the entire history, assuming opponents respond (possibly adaptively) to all past actions. This regret notion strictly generalizes external regret, policy regret, and response regret by allowing adaptive, non-stationary comparators and opponents with arbitrarily deep history dependencies, subject only to mild regularity conditions for tractability.

The authors derive necessary conditions for sublinear RP-Regret:

• Comparator variation must be sublinear: The policy comparator cannot switch arbitrarily fast between strategies.
• Imperfect recall: Both the regret minimizer and comparator (and, partially, the opponents) cannot perfectly remember the entire sequence of play (i.e., policies must be "forgetful" in a controlled sense).

To achieve tractability, the paper introduces Local RP-Regret (LRP-Regret), a linearized surrogate motivated by the one-step deviation principle. Minimizing LRP-Regret ensures no unilateral, localized policy switch at any round yields substantial cumulative utility gains.

Figural Illustration

Figure 1: Utility matrices for Prisoner's Dilemma and Stag Hunt, plus empirical demonstration that LRP-Regret minimization in Stag Hunt supports convergence to higher-utility equilibria.

The figure demonstrates that classical regret minimization leads to suboptimal, defection-prone equilibria, while minimizing LRP-Regret frequently achieves higher-payoff, cooperative equilibria.

Algorithmic Results

Direct RP-Regret Minimization

The paper proves that, under exponential decay memory conditions (EDM)—policy sensitivity to the distant past decays exponentially, parameterized by a decay factor $\gamma$—and a sublinear variation budget, RP-Regret admits minimization with a non-convex optimization oracle. However, such oracles are typically computationally infeasible except in special cases (e.g., small games or when additional exploitable structure exists).

Surrogates and Convex Relaxations

For practical computation, the authors develop:

• A projected gradient descent method for LRP-Regret, leveraging its localized convexity and the linearity of the local objective. Theoretical results (Theorem 1) establish sublinear LRP-Regret under appropriate parameter tuning (learning rate $\eta$ and memory length $m$).
• A convex RL-type transformation: With bounded-memory (Markov) strategies, the repeated game can be reformulated as an average-reward Markov game. In this space, the regret minimizer operates over occupancy measures with carefully constructed linear constraints ensuring marginal and strategic consistency.
• An online convex optimization with time-varying constraints framework, leading to efficient learning dynamics under realistic game-theoretic memory assumptions.

Theoretical Guarantees

The algorithms, under EDM and sublinear variation, achieve:

• Sublinear (in $T$) RP-Regret and LRP-Regret for repeated games with adaptive opponents and dynamic comparators.
• Exponential error decay in memory length $m$ for the bounded-memory approximations, permitting polylogarithmic memory in $T$ for constant error.
• Approximation guarantees for constraint satisfaction (feasibility of occupancy measures), with cumulative constraint violation sublinear in $T$.

Equilibria via Regret Minimization

The minimization of RP-Regret and LRP-Regret corresponds to learning new robust equilibrium concepts in repeated games, closely related to Subgame Perfect Coarse Correlated Equilibria (SPCCE) and Approximate Subgame Perfect Nash Equilibria (SPNE), but parameterized by the allowable variation budget.

• Under mild regularity, sublinear RP-Regret ensures convergence to approximate SPNE with bounded deviation, as formalized in Theorem 5 and Theorem 6.
• The class of equilibria realized via RP-Regret minimization is strictly richer than that obtained by classical external regret minimization, enabling both stability and robust cooperation even with adaptive, memory-using adversaries.

Numerical Experiments

The empirical analysis (see Figure 1, right) in Stag Hunt demonstrates that players minimizing LRP-Regret, even with very limited memory, frequently learn to cooperate and converge to the high-utility "Stag-Stag" equilibrium. This is in sharp contrast to classical no-regret dynamics, which typically select less efficient equilibria due to their myopic neglect of historical context and potential future repercussions.

Implications and Future Directions

Theoretical and Practical Considerations

• Robust multi-agent learning: The RP-Regret framework offers robust convergence guarantees in non-Markovian, strongly-coupled, and adversarially adaptive environments—scenarios where classical online learning theory is inapplicable.
• Equilibrium selection and efficiency: The connection to strong equilibrium notions may support equilibrium selection in repeated games with multiple equilibria, especially when a principal desires high-efficiency (cooperative) outcomes.
• Computational tractability: The convex reformulations and occupancy measure algorithms open avenues for scalable, multi-agent reinforcement learning in settings with complex, history-dependent opponent models, beyond oblivious or myopic adversaries.

Future Directions

• Weakening memory and adaptivity assumptions for broader applicability.
• Extending algorithms and theory to partially observed, stochastic, or continuous-action repeated games.
• Developing efficient oracle-free methods for general non-convex regret minimization in high-dimensional policy spaces.
• Further characterizing the equilibrium classes realized by various regret minimization dynamics under different information and adaptivity regimes.

Conclusion

"Regret Minimization with Adaptive Opponents in Repeated Games" (2606.06486) provides a rigorous framework for robust learning and equilibrium computation in dynamic, adaptive, and potentially adversarial multi-agent environments. By advancing both the theoretical limits (via necessary conditions, hardness results) and algorithmic tools (surrogate regret notions, convex reformulations), it establishes a new standard for counterfactual, history-aware regret minimization. The implications for multi-agent learning, game-theoretic equilibria, and algorithmic game theory are substantial, with strong prospects for practical algorithmic development and avenues for deeper theoretical investigation.