---
title: 'Regret Matching⁺: Equilibrium and Convergence'
url: https://www.emergentmind.com/topics/regret-matching
type: topic
---

# Regret Matching⁺: Equilibrium and Convergence

Regret Matching⁺ (RM⁺) is a parameter-free, online learning algorithm for normal-form and extensive-form games that maintains and updates nonnegative cumulative regrets, using only their positive parts to determine the action selection distribution. RM⁺ and its predictive extensions form the core of state-of-the-art game-solving methods, particularly within counterfactual regret minimization (CFR) frameworks. Modern variants, such as predictive RM⁺ (PRM⁺) and scale-invariant extra-gradient versions (IREG-PRM⁺), address long-standing gaps between theoretical guarantees and practical convergence behavior for equilibrium computation in large-scale zero-sum and multi-role games.

## 1. Algorithmic Foundations of Regret Matching⁺

Regret Matching⁺ replaces the traditional regret-matching probability computation with a positive-truncation operator. At each round, for a player with action set $A$ and regret vector $r^{(t)}\in\mathbb{R}^{|A|}$, the RM⁺ next-step distribution is computed as:
\[
x_{a}^{(t+1)}=
\begin{cases}
\frac{[r_{a}^{(t)}]_{+}}{\sum_{b \in A}[r_{b}^{(t)}]_{+}} & \text{if } \sum_b [r_{b}^{(t)}]_{+} > 0 \\
1/|A| & \text{otherwise.}
\end{cases}
\]
where $[r_{a}]_{+}=\max\{r_{a},0\}$.

Cumulative regrets are updated via:
\[
r^{(t+1)} = r^{(t)} + u^{(t)} - \langle u^{(t)}, x^{(t+1)} \rangle \mathbf{1}
\]
where $u^{(t)}$ is the action utility vector.

Predictive RM⁺ (PRM⁺) introduces a forecast $\hat r^{(t)}$ and uses this in the decision distribution, increasing convergence rates when system feedback is predictable [2007.14358]. IREG-PRM⁺ further enforces scale-invariance by maintaining a nondecreasing $\ell_2$-norm of the regret vector, yielding optimal convergence properties, as formalized via RVU-type bounds [2510.04407].

## 2. Connections to Online Mirror Descent and Optimistic Learning

RM⁺ is equivalent to an instance of online mirror descent (OMD) with a Euclidean potential and simplex projections, as shown in [2110.04961, 2007.14358]. The update,
\[
z^{t} = [z^{t-1} + \langle \ell^{t}, x^{t} \rangle \mathbf{1} - \ell^{t}]_{+}, \;\quad x^{t} = \frac{[z^{t-1}]_{+}}{ \|[z^{t-1}]_{+}\|_1 }
\]
matches OMD on the nonnegative orthant, with the step-size determined adaptively by the norm of the positive regret vector, thus implicitly tuning the algorithm’s reactivity to observed payoffs. This link underpins the $O(\sqrt{T})$ regret guarantees and motivates smooth and predictive modifications, as in PRM⁺ and IREG-PRM⁺.

IREG-PRM⁺ formalizes this adaptive learning rate by ensuring $\|r^{(t)}\|_2$ is nondecreasing, which acts analogously to an adaptive step-size in optimistic gradient methods. This invariance is central to matching the $O(1/T)$ average iterate rate characteristic of optimal first-order methods (mirror-prox, extragradient) [2510.04407].

## 3. Stability, Smoothing, and Convergence Pathologies

Vanilla RM⁺ and PRM⁺ can suffer from instability, including oscillating last iterates and slow convergence in pathological zero-sum games with unique Nash equilibria. RM⁺’s regret operator lacks Lipschitzness and monotonicity, precluding application of standard variational inequality theories for last-iterate convergence [2311.00676, 2305.14709]. Numerical and theoretical evidence shows that:

- RM⁺ and PRM⁺ can stagnate at $O(1/\sqrt{T})$ last-iterate duality gap in games with small action sets (e.g., $3\times3$ matrix games).
- Alternating PRM⁺ sometimes empirically exhibits last-iterate convergence, but this behavior lacks theoretical justification.

To address these deficiencies, smoothing schemes such as extragradient RM⁺ (ExRM⁺) and smooth predictive RM⁺ (SPRM⁺) introduce predictive or extragradient steps with explicit step sizes and domain projections, recovering asymptotic last-iterate convergence. Restarting approaches and projection domain modifications (“chopping off” the orthant near the origin) can guarantee stability and drive both individual and social regret to $O(T^{1/4})$ or $O(1)$, respectively [2311.00676, 2305.14709].

## 4. Extensions: Predictive, Extra-Gradient, and Scale-Invariant Methods

Table: Summary of Major RM⁺ Variants and Their Regret/Convergence Properties

| Algorithm      | Key Innovation                        | Regret / Nash Gap         |
|----------------|--------------------------------------|---------------------------|
| RM⁺            | Positive truncation                   | $O(1/\sqrt{T})$           |
| PRM⁺           | Optimistic (predicted) update         | $O\left(\sqrt{\sum \|\ell^t-m^t\|_2^2}\right)$ |
| ExRM⁺/SPRM⁺    | Extragradient/smooth projection       | Linear last-iterate with restart, $O(1/\sqrt{T})$ best-iterate |
| IREG-PRM⁺      | Norm nondecrease, scale-invariance    | $O(1/T)$ average, $O(1/\sqrt{T})$ last-iterate |

IREG-PRM⁺ [2510.04407] guarantees:
- $\|\tilde{r}^{(t)}\|_2$ nondecreasing at each iteration, acting as a step-size controller.
- RVU-type regret bounds decouple learning rate from hyperparameter tuning, facilitating $O(1/T)$ average Nash gap without sacrificing parameter-freeness or computational simplicity.
- Empirical performance matching or exceeding predictive CFR⁺ (PCFR) on games such as Kuhn poker, Leduc hold’em, Battleship, and Goofspiel.

Smoothing-based approaches (ExRM⁺, SPRM⁺) require only modest algorithmic modification—extragradient steps, prediction, or restarting—yet yield provable last-iterate convergence and practical stability, as formalized in [2311.00676].

## 5. Applications: Large-Scale Self-Play and Role-Balanced Training

RM⁺ is prominent in large-scale self-play training for generalist game AI, especially in multi-role environments. In such contexts, a generalized model trained via naive uniform self-play often displays uneven proficiency across roles. Adapting RM⁺ to operate on the N×N matrix of role-pairs (for $N$ roles), as in “Balancing the AI Strength of Roles in Self-Play Training with Regret Matching⁺” [2401.12557], enables automatic allocation of training focus:

- Positive "regret" matrices for each role-pair drive adaptive changes in the data-sampling distribution.
- An exponentially-smoothed win-rate matrix with mixing floor ensures persistent coverage of all pairs and prevents mode collapse.
- Empirical results (e.g., in a 13-character fighting game) show a ≈43% reduction in win-rate variance across roles when using RM⁺, yielding a more uniformly strong policy compared to baseline self-play.
- Complexity per update is $O(N^2)$; with grouping or subsampling practical for large $N$.

Modifications, such as smoothing factors (γ) and per-pairation mixing (η), enhance robustness and balance, retaining sublinear regret guarantees for the sampler.

## 6. Theoretical Properties and Limitations

RM⁺, PRM⁺, and IREG-PRM⁺ have been extensively analyzed:
- Standard RM⁺ (and CFR⁺) provably achieve worst-case $O(\sqrt{T})$ regret and $O(1/\sqrt{T})$ Nash-gap [2007.14358, 2110.04961].
- Predictive (optimistic) RM⁺ attains near-constant regret and accelerated equilibrium convergence when loss sequences are predictable [2007.14358].
- IREG-PRM⁺ achieves the theoretically optimal $O(1/T)$ average and $O(1/\sqrt{T})$ best-iterate duality gap, closing the gap to classic extragradient algorithms [2510.04407].
- Smoothing and restarting are necessary to achieve linear last-iterate convergence in general; without such mechanisms, raw RM⁺ dynamics may stagnate or oscillate [2311.00676, 2305.14709].
- Stability fixes (restart, orthant-chopping) guarantee improved convergence for both individual and social regret, as well as robustness in extensive-form games [2305.14709].

## 7. Empirical Performance and Practical Guidance

Empirical benchmarks demonstrate that:
- Predictive RM⁺ (PCFR) outperforms all prior non-predictive CFR variants, often by orders of magnitude in Nash gap on non-poker zero-sum benchmarks [2007.14358].
- IREG-PRM⁺ is a drop-in replacement with no new hyperparameters and matches or improves upon PRM⁺, DCFR, and adaptive OGD variants across matrix and extensive-form games [2510.04407].
- Smoothing and restarts do not harm performance and may accelerate convergence in practice [2311.00676].
- For symmetric, multi-role environments, regret-matching-based role-balancing achieves uniform competence while avoiding catastrophic neglect of under-trained roles [2401.12557].
- Parameter-freeness and scale-invariance facilitate deployment in practical, large-scale scenarios with minimal tuning.

A plausible implication is that the underlying principle—adaptive, scale-invariant control of the dual variable norm—may inform future algorithmic advances for fast, robust equilibrium computation in high-dimensional, structured games.

Source: https://www.emergentmind.com/topics/regret-matching