---
title: 'Regret Matching⁺ (RM⁺): Scale-Invariant Online Learning'
url: https://www.emergentmind.com/topics/regret-matching-rm
type: topic
---

# Regret Matching⁺ (RM⁺): Scale-Invariant Online Learning

Regret Matching⁺ (RM⁺) is a scale-invariant, parameter-free online learning algorithm that underlies the most effective approaches to large-scale zero-sum game solving and is a cornerstone of state-of-the-art counterfactual regret minimization (CFR) methods. RM⁺ extends classic regret matching by truncating negative regrets to zero on each step, enabling aggressive, yet provably no-regret, adaptation that is highly compatible with mirror descent and Blackwell approachability frameworks. Modern extensions—most notably Predictive RM⁺ (PRM⁺) and the scale-invariant IREG-PRM⁺—connect RM⁺ to recent advances in first-order zeroth-order optimization, achieving optimal average-iterate convergence rates and closing the gap between practical game-solving performance and convex-optimization theory [2510.04407, 2007.14358, 2305.14709, 2311.00676, 2510.17067].

## 1. Core Algorithmic Structure

RM⁺ operates over a finite action simplex $\Delta_n$ for $n$ actions. At iteration $t$, the algorithm updates as follows [2510.04407, 2007.14358, 2404.13891, 2305.14709]:

- **Step 1 (Counterfactual Regret):** Given the current mixed strategy $x^{(t)}$ and observed utility or loss vector $u^{(t)} \in \mathbb{R}^n$, compute the instantaneous counterfactual regret increment:
  \[
    \ell^{(t)} := u^{(t)} - \langle x^{(t)}, u^{(t)} \rangle \mathbf{1}_n
  \]

- **Step 2 (Cumulative Positive Regret, Truncation):** Cumulative regrets are updated using coordinatewise truncation:
  \[
    R^{(t)} = [ R^{(t-1)} + \ell^{(t)} ]_+, \quad \text{where } R^{(0)} = 0
  \]
  $[\cdot]_+$ denotes the coordinate-wise maximum with zero.

- **Step 3 (Strategy Update):** The next strategy is proportional to positive cumulative regrets:
  \[
    x^{(t+1)}_i = 
      \begin{cases}
        R^{(t)}_i / \| R^{(t)} \|_1 & \text{if } \| R^{(t)} \|_1 > 0\\
        1/n & \text{otherwise}
      \end{cases}
  \]

This ensures that only actions with positive cumulative regret receive probability mass. The algorithm is parameter-free and maintains scale invariance since regret vectors and probabilities are only defined up to normalization.

## 2. Theoretical Properties and Convergence

RM⁺ achieves sublinear external regret, and its integration with CFR yields scalable approximate Nash equilibria in zero-sum extensive-form games. Key properties [2510.04407, 2007.14358, 2404.13891, 1809.04040, 2305.14709, 2311.00676, 2510.17067]:

- **External Regret Bound:** For any fixed comparator $x$, RM⁺ guarantees
  \[
    \sum_{t=1}^{T} \langle u^{(t)}, x^{(t)} - x \rangle \leq O(\sqrt{T})
  \]
- **Ergodic/average-iterate convergence:** When used within CFR, the exploitability of the average strategy converges as $O(1/\sqrt{T})$ for standard RM⁺ [2007.14358, 2510.04407].

- **Optimality Gaps in Optimization:** In constrained optimization over the simplex, alternating RM⁺ achieves $\epsilon$-KKT points in $O(1/\epsilon^4)$ iterations, with possible improvement to $O(1/\epsilon^2)$ when the regret is uniformly bounded [2510.17067].

- **Exponential lower bound for vanilla RM:** Without truncation, ordinary RM can require an exponential number of steps to reach approximate equilibrium in potential games, while RM⁺ avoids stalling via regret clipping (negative regrets are immediately reset to zero) [2510.17067].

## 3. Connections with Mirror Descent and Blackwell Approachability

RM⁺ is formally equivalent to Online Mirror Descent (OMD) on the simplicial domain with a quadratic regularizer, projected onto the non-negative orthant [2007.14358, 2404.13891]. The update
\[
\tilde{x}^{(t+1)} = [\tilde{x}^{(t)} + r^{(t)}]_+
\]
with appropriate normalization, recovers the RM⁺ mechanism. The truncated regret update coincides with the Bregman projection under squared $\ell_2$ norm. This enables a direct BPV (Blackwell, Polyak, and Vygotsky) connection: in Blackwell approachability games, selecting forced halfspaces via the current non-negative regret vector and projecting yields the same iterates as OMD with quadratic regularization [2007.14358].

Furthermore, in the predictive setting, RM⁺ underlies optimistic OMD in the non-negative orthant, and PRM⁺ is the instance where the gradient predictions are one-step lagged [2404.13891].

## 4. Extensions: Predictive RM⁺, IREG-PRM⁺, and Stability

- **Predictive RM⁺ (PRM⁺):** This variant incorporates a one-step-ahead prediction $p^{(t)}$ of the upcoming regret, forming a “predicted” regret vector:
  \[
    S^{(t)} = [ R^{(t-1)} + p^{(t)} ]_+, \quad
    \pi^{(t)} = S^{(t)}/\|S^{(t)}\|_1
  \]
  The observed utility is used to update the true regret vector [2510.04407, 2404.13891]. PRM⁺ empirically accelerates convergence, especially under smoothly evolving loss sequences, but may exhibit only $O(T^{-1/2})$ worst-case rates.

- **Scale-Invariant Predictive RM⁺ (IREG-PRM⁺):** This recent variant guarantees that the $\ell_2$-norm of the regret vector is non-decreasing by shifting the prediction-plus-old regret vector by a scalar $\gamma$ at each step so that its norm is invariant [2510.04407]. This sharpens the learning dynamics and enables optimal $O(1/T)$ average-iterate and $O(1/\sqrt{T})$ last-iterate convergence in zero-sum games, matching the best-known rates for mirror-prox and optimistic mirror descent, without introducing hyperparameters.

- **Stabilization and Convergence Guarantees:** While RM⁺ and PRM⁺ achieve worst-case $O(\sqrt{T})$ regret, their updates can be unstable (large jumps in iterates), potentially destabilizing convergence when deployed in multi-agent game dynamics [2305.14709, 2311.00676]. Techniques such as restarts and “chopping” the orthant (restricting the regret vector’s $\ell_1$-norm to exceed a threshold) mitigate this, yielding $O(T^{1/4})$ individual regret and $O(1)$ social regret in normal-form games [2305.14709].

- **Smoothing for Last-Iterate Convergence:** Smoothing via extragradient techniques (ExRM⁺, Smooth Predictive RM⁺) achieves provably asymptotic and $O(1/\sqrt{T})$ best-iterate convergence in last-iterate equilibrium gap for zero-sum games, with restarts yielding linear convergence under certain conditions [2311.00676].

## 5. Role in Counterfactual Regret Minimization (CFR) and Benchmark Applications

Within the CFR framework for imperfect-information games, RM⁺ is used as the local regret minimizer at every information set. Each decision point independently updates a local regret vector, feeding counterfactual loss signals reflecting the global state and downstream values [2007.14358, 2404.13891, 1809.04040]. PRM⁺ (and its weighted or discounted versions) underlie Predictive CFR⁺ (PCFR⁺), Discounted CFR (DCFR), and, more recently, parameter-free, scale-invariant IREG-PRM⁺ [2510.04407], all empirically demonstrating multi-order-of-magnitude speedups in exploitability convergence on diverse benchmarks (e.g., Leduc Poker, Goofspiel, Battleship, Kuhn Poker, Liar’s Dice).

In generalized self-play training for games with multiple roles, RM⁺ has been used to reweight the sampling distribution over role pairs, leading to more uniformly balanced AI strength across roles and reduced win-rate variance [2401.12557].

## 6. Comparative Analysis and Limitations

The introduction of regret clipping in RM⁺ yields crucial contrasts with ordinary RM: unlike RM, which can stall for exponentially many iterations in certain potential and identical-interest games, RM⁺ ensures a monotonic increase in the $\ell_2$-norm of cumulative regret and always makes progress towards approximate KKT solutions [2510.17067]. Nonetheless, vanilla RM⁺ and PRM⁺ still exhibit suboptimal behavior in adversarial or oscillatory settings, motivating both stabilization via extragradient methods and scale-invariant modifications such as IREG-PRM⁺ to obtain best-possible convergence rates and robust last-iterate behavior [2510.04407, 2305.14709, 2311.00676].

Empirical and theoretical analyses indicate that nonuniform averaging and predictive updates further enhance performance, but only under smoothness or regularity in loss predictions [2510.04407, 2007.14358, 2404.13891]. Weighted/discounted variants of CFR built on aggressive regret discounting or weighted averaging suceed in dominated-action-heavy or deep tree regimes [1809.04040, 2404.13891].

## 7. Summary Table: RM⁺ and Variants (Selected Properties)

| Algorithm        | Regret Rate    | Averaging      | Last-Iterate    | Param-free/Scale-inv | Notable Feature                      |
|------------------|---------------|----------------|-----------------|----------------------|--------------------------------------|
| RM⁺              | $O(\sqrt{T})$ | Linear         | No guarantee    | Yes                  | Clips negative regrets               |
| PRM⁺             | $O(1/T)$*     | Linear/$t^2$   | No guarantee    | Yes                  | Predictive, “optimistic” update      |
| IREG-PRM⁺        | $O(1/T)$      | Linear/$t^2$   | $O(1/\sqrt{T})$ | Yes                  | $\ell_2$-invariant regret norm       |
| ExRM⁺/Smooth PRM⁺| $O(1/T)$      | Any            | Yes, linear     | Yes                  | Extragradient/smoothing stabilization|
| RM (no “plus”)   | Fail          | Any            | Fail            | Yes                  | Can stall exponentially              |

(*) $O(1/T)$ for average-iterate, under smooth predictable-sequence conditions [2510.04407, 2404.13891].

## References

- "Scale-Invariant Regret Matching and Online Learning with Optimal Convergence: Bridging Theory and Practice in Zero-Sum Games" [2510.04407]
- "Faster Game Solving via Predictive Blackwell Approachability: Connecting Regret Matching and Mirror Descent" [2007.14358]
- "Regret Matching+: (In)Stability and Fast Convergence in Games" [2305.14709]
- "Last-Iterate Convergence Properties of Regret-Matching Algorithms in Games" [2311.00676]
- "Convergence of Regret Matching in Potential Games and Constrained Optimization" [2510.17067]
- "Minimizing Weighted Counterfactual Regret with Optimistic Online Mirror Descent" [2404.13891]
- "Solving Imperfect-Information Games via Discounted Regret Minimization" [1809.04040]
- "Balancing the AI Strength of Roles in Self-Play Training with Regret Matching+" [2401.12557]

Source: https://www.emergentmind.com/topics/regret-matching-rm