Reward Transformation Counterfactual Regret Minimization

• RTCFR is a family of algorithms for extensive-form games that leverage reward transformation to target last-iterate convergence instead of average-strategy convergence.
• It modifies the local counterfactual loss by adding a regularization term centered on a reference strategy, turning the update into a strongly convex-concave subproblem.
• Adaptive RTCFR variants incorporate reference strategy updates and discounted regret to overcome computational and approximation challenges in equilibrium finding.

Searching arXiv for the cited RTCFR and related papers to ground the article in current literature. Reward Transformation Counterfactual Regret Minimization (RTCFR) is a family of counterfactual-regret-based algorithms for two-player zero-sum extensive-form games (EFGs) that incorporates reward transformation or regularization into the local CFR update in order to target last-iterate convergence rather than only average-strategy convergence. In the formulation developed for extensive-form games, RTCFR preserves CFR’s bottom-up, per-information-set structure while modifying the local counterfactual loss by a regularizing term centered at a reference strategy. This converts the inner problem into a strongly convex-concave problem and yields a sequence of transformed subproblems whose saddle points approach a Nash equilibrium of the original game (Ren et al., 17 Sep 2025). Subsequent work develops RTCFR$^+$ by using CFR$^+$ as the inner solver for perturbed regularized EFGs (Meng et al., 2023), and extends RTCFR to approximate Extensive-Form Perfect Equilibrium (EFPE) via perturbed strategy spaces and adaptive perturbation control (Ren et al., 11 Aug 2025).

1. Origin and problem setting

RTCFR emerged from the observation that classical regret minimization methods for normal-form games and extensive-form games typically guarantee convergence of the average iterate, not the last iterate. In EFGs, this creates two practical drawbacks that are repeatedly emphasized in the literature: computing average strategies requires significant computational or memory resources, and when strategies are represented by function approximation, averaging can introduce additional approximation error (Meng et al., 2023, Ren et al., 17 Sep 2025).

The reward-transformation approach addresses this by replacing the original equilibrium-finding problem with a sequence of strongly convex-concave optimization problems (SCCPs). At outer iteration $n$, the transformed game is centered at a reference strategy profile and regularized with weight $\mu>0$. In sequence form, the extensive-form saddle-point problem is recalled as

$$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2,$$

and the reward-transformed version is

$$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2 + \mu D_{\psi}(q_1, q_1^{r,n}) - \mu D_{\psi}(q_2, q_2^{r,n}),$$

where $q_i^{r,n}$ is the reference sequence-form strategy for the $n$-th SCCP (Ren et al., 17 Sep 2025).

In the 2023 RTCFR$^+$ formulation, the same conceptual move is stated as solving a perturbed regularized game of the form

$$\min_{\sigma_0}\max_{\sigma_1} u_1(\sigma_0,\sigma_1) +\mu \phi(\sigma_0,\sigma_n) -\mu \phi(\sigma_1,\sigma_n),$$

with the Euclidean Bregman divergence used so that CFR$^+$0 or RM$^+$1 can operate on the full strategy polytope, including boundary points (Meng et al., 2023). This establishes RTCFR not as a distinct equilibrium concept, but as an algorithmic framework that modifies regret feedback so that the current strategy profile itself is driven toward equilibrium.

A second line of development applies RTCFR to equilibrium refinement. In the EFPE setting, the target is not merely a Nash equilibrium but an $^+$2-EFPE in a perturbed game $^+$3, with behavioral lower bounds

$^+$4

In that setting, RTCFR is used to solve a sequence of perturbed regularized problems while adapting the perturbation level over time (Ren et al., 11 Aug 2025).

2. Formal construction in extensive-form games

The central technical feature of RTCFR is that it modifies the local optimization problem at each information set without abandoning CFR’s laminar decomposition. Standard CFR updates each information set independently using counterfactual losses and derives the next strategy from cumulative counterfactual regret. RTCFR keeps that local architecture, but adds a reward-transformation term that pulls the current behavioral strategy toward a reference strategy.

In the adaptive 2025 formulation, each information set $^+$5 solves a local regularized problem

$^+$6

where $^+$7 is the local counterfactual loss from the subtree below $^+$8, and the regularizer is the Euclidean Bregman divergence

$^+$9

The resulting modified counterfactual loss is

$n$0

This makes the RT term appear explicitly as a gradient-like pull toward $n$1 (Ren et al., 17 Sep 2025).

The local regret objects remain CFR-style. For transformed loss $n$2, the immediate and cumulative regrets are

$n$3

$n$4

The next strategy is then updated by regret matching or regret matching plus: $n$5 Thus RTCFR remains a local regret minimizer at each information set, but the quantity being minimized is the RT-regularized local payoff rather than the unmodified counterfactual loss (Ren et al., 17 Sep 2025).

The 2023 RTCFR$n$6 presentation emphasizes the same point from a different angle. There, the transformed local feedback is summarized as

$n$7

followed by the standard CFR$n$8 updates

$n$9

The practical intent is to keep the implementation close to CFR$\mu>0$0, modifying only the feedback via reward transformation (Meng et al., 2023).

3. Laminar decomposition, RTCFR$\mu>0$1, and perturbed-space variants

A notable technical development is the move from earlier dilated regularization to laminar regret decomposition. The adaptive 2025 paper argues that the earlier dilated-regularization version of RTCFR$\mu>0$2 was awkward in CFR because its counterfactual loss contained a complicated descendant coupling term, described as “a double-summation term in $\mu>0$3,” which complicated both implementation and convergence analysis. The laminar-decomposition formulation instead yields per-information-set optimization and keeps the update local at each information set (Ren et al., 17 Sep 2025).

This distinction matters because RTCFR is designed to retain the recursive, bottom-up structure of CFR. A particularly important implementation detail is that the 2025 RTCFR is explicitly not the dilated-regularization variant from earlier work; it is the laminar-decomposition version, which makes the per-information-set update clean and aligned with the CFR recursion (Ren et al., 17 Sep 2025).

RTCFR$\mu>0$4 occupies a closely related position in the literature. It is defined as the extensive-form extension of reward transformation in which the outer loop generates SCCPs and the inner solver is CFR$\mu>0$5. The paper characterizes the overall structure as

1. build an SCCP centered at the current reference strategy,
2. solve that SCCP using a last-iterate no-regret method,
3. use the SCCP’s saddle point as the next reference strategy,
4. repeat,

so that the sequence of SCCP saddle points converges to a Nash equilibrium of the original game (Meng et al., 2023).

A further variant appears in the EFPE setting. There, RTCFR is applied on perturbed simplexes using a basis transformation. The perturbed vertices are

$\mu>0$6

with $\mu>0$7, and the transformed coordinate is mapped back to the actual perturbed strategy by

$\mu>0$8

The transformed payoffs satisfy

$\mu>0$9

and the local RTCFR update in the perturbed game is implemented by computing counterfactual values, adding an RT correction, computing local regret in the perturbed basis, updating cumulative regret, applying regret matching, and mapping back to the original behavioral strategy (Ren et al., 11 Aug 2025).

This suggests two major architectural branches within RTCFR research: one focused on last-iterate convergence to Nash equilibrium in unperturbed EFGs, and one focused on last-iterate convergence in perturbed EFGs for approximate EFPE.

4. Adaptive RTCFR

The adaptive version of RTCFR is designed to address three bottlenecks identified in earlier reward-transformation methods: the reference strategy, the RT weight, and the regret accumulation rule. The paper states that fixed RT parameters were difficult to tune and that the exact-inner-solve assumption used in earlier theory was unrealistic, leading in practice to slow convergence, oscillations, or stagnation in local optima (Ren et al., 17 Sep 2025).

The first adaptive component is the reference strategy. Instead of fixing $$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2,$$0 for each SCCP, the algorithm periodically checks exploitability $$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2,$$1 and updates the reference when progress is sufficient. The logic is based on the metric subregularity bound

$$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2,$$2

which allows exploitability to serve as a proxy for distance to Nash equilibrium. The update rules are stated concretely: if $$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2,$$3, set $$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2,$$4 and enter an exploit phase; if $$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2,$$5 and enough iterations have elapsed, keep the strategy and reset; if no good reference is found after $$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2,$$6 iterations, move to exploration (Ren et al., 17 Sep 2025).

The second component is the adaptive RT weight. The paper proves that the SCCP saddle point exploitability satisfies

$$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2,$$7

so $$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2,$$8 simultaneously controls regularization strength and the mismatch between the transformed saddle point and the original Nash equilibrium. Rather than changing $$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2,$$9 directly, the implementation uses a scalar $$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2 + \mu D_{\psi}(q_1, q_1^{r,n}) - \mu D_{\psi}(q_2, q_2^{r,n}),$$0 in

$$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2 + \mu D_{\psi}(q_1, q_1^{r,n}) - \mu D_{\psi}(q_2, q_2^{r,n}),$$1

with

$$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2 + \mu D_{\psi}(q_1, q_1^{r,n}) - \mu D_{\psi}(q_2, q_2^{r,n}),$$2

The practical values reported are $$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2 + \mu D_{\psi}(q_1, q_1^{r,n}) - \mu D_{\psi}(q_2, q_2^{r,n}),$$3 for exploitation or acceleration, $$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2 + \mu D_{\psi}(q_1, q_1^{r,n}) - \mu D_{\psi}(q_2, q_2^{r,n}),$$4 for keeping, and $$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2 + \mu D_{\psi}(q_1, q_1^{r,n}) - \mu D_{\psi}(q_2, q_2^{r,n}),$$5 for exploration (Ren et al., 17 Sep 2025).

The third component is adaptive regret weighting or discounting, inspired by DCFR. The discounted cumulative regret is

$$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2 + \mu D_{\psi}(q_1, q_1^{r,n}) - \mu D_{\psi}(q_2, q_2^{r,n}),$$6

This introduces a “forgetfulness” mechanism so that earlier suboptimal regrets do not dominate indefinitely. The paper interprets this as a practical improvement for aligning regret minimization with last-iterate rather than average-iterate objectives (Ren et al., 17 Sep 2025).

In the EFPE literature, adaptivity appears in a different but related form: the perturbation level $$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2 + \mu D_{\psi}(q_1, q_1^{r,n}) - \mu D_{\psi}(q_2, q_2^{r,n}),$$7 itself is changed using the Information Set Nash Equilibrium (ISNE) criterion. There the algorithm reduces $$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2 + \mu D_{\psi}(q_1, q_1^{r,n}) - \mu D_{\psi}(q_2, q_2^{r,n}),$$8 and the threshold $$\min_{q_1 \in \mathcal{Q}_1} \max_{q_2 \in \mathcal{Q}_2} -q_1^\top U q_2 + \mu D_{\psi}(q_1, q_1^{r,n}) - \mu D_{\psi}(q_2, q_2^{r,n}),$$9 whenever

$q_i^{r,n}$0

according to

$q_i^{r,n}$1

with $q_i^{r,n}$2 (Ren et al., 11 Aug 2025). A plausible implication is that the adaptive RTCFR literature now includes two distinct control layers: adaptation of the RT dynamics themselves, and adaptation of the perturbation schedule in refinement problems.

5. Convergence guarantees and theoretical interpretation

The theoretical analysis of RTCFR is organized at two levels: convergence within each transformed subproblem, and convergence across the sequence of transformed subproblems. Within the $q_i^{r,n}$3-th SCCP, the adaptive 2025 paper proves for RTRMs that

$q_i^{r,n}$4

and for RTCFRs,

$q_i^{r,n}$5

These are best-iterate $q_i^{r,n}$6 statements toward the SCCP saddle point (Ren et al., 17 Sep 2025).

Across SCCPs, the same paper proves asymptotic last-iterate convergence under the idealized exact-solution assumption

$q_i^{r,n}$7

with initialization

$q_i^{r,n}$8

Under these conditions,

$q_i^{r,n}$9

and in sequence form

$n$0

The monotonicity argument is captured by

$n$1

The same paper is explicit that the exact-inner-solve assumption is the main theoretical limitation, so the guarantee functions as an asymptotic template rather than a finite-time certificate (Ren et al., 17 Sep 2025).

The 2023 RTCFR$n$2 paper gives a complementary theoretical narrative. The transformed game with Euclidean Bregman regularization is strongly convex-concave, has a unique saddle point, and RM$n$3 converges asymptotically in the last iterate to that saddle point under discrete-time feedback. Its Theorem 5.2 states: for any reference strategy profile $n$4 and $n$5, RM$n$6 enjoys asymptotic last-iterate convergence to the saddle point of the transformed problem (Meng et al., 2023). The paper does not claim a finite-time rate there, but reports nearly linear convergence empirically when $n$7 is not too small.

The EFPE-oriented RTCFR results preserve the same two-level structure. For each reward-transformed bilinear saddle-point problem, the paper states

$n$8

and across outer iterations

$n$9

It also explicitly notes that strict theoretical last-iterate convergence requires $^+$0 per RT-BSPP, although finite $^+$1 works in practice (Ren et al., 11 Aug 2025).

A common misconception is that RTCFR provides a uniform finite-time last-iterate guarantee comparable to average-strategy CFR bounds. The literature summarized here does not support that interpretation. What it supports is asymptotic last-iterate convergence to the SCCP saddle point, plus asymptotic convergence of the SCCP sequence to the target equilibrium, with finite-time best-iterate $^+$2 statements inside each transformed subproblem (Ren et al., 17 Sep 2025, Meng et al., 2023).

6. Relation to standard CFR, empirical behavior, and practical scope

RTCFR differs from standard CFR and CFR$^+$3 in both objective and training loop. Standard CFR minimizes counterfactual regret at each information set and is known for average-strategy convergence. RTCFR augments the local loss with the term

$^+$4

so the dynamics are centered on a reference strategy and target last-iterate convergence. Adaptive RTCFR further adds reference updates and discounted regret, neither of which belongs to standard CFR or CFR$^+$5 (Ren et al., 17 Sep 2025). RTCFR$^+$6 still uses positive-part cumulative regret projection, but its counterfactual losses are RT-augmented and its outer loop repeatedly changes the reference strategy (Meng et al., 2023).

The empirical literature evaluates these methods on both normal-form and extensive-form benchmarks. The adaptive 2025 paper reports experiments on random $^+$7 matrix games and on Kuhn poker, Leduc poker, Liar’s dice, and Goofspiel, using exploitability as the metric, alternating updates, and uniform initial strategies. In matrix games, adaptive RT methods significantly improve over non-adaptive RT counterparts, discounted RM variants outperform plain RM$^+$8-type methods, and convergence can reach as low as $^+$9 exploitability. In EFGs, adaptive methods are generally best across Kuhn poker, Leduc poker, and Goofspiel; on harder games like Leduc and Goofspiel, adaptive RT significantly reduces fluctuations and instability; and RT methods generally beat average-iterate methods in exploitability over time (Ren et al., 17 Sep 2025).

The 2023 RTCFR$$\min_{\sigma_0}\max_{\sigma_1} u_1(\sigma_0,\sigma_1) +\mu \phi(\sigma_0,\sigma_n) -\mu \phi(\sigma_1,\sigma_n),$$0 paper reports that RTCFR$$\min_{\sigma_0}\max_{\sigma_1} u_1(\sigma_0,\sigma_1) +\mu \phi(\sigma_0,\sigma_n) -\mu \phi(\sigma_1,\sigma_n),$$1 converges faster in all tested games, achieves much lower final exploitability, and often obtains about $$\min_{\sigma_0}\max_{\sigma_1} u_1(\sigma_0,\sigma_1) +\mu \phi(\sigma_0,\sigma_n) -\mu \phi(\sigma_1,\sigma_n),$$2 times lower exploitability than RM$$\min_{\sigma_0}\max_{\sigma_1} u_1(\sigma_0,\sigma_1) +\mu \phi(\sigma_0,\sigma_n) -\mu \phi(\sigma_1,\sigma_n),$$3/CFR$$\min_{\sigma_0}\max_{\sigma_1} u_1(\sigma_0,\sigma_1) +\mu \phi(\sigma_0,\sigma_n) -\mu \phi(\sigma_1,\sigma_n),$$4 under the same computational overhead. It is described there as the first last-iterate algorithm reported in that work to beat CFR$$\min_{\sigma_0}\max_{\sigma_1} u_1(\sigma_0,\sigma_1) +\mu \phi(\sigma_0,\sigma_n) -\mu \phi(\sigma_1,\sigma_n),$$5 on EFGs at sizes at least as large as Leduc Poker (Meng et al., 2023).

In the EFPE setting, experiments on Kuhn Poker, Leduc Poker, Goofspiel, and Liar’s Dice compare RTCFR against CFR$$\min_{\sigma_0}\max_{\sigma_1} u_1(\sigma_0,\sigma_1) +\mu \phi(\sigma_0,\sigma_n) -\mu \phi(\sigma_1,\sigma_n),$$6, EGT, RegOMWU, and perturbed variants. The reported findings are that RTCFR$$\min_{\sigma_0}\max_{\sigma_1} u_1(\sigma_0,\sigma_1) +\mu \phi(\sigma_0,\sigma_n) -\mu \phi(\sigma_1,\sigma_n),$$7 achieves the lowest exploitability and the fastest Nash-equilibrium convergence, often exhibits near-$$\min_{\sigma_0}\max_{\sigma_1} u_1(\sigma_0,\sigma_1) +\mu \phi(\sigma_0,\sigma_n) -\mu \phi(\sigma_1,\sigma_n),$$8 practical behavior, performs much better than fixed-perturbation baselines on maximum information set regret, and uses one tree traversal per iteration, whereas some baselines require more traversals (Ren et al., 11 Aug 2025).

Practical guidance in the adaptive NE-setting paper is explicit. Fixed $$\min_{\sigma_0}\max_{\sigma_1} u_1(\sigma_0,\sigma_1) +\mu \phi(\sigma_0,\sigma_n) -\mu \phi(\sigma_1,\sigma_n),$$9 and fixed $^+$00 are described as fragile; adaptive updates are recommended. Exploitability checks need not be done every iteration, but can be performed every $^+$01 steps, with default $^+$02. Recommended values are $^+$03 for acceleration, $^+$04 for stable continuation, and $^+$05 for exploration. For RTDRM and RTDCFR, $^+$06 is reported as a good compromise, since smaller $^+$07 makes convergence too cyclic and unstable, while overly aggressive discounting can collapse behavior back toward the less effective RTCFR$^+$08 regime (Ren et al., 17 Sep 2025).

A related but distinct line of work concerns model-free CFR-style learning under interactive bandit feedback. That literature uses an importance-weighted transformed reward signal,

$^+$09

to construct CFR feedback from partial observations, but it does not define or name RTCFR as such (Farina et al., 2021). This suggests a broader historical context in which “reward transformation” can refer either to regularization around a reference strategy, as in RTCFR proper, or to importance-weighted transformation of observed payoffs under partial feedback. The published RTCFR line in EFGs is specifically the former.