Double Neural CFR

• Double Neural CFR is a neural generalization of the classic Counterfactual Regret Minimization, replacing tabular records with dual networks for efficient strategy learning.
• It utilizes recurrent architectures with attention and mini-batch MCCFR to manage massive state spaces while achieving convergence rates similar to traditional methods.
• Empirical evaluations show that the method attains near-optimal exploitability with dramatically lower memory and computational demands compared to tabular CFR and deep RL.

Double Neural Counterfactual Regret Minimization (Double Neural CFR) is a neural generalization of the Counterfactual Regret Minimization (CFR) framework for solving large-scale imperfect information extensive-form games. The central innovation is the replacement of tabular cumulative regret and strategy representations with two separate neural networks, enabling efficient learning and generalization over massive or continuous state spaces. By leveraging recurrent neural architectures with attention, robust sampling strategies, and mini-batch Monte Carlo variants of CFR (MCCFR), Double Neural CFR attains convergence rates and solution qualities that match or outperform classic tabular CFR, while requiring drastically reduced memory and computation (1812.10607).

1. Principles of Counterfactual Regret Minimization

CFR provides a principled iterative algorithm for computing Nash equilibria in multi-agent imperfect information extensive-form games. Key elements include:

• Information Sets and Strategies: Each player $i$ observes information sets $I_i$, partitioning histories $h \in H$ where that player acts. A strategy $\sigma_i$ assigns distributions over legal actions at each $I_i$.
• Reach Probabilities: Under joint strategy profile $\sigma$, the reach probability for a history $h$ is $\pi^\sigma(h)$, with contributions factored into each player's portion and opponents' portions.
• Counterfactual Values: The counterfactual value of $I_i$ for player $i$ is the average utility received if $I_i$0 reaches $I_i$1 and the rest of the game is played according to $I_i$2; precise definition:

$I_i$3

• Regret Calculation and Matching: After each iteration, instantaneous regrets $I_i$4 are computed for all actions $I_i$5. These are accumulated into $I_i$6. The (behavioral) strategy for the next iteration is given by regret matching:

$I_i$7

• Averaged Strategies: The average strategy up to iteration $I_i$8 is the reach-weighted average of past strategies.

The standard CFR machinery is efficient but fundamentally constrained by the necessity of tabular representations, which scale poorly in games with vast numbers of information sets or actions.

2. Double Neural Representation Architecture

Double Neural CFR introduces a dual-network paradigm:

• Regret-Sum Network (RSN): Parameterized by $I_i$9, RSN encodes cumulative regret values $h \in H$0 over the joint space of information sets and actions.
• Avg-Strategy Network (ASN): Parameterized by $h \in H$1, ASN outputs cumulative strategy numerators $h \in H$2, which are required for reach-weighted averaging.

Encoding Information Sets:

Each information set is represented as a variable-length sequence of private information, public observations, and actions. This sequence is processed by a recurrent neural network (e.g., LSTM or GRU), producing hidden states, which are then fed through an attention mechanism to produce a fixed-dimensional embedding $h \in H$3. A final output head maps $h \in H$4 to a $h \in H$5-dimensional vector representing the regrets or average strategy numerators for all actions at that information set.

Network Fitting Process:

After each sample-based CFR iteration, two empirical memories $h \in H$6 and $h \in H$7 are assembled. These are used to perform regression updates: $h \in H$8

$h \in H$9

Standard minibatch SGD (e.g., Adam) is used for this fitting procedure.

3. Stochastic Optimization and Mini-batch Variants

The neural networks are trained using data from mini-batch Monte Carlo Counterfactual Regret Minimization (MCCFR) and its positive variant MCCFR$\sigma_i$0:

• Robust Sampling: At decision nodes, $\sigma_i$1 actions are sampled uniformly without replacement. Setting $\sigma_i$2 matches outcome sampling; $\sigma_i$3 recovers external sampling. The robust policy $\sigma_i$4 yields importance weights for terminal histories, with favorable bias-variance trade-offs especially at intermediate $\sigma_i$5.
• Mini-batch MCCFR: $\sigma_i$6 independent trajectories are sampled per iteration. For an information set $\sigma_i$7, the mini-batch counterfactual value estimate is:

$\sigma_i$8

• Mini-batch MCCFR$\sigma_i$9: The cumulative regret update is replaced by the positive-part version:

$I_i$0

This allows efficient exploitation of variance-reduced updates and improved convergence.

4. End-to-End Algorithmic Workflow

The procedure follows three main stages per iteration:

1. Strategy Extraction: At each information set, extract $I_i$1 by regret-matching on $I_i$2 from the RSN, following the formula from CFR.
2. Sampling and Data Collection: Using the current policy, sample $I_i$3 robust trajectories to generate empirical memories:
   • $I_i$4: ((information set, action), instantaneous sampled regret)
   • $I_i$5: ((information set, action), $I_i$6)
3. Network Training: Update RSN and ASN parameters using least-squares regression against cumulative targets, as described above.

Training employs minibatch optimizers, gradient clipping, and early stopping on MSE loss. Initialization of network weights can be random or performed by warm-starting from a tabular solution.

Pseudocode Summary (abbreviated):

$\sigma$6

5. Empirical Evaluation and Performance

Double Neural CFR has been evaluated in One-Card Poker and No-Limit Leduc Hold'em environments with state spaces exceeding $I_i$7 configurations. Key empirical findings include:

• Achieves exploitability $I_i$8 in $I_i$9–$\sigma$0 iterations, on par with classic tabular CFR/CFR$\sigma$1 but using only $\sigma$2 network parameters, in contrast to $\sigma$3–$\sigma$4 for dense tabular approaches.
• Dramatically outpaces deep RL-based fictitious self-play (NFSP), which requires around $\sigma$5 iterations to