Deep Bayesian CFR: Neural Equilibrium

• Deep Bayesian CFR is a neural extension of CFR that integrates kernel density belief updates with type-conditioned regret estimation to approximate Bayesian equilibria.
• It combines online posterior estimation with neural network approximations, enhancing scalability and maintaining low exploitability in multi-agent settings.
• Empirical results in Texas Hold’em poker demonstrate Deep Bayesian CFR’s effectiveness, achieving exploitability metrics competitive with state-of-the-art methods.

Deep Bayesian Counterfactual Regret Minimization (Deep Bayesian CFR, or Deep BCFR) is a neural-network-based extension of Counterfactual Regret Minimization for computing approximate Bayesian Nash equilibria in extensive-form Bayesian games with incomplete information. Deep BCFR combines online posterior belief updates via kernel density estimation with neural approximation of instantaneous, type-conditioned counterfactual regrets and policies. This approach achieves low Bayesian exploitability in environments where latent types or payoffs of agents are not known a priori and must be inferred from observed histories. Deep BCFR builds upon the Bayesian-CFR framework, integrating neural networks for scalability while maintaining theoretical guarantees and empirical performance advantages in complex multi-agent settings such as Texas Hold’em poker (Zhang et al., 2024).

1. Foundations: CFR and Bayesian-CFR

Counterfactual Regret Minimization (CFR) is a foundational algorithm for computing Nash equilibria in extensive-form games. For a player $i$, at each information set $I$, the instantaneous counterfactual regret for action $a\in A(I)$ at iteration $t$ is defined as

$$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t, I)$$

where $\sigma^t_{I\to a}$ is the strategy profile that deviates to $a$ at $I$, and $u_i(\cdot, I)$ denotes the expected utility. The cumulative regret is

$$R^T(I, a) = \sum_{t=1}^T r^t(I, a), \quad R^T_+(I, a) = \max\{R^T(I,a), 0\}$$

Regret-matching sets

$I$0

yielding average regret $I$1.

In the Bayesian setting, each player receives a private type $I$2 from an (unknown) prior $I$3. Each player maintains a posterior $I$4 based on observation history $I$5. Bayesian-CFR generalizes CFR by weighting regrets by $I$6 and uses kernel-density estimation for posterior updates.

2. Bayesian Regret and Theoretical Properties

The Bayesian average overall regret for player $I$7 after $I$8 rounds is

$I$9

where $a\in A(I)$0 is the joint strategy profile at round $a\in A(I)$1 for type profile $a\in A(I)$2. If all immediate Bayesian counterfactual regrets

$a\in A(I)$3

are minimized via regret-matching+, then $a\in A(I)$4 as $a\in A(I)$5.

Theoretical regret bounds (Theorem 3.3 of (Zhang et al., 2024)):

$a\in A(I)$6

with

$a\in A(I)$7

3. Deep Bayesian CFR: Neural Representations

Deep BCFR extends Bayesian CFR by using neural networks to approximate the type-conditioned instantaneous counterfactual regrets and average policies:

• For each player $a\in A(I)$8 and each type $a\in A(I)$9, an advantage network $t$0 takes a feature representation of $t$1 and a one-hot encoding of $t$2, merged in the network's second hidden layer.
• A policy network $t$3, using only public features of $t$4, outputs a distribution over possible actions.

Neural training alternates two replay-based procedures:

• Value-memory update: During each of $t$5 external-sampling traversals per iteration, tuples $t$6 of sampled regrets are written to a replay buffer $t$7. Training of $t$8 proceeds by minimizing the MSE

$t$9

• Policy-memory update: Each traversal stores $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t, I)$$0 into $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t, I)$$1. The policy network $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t, I)$$2 is periodically trained via

$$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t, I)$$3

4. Online Kernel-Density Belief Updating

Posterior beliefs over latent types $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t, I)$$4 are updated at each iteration using kernel-density estimation (KDE):

Given observed histories $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t, I)$$5 and a set of reference type–history samples $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t, I)$$6, define kernels $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t, I)$$7 (histories) and $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t, I)$$8 (types) with bandwidth parameters $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t, I)$$9. The likelihood estimate

$\sigma^t_{I\to a}$0

yields the posterior by Bayes’ rule:

$\sigma^t_{I\to a}$1

As $\sigma^t_{I\to a}$2, this KDE-based posterior is proven (Lemma 2.1 and Theorem 2.2) to converge (in $\sigma^t_{I\to a}$3) to the true posterior over an equivalence class of types (Zhang et al., 2024).

5. Algorithmic Workflow and Pseudocode

The Deep BCFR workflow integrates posterior updating, neural value estimation, and external sampling:

$a$5

6. Regret Guarantees and Convergence Rates

For Deep BCFR, Theorem 4.4 provides the following high-probability bound on Bayesian regret:

$\sigma^t_{I\to a}$4

where $\sigma^t_{I\to a}$5, $\sigma^t_{I\to a}$6 is the number of external samples per iteration, $\sigma^t_{I\to a}$7 bounds the per-iteration MSE in the advantage network, and $\sigma^t_{I\to a}$8 is a probability parameter. As $\sigma^t_{I\to a}$9, the average regret satisfies

$a$0

Hence, Deep BCFR converges to a Bayesian equilibrium up to the neural network MSE error, maintaining the statistical properties of Bayesian-CFR+ in the presence of function approximation.

7. Empirical Results and Practical Considerations

Deep BCFR was evaluated in two-player heads-up Texas Hold’em using RLCard. Player types $a$1 comprise Normal, Conservative, Aggressive (pure types), and nine mixtures (e.g., $a$2). Key experimental settings:

• Hardware: AMD EPYC 7513 CPU, NVIDIA RTX A6000 GPU
• Metric: exploitability (milli-big blinds/game; lower is better)
• Hyperparameters: $a$3 traversals per iteration; replay buffer $a$4; advantage and policy nets as two-layer MLPs (width 128); Adam optimizer (lr=0.001, batch 64)

Performance metrics (averaged over all type mixtures):

Algorithm	Exploitability (mbb/g)
CFR	0.28
CFR+	0.07
Deep CFR	0.36
MCCFR	0.35
DQN	1.34
BCFR	0.17
BCFR+	0.02
Deep BCFR	0.08

Bayesian opponent modeling confers substantial benefit over type-agnostic CFR. BCFR+ achieves the best exploitability (0.02 mbb/g), and Deep BCFR outperforms Deep CFR at modest neural cost. Ablation shows that omitting the posterior-update module degrades BCFR performance (exploitability increases from 0.19 to 0.27), confirming the importance of online kernel-density belief estimation. This suggests that belief updates are a principal driver of Deep BCFR's improvement relative to standard methods (Zhang et al., 2024).