Bayesian-CFR+: Deep Bayesian Regret Minimization

• Bayesian-CFR+ is an extension of CFR that integrates Bayesian belief updates to handle latent player types and incomplete information.
• It employs neural networks to approximate counterfactual regrets and strategies, enabling adaptive learning through value and policy memory updates.
• Empirical results show reduced exploitability in games, highlighting the effectiveness of Bayesian type inference in achieving equilibrium convergence.

Deep Bayesian Counterfactual Regret Minimization (Deep BCFR) is a neural-augmented algorithm for computing approximate Bayesian Nash equilibria in extensive-form games with incomplete information. Extending the classical Counterfactual Regret Minimization (CFR) framework, Deep BCFR incorporates online Bayesian belief updates regarding players’ latent types, using kernel-density estimation and regret-driven neural policy/value networks. It generalizes CFR to the Bayesian setting, ensuring convergence to equilibrium in the presence of player types and hidden payoff information (Zhang et al., 2024).

1. Foundations: CFR and Bayesian CFR

Counterfactual Regret Minimization (CFR) is an iterative algorithm for computing Nash equilibria in extensive-form games of imperfect information. At each iteration $t$, each player $i$ performs regret-matching to update their strategy $\sigma_i^t$ at every information set $I$. The instantaneous counterfactual regret for action $a \in A(I)$ is

$$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t,I),$$

where $u_i(\sigma,h)$ denotes the expected utility for player $i$ when deviating to action $a$ at $I$ and all players subsequently play according to $i$0. The cumulative and positive cumulative regrets are, respectively,

$i$1

The updated policy is set by regret-matching as

$i$2

CFR ensures vanishing average regret, $i$3.

In a Bayesian game, players’ payoffs and strategies are functions of a latent type $i$4 drawn from a prior $i$5. A player maintains a posterior belief $i$6 given their observation history $i$7. Bayesian CFR modifies CFR by weighting all counterfactual regrets at each iteration by the player's posterior distribution and updating these posteriors with kernel-density estimates.

2. Bayesian Regret and Updating Procedures

Bayesian average regret for player $i$8 over $i$9 iterations is defined as

$\sigma_i^t$0

Immediate Bayesian counterfactual regrets at each information set are

$\sigma_i^t$1

and Bayesian regret-matching policy updates follow an analogous structure to conventional CFR.

If every immediate Bayesian regret is driven to zero by the algorithm, overall Bayesian average regret converges to zero, ensuring convergence to a Bayesian equilibrium.

3. Neural Network Integration in Deep Bayesian CFR

Deep BCFR employs two principal neural network modules for each player $\sigma_i^t$2 and each type $\sigma_i^t$3:

• Advantage Network ($\sigma_i^t$4): Approximates immediate (type-conditioned) counterfactual regrets. The input consists of an encoding of the information set $\sigma_i^t$5 concatenated with a one-hot vector for $\sigma_i^t$6, merged in the network’s second hidden layer.
• Policy Network ($\sigma_i^t$7): Approximates the player’s average strategy, taking as input only public information.

Both networks are trained with experience replay:

• Value-memory update: For each of $\sigma_i^t$8 external-sampling traversals per iteration, sampled regrets $\sigma_i^t$9 at each visited infoset are stored in a replay memory $I$0. The advantage network parameters $I$1 are trained to minimize the mean-square error loss

$I$2

where $I$3.

• Policy-memory update: The regret-matching policy samples at each infoset are stored in $I$4 and used to update $I$5 by MSE loss minimization

$I$6

4. Bayesian Posterior Estimation via Kernel Density

Players maintain a reference set $I$7 of type–history samples. To update beliefs, Deep BCFR uses kernel density estimation with distinct kernel functions $I$8 for histories and $I$9 for types, yielding an estimated likelihood

$a \in A(I)$0

where $a \in A(I)$1 and $a \in A(I)$2 are suitable metrics and $a \in A(I)$3 are kernel bandwidths.

The posterior is then updated via Bayes’ rule: $a \in A(I)$4 As $a \in A(I)$5, this KDE-based posterior converges (in $a \in A(I)$6) to the true posterior on equivalence classes of types. This suggests that the kernel-density update is crucial for accurate type inference and equilibrium convergence.

5. Algorithmic Workflow

The high-level procedure for Deep BCFR is:

1. For each iteration, update observation histories and posterior beliefs over types using the kernel-density rule.
2. Sample candidate type $a \in A(I)$7 from the current posterior.
3. For each player and for $a \in A(I)$8 external-sampling traversals:
   • Traverse the game tree, compute sampled regrets, and store $a \in A(I)$9 in $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t,I),$$0.
   • Store policy samples $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t,I),$$1 in $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t,I),$$2.
4. Update advantage networks via stochastic gradient descent on the MSE loss.
5. Periodically update the policy network using sampled policies.

Neural networks are initialized per player and type, and learning proceeds with standard optimizer choices such as Adam.

6. Theoretical Guarantees

Deep BCFR inherits the Bayesian-approximately vanishing regret guarantees of Bayesian CFR+. Formally, with $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t,I),$$3 external samples and per-iteration MSE of the advantage approximator bounded by $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t,I),$$4, for probability at least $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t,I),$$5,

$$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t,I),$$6

where $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t,I),$$7. Therefore, as $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t,I),$$8, the average regret converges to $$r^t(I,a) = u_i(\sigma^t_{I\to a},I) - u_i(\sigma^t,I),$$9, i.e., Deep BCFR converges to a Bayesian equilibrium up to neural function approximation error.

7. Empirical Evaluation and Significance

In experiments using two-player heads-up Texas Hold’em (RLCard environment), Deep BCFR was evaluated alongside baseline algorithms including CFR, CFR+, MCCFR, Deep CFR, DQN, BCFR, and BCFR+. Each player’s latent payoff type $u_i(\sigma,h)$0 was sampled among three pure types (Normal, Conservative, Aggressive) and nine mixture types. Results were measured in exploitability (milli-big blinds per game):

Algorithm	Exploitability (mbb/g)
CFR	0.28
CFR+	0.07
Deep CFR	0.36
MCCFR	0.35
DQN	1.34
BCFR	0.17
BCFR+	0.02
Deep BCFR	0.08

Bayesian modeling of opponent types offered significant reduction in exploitability. BCFR+ achieved the best exploitability (0.02 mbb/g), nearly matching the complete-information lower bound (0.16 mbb/g). Deep BCFR outperformed Deep CFR by incorporating Bayesian type inference into the regret and value functions with modest neural network overhead.

Ablation studies confirmed the critical role of the online posterior update module: omitting it increased exploitability from 0.19 to 0.27, while vanilla CFR (without type modeling) yielded 0.31. A plausible implication is that integrated Bayesian belief updating is the principal driver of performance in Deep BCFR (Zhang et al., 2024).