---
title: Bayesian Strong Stackelberg Q-learning
url: https://www.emergentmind.com/topics/bayesian-strong-stackelberg-q-learning-bss-q
type: topic
---

# Bayesian Strong Stackelberg Q-learning

Bayesian Strong Stackelberg Q-learning (BSS-Q) is a multi-agent reinforcement learning (MARL) methodology designed to learn optimal leader policies in the setting of Bayesian Stackelberg Markov Games (BSMGs). Within this framework, strategic leader-follower (defender-attacker) interactions occur under uncertainty regarding the follower’s (attacker’s) type, reflecting critical requirements in domains such as Moving Target Defense (MTD) for cybersecurity. BSS-Q provides the first scalable, provably convergent approach for learning Strong Stackelberg Equilibrium (SSE) policies in sequential incomplete-information settings, operating without prior knowledge of model rewards or transitions [2007.10457].

## 1. Bayesian Stackelberg Markov Games: Preliminaries

A Bayesian Stackelberg Markov Game (BSMG) is a two-player, turn-based stochastic game formalized as the tuple 
$$ \bigl\langle S,\; A_L,\; A_F,\; \Theta,\; T,\; R_L,\; R_F,\; P_0,\;\gamma\bigr\rangle $$
with the following elements:

- $S$: Finite set of states $\{s_1,\ldots,s_{|S|}\}$.
- $\Theta$: Finite set of attacker (follower) types $\{\theta_1,\ldots,\theta_t\}$; defender maintains a state-conditional belief $\theta(s)\in\Delta(\Theta)$ with $\theta_i(s)=\Pr(\text{attacker}=i|s)$.
- $A_L(s)$, $A_F^i(s)$: Defender (leader) and attacker type-$i$ action sets at state $s$; $A_F(s)=\bigcup_iA_F^i(s)$.
- $T(s,a_L,a_F^i,s')$: Stochastic transition kernel.
- $R_L(s,a_L,a_F^i)$, $R_F^i(s,a_L,a_F^i)$: Immediate rewards for leader and follower type $i$.
- $P_0\in\Delta(S\times\Theta)$: Prior on start state and follower type.
- $\gamma\in[0,1)$: Discount factor.

The leader selects a mixed strategy $x(\cdot|s)$; the follower observes this commitment, selects a pure best response, and transitions follow $T$. Payoffs for both agents are discounted sum utilities.

## 2. Strong Stackelberg Equilibrium in BSMGs

The Strong Stackelberg Equilibrium (SSE) for BSMGs comprises stationary Markovian policies:

- $x^*=\{x^*(\cdot|s)\}_{s\in S}$: leader’s mixed strategies.
- $q^*_i=\{q^*_i(s)\}_{s\in S}$: type-$i$ follower’s pure strategies.

The SSE conditions at each state $s$ are:

a) **Best Response**: For each type $i$, $q^*_i(s)\in\arg\max_{b\in A_F^i(s)} V^i_F(s;x^*,b)$, i.e., optimal for follower type-$i$ given leader’s $x^*$.

b) **Tie-Breaking Favoring Leader**: If multiple best responses exist ($R^i(x^*(\cdot|s))$), select $q^*_i(s)\in\arg\max_{b\in R^i(x^*(\cdot|s))} V_L(s;x^*,b)$. 

Existence and uniqueness of the leader’s equilibrium value are guaranteed for finite $S$, $A$, and $\gamma<1$. The SSE characterization ensures strategic leader commitment accounting for best-responding followers under type uncertainty.

## 3. The Bayesian Strong Stackelberg Q-learning Algorithm

BSS-Q is a joint learning procedure that iteratively approximates SSE policies and value functions through the interaction loop. Main components:

- **Type-Conditional Q-Tables for Leader**: $Q^L_i(s,a_L,a_F)$, estimating discounted rewards when the leader plays $a_L$ and the type-$i$ follower plays $a_F$.
- **Follower Q-Tables**: $Q^F_i(s,a_L,a_F)$ for each attacker type.
- **Belief Tracking**: $\theta_t(s)$, updated online via Bayes’ rule if required.

The learning proceeds as follows:

1. Observe $s$, sample attacker type $i\sim\theta(s)$.
2. Actions: $a_L$ via $\epsilon$-greedy w.r.t. leader strategy $x_t(s)$; $a_F$ via $\epsilon$-greedy w.r.t. follower policy $q_{i,t}(s)$.
3. Execute $(a_L,a_F)$, observe rewards $(r_L,r_F)$ and next state $s'$.
4. Compute the stage-game SSE at $s'$ using current Q-tables as payoff matrices; obtain $(x_{t+1}(s'), q_{i,t+1}(s'))$ and equilibrium values $(V^L(s'), V^F_i(s'))$.
5. Q-Table Updates:
   $$
   Q_{t+1}^{L,i}(s,a_L,a_F)
   = (1-\alpha_t)Q^{L,i}_t(s,a_L,a_F) + \alpha_t\left(r_L + \gamma V^L(s')\right)
   $$
   $$
   Q_{t+1}^{F,i}(s,a_L,a_F)
   = (1-\alpha_t)Q^{F,i}_t(s,a_L,a_F) + \alpha_t\left(r_F + \gamma V^{F,i}(s')\right)
   $$
6. Optionally update belief $\theta_{t+1}(\cdot)$ by Bayes’ rule using observed $a_F$.

This iterative process enables principled policy improvement in adversarial partially observable settings.

## 4. Convergence Properties

Under standard stochastic-approximation conditions—step-sizes $(\alpha_t)$ with $\sum_t\alpha_t=\infty$, $\sum_t\alpha_t^2<\infty$; infinite visitation to each $(s,a_L,a_F,i)$-tuple; $\gamma<1$—BSS-Q's Q-tables converge almost surely to the unique fixed point of the Bellman operator defined by the stage-game SSE. The policies thus induced converge to a stationary SSE of the underlying BSMG.

The underlying Bellman operator for leader-type $i$ is:
$$
\Omega(Q^L_i)(s,a_L,a_F) = R_L(s,a_L,a_F) + \gamma V^L_{Q}(s')
$$
where $V^L_Q(s')$ is derived from the Q-tables via SSE computation. The operator $\Omega$ is a $\gamma$-contraction in the sup-norm due to discounting and leader-favoring tie-breaking. Standard stochastic approximation guarantees convergence to fixed-point policies $x^*, q_i^*$.

## 5. Computational Complexity and Scalability

BSS-Q's storage and computation are dominated by:

- **Q-table Storage**: $|\Theta|$ Q-tables of size $|S|\times|A_L|\times\max_i|A^i_F|$.
- **Per-Step SSE Computation**: At each transition, a compact Bayesian Stackelberg normal-form (stage-)game of size $|A_L|\times\sum_i|A^i_F|$ must be solved. Exact SSE computation is NP-hard in the number of types and follower actions, but practical encoding as a MILP ($O(|A_L|\sum_i|A^i_F|)$) is tractable up to hundreds of actions, yielding solutions in milliseconds to seconds.

Compared to single-agent RL—with $O(|S||A_L|)$ storage and $O(1)$ policy improvement per-step—BSS-Q adds game-solving overhead per step. Empirically, BSS-Q is feasible for cybersecurity MTD domains where $|S|\lesssim 10^2$ and total attacker actions $\lesssim 10^3$.

## 6. Empirical Evaluation

BSS-Q was evaluated in two MTD scenarios within an OpenAI Gym-style environment, without prior knowledge of the transition or reward model.

**A. Web-Application Stack Defense**
- States: 4 configurations ($S = \{\text{language}\times\text{database}\}$).
- 3 attacker types: DB-expert (269 actions), ScriptKiddie (34), BlackHat (48), action sets derived from CVEs.
- Defender actions: 4 configurations; switching and attack costs from empirical latency and CVSS scores.

**B. Cloud-Network IDS Placement**
- States correspond to attack graph levels ($|S|=4$).
- Single attacker type ($O(10$–$50)$ actions per state).
- Defender actions: host-based or network-based IDS placement/removal.
- Rewards from CVSS; transitions from attack graph structure.

**Baselines**
- Uniform Random Strategy (URS).
- Bayesian EXP-Q (bandit approach ignoring strategic best responses).
- Nash-Q (computes Nash equilibria at each stage).
- State-agnostic “optimal” (S-OPT) from a two-stage BSG formulation.

**Metrics & Results**

| Scenario              | States | # Attacker Types | BSS-Q Performance        | Baselines Performance                | Planning Time                         |
|-----------------------|--------|------------------|--------------------------|--------------------------------------|---------------------------------------|
| Web-Application Stack | 4      | 3                | Converged near FI-SSE,   | URS, B-EXP-Q, S-OPT outperformed in  | BSS-Q: $\sim$150 s/episode            |
|                       |        |                  | eliminated cycles        | at least 2 states; SA-RL cycles      | B-EXP-Q: $\sim$220 s/episode          |
| Cloud-Network IDS     | 4      | 1                | Outperformed URS, EXP-Q  | Nash-Q matched/slighly exceeded,     | URS: $\lesssim$100 s/episode;         |
|                       |        |                  | in 2/3 non-terminal      | Nash-Q sensitive to equiv.           | Nash-Q failed to scale                |
|                       |        |                  | states, matched Nash-Q   |                                      |                                       |

BSS-Q established rapid convergence to SSE-like rewards, improved state-of-the-art MTD performance, and eliminated exploitable cycles characteristic of single-agent RL or myopic bandit-style baselines.

## 7. Significance and Application Scope

BSS-Q provides a theoretically justified, scalable solution for adaptive decision-making in sequential adversarial environments with incomplete information, specifically suited for cybersecurity MTD domains. It bridges MARL and game-theoretic learning for environments where the defender must account for type uncertainty and commit to strategies robust against strategic, best-responding attackers. The framework extends to settings lacking prior distributional knowledge over state transitions or rewards, addressing key challenges in practical security defense planning [2007.10457].

Source: https://www.emergentmind.com/topics/bayesian-strong-stackelberg-q-learning-bss-q