---
title: Count-Based Soft Q-Learning (CBSQL)
url: https://www.emergentmind.com/topics/count-based-soft-q-learning-cbsql
type: topic
---

# Count-Based Soft Q-Learning (CBSQL)

Count-Based Soft Q-Learning (CBSQL) is an algorithmic modification of Soft Q-Learning (SQL) in the maximum entropy reinforcement learning (MaxEnt RL) paradigm. CBSQL replaces the conventional constant inverse-temperature parameter with a state-dependent schedule that adapts dynamically according to state visit counts or density-model pseudo-counts, resulting in an adaptive tradeoff between exploration and exploitation throughout the learning process [2111.14204].

## 1. MaxEnt RL, Soft Q-Learning, and the β Parameter

Traditional reinforcement learning optimizes the expected discounted reward:
$$
\pi^* = \arg\max_\pi\, \mathbb{E}_{s\sim p_\pi^\gamma}\left[\mathbb{E}_{a\sim\pi(\cdot|s)}[r(s,a)]\right]
$$
Maximum entropy RL augments this with an entropy bonus, formulated as:
$$
\pi^* = \arg\max_\pi\, \mathbb{E}_{s\sim p_\pi^\gamma}\left[\,\mathbb{E}_{a\sim\pi(\cdot|s)}[r(s,a)] + \frac{1}{\beta} H[\pi(\cdot|s)]\,\right]
$$
where $H[\pi(\cdot|s)] = -\sum_a \pi(a|s) \log \pi(a|s)$, and $\beta > 0$ serves as an inverse-temperature. As $\beta \rightarrow \infty$, the entropy term vanishes and standard RL is recovered; $\beta \rightarrow 0$ yields a uniform policy.

SQL applies a “soft” Bellman operator:
$$
\mathcal{B}_\beta\, Q(s,a) = r(s,a) + \gamma\, \mathbb{E}_{s'}\left[\,\frac{1}{\beta} \log \sum_{a'} \exp(\beta Q(s',a'))\,\right]
$$
The $\log$–sum–exp is the mellowmax operator, mediating between greedy ($\beta \to \infty$) and random ($\beta \to 0$) value backups, effecting an entropy-regularized update.

## 2. Motivation for State-Dependent Temperature Scheduling

Empirical and theoretical considerations indicate that the optimal entropy-regularization strategy is nonstationary and state-dependent. Early in training, Q-estimates are noisy; a high temperature (low $\beta$) promotes valuable exploration. As learning progresses and Q-estimates for frequently visited states become accurate, reducing the temperature (raising $\beta$) shrinks policy entropy, encouraging exploitation. This “confidence” in Q-values is inherently state-specific, tracking the effective evidence available. Fixed or globally-annealed $\beta$ cannot properly capture this local learning progress, leading to undesirable tradeoffs in nonstationary or heterogeneous domains [2111.14204].

## 3. Count-Based Temperature Scheduling: Derivation and Properties

CBSQL introduces a statewise inverse-temperature parameter. With $n_k(s)$ the effective visit-count for state $s$ at update $k$, the statewise parameter is
$$
\beta(s) = \kappa \cdot n_k(s)
$$
with $\kappa > 0$ a tunable constant. For tabular domains, $n_k(s)$ is the Q-update count for $s$. In large or continuous state spaces, CBSQL employs pseudo-counts from a density model $\rho$:
$$
n_k(s) = \frac{\rho_k(s)\,(1-\rho'_k(s))}{\rho'_k(s) - \rho_k(s)}
$$
where $\rho_k(s)$ is the model probability after $k$ updates and $\rho'_k(s)$ after an additional update on $s$. This pseudo-count increases monotonically and behaves like a visit-count.

The corresponding temperature parameter is $\tau(s) = 1/\beta(s) = 1/(\kappa n_k(s))$. As $k \to \infty$ for recurrent states, $n_k(s) \to \infty$, $\beta(s) \to \infty$, and $\tau(s) \to 0$, yielding greedy backups in well-known states. For novel or rarely visited states, the pseudo-count is small, maintaining high temperature and exploration.

## 4. Algorithmic Formulation and Soft Bellman Backup

With adaptive temperatures, the soft Bellman backup becomes:
$$
\mathcal{B}_{\tau} Q(s,a) = r(s,a) + \gamma\, \mathbb{E}_{s'|s,a}\left[\,\tau(s') \log \sum_{a'} \exp\left(\frac{Q(s',a')}{\tau(s')}\right)\,\right]
$$
or equivalently in terms of $\beta(s')$:
$$
\mathcal{B}_\beta Q(s,a) = r(s,a) + \gamma\, \mathbb{E}_{s'}\left[\,\frac{1}{\beta(s')} \log \sum_{a'} \exp\big(\beta(s') Q(s',a')\big)\,\right]
$$

Pseudocode outlines the integration of count-based temperature scheduling into deep SQL, including experience replay, density model updates, and periodic target network synchronization. The key differentiators are lines introducing (pseudo-)count calculation and the corresponding update to $\beta(s)$ before constructing the soft Q-target.

## 5. Hyperparameters, Theoretical Considerations, and Practicalities

Major hyperparameters include:

- $\kappa$ (temperature-growth coefficient): Controls the rate at which $\beta(s)$ increases. Typical working value is $\kappa \approx 0.01$.
- Density model parameters for pseudo-count computation (e.g., context-tree switching (CTS) $\alpha$s, context depths), which regulate how rapidly pseudo-counts accumulate.
- Standard deep RL settings (learning rate $\eta$, replay buffer size, batch size, discount factor $\gamma$, $\epsilon$-greedy schedule, network architecture) follow DQN/SQL practices.

For any fixed $\beta(s)$, the soft Bellman operator is a contraction (in sup-norm), ensuring a unique fixed point. As statewise $\beta(s)$ increases over time, the CBSQL operator interpolates between an entropy-dominated regime and the pure max-operator, resulting in stable early exploration and convergence to greedy optimality for recurrent states [2111.14204].

## 6. Empirical Evaluations and Comparative Performance

CBSQL was assessed in both tabular and deep RL settings:

**Noisy Chain Toy Domain**:
- Five states linearly arranged, two actions per state, stochastic rewards.
- Only the “go right” action at the terminal state yields positive reward.
- CBSQL with true counts converged more rapidly and reliably to near-optimal policy (by ≈100 episodes) than all fixed-$\beta$ SQL baselines and standard Q-learning.

**Atari 2600 (Deep RL)**:
- Six games: Breakout, Freeway, Pong, Q*bert, Seaquest, Space Invaders.
- Preprocessing: grayscale 84×84 images, four-frame stack, reward clipping $[-1,1]$, $\gamma=0.99$.
- Training: $3\times 10^6$ frames, $\epsilon$-greedy annealed to 0.1, Adam optimizer, replay buffer size $10^6$, batch size 32.
- Baselines: DQN, SQL with $\beta=100$ and $\beta=1000$.

Summary performance (mean $\pm$ std over three seeds; metric: average reward over last 100 test episodes):

| Game        | DQN          | SQL(100)     | SQL(1000)     | CBSQL        |
|-------------|--------------|--------------|---------------|--------------|
| Breakout    | 5.9 ± 5.9    | 5.9 ± 4.5    | 5.1 ± 4.7     | 8.2 ± 6.1    |
| Freeway     | 21.0 ± 1.5   | 14.6 ± 8.5   | 22.6 ± 4.7    | 25.8 ± 4.9   |
| Pong        | 1.9 ± 2.6    | 17.8 ± 2.2   | 16.3 ± 2.7    | 17.6 ± 2.0   |
| Q*bert      | 568.4 ± 1101 | 828 ± 1412   | 564.5 ± 1098  | 875.3 ± 1255 |
| Seaquest    | 13.5 ± 24.1  | 4.0 ± 60.2   | 17.2 ± 24.0   | 84.6 ± 60.2  |
| SpaceInv    | 132.7 ± 113  | 158.9 ± 128.5| 132.3 ± 118.4 | 138.9 ± 112.8|

CBSQL displayed consistent improvement over both SQL (with fixed $\beta$) and DQN baselines. Integration with Rainbow’s extensions (double-DQN, prioritized replay, dueling nets, noisy nets, distributional, multi-step) yielded further performance increases (e.g., Breakout: CBSQL+Rainbow 39.9 vs Rainbow-DQN 10.2 after 500K frames).

## 7. Empirical Observations: Ablation, Stability, and Exploration-Exploitation Tradeoff

Temperature as a function of visit-count decays as $\tau(s)=1/(\kappa n)$, providing high initial exploration ($\tau(s)\to\infty$ as $n\to0$) and automatic decay as learning accrues. Empirical learning curves indicate that constant $\beta$ values set too high impede early learning, while CBSQL’s adaptive scheme finds an effective balance. Early high $\tau(s)$ stabilizes soft Q-updates by tempering the influence of noisy value estimates, and later low $\tau(s)$ focuses updates on exploitation. CBSQL stabilizes learning trajectories while eliminating the need for extra schedule tuning or domain-specific parameters [2111.14204].

Source: https://www.emergentmind.com/topics/count-based-soft-q-learning-cbsql