---
title: Coarse-to-Fine Q-Network (CQN) in RL
url: https://www.emergentmind.com/topics/coarse-to-fine-q-network-cqn
type: topic
---

# Coarse-to-Fine Q-Network (CQN) in RL

The Coarse-to-Fine Q-Network (CQN), sometimes called Growing Q-Network (GQN), is a family of value-based reinforcement learning algorithms that solve continuous control problems by dynamically discretizing the action space at multiple resolutions. CQN leverages hierarchical or adaptive action discretization, decoupled Q-value architectures, and techniques including distributional Q-learning, behavior cloning regularization, and prioritized replay to achieve data efficiency, scalability, and strong empirical performance in both simulated and real-world robotic tasks. Recent work demonstrates that CQN and its variants outperform both static discretization and standard actor–critic baselines, especially in high-dimensional, sparse-reward, or long-horizon settings [2404.04253][2407.07787][2411.12155][2204.12471].

## 1. Adaptive and Hierarchical Action Discretization

CQN restructures control in continuous action spaces—the standard setting in robotics and physical simulation—into a sequence or schedule of discretization steps. The action space $A \subset \mathbb{R}^M$ is covered by a grid, with each dimension partitioned into $G$ bins at the highest resolution. At any stage, only a subgrid $A^g$ (with $g$ bins per dimension, $g < G$) is active; the agent chooses actions from this coarser mask and the Bellman backup maximizes only over this subgrid.

Adaptation of control resolution is achieved by:
- A fixed linear schedule: increment $g$ every $E_{\mathrm{total}}/(G - 1)$ episodes.
- An adaptive criterion: increment $g$ when the running mean return $\mu_{\mathrm{MA}}$ and standard deviation $\sigma_{\mathrm{MA}}$ fail to improve beyond a threshold $G_{\mathrm{thresh}} = (1.00-0.05\,\mathrm{sgn}(\mu_{\mathrm{MA}}))\,\mu_{\mathrm{MA}} + 0.90 \,\sigma_{\mathrm{MA}}$.

Alternative CQN frameworks use a hierarchical “zoom-in” progression: for each of $L$ discretization levels, each dimension is iteratively subdivided into $B$ bins, and the highest-valued subinterval is recursively refined. The process either finishes at the finest level or dynamically adapts to task learning progress [2404.04253][2407.07787][2411.12155].

## 2. Q-Learning Updates and Decoupled Value Decomposition

CQN algorithms perform temporal-difference (TD) learning with n-step targets and double Q-learning, using a target network ($\theta^{-}$):

$$
y_t = \sum_{k=0}^{n-1} \gamma^k r_{t+k} + \gamma^n Q_{\theta^-}\bigl(s_{t+n},\,\arg\max_{a'\in A^g} Q_\theta(s_{t+n},a') \bigr)
$$

To make training tractable in high action dimensions, decoupled Q-networks (DecQN) decompose the Q-function as:

$$
Q_\theta(s, a) = \frac{1}{M} \sum_{j=1}^M Q_\theta^j(s, a^j)
$$

$Q_\theta^j$ is a univariate head for the $j$-th action dimension, each outputting a vector of $G$ Q-values. At any epoch, only the first $g$ bins are unmasked. The policy is derived as a sum of independent utilities, providing scalability to $M \gg 1$ dimensions [2404.04253].

The CQN architecture also extends to action-sequence Q-values (CQN-AS): the critic outputs $Q$-values for a sequence (or block) of $K$ future actions, enabling richer temporal credit assignment and regularization [2411.12155].

## 3. Learning Objectives and Auxiliary Losses

CQN combines several objective components:

- n-step TD loss (with distributional or standard regression): per minibatch, minimize $\mathrm{Huber}(y - Q_\theta(s, a))$ or its distributional equivalent.
- Behavior cloning (BC) loss on demonstration data: a large-margin hinge or MSE term that encourages the Q-value of expert actions to surpass other actions by a margin. For example,

$$
L_{\mathrm{BC}}(\theta) = \sum_{l=1}^L \sum_{n=1}^N \max_{j \neq j^*} \big[ Q_{l,n}(s_t, a_{l,n,j}) + m - Q_{l,n}(s_t, a_{l,n,j^*}) \big]_+
$$

The overall loss is weighted,

$$
L(\theta) = w_{RL} L_{RL}(\theta) + w_{BC} L_{BC}(\theta)
$$

with typical weights $w_{BC}=1.0$, $w_{RL}=0.1$ [2407.07787].

Other enhancements include prioritized experience replay, target network Polyak averaging (e.g., $\tau=0.02$), and exploration via $\epsilon$-greedy or policy noise.

## 4. Network Architecture and Implementation

The canonical CQN employs:

- A shared multi-layer perceptron backbone (2 layers, 512 ReLU/SiLU units, LayerNorm).
- Per-action-dimension output heads, linear over $G$ bins.
- Input: concatenation of visual encoded features, proprioception, previous chosen actions, one-hot discretization level encoding.
- Typical settings: $L=3$ levels, $B=3$–$5$ bins, total action dimensions $M=7$–$39$ [2404.04253][2407.07787].

For CQN-AS, block inputs at each step $k$ in the sequence comprise a GRU-RNN embedding to share temporal context across the action sequence [2411.12155].

Replay buffers mix online experiences and expert demonstrations; training augments the demo buffer with successful online episodes. Gradients are updated via Adam or AdamW ($\mathrm{lr}=3 \times 10^{-4}$ or $5 \times 10^{-5}$), batch size $B=128$–$256$, target network update every 1k steps.

## 5. Empirical Performance and Comparative Evaluation

CQN and its variants have been extensively evaluated in:

- DeepMind Control Suite (continuous control, up to $M=38$): Growing Q-Network outperforms static DecQN discretizations, matching continuous D4PG and DMPO. Adaptive scheduling converges faster and more stably [2404.04253].
- RLBench and real-world robotic manipulation: CQN attains $\sim$80% success in $2 \times 10^{4}$ steps, exceeding DrQ-v2+ and behavior cloning-only baselines. With sufficient demos and distributional critics, CQN solves 4 real-world tasks in $600$–$1000$ steps (5–8 minutes) [2407.07787].
- MetaWorld and MyoSuite (biomechanical, $M=39$): Larger MLPs and coarse-to-fine schedules enable scaling to very high dimensions. GQN-$2\rightarrow65$ achieves high final success [2404.04253].
- Hard, sparse, or long-horizon tasks: CQN-AS surpasses ACT (action-chunking BC), vanilla CQN (no-sequence), and SAC on BiGym and RLBench, particularly where long sequences or global–local strategy is required [2411.12155].
- Ablations confirm that CQN’s adaptive/coarse-to-fine schedule outperforms static binning, and that BC loss and distributional critics are critical for stability and performance.

## 6. Theoretical Rationale and Insights

Coarse discretization (e.g., two bins—bang-bang control) incentivizes wide exploration by covering large regions of action space, evading local optima introduced by action penalties. Progressive refinement (“coarse-to-fine”) introduces fine granularity only as the task requires, yielding precision and actuation smoothness.

Decoupled (dimensionwise) Q-learning transforms the exponential joint space $|\mathcal{A}|^M$ into linearly-scaling heads, permitting tractable learning and inference in high dimensions [2404.04253]. Adaptive schedules using performance-based thresholds prevent premature over-refinement, dynamically trading off exploration and exploitation.

In action-sequence CQN (CQN-AS), sequence prediction structures the value function for multi-step credit assignment and regularizes away mode collapse or excessive temporal variance [2411.12155].

## 7. Representative Algorithm and Configuration Table

| Variant         | Key Ideas                          | Empirical Setting                |
|-----------------|-----------------------------------|----------------------------------|
| GQN (CQN)       | Grow active bin set; DecQN heads  | DeepMind Control Suite, M=38     |
| CQN [2407.07787]| Hierarchical “zoom-in” via L,B    | RLBench/Real-World, L=3, B=5     |
| CQN-AS          | Sequence value prediction          | BiGym (K=16), RLBench (K=4)      |

Each variant implements hierarchical or adaptive bin discretization, dimensionwise decoupling, and multi-task deployment with shared architecture and replay. Hyperparameters including levels $L$, bins $B$, n-step return, replay weighting, and network size are selected empirically and tuned per benchmark.

## 8. References

- “Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution” [2404.04253]
- “Continuous Control with Coarse-to-fine Reinforcement Learning” [2407.07787]
- “Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Robot Learning” [2411.12155]
- “Coarse-to-fine Q-attention with Tree Expansion” [2204.12471]

Source: https://www.emergentmind.com/topics/coarse-to-fine-q-network-cqn