Coarse-to-Fine Q-Network (CQN) in RL

• Coarse-to-Fine Q-Network (CQN) is a reinforcement learning method that uses adaptive hierarchical action discretization to solve continuous control tasks.
• It employs decoupled Q-learning with dimension-wise value decomposition and n-step temporal-difference updates to manage high-dimensional continuous spaces.
• CQN integrates behavior cloning, prioritized replay, and sequence-based value prediction to enhance data efficiency and achieve robust empirical performance.

The Coarse-to-Fine Q-Network (CQN), sometimes called Growing Q-Network (GQN), is a family of value-based reinforcement learning algorithms that solve continuous control problems by dynamically discretizing the action space at multiple resolutions. CQN leverages hierarchical or adaptive action discretization, decoupled Q-value architectures, and techniques including distributional Q-learning, behavior cloning regularization, and prioritized replay to achieve data efficiency, scalability, and strong empirical performance in both simulated and real-world robotic tasks. Recent work demonstrates that CQN and its variants outperform both static discretization and standard actor–critic baselines, especially in high-dimensional, sparse-reward, or long-horizon settings (Seyde et al., 2024, Seo et al., 2024, Seo et al., 2024, James et al., 2022).

1. Adaptive and Hierarchical Action Discretization

CQN restructures control in continuous action spaces—the standard setting in robotics and physical simulation—into a sequence or schedule of discretization steps. The action space $A \subset \mathbb{R}^M$ is covered by a grid, with each dimension partitioned into $G$ bins at the highest resolution. At any stage, only a subgrid $A^g$ (with $g$ bins per dimension, $g < G$) is active; the agent chooses actions from this coarser mask and the Bellman backup maximizes only over this subgrid.

Adaptation of control resolution is achieved by:

• A fixed linear schedule: increment $g$ every $E_{\mathrm{total}}/(G - 1)$ episodes.
• An adaptive criterion: increment $g$ when the running mean return $\mu_{\mathrm{MA}}$ and standard deviation $\sigma_{\mathrm{MA}}$ fail to improve beyond a threshold $$G_{\mathrm{thresh}} = (1.00-0.05\,\mathrm{sgn}(\mu_{\mathrm{MA}}))\,\mu_{\mathrm{MA}} + 0.90 \,\sigma_{\mathrm{MA}}$$.

Alternative CQN frameworks use a hierarchical “zoom-in” progression: for each of $L$ discretization levels, each dimension is iteratively subdivided into $B$ bins, and the highest-valued subinterval is recursively refined. The process either finishes at the finest level or dynamically adapts to task learning progress (Seyde et al., 2024, Seo et al., 2024, Seo et al., 2024).

2. Q-Learning Updates and Decoupled Value Decomposition

CQN algorithms perform temporal-difference (TD) learning with n-step targets and double Q-learning, using a target network ($\theta^{-}$):

$$y_t = \sum_{k=0}^{n-1} \gamma^k r_{t+k} + \gamma^n Q_{\theta^-}\bigl(s_{t+n},\,\arg\max_{a'\in A^g} Q_\theta(s_{t+n},a') \bigr)$$

To make training tractable in high action dimensions, decoupled Q-networks (DecQN) decompose the Q-function as:

$$Q_\theta(s, a) = \frac{1}{M} \sum_{j=1}^M Q_\theta^j(s, a^j)$$

$Q_\theta^j$ is a univariate head for the $j$-th action dimension, each outputting a vector of $G$ Q-values. At any epoch, only the first $g$ bins are unmasked. The policy is derived as a sum of independent utilities, providing scalability to $M \gg 1$ dimensions (Seyde et al., 2024).

The CQN architecture also extends to action-sequence Q-values (CQN-AS): the critic outputs $Q$-values for a sequence (or block) of $K$ future actions, enabling richer temporal credit assignment and regularization (Seo et al., 2024).

3. Learning Objectives and Auxiliary Losses

CQN combines several objective components:

• n-step TD loss (with distributional or standard regression): per minibatch, minimize $\mathrm{Huber}(y - Q_\theta(s, a))$ or its distributional equivalent.
• Behavior cloning (BC) loss on demonstration data: a large-margin hinge or MSE term that encourages the Q-value of expert actions to surpass other actions by a margin. For example,

$$L_{\mathrm{BC}}(\theta) = \sum_{l=1}^L \sum_{n=1}^N \max_{j \neq j^*} \big[ Q_{l,n}(s_t, a_{l,n,j}) + m - Q_{l,n}(s_t, a_{l,n,j^*}) \big]_+$$

The overall loss is weighted,

$$L(\theta) = w_{RL} L_{RL}(\theta) + w_{BC} L_{BC}(\theta)$$

with typical weights $w_{BC}=1.0$, $w_{RL}=0.1$ (Seo et al., 2024).

Other enhancements include prioritized experience replay, target network Polyak averaging (e.g., $\tau=0.02$), and exploration via $\epsilon$-greedy or policy noise.

4. Network Architecture and Implementation

The canonical CQN employs:

• A shared multi-layer perceptron backbone (2 layers, 512 ReLU/SiLU units, LayerNorm).
• Per-action-dimension output heads, linear over $G$ bins.
• Input: concatenation of visual encoded features, proprioception, previous chosen actions, one-hot discretization level encoding.
• Typical settings: $L=3$ levels, $B=3$–$5$ bins, total action dimensions $M=7$–$39$ (Seyde et al., 2024, Seo et al., 2024).

For CQN-AS, block inputs at each step $k$ in the sequence comprise a GRU-RNN embedding to share temporal context across the action sequence (Seo et al., 2024).

Replay buffers mix online experiences and expert demonstrations; training augments the demo buffer with successful online episodes. Gradients are updated via Adam or AdamW ($\mathrm{lr}=3 \times 10^{-4}$ or $5 \times 10^{-5}$), batch size $B=128$–$256$, target network update every 1k steps.

5. Empirical Performance and Comparative Evaluation

CQN and its variants have been extensively evaluated in:

• DeepMind Control Suite (continuous control, up to $M=38$): Growing Q-Network outperforms static DecQN discretizations, matching continuous D4PG and DMPO. Adaptive scheduling converges faster and more stably (Seyde et al., 2024).
• RLBench and real-world robotic manipulation: CQN attains $\sim$80% success in $2 \times 10^{4}$ steps, exceeding DrQ-v2+ and behavior cloning-only baselines. With sufficient demos and distributional critics, CQN solves 4 real-world tasks in $600$–$1000$ steps (5–8 minutes) (Seo et al., 2024).
• MetaWorld and MyoSuite (biomechanical, $M=39$): Larger MLPs and coarse-to-fine schedules enable scaling to very high dimensions. GQN-$2\rightarrow65$ achieves high final success (Seyde et al., 2024).
• Hard, sparse, or long-horizon tasks: CQN-AS surpasses ACT (action-chunking BC), vanilla CQN (no-sequence), and SAC on BiGym and RLBench, particularly where long sequences or global–local strategy is required (Seo et al., 2024).
• Ablations confirm that CQN’s adaptive/coarse-to-fine schedule outperforms static binning, and that BC loss and distributional critics are critical for stability and performance.

6. Theoretical Rationale and Insights

Coarse discretization (e.g., two bins—bang-bang control) incentivizes wide exploration by covering large regions of action space, evading local optima introduced by action penalties. Progressive refinement (“coarse-to-fine”) introduces fine granularity only as the task requires, yielding precision and actuation smoothness.

Decoupled (dimensionwise) Q-learning transforms the exponential joint space $|\mathcal{A}|^M$ into linearly-scaling heads, permitting tractable learning and inference in high dimensions (Seyde et al., 2024). Adaptive schedules using performance-based thresholds prevent premature over-refinement, dynamically trading off exploration and exploitation.

In action-sequence CQN (CQN-AS), sequence prediction structures the value function for multi-step credit assignment and regularizes away mode collapse or excessive temporal variance (Seo et al., 2024).

7. Representative Algorithm and Configuration Table

Variant	Key Ideas	Empirical Setting
GQN (CQN)	Grow active bin set; DecQN heads	DeepMind Control Suite, M=38
CQN (Seo et al., 2024)	Hierarchical “zoom-in” via L,B	RLBench/Real-World, L=3, B=5
CQN-AS	Sequence value prediction	BiGym (K=16), RLBench (K=4)

Each variant implements hierarchical or adaptive bin discretization, dimensionwise decoupling, and multi-task deployment with shared architecture and replay. Hyperparameters including levels $L$, bins $B$, n-step return, replay weighting, and network size are selected empirically and tuned per benchmark.

• “Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution” (Seyde et al., 2024)
• “Continuous Control with Coarse-to-fine Reinforcement Learning” (Seo et al., 2024)
• “Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Robot Learning” (Seo et al., 2024)
• “Coarse-to-fine Q-attention with Tree Expansion” (James et al., 2022)