Decoupled Q-Chunking (2512.10926v1)

Abstract: Temporal-difference (TD) methods learn state and action values efficiently by bootstrapping from their own future value predictions, but such a self-bootstrapping mechanism is prone to bootstrapping bias, where the errors in the value targets accumulate across steps and result in biased value estimates. Recent work has proposed to use chunked critics, which estimate the value of short action sequences ("chunks") rather than individual actions, speeding up value backup. However, extracting policies from chunked critics is challenging: policies must output the entire action chunk open-loop, which can be sub-optimal for environments that require policy reactivity and also challenging to model especially when the chunk length grows. Our key insight is to decouple the chunk length of the critic from that of the policy, allowing the policy to operate over shorter action chunks. We propose a novel algorithm that achieves this by optimizing the policy against a distilled critic for partial action chunks, constructed by optimistically backing up from the original chunked critic to approximate the maximum value achievable when a partial action chunk is extended to a complete one. This design retains the benefits of multi-step value propagation while sidestepping both the open-loop sub-optimality and the difficulty of learning action chunking policies for long action chunks. We evaluate our method on challenging, long-horizon offline goal-conditioned tasks and show that it reliably outperforms prior methods. Code: github.com/ColinQiyangLi/dqc.

• The paper introduces DQC, decoupling the critic's large action chunk size from the policy's smaller one to mitigate bootstrapping bias in RL.
• It employs a theoretical analysis of open-loop consistency and a distilled partial chunk critic with best-of-N sampling for efficient value learning.
• Empirical results on OGBench show improved performance on long-horizon tasks, validating the method's theoretical and practical advantages.

Decoupled Q-Chunking: Theoretical and Empirical Advances in Temporal Abstraction for RL

Introduction and Motivation

Temporal-difference (TD) methods are a cornerstone of reinforcement learning (RL) for both online and offline regimes but are susceptible to compounding bootstrapping bias, especially apparent in long-horizon, sparse-reward tasks. Previous attempts to mitigate this bias at scale have included multi-step return backups and, recently, action chunking critics: critics that estimate the value of entire action sequences ("chunks") rather than individual actions. However, extracting performant policies from chunked critics is difficult since the policy must output entire action chunks in an open-loop fashion, exacerbating sample complexity and limiting adaptivity in reactive environments.

The authors introduce Decoupled Q-Chunking (DQC), whose core idea is to decouple the action chunk size of the critic from the chunk size of the policy. DQC proposes that the critic uses large action chunks for value learning while the policy operates with shorter (and thus more reactive) action chunks. The paper establishes a theory of action chunking in Q-learning, introduces the DQC algorithm, and demonstrates significant empirical advantage on the most challenging environments from OGBench.

Figure 1: Decoupled Q-chunking (DQC) decouples the critic's chunk size from the policy's, enabling efficient value learning with a large critic chunk while retaining tractable and reactive policy learning via short policy chunks; right: DQC surpasses baselines on six hardest OGBench tasks.

Theoretical Foundations of Q-Chunking

The paper thoroughly investigates the theoretical underpinnings of Q-learning with chunked critics. The value learning bias is formalized via the notion of open-loop consistency (OLC), which captures the distributional discrepancy between open-loop replay of action chunks and their occurrence within data. Two flavors—weak and strong OLC—are defined, bounding the value estimation bias and policy suboptimality, respectively.

Under weak OLC, the worst-case value estimation bias of a chunked critic, $\varepsilon_h$, is shown to scale as:

$\Theta(\varepsilon_h H\bar{H}),$

where $H=1/(1-\gamma)$ and $\bar{H}=1/(1-\gamma^h)$. This bias is unavoidable if the data distribution exhibits significant closed-loop feedback not reflected in open-loop execution.

More critically, the optimality gap between the learned action-chunking policy and the optimal closed-loop policy is shown to also scale with the OLC constant. The authors demonstrate both upper and lower bounds on this gap and, importantly, show that with only weak OLC there exist degenerate cases where the learned chunked policy can be arbitrarily suboptimal—a significant and contradictory result relative to prior intuition.

With strong OLC, the policy suboptimality of chunked Q-learning can be tightly bounded as:

$$V^\star(s) - V^\mathrm{ac}(s) \leq 3\varepsilon_h H\bar{H},$$

guaranteeing near-optimality. The theory further contrasts chunking with standard $n$-step backup: as data becomes more suboptimal, chunking Q-learning can provably outperform $n$-step methods, formalized as a function of the empirical suboptimality $\delta_n$.

Policy Reactivity and Policy Extraction

Extracting policies from chunked critics is the central practical difficulty—policies must produce open-loop action chunks, which is undesirable in environments that benefit from closed-loop adaptivity. The authors analyze the scenario where the learned chunked policy is instead executed in closed-loop: only the first action of each chunk is used, and the chunking policy is recomputed at each step. Theoretical results indicate that, under reasonable conditions (bounded optimality variability or absence of stochastic shortcuts), the resultant closed-loop policy remains near-optimal, with the optimality gap scaling at most with the open-loop consistency and variability of the data.

The DQC Algorithm

Building on the theoretical insights, the paper proposes Decoupled Q-Chunking (DQC), which realizes the policy-critic decoupling paradigm in practice. DQC trains a value critic $Q_\phi$ via temporally extended (chunked) TD backup with chunk size $h$, and a policy $\pi$ operating with a smaller chunk size $h_a \ll h$.

The keystone of DQC is a distilled partial chunk critic $Q^P_\psi$, which predicts the maximal Q-value attainable by optimally extending a given short action chunk. The policy $\pi(a_{t:t+h_a}|s_t)$ is then trained to maximize $Q^P_\psi(s_t, a_{t:t+h_a})$ via best-of-N sampling over a flow-matching–trained behavior prior $\pi_\beta$—bypassing the pathologies of direct chunked policy learning.

Figure 2: Comparison of DQC (decoupled chunk lengths) with strong Q-chunking and $n$-step baselines on long-horizon goal-conditioned tasks.

This design efficiently leverages the benefits of chunked critics (faster value propagation/blunted bootstrapping bias) while maintaining the tractability and reactivity of short-chunk policy learning.

Empirical Results

DQC is evaluated on the most challenging tasks of OGBench, encompassing dexterous manipulation, combinatorial puzzles, and high-dimensional locomotion. Its empirical advantage is unambiguous, with mean and confidence intervals consistently exceeding prior state-of-the-art methods, including $n$-step and direct chunking approaches as well as strong GCRL baselines.

Figure 3: DQC achieves superior mean aggregated score across six difficult OGBench environments.

Crucially, ablations demonstrate that training a separate distilled critic for partial chunks is necessary for strong performance (Figure 4), outperforming "naive" decoupling or mere $n$-step return methods.

Figure 4: DQC consistently matches or exceeds non-distilled ablations across multiple chunk configuration settings.

The analysis of hyperparameter sensitivity (Figure 5) confirms robustness to expectile/quantile loss settings and policy extraction details, with performance scaling primarily with batch size and the depth of best-of-N sampling.

Figure 5: DQC's success rates remain robust for optimistic backup hyperparameters; performance scales with batch size and number of samples in best-of-N extraction.

Batch size remains a decisive factor for hard tasks, as demonstrated in Figure 6.

Figure 6: DQC requires large batch sizes for maximal performance in the most complex long-horizon problems.

Practical and Theoretical Implications

Practical Impact: DQC establishes a viable and scalable approach to value and policy learning in the presence of temporal abstraction. This is directly relevant for long-horizon, goal-conditioned, and sparse-reward domains typical in robotics and complex planning. The design is both theoretically justified and empirically validated to outperform previous Q-learning variants in offline RL.

Theoretical Impact: The paper provides the first formal analysis of bootstrapping bias, suboptimality gaps, and policy extraction strategies in Q-learning with temporally abstracted (chunked) critics. The identification and tight bounding of the OLC condition and the demonstration that naive chunked policy extraction can be arbitrarily suboptimal constitute strong and potentially contradictory results with respect to prior expectations within hierarchical RL.

Figure 7: Representative long-horizon manipulation task (cube-triple) example from OGBench.

Prospects and Future Directions

As RL systems scale to more variable and compositional environments, state-conditional adaptation of chunk sizes and integration with richer hierarchical policies are natural progressions. The DQC paradigm can serve as a theoretical and practical foundation for future work on flexible and efficient temporal abstraction, hierarchical policy extraction, and principled long-horizon value estimation in both offline and online RL.

Conclusion

Decoupled Q-Chunking (DQC) advances the state of the art in temporal abstraction for RL by establishing both a rigorous theoretical analysis for action chunking Q-learning and an effective algorithm for learning with decoupled policy and critic chunk sizes. DQC demonstrates strong empirical advantages on difficult, long-horizon tasks and provides new foundations for future research in value-based temporal abstraction, bootstrapping bias reduction, and hierarchical control and policy learning.