Accelerating Q-learning through Efficient Value-Sharing across Actions

Abstract: Action-values are foundational to many control algorithms such as Q-learning. Therefore learning action-values efficiently is central to reinforcement learning (RL). However, learning them can be slow, requiring many updates to move values from their initialization, typically near zero, to their true values, which may be far from zero. Moreover, action-value learning algorithms typically update each state-action pair independently, without learning shared value structure across actions within a state. In this paper, we address these inefficiencies by introducing the mean-expansion layer, which accelerates action-value learning by sharing values across actions within a state and by changing the problem from directly learning potentially large action-values to learning a lower-norm representation of them. In deep RL, this layer can be applied as a parameter-free addition to Q-network architectures without altering the underlying algorithm. Applied to deep Q-networks and implicit quantile networks, it improves aggregate performance across 57 Atari games while increasing action gaps and dramatically reducing value overestimation.

• The paper introduces the mean-expansion layer to enable efficient value sharing across actions, thereby accelerating credit propagation and reducing learning delays.
• It reparameterizes the action-value vector to distribute TD-error updates uniformly, which improves sample efficiency in both deep and tabular Q-learning.
• Empirical results on Atari-57 and gridworld benchmarks demonstrate significant performance gains, reduced overestimation, and increased relative action gaps compared to traditional methods.

Accelerating Q-learning via Mean-Expansion: A Technical Perspective

Introduction

The paper "Accelerating Q-learning through Efficient Value-Sharing across Actions" (2606.29806) introduces the mean-expansion layer (ME layer) as a mechanism to address two entrenched inefficiencies in value-based reinforcement learning: the slow propagation of large-magnitude Q-values from initialization and the lack of intra-state value sharing. This approach reframes the parameterization of action-value vectors in both tabular and deep RL regimes, facilitating improved sample efficiency, reduced overestimation, and larger action gaps, all without additional learnable parameters or architectural complexity.

Value-Sharing and Norm Minimization

Traditional Q-learning architectures independently estimate action-values, $\mathbf{q} \in \mathbb{R}^n$, for each action in a given state. When reward or value scales are large, the network must migrate from an initialization near zero to potentially high norm targets, incurring a significant learning delay. Additionally, such architectures do not leverage the statistical regularity that, within a state, action-values are typically highly correlated (often with similar sign and magnitude).

The mean-expansion layer performs a linear, invertible transformation, projecting any vector $\mathbf{z}$ onto the all-ones vector, scaling this mean projection, and reconstituting the action-value vector with an implicit shared baseline. This mechanism avoids explicit baseline parameterization, as required by dueling architectures, and instead propagates the baseline through the parameter-free operation. Formally, for residual input $\mathbf{z}$ and scaling parameter $k$, the output is:

$$\mathbf{q} = \mathbf{z} + \frac{k}{n}\mathbf{1}\mathbf{1}^\top\mathbf{z}$$

where $n$ is the number of actions. The resultant representation compresses the overall norm (relative to direct value parameterization), accelerating gradient-driven training.

Figure 1: The mean-expansion layer projects the input vector onto the mean direction, scales this mean by $k$, and reconstructs the output by summing the residual and implicit baseline components.

Theoretical analysis shows that for any nonzero mean $\mu$ of the input vector, choosing $k \in (0,n]$ produces strictly norm-reducing decompositions. The value of $k$ parameterizes the degree of mean scaling, allowing practitioners to balance optimization condition number and value-sharing intensity.

Implementation in Deep and Tabular RL

When applied to deep Q-networks (DQN) and implicit quantile networks (IQN), the mean-expansion layer is inserted as a final operation preceding the value output. All preceding network layers operate in the residual space, simplifying the value output space and ensuring the main model capacity is devoted to modeling relative differences, not the common offset.

In tabular Q-learning, storing only residuals and reconstructing Q-values with the mean-expansion layer yields a simple yet effective update rule. All entries in the residual vector are incremented proportionally to the TD-error and $\mathbf{z}$0, efficiently sharing update credit across actions.

Empirical Results

Gridworld: Sample Efficiency

In a pedagogical stochastic gridworld, the implicit-baseline tabular Q-learning (IBQ) consistently achieves faster learning compared to traditional Q-learning, especially in low-sample regimes, with over 20% more episodes completed at 1k timesteps for most $\mathbf{z}$1. As the environment is fully learned, this advantage attenuates.

Figure 2: IBQ with higher $\mathbf{z}$2 values yields a marked sample efficiency improvement over Q-learning in a 5x5 gridworld, especially early in learning.

Atari-57: Sample Efficiency, Performance, and Scaling

On the Atari-57 suite, mean-expansion enables significant improvements in learning speed and aggregate interquartile mean of human-normalized scores. IB-DQN($\mathbf{z}$3) matches the best DQN score with less than 40% of the total training time and generally surpasses both standard and dueling DQN. Extension to IQN (IB-IQN) also yields systematic aggregate performance gains. Notably, the empirical advantage is robust to optimizer and loss function choices, as confirmed via RMSprop-Huber runs.

Figure 3: Mean-expansion, as denoted by dashed curves, improves the interquartile mean of human-normalized score across 57 Atari games, with consistently better area-under-curve metrics.

Figure 4: Mean score curves show that IB-DQN and IB-IQN achieve superior or earlier performance relative to standard baselines in per-game longitudinal evaluation.

Overestimation and Action Gap Analysis

IB-DQN systematically reduces value overestimation relative to DQN. For all games, the mean-expansion layer reduces the average overestimation area under the curve during training. In many cases, overestimation is nearly eliminated or even inverted (i.e., to slight underestimation), a highly desirable property for reducing instability in deep RL.

Moreover, IB-DQN increases relative action gap—the normalized difference between top two action-values—in almost all environments. This supports more stable policy selection since actions are less likely to switch due to minor fluctuations in Q-values.

Figure 5: Across all games, IB-DQN reduces overestimation (left) and increases the relative action gap (right) relative to DQN (values clipped at 0.01 for clarity).

Sensitivity to the Mean-Scaling Parameter $\mathbf{z}$4

The effect of $\mathbf{z}$5 was assessed across several domains. Small to moderate values of $\mathbf{z}$6 outperform $\mathbf{z}$7 (i.e., no mean-expansion), while excessively large $\mathbf{z}$8 induces numerical instability and performance degradation. The results empirically validate theoretical claims regarding the tradeoff between norm reduction and instability.

Figure 6: Sensitivity analysis shows performance gains across a range of $\mathbf{z}$9 values, with degradation as $\mathbf{z}$0 grows large, especially on LunarLander-v3.

Comparison to Dueling Architectures and Prior Work

While dueling DQN and regularized dueling Q-learning (RDQ) decompose value functions into baseline and residual components, only the mean-expansion layer achieves this implicitly with no extra learnable parameters or auxiliary streams. The dueling baseline stream is strictly equivalent to an explicit mean-residual decomposition; mean-expansion instead leverages a linear, differentiable operation. RDQ's gain comes at the cost of architectural and loss-function complexity.

Theoretical and Practical Implications

The mean-expansion approach reframes the value function learning problem, particularly in deep RL, as learning relative differences instead of large absolute returns. This enables:

• Accelerated credit propagation: TD-error updates are broadcast across all actions, accelerating convergence of shared baseline components and reducing learning time in early regimes.
• Norm minimization: Lower norms facilitate faster optimization and mitigate instabilities related to gradient explosion or overestimation bias.
• Overestimation mitigation and policy stability: Empirically, it produces lower overestimation and better-separated action values, reducing the likelihood of policy flapping and unreliable Q-targets.
• Parameter-free augmentation: The layer can be universally deployed in DQN-style architectures with zero increase in learnable parameters and no algorithmic modifications.

However, the efficacy of the layer degrades for extreme $\mathbf{z}$1 in very large action spaces, and there is an open gap regarding extension to continuous control and a full understanding of long-term impacts on convergence and bias.

Prospects for Future Research

Key open questions include: precise characterization of the learning path alteration in overparameterized networks, formal convergence proofs in the presence of deep non-linear function approximation, mechanisms by which mean-expansion reduces overestimation so robustly, interaction with recent advances in distributional or offline RL, and methods for generalizing to continuous-valued action spaces.

Conclusion

The mean-expansion layer provides an efficient, theoretically sound, and practically simple approach to value sharing in Q-learning. By recasting the value learning task in a lower-norm, baseline-residual space without architectural overhead, it achieves better sample efficiency, lower overestimation, and increased policy stability in standard RL benchmarks like Atari-57. The introduced abstraction provides a foundation for future advances in value-based RL methods with shared structure across the action space.