Maxmin Q-learning: Ensemble Bias Correction

• Maxmin Q-learning is an ensemble-based RL method that mitigates overestimation by taking the minimum over multiple Q-value estimates.
• It employs a tunable bias-variance trade-off, reducing variance while controlling underestimation through adjustable ensemble size.
• Proven effective in noisy, stochastic environments, it enhances training stability and outperforms classical Q-learning variants.

Maxmin Q-learning is an ensemble-based reinforcement learning (RL) algorithm developed to address the overestimation bias intrinsic to classical Q-learning and its deep variants. By leveraging multiple independent estimators and bootstrapping from their minimum, Maxmin Q-learning provides tunable bias control, reduced variance in target estimation, and improved stability in training, especially in stochastic and high-noise environments (Lan et al., 2020, Sheikh et al., 2020, Lee et al., 2023). The algorithm is foundational to a family of modern RL approaches employing ensemble statistics for bias correction.

1. Motivation: Overestimation Bias and Ensemble Correction

Classical Q-learning, and by extension DQN, forms its Bellman target as

$Y_t = R_t + \gamma \max_{a'} Q(s_{t+1}, a')$

The use of the $\max$ operator over noisy Q-value estimates induces a systematic overestimation bias, since the maximum is more likely to pick overestimated values. This max-operator bias accumulates through function approximation and correlates with learning instability and suboptimal policy selection (Lan et al., 2020, Sheikh et al., 2020). Maxmin Q-learning counters this by structuring the target to take the minimum across several Q-estimators, thereby systematically introducing a conservative (underestimated) correction.

2. Algorithmic Structure and Update Rule

Maxmin Q-learning maintains an ensemble of $N$ Q-function approximators, $\{Q^1, ..., Q^N\}$, each parameterized independently (either as tables in the tabular case or as neural networks in deep RL). For any state-action pair $(s, a)$, the ensemble computes $N$ distinct values. The minimum across the ensemble is used to form the backup target. The update steps are as follows (Lan et al., 2020, Sheikh et al., 2020, Lee et al., 2023):

1. Sample a transition $(s_t, a_t, r_t, s_{t+1})$.
2. Compute ensemble values $Q^i(s_{t+1}, a'), \; i = 1, ..., N$ for all $a' \in \mathcal{A}$.
3. Define

$Q^{\min}(s, a) = \min_{i=1,...,N} Q^i(s, a)$

1. Compute the target:

$\max$0

2. For each $\max$1 (or for a randomly sampled $\max$2), update:

$\max$3

In deep RL, this update is performed over minibatches and with target networks as in DQN (Sheikh et al., 2020).

3. Bias-Variance Tradeoff and Order Statistic Analysis

The core effect of the min-ensemble target is to systematically reduce overestimation by making positive errors less likely to dominate the Bellman backup. If estimation errors are independent among the $\max$4 members, the minimum tends to amplify underestimation. Order statistic theory provides quantitative characterization (Lan et al., 2020):

• Bias: For $\max$5 actions and $\max$6 estimators, if each $\max$7 is perturbed by i.i.d. zero-mean noise, the expected bias of $\max$8 is negative and increases in magnitude with $\max$9. As $N$0, the target becomes maximally pessimistic.
• Variance: The variance of the min-ensemble estimator decreases with $N$1:

$N$2

where $N$3 quantifies the noise range. This is strictly smaller than single-estimator Q learning for $N$4, even accounting for divided samples.

This provides a tunable bias-variance mechanism: increasing $N$5 interpolates between optimistic (standard Q-learning, $N$6) and highly conservative (large $N$7). The parameter $N$8 can thus be tuned to achieve approximately unbiased estimation ($N$9).

4. Theoretical Properties and Convergence

Maxmin Q-learning fits into the Generalized Q-learning framework, which leverages contraction mappings in the (weighted) sup norm to guarantee almost sure convergence in the tabular setting (Lan et al., 2020, Lee et al., 2023). Specifically:

• The update operator assimilating the min-ensemble over Q-functions is a contraction if the underlying MDP is finite and discount $\{Q^1, ..., Q^N\}$0.
• Step-sizes must satisfy usual Robbins-Monro conditions.
• Under sufficient exploration, every $\{Q^1, ..., Q^N\}$1 converges to the unique fixed point.
• This extends to the asynchronous variant where only one Q-function is updated per step (Lee et al., 2023).

This structure also reveals that Maxmin Q-learning is equivalent to minimax Q-learning on a two-player zero-sum Markov game with a dummy adversarial player selecting the worst among $\{Q^1, ..., Q^N\}$2 critics (Lee et al., 2023).

5. Relationship to Other Ensemble and Bias-Correction Algorithms

Maxmin Q-learning stands in direct contrast to the following ensemble-based methods:

Algorithm	Target Structure	Bias Direction
Q-learning	$\{Q^1, ..., Q^N\}$3	Overestimation
Double Q-learning	Decoupled $\{Q^1, ..., Q^N\}$4 and value in two nets	Reduced (but not eliminated)
Ensemble DQN	$\{Q^1, ..., Q^N\}$5	Reduced variance, still over
Maxmin Q-learning	$\{Q^1, ..., Q^N\}$6	Controlled underestimation

Unlike simple averaging, the min-ensemble produces a guaranteed pessimistic correction, which explicitly counters the max-induced overestimation in RL. Double Q-learning corresponds to $\{Q^1, ..., Q^N\}$7 with cross-updates but applies the max operator for policy selection, failing to fully suppress bias (Lan et al., 2020, Sheikh et al., 2020). Maxmin Q-learning strictly generalizes Double Q-learning and allows continuous tuning.

6. Empirical Performance and Practical Considerations

Across classic and deep RL benchmarks, Maxmin Q-learning with moderate $\{Q^1, ..., Q^N\}$8 yields:

• Lower bias and variance in value estimates in noisy environments (Lan et al., 2020).
• Greater stability and robustness to high-variance rewards or stochastic transitions (e.g., in Mountain Car with $\{Q^1, ..., Q^N\}$9) (Lan et al., 2020).
• Competitive or superior learning curves compared to DQN, Double DQN, and Averaged-DQN in environments with substantial noise or reward randomness (Sheikh et al., 2020, Lan et al., 2020).
• In practical settings, moderate $(s, a)$0 (e.g., 4 or 8 in deep RL) balances computational burden and bias correction (Lan et al., 2020).
• In exploration-harder tasks, keeping $(s, a)$1 small preserves optimism; in noisy settings, raising $(s, a)$2 improves learning stability.

Maxmin Q-learning's effectiveness depends on ensemble diversity: if all $(s, a)$3 collapse to similar representations, bias reduction degrades. Representation diversity maximization is proposed as a regularization strategy to counteract ensemble collapse (Sheikh et al., 2020).

7. Extensions, Unified Frameworks, and Limitations

Maxmin Q-learning has been placed within a broader unifying framework—dummy adversarial Q-learning (DAQ)—that interprets the min-ensemble as a zero-sum game between the agent and a dummy adversary selecting the worst estimator. This approach generalizes and connects several bias-corrected Q-learning algorithms, making explicit the adversarial correction mechanism (Lee et al., 2023). Recent work further explores regularization strategies to maintain functional diversity within the ensemble (Sheikh et al., 2020).

The principal limitation, confirmed by both theoretical and empirical investigation, is that excessive pessimism (too large $(s, a)$4) can induce slow learning or lead to conservative policies where optimism is beneficial for exploration. Proper tuning, often via small-scale ablation or based on action-space size and task stochasticity, is recommended (Lan et al., 2020).

---

Key References:

• "Maxmin Q-learning: Controlling the Estimation Bias of Q-learning" (Lan et al., 2020)
• "Preventing Value Function Collapse in Ensemble Q-Learning by Maximizing Representation Diversity" (Sheikh et al., 2020)
• "Suppressing Overestimation in Q-Learning through Adversarial Behaviors" (Lee et al., 2023)