Balanced Q-learning: Regulating Bias in RL

• Balanced Q-learning is an adaptive algorithm that blends max and min Q-learning targets using an online beta coefficient to balance estimation bias.
• It improves convergence by dynamically regulating optimism and pessimism based on state-action risk profiles, leading to faster self-correction.
• The method has spurred related approaches, such as self-correcting and simultaneous double Q-learning, to address bias control in reinforcement learning.

Searching arXiv for relevant papers on Balanced Q-learning and closely related bias-balancing variants. Balanced Q-learning denotes a class of Q-learning modifications that regulate the bias induced by max-based bootstrapping, and in its most specific sense refers to the algorithm introduced in “Balanced Q-learning: Combining the Influence of Optimistic and Pessimistic Targets” (Karimpanal et al., 2021). In that formulation, the target is a convex combination of an optimistic max term and a pessimistic min term, with the mixing weight determined online and analytically. The underlying premise is that bias is not uniformly detrimental: optimistic targets can be useful in high-value, low-risk regions, whereas pessimistic targets can be useful in low-value, high-risk regions (Karimpanal et al., 2021). Related work uses the same balancing intuition in different ways, including symmetric coupling of two estimators, self-correcting action selection, adversarial reformulations, value-guided regularization, and bounded-rational soft updates (Na et al., 2024, Zhu et al., 2020, Lee et al., 2023, Park et al., 2024, Grau-Moya et al., 2018).

1. Conceptual basis and motivation

Standard Q-learning uses the target

$Q_{T_n}=r(s,a)+\gamma \max_{a'} Q_n(s',a').$

This target is optimistic by construction, and the associated estimation bias is written as

$$Z=\gamma(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a')).$$

The expectation of this bias is positive, which is the classical overestimation problem emphasized in the balanced-Q-learning literature (Karimpanal et al., 2021).

The pessimistic extreme replaces the max with a min,

$Q_{T_n}=r(s,a)+\gamma \min_{a'} Q_n(s',a'),$

with bias

$$Z=\gamma(\underset{a'}{min\thinspace}Q_{est}(s',a')-\underset{a'}{min\thinspace}Q_{true}(s',a')).$$

This produces negative expected bias (Karimpanal et al., 2021). Balanced Q-learning is motivated by the claim that neither overestimation nor underestimation is universally preferable. In risky MDPs, optimism may accelerate movement toward high-return regions, while pessimism may discourage repeated selection of dangerous or low-value actions (Karimpanal et al., 2021).

A closely related motivation appears throughout later work on bias control in Q-learning. Self-correcting Q-learning explicitly characterizes Q-learning as “too optimistic,” Double Q-learning as “too pessimistic,” and its own estimator as lying between those extremes (Zhu et al., 2020). Simultaneous double Q-learning describes balance in terms of two estimators $Q^A$ and $Q^B$ that play complementary, symmetric roles rather than being randomly selected and updated one at a time (Na et al., 2024). Dummy adversarial Q-learning uses a dummy adversarial player to induce a countervailing underestimation tendency that suppresses overestimation (Lee et al., 2023). This suggests that “balanced Q-learning” has both a narrow meaning—the specific adaptive max/min target of (Karimpanal et al., 2021)—and a broader literature-level meaning encompassing methods that regulate optimism and pessimism rather than eliminating bias by a single fixed correction.

2. Core formulation of Balanced Q-learning

The defining target in Balanced Q-learning is

$$Q_{T_n}=r(s,a)+\gamma\left[\beta(s,a)\underset{a'}{max\thinspace}Q_{n}(s',a')+(1-\beta(s,a))\underset{a'}{min\thinspace}Q_{n}(s',a')\right],$$

with

$0\leq \beta(s,a)\leq 1.$

The corresponding estimation bias is

$$Z=\gamma\beta(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a'))$$

$$+\gamma(1-\beta)(\underset{a'}{min\thinspace}Q_{est}(s',a')-\underset{a'}{min\thinspace}Q_{true}(s',a')).$$

Varying $$Z=\gamma(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a')).$$0 from $$Z=\gamma(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a')).$$1 to $$Z=\gamma(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a')).$$2 therefore moves continuously from maximally optimistic to maximally pessimistic bootstrapping (Karimpanal et al., 2021).

The paper introduces the target difference

$$Z=\gamma(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a')).$$3

and derives

$$Z=\gamma(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a')).$$4

This expression is the basis for the claim that the preferred sign of the target difference depends on whether the current estimate is below or above the optimum (Karimpanal et al., 2021).

The online balancing coefficient is adjusted through a sample-specific update. Starting from a modified target $$Z=\gamma(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a')).$$5, the paper derives

$$Z=\gamma(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a')).$$6

where the TD error is

$$Z=\gamma(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a')).$$7

or equivalently,

$$Z=\gamma(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a')).$$8

The running balancing factor is then updated as

$$Z=\gamma(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a')).$$9

In the DQN-style implementation, the paper initializes $Q_{T_n}=r(s,a)+\gamma \min_{a'} Q_n(s',a'),$0, clips samplewise $Q_{T_n}=r(s,a)+\gamma \min_{a'} Q_n(s',a'),$1 to $Q_{T_n}=r(s,a)+\gamma \min_{a'} Q_n(s',a'),$2, averages them within the batch, and then updates the running estimate (Karimpanal et al., 2021).

The intended interpretation is explicit. If $Q_{T_n}=r(s,a)+\gamma \min_{a'} Q_n(s',a'),$3, $Q_{T_n}=r(s,a)+\gamma \min_{a'} Q_n(s',a'),$4 increases and the next target becomes more optimistic; if $Q_{T_n}=r(s,a)+\gamma \min_{a'} Q_n(s',a'),$5, $Q_{T_n}=r(s,a)+\gamma \min_{a'} Q_n(s',a'),$6 decreases and the next target becomes more pessimistic; if $Q_{T_n}=r(s,a)+\gamma \min_{a'} Q_n(s',a'),$7, the current $Q_{T_n}=r(s,a)+\gamma \min_{a'} Q_n(s',a'),$8 is already appropriate (Karimpanal et al., 2021). A common misconception is that Balanced Q-learning is a bias-elimination method. In fact, its defining formulation treats bias as a controllable learning signal rather than a quantity that should always be removed.

3. Theoretical interpretation and convergence properties

The theoretical argument in (Karimpanal et al., 2021) is organized around the sign of the target difference term. In high-value, low-risk regions, the paper associates learning with the regime $Q_{T_n}=r(s,a)+\gamma \min_{a'} Q_n(s',a'),$9, for which positive target differences are helpful; larger $$Z=\gamma(\underset{a'}{min\thinspace}Q_{est}(s',a')-\underset{a'}{min\thinspace}Q_{true}(s',a')).$$0 is then preferred. In low-value, high-risk regions, the paper associates learning with $$Z=\gamma(\underset{a'}{min\thinspace}Q_{est}(s',a')-\underset{a'}{min\thinspace}Q_{true}(s',a')).$$1, for which negative target differences are preferable; smaller $$Z=\gamma(\underset{a'}{min\thinspace}Q_{est}(s',a')-\underset{a'}{min\thinspace}Q_{true}(s',a')).$$2 is then preferred (Karimpanal et al., 2021). The central theoretical claim is therefore not that one form of bias dominates globally, but that the utility of bias is state-action dependent.

For tabular Balanced Q-learning, convergence is stated for finite MDPs under the update

$$Z=\gamma(\underset{a'}{min\thinspace}Q_{est}(s',a')-\underset{a'}{min\thinspace}Q_{true}(s',a')).$$3

with

$$Z=\gamma(\underset{a'}{min\thinspace}Q_{est}(s',a')-\underset{a'}{min\thinspace}Q_{true}(s',a')).$$4

The theorem states that Balanced Q-learning converges with probability $$Z=\gamma(\underset{a'}{min\thinspace}Q_{est}(s',a')-\underset{a'}{min\thinspace}Q_{true}(s',a')).$$5 as long as

$$Z=\gamma(\underset{a'}{min\thinspace}Q_{est}(s',a')-\underset{a'}{min\thinspace}Q_{true}(s',a')).$$6

$$Z=\gamma(\underset{a'}{min\thinspace}Q_{est}(s',a')-\underset{a'}{min\thinspace}Q_{true}(s',a')).$$7

for all $$Z=\gamma(\underset{a'}{min\thinspace}Q_{est}(s',a')-\underset{a'}{min\thinspace}Q_{true}(s',a')).$$8 (Karimpanal et al., 2021). The paper also notes that because the target includes a min term, the limiting fixed point is more risk-averse than the standard Q-learning fixed point.

Later balancing variants retain this convergence-oriented perspective but alter the mechanism. Self-correcting Q-learning proves convergence to $$Z=\gamma(\underset{a'}{min\thinspace}Q_{est}(s',a')-\underset{a'}{min\thinspace}Q_{true}(s',a')).$$9 with probability $Q^A$0 under the same standard assumptions used for Q-learning, while modifying only the action-selection step through a corrected selector (Zhu et al., 2020). Simultaneous double Q-learning derives a finite-time expected error bound under assumptions including $Q^A$1, constant step size $Q^A$2, bounded rewards, bounded initialization, i.i.d. sampling from the stationary distribution induced by the behavior policy, and ergodicity of the MDP (Na et al., 2024). Dummy adversarial Q-learning provides finite-time convergence for asynchronous DAQ by reduction to minimax Q-learning, with exponentially decaying terms such as $Q^A$3 and $Q^A$4 plus a constant bias term due to constant step size (Lee et al., 2023). These results do not define Balanced Q-learning itself, but they show that balancing optimism and pessimism has become a broader analytical theme in finite-time RL theory.

4. Relation to adjacent bias-control methods

The following variants are closely related to balanced Q-learning in objective, but they differ in mechanism.

Method	Balancing mechanism	Setting
Balanced Q-learning	Convex combination of max and min targets with online $Q^A$5	Tabular and DQN-style (Karimpanal et al., 2021)
Self-correcting Q-learning	Corrected action selection using current and previous Q estimates	Tabular and DQN (Zhu et al., 2020)
Simultaneous double Q-learning	Symmetric coupling of $Q^A$6 and $Q^A$7, updated simultaneously	Tabular finite-time analysis (Na et al., 2024)
Dummy adversarial Q-learning	Dummy adversarial player with reward shift $Q^A$8 in a zero-sum formulation	Tabular bias control (Lee et al., 2023)
DIAR	Q-network anchored by a value function trained with expectile-style asymmetry	Offline RL (Park et al., 2024)
DecQN	Mean-aggregated per-dimension utilities after bang-bang discretization	Continuous control (Seyde et al., 2022)

Self-correcting Q-learning modifies vanilla Q-learning through

$Q^A$9

followed by

$Q^B$0

Its key lemma places the resulting estimator between the positively biased single estimator and the negatively biased double estimator (Zhu et al., 2020). This is a balancing construction, but it is not the same as the adaptive max/min interpolation of (Karimpanal et al., 2021).

Simultaneous double Q-learning uses the updates

$Q^B$1

$Q^B$2

Its “balancing” consists in symmetrically coupling two estimators so that each uses the other for action selection but bootstraps itself (Na et al., 2024). The paper explicitly notes a caveat: if $Q^B$3, then the two estimators remain identical and SDQ collapses to standard Q-learning.

Dummy adversarial Q-learning introduces a dummy action $Q^B$4 and reward shifts $Q^B$5, leading to targets such as

$Q^B$6

or

$Q^B$7

The paper states that this regulates overestimation through dummy adversarial behaviors while preserving the optimal policy because constant reward shifts do not change greedy action preferences at a given state (Lee et al., 2023).

A further misconception is that balanced Q-learning must use multiple critics. The exact Balanced Q-learning algorithm in (Karimpanal et al., 2021) does not require estimator ensembles; its distinguishing feature is online control of the bias direction through $Q^B$8. Multi-estimator methods belong to related, not identical, lines of work.

5. Extensions beyond the original tabular formulation

In continuous control, “Solving Continuous Control via Q-learning” presents Decoupled Q-Networks (DecQN), which can be read as a structurally balanced Q-learning scheme because the global action-value function is decomposed equally across action dimensions: $Q^B$9 The argmax then factorizes into per-dimension local maximizations, and the paper emphasizes that the mean aggregation $$Q_{T_n}=r(s,a)+\gamma\left[\beta(s,a)\underset{a'}{max\thinspace}Q_{n}(s',a')+(1-\beta(s,a))\underset{a'}{min\thinspace}Q_{n}(s',a')\right],$$0 was empirically more stable and better scaled than a pure sum (Seyde et al., 2022). This is not Balanced Q-learning in the named sense of (Karimpanal et al., 2021), but it uses “balanced” to describe equalized per-dimension contribution within a critic-only method.

In offline RL, DIAR defines a different balance: Q-learning is anchored by a state-value network trained with expectile-style asymmetry. Its Q-objective is

$$Q_{T_n}=r(s,a)+\gamma\left[\beta(s,a)\underset{a'}{max\thinspace}Q_{n}(s',a')+(1-\beta(s,a))\underset{a'}{min\thinspace}Q_{n}(s',a')\right],$$1

and the value network is trained with

$$Q_{T_n}=r(s,a)+\gamma\left[\beta(s,a)\underset{a'}{max\thinspace}Q_{n}(s',a')+(1-\beta(s,a))\underset{a'}{min\thinspace}Q_{n}(s',a')\right],$$2

The paper explicitly describes this as making decision-making more balanced and adaptive, especially for long-horizon sparse-reward problems, by constraining Q-values with a value function and by adaptive revaluation based on whether $$Q_{T_n}=r(s,a)+\gamma\left[\beta(s,a)\underset{a'}{max\thinspace}Q_{n}(s',a')+(1-\beta(s,a))\underset{a'}{min\thinspace}Q_{n}(s',a')\right],$$3 (Park et al., 2024).

In multi-agent stochastic games, “Balancing Two-Player Stochastic Games with Soft Q-Learning” extends soft Q-learning by imposing one KL-divergence constraint for the player and one for the opponent. The free-energy objective is

$$Q_{T_n}=r(s,a)+\gamma\left[\beta(s,a)\underset{a'}{max\thinspace}Q_{n}(s',a')+(1-\beta(s,a))\underset{a'}{min\thinspace}Q_{n}(s',a')\right],$$4

Here balance means tuneable strategic behavior, with $$Q_{T_n}=r(s,a)+\gamma\left[\beta(s,a)\underset{a'}{max\thinspace}Q_{n}(s',a')+(1-\beta(s,a))\underset{a'}{min\thinspace}Q_{n}(s',a')\right],$$5 yielding zero-sum behavior, $$Q_{T_n}=r(s,a)+\gamma\left[\beta(s,a)\underset{a'}{max\thinspace}Q_{n}(s',a')+(1-\beta(s,a))\underset{a'}{min\thinspace}Q_{n}(s',a')\right],$$6 yielding team-game behavior, and finite $$Q_{T_n}=r(s,a)+\gamma\left[\beta(s,a)\underset{a'}{max\thinspace}Q_{n}(s',a')+(1-\beta(s,a))\underset{a'}{min\thinspace}Q_{n}(s',a')\right],$$7 spanning intermediate bounded-performance opponents (Grau-Moya et al., 2018). This is a distinct use of the term “balancing,” but it extends the idea of controlled optimism and pessimism to interactive settings.

The broader literature therefore uses “balanced” in several technically different senses: balancing max and min targets (Karimpanal et al., 2021), balancing two estimators symmetrically (Na et al., 2024), balancing over- and underestimation via a correction term (Zhu et al., 2020), balancing optimism with an adversarial counterforce (Lee et al., 2023), balancing Q and value guidance in offline RL (Park et al., 2024), and balancing actuator-wise value contributions in continuous control (Seyde et al., 2022).

6. Empirical behavior, applications, and limitations

The empirical results reported for Balanced Q-learning proper are organized around environments with different reward-risk structures. In the “line world” experiment, when $$Q_{T_n}=r(s,a)+\gamma\left[\beta(s,a)\underset{a'}{max\thinspace}Q_{n}(s',a')+(1-\beta(s,a))\underset{a'}{min\thinspace}Q_{n}(s',a')\right],$$8, the left action is high-value and optimism is helpful; Balanced Q-learning matches standard Q-learning. When $$Q_{T_n}=r(s,a)+\gamma\left[\beta(s,a)\underset{a'}{max\thinspace}Q_{n}(s',a')+(1-\beta(s,a))\underset{a'}{min\thinspace}Q_{n}(s',a')\right],$$9, standard Q-learning over-selects the left action because of optimism and corrects slowly, whereas Balanced Q-learning self-corrects much faster—about an order of magnitude faster according to the paper (Karimpanal et al., 2021). As $0\leq \beta(s,a)\leq 1.$0 varies, high $0\leq \beta(s,a)\leq 1.$1 leads to high $0\leq \beta(s,a)\leq 1.$2 and low $0\leq \beta(s,a)\leq 1.$3 leads to low $0\leq \beta(s,a)\leq 1.$4, which the paper interprets as learned environment-dependent risk sensitivity.

In MinAtar and related benchmarks—Breakout, Space Invaders, Asterix, Seaquest, Freeway, Island Navigation, CartPole-v0, and Tabular Navigation—the paper compares Balanced DQN with DQN, Double DQN, Maxmin DQN, Averaged DQN, and REDQ. It reports that Balanced DQN achieves the best mean return in almost all of the listed environments, including $0\leq \beta(s,a)\leq 1.$5 on Breakout, $0\leq \beta(s,a)\leq 1.$6 on Space Invaders, and $0\leq \beta(s,a)\leq 1.$7 on Asterix (Karimpanal et al., 2021). In Cliff world, with cliff states giving reward $0\leq \beta(s,a)\leq 1.$8, all other transitions giving reward $0\leq \beta(s,a)\leq 1.$9, $$Z=\gamma\beta(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a'))$$0, $$Z=\gamma\beta(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a'))$$1, $$Z=\gamma\beta(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a'))$$2, and $$Z=\gamma\beta(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a'))$$3, visitation maps show that Balanced Q-learning learns more conservative paths and avoids states near the cliff (Karimpanal et al., 2021).

Related balancing variants exhibit analogous empirical patterns. Simultaneous double Q-learning is reported to converge faster and perform better overall than standard double Q-learning on FrozenLake, CliffWalking, and Taxi in OpenAI Gym, using $$Z=\gamma\beta(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a'))$$4-greedy exploration, $$Z=\gamma\beta(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a'))$$5, $$Z=\gamma\beta(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a'))$$6, and 30 independent runs (Na et al., 2024). Self-correcting Q-learning performs best or near-best in high-variance domains and is often faster than Q-learning in low-variance domains; the paper reports that $$Z=\gamma\beta(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a'))$$7 to $$Z=\gamma\beta(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a'))$$8 is usually effective (Zhu et al., 2020). DIAR is reported to outperform state-of-the-art algorithms on Maze2D, AntMaze, and Kitchen (Park et al., 2024). DecQN matches or sometimes exceeds strong actor-critic baselines such as D4PG and DMPO on state-based tasks, and remains highly competitive against DrQ-v2 and Dreamer-v2 on several pixel-based environments (Seyde et al., 2022).

The limitations stated in the literature are method-specific. Balanced Q-learning includes a hyperparameter $$Z=\gamma\beta(\underset{a'}{max\thinspace}Q_{est}(s',a')-\underset{a'}{max\thinspace}Q_{true}(s',a'))$$9, though the paper reports that $$+\gamma(1-\beta)(\underset{a'}{min\thinspace}Q_{est}(s',a')-\underset{a'}{min\thinspace}Q_{true}(s',a')).$$0 worked consistently well across environments (Karimpanal et al., 2021). SDQ requires nonidentical initialization to avoid collapse to standard Q-learning (Na et al., 2024). DecQN works best when discretized boundary actions are sufficient and when state information can resolve coordination among action dimensions (Seyde et al., 2022). DIAR is designed for long-horizon sparse-reward offline RL and depends on latent trajectory modeling and adaptive revaluation (Park et al., 2024). These constraints suggest that “balanced Q-learning” is best understood not as a single implementation template, but as a family of methods that regulate target bias, estimator coupling, or planning commitment in a problem-dependent manner.