Mildly Conservative Regularized Q-Learning

• The paper introduces MCRQ as an offline RL framework that keeps Q-values for unsupported actions below the best in-dataset levels to prevent harmful overestimation.
• It combines temporal-difference calibration with behavior cloning in the Bellman backup, ensuring a balanced trade-off between safety and effective generalization.
• Empirical results on D4RL MuJoCo benchmarks demonstrate higher mean scores and reduced variance compared to methods like CQL and IQL.

Mildly Conservative Regularized Q-Learning (MCRQ) denotes a family of offline reinforcement learning methods that seek to keep value estimates conservative enough to prevent harmful out-of-distribution (OOD) overestimation, while avoiding the severe pessimism that can suppress policy improvement and generalization. In the cited literature, the term is used in two closely related senses. First, “Mildly Conservative Q-learning” (MCQ) is explicitly described as conceptually a form of mildly conservative regularized Q-learning, because it regularizes the Q-function so that OOD actions are kept below the best in-dataset action without being driven to uniformly low values (Lyu et al., 2022). Second, “Mildly Conservative Regularized Q-learning” (MCRQ) is introduced as a concrete algorithm built from a mildly conservative regularized evaluation operator that combines temporal-difference calibration with behavior cloning inside the Bellman backup (Chen et al., 8 Aug 2025).

1. Terminology and conceptual scope

In offline RL, training proceeds from a fixed dataset

$\mathcal{D} = \{(s,a,r,s')\},$

collected by an unknown behavior policy, and the learner is not allowed further environment interaction. The central difficulty is distribution shift: the learned policy queries actions that may be rare or absent in the dataset, so the critic must extrapolate on unsupported state–action pairs. MCRQ-style methods address this by imposing conservatism at the level of value estimation rather than relying solely on explicit policy constraints (Lyu et al., 2022).

Within this literature, “mildly conservative” has a specific technical meaning. It does not mean abandoning pessimism, nor does it mean simple behavior cloning. Instead, it denotes a regime in which OOD actions are prevented from overtaking the best supported actions, but are not globally forced to extremely low values. A concise idealization appears in MCQ: $$Q(s,a^{\rm ood}) < \max_{a\in \operatorname{Support}(\mu(\cdot\mid s))} Q(s,a),$$ where $\mu$ is the behavior policy. This preserves safe action ranking while allowing a smoother Q-landscape outside exact data support (Lyu et al., 2022).

The same design principle reappears in later work under different operator choices. The explicit MCRQ algorithm builds conservatism directly into policy evaluation by combining a standard empirical Bellman operator, a TD-based correction term, and a behavior cloning penalty. Related methods such as CPQL, ACL-QL, and SCQ pursue the same high-level objective—conservative but not over-pessimistic offline control—through multi-step operators, adaptive per-transition conservative weights, or selective penalization of hard OOD regions (Chen et al., 8 Aug 2025, Kim et al., 14 May 2026, Wu et al., 2024, Shimizu et al., 2024).

2. Offline RL rationale for mild conservatism

The standard offline RL setting is a discounted MDP

$$\mathcal{M}=\langle \mathcal{S},\mathcal{A},r,\rho_0,p,\gamma\rangle$$

with bounded rewards and objective

$$J(\pi) = \mathbb{E}\Big[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t)\Big].$$

Classical Q-learning relies on Bellman targets of the form

$y = r(s,a) + \gamma \max_{a'} Q(s',a'),$

but offline datasets only constrain $Q$ on actions drawn by the behavior policy. When $\max_{a'}$ evaluates unseen actions, neural approximators may assign arbitrary values to them, and bootstrapping can amplify these errors into severe OOD overestimation (Lyu et al., 2022).

The literature described here distinguishes three broad responses to this problem. Policy-constraint or behavior-regularization methods such as TD3+BC, BEAR, BCQ, and related variants keep the learned policy close to the behavior policy, which limits distribution shift but can strongly limit improvement when the dataset is suboptimal. Value-penalization methods such as CQL explicitly depress OOD Q-values, which provides strong conservative guarantees but often becomes overly pessimistic even near the support boundary. Methods that avoid querying OOD actions, such as IQL-style approaches, remove extrapolation from unseen actions but remain tightly limited by the dataset’s support (Lyu et al., 2022).

MCRQ-style methods are motivated by the claim that the appropriate conservatism should be “as mild as possible while still preventing harmful overestimation.” In this view, excessive pessimism flattens the Q-function outside the dataset support, creates artificial barriers to stitching together useful sub-trajectories, and degrades transfer from offline pretraining to online fine-tuning. A plausible implication is that mild conservatism is best understood as a ranking condition—unsupported actions should not become falsely optimal—rather than a demand that all unsupported actions receive uniformly low values (Lyu et al., 2022).

3. MCQ as a foundational MCRQ formulation

MCQ provides one of the clearest formulations of mild conservatism at the Q-function level. In the tabular setting, its analytical object is the Mildly Conservative Bellman (MCB) operator. For in-support actions, the update follows the standard optimal Bellman backup. For OOD actions, MCQ uses a special self-backup together with a mildly conservative projection that sets the OOD value to the maximum supported Q-value at the state minus a small $\delta>0$. The resulting fixed point $Q_{\mathrm{MCB}}$ induces a greedy policy that cannot prefer unsupported actions over the best supported one (Lyu et al., 2022).

In continuous domains, MCQ replaces exact support computations with a learned behavior model $$Q(s,a^{\rm ood}) < \max_{a\in \operatorname{Support}(\mu(\cdot\mid s))} Q(s,a),$$0 trained as a conditional VAE. It approximates the in-support maximum by sampling $$Q(s,a^{\rm ood}) < \max_{a\in \operatorname{Support}(\mu(\cdot\mid s))} Q(s,a),$$1 actions from $$Q(s,a^{\rm ood}) < \max_{a\in \operatorname{Support}(\mu(\cdot\mid s))} Q(s,a),$$2, evaluating their Q-values, and using the maximum over samples as a pseudo target. The practical critic objective combines ordinary Bellman regression on dataset transitions with an auxiliary loss on policy-sampled actions: $$Q(s,a^{\rm ood}) < \max_{a\in \operatorname{Support}(\mu(\cdot\mid s))} Q(s,a),$$3 Here $$Q(s,a^{\rm ood}) < \max_{a\in \operatorname{Support}(\mu(\cdot\mid s))} Q(s,a),$$4 is the standard SAC-style target with clipped double Q, while $$Q(s,a^{\rm ood}) < \max_{a\in \operatorname{Support}(\mu(\cdot\mid s))} Q(s,a),$$5 is a pseudo target derived from behavior-model samples and a minimum over two critics (Lyu et al., 2022).

This construction is the reason MCQ is described as a form of MCRQ. The regularization is not an explicit policy-distance penalty. Instead, it is an auxiliary Q-loss that shapes values for policy-sampled, potentially OOD actions so that they remain tied to estimated in-support maxima. In theory, OOD actions are those with $$Q(s,a^{\rm ood}) < \max_{a\in \operatorname{Support}(\mu(\cdot\mid s))} Q(s,a),$$6; in practice, MCQ treats actions sampled from the current policy at dataset states as candidate OOD actions. This is justified on the grounds that if a sampled action is actually in-support, training it toward the batch-optimal Q-value is still consistent with the MCB fixed point (Lyu et al., 2022).

The pseudo-target mechanism is intentionally mild. It does not subtract an explicit $$Q(s,a^{\rm ood}) < \max_{a\in \operatorname{Support}(\mu(\cdot\mid s))} Q(s,a),$$7 in practice, because the expectation over finitely many samples from $$Q(s,a^{\rm ood}) < \max_{a\in \operatorname{Support}(\mu(\cdot\mid s))} Q(s,a),$$8 is typically slightly below the true in-support maximum. The paper also gives a bound: if

$$Q(s,a^{\rm ood}) < \max_{a\in \operatorname{Support}(\mu(\cdot\mid s))} Q(s,a),$$9

then

$\mu$0

This suggests that the optimism of the pseudo target is bounded by behavior-model error and the number of behavior samples (Lyu et al., 2022).

4. MCRE and the explicit MCRQ algorithm

The explicitly named MCRQ algorithm is built on mildly conservative regularized evaluation (MCRE). Instead of using the empirical Bellman operator $\mu$1 alone, MCRE evaluates a policy with a modified operator

$\mu$2

where $\mu$3 controls a TD-based correction operator and

$\mu$4

is a behavior cloning penalty for deterministic policies (Chen et al., 8 Aug 2025).

The correction operator is

$\mu$5

Its role is calibrative rather than uniformly pessimistic. If the TD target is larger than the current estimate at $\mu$6, the operator reduces the target; if the TD target is smaller, it increases it. The behavior cloning term then reduces the backup when the learned action deviates strongly from the dataset action. This produces local, backup-level conservatism rather than global suppression of all OOD actions (Chen et al., 8 Aug 2025).

MCRQ instantiates MCRE in a TD3-style actor–critic architecture with two critics and one actor. The target action is $\mu$7. The standard TD3 target is

$\mu$8

and the correction target is

$\mu$9

The final target is

$$\mathcal{M}=\langle \mathcal{S},\mathcal{A},r,\rho_0,p,\gamma\rangle$$0

Each critic minimizes $$\mathcal{M}=\langle \mathcal{S},\mathcal{A},r,\rho_0,p,\gamma\rangle$$1 (Chen et al., 8 Aug 2025).

The actor combines Q maximization and explicit behavior cloning: $$\mathcal{M}=\langle \mathcal{S},\mathcal{A},r,\rho_0,p,\gamma\rangle$$2 Thus, the named MCRQ algorithm regularizes both critic and actor, but in different ways: critic-side behavior cloning enters the Bellman target itself, while actor-side behavior cloning constrains policy deviation from the data (Chen et al., 8 Aug 2025).

5. Guarantees, bounds, and failure modes

Theoretical analysis for MCQ and MCRQ centers on safe improvement and bounded optimism. For MCQ, the MCB operator is a $$\mathcal{M}=\langle \mathcal{S},\mathcal{A},r,\rho_0,p,\gamma\rangle$$3-contraction on $$\mathcal{M}=\langle \mathcal{S},\mathcal{A},r,\rho_0,p,\gamma\rangle$$4, so it converges to a unique fixed point $$\mathcal{M}=\langle \mathcal{S},\mathcal{A},r,\rho_0,p,\gamma\rangle$$5. On supported state–action pairs, the paper proves

$$\mathcal{M}=\langle \mathcal{S},\mathcal{A},r,\rho_0,p,\gamma\rangle$$6

where $$\mathcal{M}=\langle \mathcal{S},\mathcal{A},r,\rho_0,p,\gamma\rangle$$7 is the behavior-policy value and $$\mathcal{M}=\langle \mathcal{S},\mathcal{A},r,\rho_0,p,\gamma\rangle$$8 is the batch-optimal value constrained to supported actions. Consequently,

$$\mathcal{M}=\langle \mathcal{S},\mathcal{A},r,\rho_0,p,\gamma\rangle$$9

so the induced policy behaves at least as well as the behavior policy. The same analysis is presented as tighter than policy-constraint guarantees of the form

$$J(\pi) = \mathbb{E}\Big[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t)\Big].$$0

and tighter than the pessimistic lower bound stated for CQL (Lyu et al., 2022).

For MCRE, the modified evaluation operator is a contraction in $$J(\pi) = \mathbb{E}\Big[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t)\Big].$$1, with coefficient

$$J(\pi) = \mathbb{E}\Big[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t)\Big].$$2

Therefore, for any initial $$J(\pi) = \mathbb{E}\Big[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t)\Big].$$3, the iteration $$J(\pi) = \mathbb{E}\Big[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t)\Big].$$4 converges to a unique fixed point $$J(\pi) = \mathbb{E}\Big[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t)\Big].$$5. In the no-sampling-error setting, the paper states that

$$J(\pi) = \mathbb{E}\Big[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t)\Big].$$6

provided

$$J(\pi) = \mathbb{E}\Big[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t)\Big].$$7

and additionally reports

$$J(\pi) = \mathbb{E}\Big[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t)\Big].$$8

With sampling error, the bound acquires an extra term scaling as $$J(\pi) = \mathbb{E}\Big[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t)\Big].$$9, where $y = r(s,a) + \gamma \max_{a'} Q(s',a'),$0 is the state–action count (Chen et al., 8 Aug 2025).

These guarantees also identify concrete failure modes. In MCQ, if $y = r(s,a) + \gamma \max_{a'} Q(s',a'),$1 in the critic loss is too small, the critic becomes dominated by OOD training, value underestimation occurs, and Q can collapse. Poor behavior modeling by the CVAE can bias the pseudo maximum and hence the pseudo Q-values. In the named MCRQ algorithm, overly large $y = r(s,a) + \gamma \max_{a'} Q(s',a'),$2 increases critic-side conservatism and can degrade performance, while large $y = r(s,a) + \gamma \max_{a'} Q(s',a'),$3 can also be harmful, especially on expert datasets. More generally, both formulations remain bounded by dataset coverage: tiny datasets or extremely narrow support limit what any offline method can recover (Lyu et al., 2022, Chen et al., 8 Aug 2025).

6. Empirical profile, related methods, and open directions

MCQ is evaluated mainly on D4RL MuJoCo tasks—HalfCheetah, Hopper, and Walker2d, with random, medium, medium-replay, medium-expert, and expert datasets—and compared against BC, SAC, CQL, UWAC, TD3+BC, IQL, and others. On the 12 non-expert MuJoCo datasets, MCQ reports an average normalized score of approximately $y = r(s,a) + \gamma \max_{a'} Q(s',a'),$4, compared with approximately $y = r(s,a) + \gamma \max_{a'} Q(s',a'),$5 for the best baseline, IQL. Across all 15 MuJoCo tasks, MCQ reports approximately $y = r(s,a) + \gamma \max_{a'} Q(s',a'),$6, versus approximately $y = r(s,a) + \gamma \max_{a'} Q(s',a'),$7 for IQL and approximately $y = r(s,a) + \gamma \max_{a'} Q(s',a'),$8–$y = r(s,a) + \gamma \max_{a'} Q(s',a'),$9 for CQL and TD3+BC. The same work also studies offline-to-online transfer with 1M offline steps followed by 100k online steps, reporting faster adaptation and higher final returns than CQL, TD3+BC, IQL, and AWAC on challenging non-expert datasets (Lyu et al., 2022).

The explicitly named MCRQ algorithm is evaluated on the 15 D4RL MuJoCo datasets and compared with BEAR, UWAC, BC, CDC, AWAC, BCQ, OneStep, TD3_BC, CQL, IQL, PBRL, and more recent state-of-the-art methods. The paper reports that MCRQ has the highest mean and the smallest variance over the 15 row-normalized scores, and also reports lower computational cost than CQL and IQL on HalfCheetah datasets. KL-divergence and t-SNE analyses are used to argue that MCRQ remains closer to the behavior distribution than BCQ or TD3_BC on random data, while being less conservative than CQL (Chen et al., 8 Aug 2025).

Several adjacent methods clarify what is distinctive about MCRQ. CPQL replaces the single-step Bellman backup by a multi-step Peng’s $Q$0 operator while retaining a CQL-style conservative term; its fixed point corresponds to the mixture policy $Q$1, which yields implicit behavior regularization and a safe-improvement guarantee relative to the behavior policy (Kim et al., 14 May 2026). ACL-QL introduces adaptive per-transition conservative weights $Q$2 and $Q$3, together with monotonicity and band constraints that keep learned Q-values between ordinary Q-learning and CQL in a transition-dependent “mild range” (Wu et al., 2024). SCQ distinguishes “easy OOD” from “hard OOD” regions using CVAE-based reconstruction error and penalizes only hard OOD actions, obtaining value estimates that are conservative but potentially much less so than CQL (Shimizu et al., 2024).

A recurrent misconception is that mild conservatism is merely weaker behavior cloning. The cited methods do not support that reduction. MCQ regularizes primarily at the Q-function level through pseudo Q-targets for policy-sampled actions, not through an explicit policy-distance penalty. The named MCRQ algorithm regularizes within the Bellman backup and only then adds actor-side behavior cloning. A second misconception is that mild conservatism abandons safety guarantees. On the contrary, the major formulations in this line aim to preserve at least one of the following: no harmful OOD domination, performance no worse than the behavior policy, or a calibrated lower-bounding relation to the target value function (Lyu et al., 2022, Chen et al., 8 Aug 2025).

Open directions identified in the cited works are consistent. They include reducing sensitivity to conservatism hyperparameters, improving behavior-model quality, extending analysis beyond deterministic policies and continuous-control settings, and developing adaptive schemes that tune conservatism from uncertainty or empirical error signals rather than fixed global coefficients. This suggests that MCRQ is best viewed not as a single settled recipe, but as an operator-design and regularization principle for offline Q-learning: inject enough pessimism to avoid catastrophic OOD overestimation, but not enough to erase the generalization and policy-improvement capacity of the critic (Chen et al., 8 Aug 2025, Kim et al., 14 May 2026, Wu et al., 2024).