---
title: Critic Regularized Regression (CRR)
url: https://www.emergentmind.com/topics/critic-regularized-regression-crr
type: topic
---

# Critic Regularized Regression (CRR)

Critic Regularized Regression (CRR) is a framework for offline (batch) reinforcement learning (RL) that employs a critic-driven regularization in the policy learning phase to address extrapolation error and improve stability in policy optimization from static datasets. It combines policy evaluation under a fixed dataset with a value-filtered regression objective for the policy, leveraging a learned Q-function (the critic) to regularize which actions are imitated. CRR is closely related to a broader family of critic-regularized methods, with theoretical and empirical links to conservative Q-learning (CQL) and one-step advantage-weighted regression. Its design offers a practical and robust approach for learning policies from offline data, especially in high-dimensional and real-world RL tasks [2006.15134, 2307.12968].

## 1. Mathematical Formulation

CRR proceeds by alternately updating:
- The critic (Q-function), via distributional temporal-difference (TD) loss,
- The policy (actor), via a value-filtered regression objective that emphasizes actions supported by high Q-values.

For a fixed dataset $\mathcal{B}$ of transitions $(s, a, r, s')$ sampled under an unknown behavioral policy $\beta(a|s)$, CRR employs:

**Critic update**:
\[
\mathcal{L}_{\rm critic}(\theta) = \mathbb{E}_{(s,a,r,s')\sim\mathcal{B}} \left[ D\big(Q_\theta(s,a),~ r + \gamma\, \mathbb{E}_{a'\sim\pi_{\phi'}(s')}\left[Q_{\theta'}(s',a')\right]\big) \right]
\]
where $D(\cdot,\cdot)$ is typically a cross-entropy or MSE, and $(\theta', \phi')$ are target network parameters.

**Policy (actor) update**:
\[
\max_{\phi}\; \mathbb{E}_{(s,a)\sim\mathcal{B}}\left[ f\left(\hat A_\theta(s,a)\right)\, \log\pi_\phi(a|s) \right]
\]
where $\hat A_\theta(s,a) = Q_\theta(s,a) - \frac{1}{m}\sum_{j=1}^m Q_\theta(s, \tilde a_j)$ with actions $\tilde a_j$ sampled from $\pi_\phi(\cdot|s)$. The filter $f(\cdot)$ is crucial and may be binary ($\mathbf{1}[\hat A_\theta > 0]$) or exponential ($\exp(\hat A_\theta/\beta)$). This weighting restricts imitation to actions judged superior by the critic [2006.15134].

## 2. Regularization Mechanism and Stability

CRR's core innovation is using the critic as a regularizer in the actor update, selectively copying actions from the dataset based on the critic's advantage estimate. This approach prevents the policy from imitating sub-optimal or out-of-distribution actions, thus mitigating extrapolation error commonly arising in offline RL, where the policy may propose actions not supported by the data, leading to uncontrolled and often over-optimistic Q-value estimates. Compared to standard policy gradients or behavioral cloning, CRR's value-weighted regression targets high-value actions, while ignoring (or downweighting) poor or out-of-distribution actions, naturally constraining the learned policy to regions well-supported by the dataset [2006.15134].

## 3. Policy Update Derivation and Algorithm

The exponential-filtered policy update in CRR represents a regularized policy optimization step:
\[
\max_{q(\cdot|s)}\; \mathbb{E}_{a\sim q}[Q_\theta(s,a)] - \beta\, {\rm KL}(q(\cdot|s)\;\|\;\mu_{\mathcal{B}}(\cdot|s))
\]
where $\mu_\mathcal{B}$ is the empirical behavior policy. The analytic solution is
\[
q^*(a|s) \propto \exp(Q_\theta(s,a)/\beta) \, \mu_\mathcal{B}(a|s)
\]
and the parametric policy $\pi_\phi$ is optimized to minimize the cross-entropy between $q^*$ and $\pi_\phi$, implemented via samples weighted by $f_{\exp}$ or $f_{\rm bin}$ as described above [2006.15134]. No additional regularization penalty is applied—critic-driven filtering acts as the only regularizer.

**Algorithmic structure:**  
Each iteration consists of (1) sampling a minibatch, (2) one gradient step on the actor loss via filtered regression, (3) critic TD update, and (4) periodic target network synchronization.

## 4. Comparison with Other Critic Regularization Methods

CRR formalizes one approach within the broader genre of critic-regularized algorithms, with CQL as a prototypical example [2307.12968]. In CQL, the critic update includes an explicit penalty:
\[
\lambda\left(\mathbb{E}_{s, a\sim \pi(\cdot|s)}[Q(s,a)] - \mathbb{E}_{s, a\sim \beta(\cdot|s)}[Q(s,a)] \right)
\]
where $\lambda$ controls the regularization strength. The CRR and CQL scheme are tightly linked: for $\lambda=1$, CQL’s solution reduces to a one-step reverse-KL–like regularization equivalent to CRR’s policy objective. For intermediate $\lambda$, the resulting policies are nearly indistinguishable from those produced by one-step RL (argmax match >95% in discrete tabular domains for $\lambda \approx 1-10$).

A summary of the connections:

| Method     | Regularization Mechanism       | Relation to CRR                                  |
|------------|-------------------------------|--------------------------------------------------|
| CQL        | Critic penalized for OOD actions | Equivalent to CRR for $\lambda\approx1$          |
| One-step RL | Single policy update, strong regularization | Special case of CQL/CRR, recovers similar policy |
| CRR        | Value-filtered regression      | General framework                                |

CRR avoids direct Q maximization via policy gradients and, through filtering, can be interpreted as a regularized policy improvement (with either hard or soft filters) [2006.15134, 2307.12968].

## 5. Network Architectures and Hyperparameters

CRR is implemented in deep RL settings using flexible architectures suited to the high-dimensional and partially observable tasks found in common benchmarks:
- **Vision/proprioceptive stack:** Small ResNet processing $64\times64$ camera views, concatenated with proprioceptive data, followed by a 4-block residual MLP (hidden size 1024, layernorm, ReLU)
- **Value (critic) head:** Linear output producing distributional $Q$-values (21 atoms over $[0,100]$)
- **Policy head:** Mixture of Gaussians (5 components, mean/diagonal covariance); mean used at evaluation
- **Recurrence:** LSTM layers (size 1024) for egocentric/partially observable settings

Key hyperparameters:
- Adam optimizer, learning rates $10^{-4}$ (actor and critic)
- Batch size: 1024 (feed-forward), 128 (recurrent)
- Filter temperature $\beta=1.0$
- Advantage sampler $m=4$
- Critic atoms: 21 (CRR); target update every 100 steps

Filter selection is task-dependent: binary and binary-max excel on simple tasks, while the exponential filter is advantageous on complex, high-dimensional environments [2006.15134].

## 6. Empirical Performance and Ablations

CRR outperforms several state-of-the-art offline RL algorithms in both low- and high-dimensional domains:

- **DeepMind Control Suite:** CRR achieves normalized returns competitive with D4PG and ABM, surpassing BCQ and BC on control, manipulation, and locomotion tasks.
- **Locomotion and Manipulation:** CRR-variant returns (exp and bin) show significant gains, with mean success counts substantially higher than D4PG and BCQ, especially on vision-based and high-DoF manipulation.
- **Qualitative ablations:**
  - Binary filters aggressively remove suboptimal actions, excelling on simple tasks with good coverage
  - Exponential filter is more permissive; preferable with abundant, high-quality data in large state/action spaces
  - Turning off policy noise at evaluation consistently improves final returns, especially with termination-sensitive tasks
  - Critic-Weighted Policy (CWP): At test time, sampling actions according to $\exp(Q/\beta)$ yields 2–5% further performance boosts
  - K-step returns for advantage estimation or noise injection during evaluation can degrade offline policy quality

## 7. Lower Bounds and Theoretical Guarantees

CRR enjoys pessimistic value estimation guarantees, similar to those proven for CQL [2307.12968]:
\[
Q_\theta(s,a) \le Q^{\pi_\theta}(s,a)\quad \forall s,a
\]
given suitable realizability and convergence assumptions, meaning that the learned Q-function does not overestimate the value of the current policy. For $\lambda=1$, CRR recovers a behavior-regularized Q (i.e., $Q^\beta$), which is a lower bound for any policy $\pi$ that remains close to $\beta$. This property is essential to ensuring safe and robust deployment in offline settings prone to value overestimation. Empirically, in tasks demanding strong regularization or where extrapolation errors are prevalent, CRR and CQL produce similar and competitive performance [2307.12968].

---

**References:**  
- "Critic Regularized Regression" [2006.15134]  
- "A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning" [2307.12968]

Source: https://www.emergentmind.com/topics/critic-regularized-regression-crr