---
title: 'MR.Q: Model-based Representations for Q-Learning'
url: https://www.emergentmind.com/topics/mr-q
type: topic
---

# MR.Q: Model-based Representations for Q-Learning

Searching arXiv for the MR.Q reinforcement-learning papers and closely related work.
MR.Q, expanded in the appendix as **Model-based Representations for Q-learning**, is a reinforcement-learning method that uses **model-based training signals** to learn latent representations while keeping **control itself model-free**. In the literature, it is described both as a **general-purpose model-free deep RL algorithm** across continuous and discrete action spaces, vector and pixel observations, and as a **purely model-free multitask actor–critic agent that borrows model-based prediction only for representation learning, not for planning** [2501.16142; 2606.05555]. Its central premise is that much of the empirical benefit often attributed to world models may derive from **predictive representation learning** rather than from planning or imagined rollouts. MR.Q therefore combines an off-policy actor–critic backbone with auxiliary prediction of latent dynamics, reward, and termination, seeking the sample efficiency and representational structure of model-based methods without model-predictive control, latent rollout planning, or imagination-based policy optimization [2501.16142; 2606.05555].

## 1. Conceptual definition and problem setting

MR.Q was proposed against two related backgrounds. The first is the search for a **single model-free method** that can address “a diverse class of domains and problem settings” with **one hyperparameter set** rather than benchmark-specific retuning [2501.16142]. The second is the multitask RL question of whether the scalability of recent world-model systems is driven primarily by planning, or by the **representations induced by predictive learning** [2606.05555].

The method is explicitly positioned between standard model-free and model-based RL. On the control side, policy improvement is done by **actor–critic learning**, with **twin critics** in the TD3 style. On the auxiliary side, MR.Q learns predictive models of **next representation**, **reward**, and **termination**, but these are used only to shape the latent space. It is therefore **not** a world-model planner like Dreamer, TD-MPC2, or Newt, and it does **not** perform latent rollouts for action selection [2501.16142; 2606.05555].

In the multitask setting, training is over tasks $\tau \sim p(\tau)$ with objective
\[
\mathbb{E}_{\tau \sim p(\tau), \pi}\left[\sum_{t=0}^{\infty}\gamma^t r_t\right].
\]
The motivation given for this setting is that scaling RL across many tasks is hindered by **non-stationary replay data**, **task interference**, **unstable bootstrapping**, **poor representation quality**, and **underuse of model capacity**. The proposed remedy is not merely to increase parameter count, but to supply larger networks with denser predictive supervision so that they learn “stable, reusable features” [2606.05555].

A recurrent misconception addressed in the literature is that MR.Q is simply TD3 with extra heads. That is incomplete. The method’s defining claim is that latent dynamics and reward prediction can make value estimation easier by learning a representation in which the value function is **closer to linear**, while the final critic remains nonlinear in practice [2501.16142].

## 2. Theoretical basis: approximate linearization of value

The original theoretical framing introduces state and state-action embeddings
\[
f_\omega: s \rightarrow \mathbf{z}_s, \qquad g_\omega: (s,a) \rightarrow \mathbf{z}_{sa},
\]
and motivates them through a linear decomposition
\[
Q(s,a) = \mathbf{z}_{sa}^\top \mathbf{w}.
\]
The point is not that MR.Q ultimately uses a linear critic; the paper explicitly shows that replacing the nonlinear value network with a linear one is “disastrous.” Rather, the claim is that a good representation should make value **approximately linear** in latent space [2501.16142].

The original analysis compares a **model-free linear TD update** with a **model-based linear solution** obtained from linear reward prediction and linear latent dynamics. Theorem 1 states that “the fixed point of the model-free approach and the solution of the model-based approach are the same” in linear space. Theorem 2 then bounds value error by reward and dynamics prediction errors. In the notation of the paper, the model-based feature-learning objective is
\[
\mathcal{L}(\mathbf{z}_{sa}, \mathbf{w}_r, W_p) =
\underbrace{\mathbb{E}_D \left[ \left( \mathbf{z}_{sa}^\top \mathbf{w}_r - r \right)^2 \right]}_{\text{reward learning}}
+
\underbrace{\lambda \mathbb{E}_D \left[ \left( \mathbf{z}_{sa}^\top W_p - \mathbf{z}_{s'a'} \right)^2 \right]}_{\text{dynamics learning}}.
\]
This is used to argue that if latent reward and transition structure are easy to predict, then downstream value prediction should also become easier [2501.16142].

The practical algorithm deliberately relaxes this theory. The next-state target is changed from a **state-action target** $\mathbf{z}_{s'a'}$ to a **state target** $\bar{\mathbf{z}}_{s'}$, making the target independent of the current policy and less non-stationary. The paper writes the relaxed objective as
\[
\mathcal{L}(\mathbf{z}_{sa}, \mathbf{w}_r, W_p)
=
\mathbb{E}_D \left[ \left( \mathbf{z}_{sa}^\top \mathbf{w}_r - r \right)^2 \right]
+
\lambda \mathbb{E}_D \left[ \left( \mathbf{z}_{sa}^\top W_p - \bar{\mathbf{z}}_{s'} \right)^2 \right].
\]
A target encoder is introduced to stabilize $\bar{\mathbf{z}}_{s'}$. The exact linearity guarantees are then dropped, and the method shifts to the weaker but operationally central claim that predictive representation learning can **approximately linearize value** well enough for a high-capacity nonlinear critic to exploit [2501.16142].

This theoretical construction is echoed by later multitask analysis. There, MR.Q is treated as the central example of a system in which “predictive representation learning + high-capacity value functions” is sufficient to scale well, even “without planning” [2606.05555].

## 3. Architecture and objectives

The core MR.Q architecture consists of a **state encoder**, a **state-action encoder**, a **latent predictive model**, a **policy**, and **twin Q-networks** [2501.16142]. In the multitask formulation, observations and optional task information are first mapped to a latent state
\[
z_t = \phi_\xi(s_t, \tau),
\]
with task conditioning implemented through a learned task embedding $e(\tau)$ when task information is available. In the MMBench experiments, this is a **language instruction embedding** concatenated with state or latent features [2606.05555].

The actor and critics operate in latent space. The multitask paper writes these as
\[
\pi_\psi(a \mid z), \qquad Q_{\theta_1}(z,a), \qquad Q_{\theta_2}(z,a),
\]
while the original paper uses the pair $(\mathbf{z}_s,\mathbf{z}_{sa})$ with downstream policy and value functions on top. The original architecture specifies a 512-dimensional latent for both $\mathbf{z}_s$ and $\mathbf{z}_{sa}$, with modality-specific encoders: a convolutional encoder for images and a 3-layer MLP for vector observations [2501.16142]. The multitask representation analysis also explicitly discusses a **512-dimensional latent space** [2606.05555].

MR.Q augments actor–critic learning with predictive heads that take $(z_t,a_t)$ and predict
\[
\hat{z}_{t+1}, \qquad \hat{r}_t, \qquad \hat{d}_t.
\]
These are the **latent dynamics**, **reward**, and **termination** targets. The multitask paper summarizes the total objective as
\[
\mathcal{L}_{\text{MR.Q}} = \mathcal{L}_{\text{TD}} + \lambda \mathcal{L}_{\text{predictive}},
\]
where $\mathcal{L}_{\text{predictive}}$ decomposes into losses for dynamics, reward, and termination prediction [2606.05555].

The original paper specifies the encoder loss more explicitly through short latent rollouts:
\[
\mathcal{L}_{\text{Encoder}}(f,g,\mathbf{m})
:=
\sum_{t=1}^{H_{\text{Enc}}}
\lambda_{\text{Reward}} \mathcal{L}_{\text{Reward}}
+
\lambda_{\text{Dynamics}} \mathcal{L}_{\text{Dynamics}}
+
\lambda_{\text{Terminal}} \mathcal{L}_{\text{Terminal}}.
\]
The reward loss uses categorical prediction with **Two-Hot** targets and **symexp** spacing; the dynamics loss is squared error to a target-encoder latent; the terminal loss is squared error on the done flag. In the practical algorithm, the actor and critic use the learned embeddings “without propagating gradients” back through the encoder, which the paper presents as part of its stabilization strategy [2501.16142].

Several architectural choices are emphasized as essential rather than incidental. The **target encoder** matters substantially, the **state target** is better than a state-action target, **multi-step returns** help, and a **nonlinear critic** remains necessary even if the representation approximately linearizes value [2501.16142].

## 4. Training procedure and implementation profile

MR.Q is trained **off-policy** with a replay buffer, target networks, and prioritized replay. In the multitask formulation, the shared replay stores tuples
\[
(s_t, a_t, r_t, d_t, s_{t+1}, \tau),
\]
and minibatches are sampled uniformly from the shared replay memory to jointly optimize the actor, twin critics, encoder, and auxiliary predictive heads [2606.05555]. In the general-purpose formulation, the replay buffer has capacity **1M** and uses **LAP** prioritized sampling with replay ratio **1** and batch size **256** [2501.16142].

The training loop is structurally simple. Interaction data are collected with noisy policy actions and stored in replay. Sampled batches are encoded into latent states, predictive targets are formed, twin critics are updated with TD learning, the policy is updated to maximize critic values, and target networks are **slowly updated** or periodically copied in TD3 style [2501.16142; 2606.05555]. The original paper makes this schedule unusually explicit: every
\[
T_{\text{target}} = 250
\]
environment steps, encoder, critic, and actor targets are synchronized, and reward-scale statistics are updated [2501.16142].

The method is designed to cover both **continuous** and **discrete** action spaces. For continuous actions, target policy smoothing uses clipped Gaussian noise and actions are clipped to range. For discrete actions, actions are represented internally as one-hot vectors and the target action is an $\arg\max$ after noisy logits. The policy activation is **$\tanh$** for continuous actions and **Gumbel-Softmax** for discrete actions [2501.16142].

The published implementation profile is notable because it is intentionally fixed across benchmarks. The original paper reports the following shared hyperparameters: $\lambda_{\text{Dynamics}}=1$, $\lambda_{\text{Reward}}=0.1$, $\lambda_{\text{Terminal}}=0.1$, $\lambda_{\text{pre-activ}}=10^{-5}$, $H_{\text{Enc}}=5$, $H_Q=3$, target policy noise $\sigma=0.2$, target noise clip $c=0.3$, LAP smoothing $\alpha=0.4$, discount $\gamma=0.99$, replay buffer $=1$M, batch size $=256$, and target update frequency $T_{\text{target}}=250$ [2501.16142]. The multitask paper does not reproduce a full hyperparameter table, but it states that MR.Q extends TD3, uses a shared multitask replay buffer, conditions on language embeddings when task information is available, and employs a **frozen DINOv2 encoder** in the pixel experiments [2606.05555].

## 5. Empirical performance and representational analysis

The original general-purpose evaluation spans **118 environments** from four benchmark suites: **5 Gym locomotion tasks**, **28 DMC proprioceptive tasks**, **28 DMC visual tasks**, and **57 Atari games**, with **10 seeds** in the main results [2501.16142]. On Gym locomotion, MR.Q attains a TD3-normalized aggregate mean of **1.46**, below **TD7: 1.57** but above **TD-MPC2: 1.04**, **DreamerV3: 0.76**, and **PPO: 0.45**. On DMC proprioceptive control, it is the best aggregate method at **835.95**, exceeding **TD-MPC2: 783.57**, **TD7: 566.97**, **DreamerV3: 530.33**, and **PPO: 254.04**. On DMC visual control, it again leads with **602.28**, ahead of **DrQ-v2: 510.81**, **TD-MPC2: 479.87**, **DreamerV3: 463.48**, and **PPO: 110.82**. On Atari, MR.Q reaches a human-normalized score of **2.54**, below **DreamerV3: 3.74** but above **Rainbow: 1.08**, **DQN: 0.25**, and **PPO: 0** [2501.16142].

The same paper also emphasizes computational profile. On Gym HalfCheetah-v4, MR.Q uses **4.1M** parameters, versus **5.5M** for TD-MPC2 and **9.7M** for DreamerV3; its training speed is **49 FPS** versus **18** and **14**, and its evaluation speed is **1.9k FPS** versus **236** and **27**. On Atari, MR.Q uses **4.4M** parameters, whereas DreamerV3 uses **187.3M** [2501.16142]. This makes computational efficiency a substantive part of the method’s identity rather than an incidental implementation detail.

Later work re-evaluates MR.Q in the large-scale multitask regime and uses it to argue that **representation learning**, not planning, is the primary driver of scalability [2606.05555]. On the 200-task “soup” benchmark at **2M environment steps**, MR.Q obtains a normalized score of **0.11** versus **0.08** for Newt, reported as a **37% relative advantage** early in training. The lead remains around **5–8%** across training, although both methods converge to similar final performance. In held-out-task transfer after pretraining on **200 tasks for 10M steps** and finetuning on **28 held-out tasks for 200k steps**, MR.Q records **0.13** versus **0.09** zero-shot, **0.55** versus **0.48** at **100k** finetuning steps, and **0.62** versus **0.55** at **200k** steps, beating Newt on **17/28 tasks (61%)** at the end of finetuning [2606.05555].

The representational analysis in the multitask paper is equally central. Compared with an encoder-free baseline, MR.Q is reported to have **higher SRank**, **higher effective dimensionality**, **less latent collapse**, **fewer dormant neurons**, and **lower value loss**. In PCA-based analysis, removing predictive representation learning reduces the $95\%$-variance dimensionality from **89** to **21** on DMControl-Ext and from **66** to **15** on MuJoCo [2606.05555]. These results are used to support the claim that predictive supervision “prevents collapse and preserves expressive capacity” in multitask settings.

Ablations in the original paper identify what matters most. Removing model-based representation learning (“No MR”) hurts across Gym, DMC-prop, DMC-visual, and Atari; removing the target encoder hurts substantially; using 1-step returns instead of multi-step returns degrades performance; and replacing the nonlinear critic with a linear one causes catastrophic drops on every benchmark [2501.16142]. A plausible implication is that MR.Q’s empirical strength is not attributable to any single auxiliary prediction head, but to the joint effect of predictive representation learning, stabilized targets, and a high-capacity nonlinear value function.

## 6. Relations, extensions, and limitations

MR.Q occupies a specific position in the RL method landscape. It is most closely related, in the original paper’s own account, to **TD7**, but differs by discarding the original input for downstream actor and critic, adding **reward and termination losses**, and making implementation choices such as **multi-step returns**, **unrolled dynamics**, and **categorical reward losses** that are aimed at universality [2501.16142]. Relative to model-based systems such as DreamerV3, TD-MPC2, and Newt, its defining omission is **planning**. The later multitask study sharpens this contrast by writing down Newt’s world-model loss and latent-space planning objective, then stressing that MR.Q never optimizes actions through such a planning problem [2606.05555].

Subsequent work has treated MR.Q as a base representation-learning framework to extend rather than replace. **DR.Q**, or **Debiased model-based Representations for Q-learning**, is described explicitly as “MR.Q + InfoNCE + faded prioritized replay,” retaining the model-based representation framework and TD3-style backbone while adding a mutual-information objective and recency-aware prioritization [2605.11711]. That paper reports that DR.Q **consistently outperforms MR.Q across all benchmarks**, including a **26.8%** gain over MR.Q on **DMC-Visual**, and interprets the gap as evidence that latent consistency alone is not sufficient: information preservation and replay debiasing also matter [2605.11711]. This suggests that MR.Q established a reusable architectural template whose main open questions concern the quality and bias of the learned representation rather than the viability of the overall approach.

The limitations stated in the literature are specific. The original paper says MR.Q is **not equipped for hard exploration** and does not address **non-Markovian environments** [2501.16142]. The multitask study notes that most experiments are on **multitask continuous-control benchmarks**, that the **exact mechanisms** by which predictive objectives help are “not fully understood,” and that planning may still help in some settings even if it is “not necessary” to obtain strong scalability there [2606.05555]. Both accounts therefore resist the stronger conclusion that model-based control is obsolete. The narrower claim is that, across the reported domains, **predictive representation learning may matter more than planning itself** [2606.05555].

In that sense, MR.Q is best understood not merely as a particular off-policy algorithm, but as a methodological argument about what should be retained from model-based RL. It keeps **latent dynamics, reward, and termination prediction** as dense supervisory signals; it rejects **planning, search, and imagined rollouts for control**; and it argues, through both theory and experiment, that this combination can yield a broadly competitive and computationally lighter agent across diverse benchmarks [2501.16142; 2606.05555].

Source: https://www.emergentmind.com/topics/mr-q