MR.Q: Model-based Representations for Q-Learning

• MR.Q is a reinforcement learning method that uses model-based training signals for auxiliary predictive tasks to shape latent representations without employing planning.
• It integrates an actor–critic framework with twin Q-networks and predictive heads for dynamics, reward, and termination, achieving improved sample efficiency across various benchmarks.
• Empirical results demonstrate that MR.Q achieves competitive performance and computational efficiency on diverse tasks such as Gym locomotion, DMC, and Atari.

Searching arXiv for the MR.Q reinforcement-learning papers and closely related work. MR.Q, expanded in the appendix as Model-based Representations for Q-learning, is a reinforcement-learning method that uses model-based training signals to learn latent representations while keeping control itself model-free. In the literature, it is described both as a general-purpose model-free deep RL algorithm across continuous and discrete action spaces, vector and pixel observations, and as a purely model-free multitask actor–critic agent that borrows model-based prediction only for representation learning, not for planning (Fujimoto et al., 27 Jan 2025, Obando-Ceron et al., 4 Jun 2026). Its central premise is that much of the empirical benefit often attributed to world models may derive from predictive representation learning rather than from planning or imagined rollouts. MR.Q therefore combines an off-policy actor–critic backbone with auxiliary prediction of latent dynamics, reward, and termination, seeking the sample efficiency and representational structure of model-based methods without model-predictive control, latent rollout planning, or imagination-based policy optimization (Fujimoto et al., 27 Jan 2025, Obando-Ceron et al., 4 Jun 2026).

1. Conceptual definition and problem setting

MR.Q was proposed against two related backgrounds. The first is the search for a single model-free method that can address “a diverse class of domains and problem settings” with one hyperparameter set rather than benchmark-specific retuning (Fujimoto et al., 27 Jan 2025). The second is the multitask RL question of whether the scalability of recent world-model systems is driven primarily by planning, or by the representations induced by predictive learning (Obando-Ceron et al., 4 Jun 2026).

The method is explicitly positioned between standard model-free and model-based RL. On the control side, policy improvement is done by actor–critic learning, with twin critics in the TD3 style. On the auxiliary side, MR.Q learns predictive models of next representation, reward, and termination, but these are used only to shape the latent space. It is therefore not a world-model planner like Dreamer, TD-MPC2, or Newt, and it does not perform latent rollouts for action selection (Fujimoto et al., 27 Jan 2025, Obando-Ceron et al., 4 Jun 2026).

In the multitask setting, training is over tasks $\tau \sim p(\tau)$ with objective

$$\mathbb{E}_{\tau \sim p(\tau), \pi}\left[\sum_{t=0}^{\infty}\gamma^t r_t\right].$$

The motivation given for this setting is that scaling RL across many tasks is hindered by non-stationary replay data, task interference, unstable bootstrapping, poor representation quality, and underuse of model capacity. The proposed remedy is not merely to increase parameter count, but to supply larger networks with denser predictive supervision so that they learn “stable, reusable features” (Obando-Ceron et al., 4 Jun 2026).

A recurrent misconception addressed in the literature is that MR.Q is simply TD3 with extra heads. That is incomplete. The method’s defining claim is that latent dynamics and reward prediction can make value estimation easier by learning a representation in which the value function is closer to linear, while the final critic remains nonlinear in practice (Fujimoto et al., 27 Jan 2025).

2. Theoretical basis: approximate linearization of value

The original theoretical framing introduces state and state-action embeddings

$$f_\omega: s \rightarrow \mathbf{z}_s, \qquad g_\omega: (s,a) \rightarrow \mathbf{z}_{sa},$$

and motivates them through a linear decomposition

$Q(s,a) = \mathbf{z}_{sa}^\top \mathbf{w}.$

The point is not that MR.Q ultimately uses a linear critic; the paper explicitly shows that replacing the nonlinear value network with a linear one is “disastrous.” Rather, the claim is that a good representation should make value approximately linear in latent space (Fujimoto et al., 27 Jan 2025).

The original analysis compares a model-free linear TD update with a model-based linear solution obtained from linear reward prediction and linear latent dynamics. Theorem 1 states that “the fixed point of the model-free approach and the solution of the model-based approach are the same” in linear space. Theorem 2 then bounds value error by reward and dynamics prediction errors. In the notation of the paper, the model-based feature-learning objective is

$$\mathcal{L}(\mathbf{z}_{sa}, \mathbf{w}_r, W_p) = \underbrace{\mathbb{E}_D \left[ \left( \mathbf{z}_{sa}^\top \mathbf{w}_r - r \right)^2 \right]}_{\text{reward learning}} + \underbrace{\lambda \mathbb{E}_D \left[ \left( \mathbf{z}_{sa}^\top W_p - \mathbf{z}_{s'a'} \right)^2 \right]}_{\text{dynamics learning}}.$$

This is used to argue that if latent reward and transition structure are easy to predict, then downstream value prediction should also become easier (Fujimoto et al., 27 Jan 2025).

The practical algorithm deliberately relaxes this theory. The next-state target is changed from a state-action target $\mathbf{z}_{s'a'}$ to a state target $\bar{\mathbf{z}}_{s'}$, making the target independent of the current policy and less non-stationary. The paper writes the relaxed objective as

$$\mathcal{L}(\mathbf{z}_{sa}, \mathbf{w}_r, W_p) = \mathbb{E}_D \left[ \left( \mathbf{z}_{sa}^\top \mathbf{w}_r - r \right)^2 \right] + \lambda \mathbb{E}_D \left[ \left( \mathbf{z}_{sa}^\top W_p - \bar{\mathbf{z}}_{s'} \right)^2 \right].$$

A target encoder is introduced to stabilize $\bar{\mathbf{z}}_{s'}$. The exact linearity guarantees are then dropped, and the method shifts to the weaker but operationally central claim that predictive representation learning can approximately linearize value well enough for a high-capacity nonlinear critic to exploit (Fujimoto et al., 27 Jan 2025).

This theoretical construction is echoed by later multitask analysis. There, MR.Q is treated as the central example of a system in which “predictive representation learning + high-capacity value functions” is sufficient to scale well, even “without planning” (Obando-Ceron et al., 4 Jun 2026).

3. Architecture and objectives

The core MR.Q architecture consists of a state encoder, a state-action encoder, a latent predictive model, a policy, and twin Q-networks (Fujimoto et al., 27 Jan 2025). In the multitask formulation, observations and optional task information are first mapped to a latent state

$z_t = \phi_\xi(s_t, \tau),$

with task conditioning implemented through a learned task embedding $$\mathbb{E}_{\tau \sim p(\tau), \pi}\left[\sum_{t=0}^{\infty}\gamma^t r_t\right].$$0 when task information is available. In the MMBench experiments, this is a language instruction embedding concatenated with state or latent features (Obando-Ceron et al., 4 Jun 2026).

The actor and critics operate in latent space. The multitask paper writes these as

$$\mathbb{E}_{\tau \sim p(\tau), \pi}\left[\sum_{t=0}^{\infty}\gamma^t r_t\right].$$1

while the original paper uses the pair $$\mathbb{E}_{\tau \sim p(\tau), \pi}\left[\sum_{t=0}^{\infty}\gamma^t r_t\right].$$2 with downstream policy and value functions on top. The original architecture specifies a 512-dimensional latent for both $$\mathbb{E}_{\tau \sim p(\tau), \pi}\left[\sum_{t=0}^{\infty}\gamma^t r_t\right].$$3 and $$\mathbb{E}_{\tau \sim p(\tau), \pi}\left[\sum_{t=0}^{\infty}\gamma^t r_t\right].$$4, with modality-specific encoders: a convolutional encoder for images and a 3-layer MLP for vector observations (Fujimoto et al., 27 Jan 2025). The multitask representation analysis also explicitly discusses a 512-dimensional latent space (Obando-Ceron et al., 4 Jun 2026).

MR.Q augments actor–critic learning with predictive heads that take $$\mathbb{E}_{\tau \sim p(\tau), \pi}\left[\sum_{t=0}^{\infty}\gamma^t r_t\right].$$5 and predict

$$\mathbb{E}_{\tau \sim p(\tau), \pi}\left[\sum_{t=0}^{\infty}\gamma^t r_t\right].$$6

These are the latent dynamics, reward, and termination targets. The multitask paper summarizes the total objective as

$$\mathbb{E}_{\tau \sim p(\tau), \pi}\left[\sum_{t=0}^{\infty}\gamma^t r_t\right].$$7

where $$\mathbb{E}_{\tau \sim p(\tau), \pi}\left[\sum_{t=0}^{\infty}\gamma^t r_t\right].$$8 decomposes into losses for dynamics, reward, and termination prediction (Obando-Ceron et al., 4 Jun 2026).

The original paper specifies the encoder loss more explicitly through short latent rollouts: $$\mathbb{E}_{\tau \sim p(\tau), \pi}\left[\sum_{t=0}^{\infty}\gamma^t r_t\right].$$9 The reward loss uses categorical prediction with Two-Hot targets and symexp spacing; the dynamics loss is squared error to a target-encoder latent; the terminal loss is squared error on the done flag. In the practical algorithm, the actor and critic use the learned embeddings “without propagating gradients” back through the encoder, which the paper presents as part of its stabilization strategy (Fujimoto et al., 27 Jan 2025).

Several architectural choices are emphasized as essential rather than incidental. The target encoder matters substantially, the state target is better than a state-action target, multi-step returns help, and a nonlinear critic remains necessary even if the representation approximately linearizes value (Fujimoto et al., 27 Jan 2025).

4. Training procedure and implementation profile

MR.Q is trained off-policy with a replay buffer, target networks, and prioritized replay. In the multitask formulation, the shared replay stores tuples

$$f_\omega: s \rightarrow \mathbf{z}_s, \qquad g_\omega: (s,a) \rightarrow \mathbf{z}_{sa},$$0

and minibatches are sampled uniformly from the shared replay memory to jointly optimize the actor, twin critics, encoder, and auxiliary predictive heads (Obando-Ceron et al., 4 Jun 2026). In the general-purpose formulation, the replay buffer has capacity 1M and uses LAP prioritized sampling with replay ratio 1 and batch size 256 (Fujimoto et al., 27 Jan 2025).

The training loop is structurally simple. Interaction data are collected with noisy policy actions and stored in replay. Sampled batches are encoded into latent states, predictive targets are formed, twin critics are updated with TD learning, the policy is updated to maximize critic values, and target networks are slowly updated or periodically copied in TD3 style (Fujimoto et al., 27 Jan 2025, Obando-Ceron et al., 4 Jun 2026). The original paper makes this schedule unusually explicit: every

$$f_\omega: s \rightarrow \mathbf{z}_s, \qquad g_\omega: (s,a) \rightarrow \mathbf{z}_{sa},$$1

environment steps, encoder, critic, and actor targets are synchronized, and reward-scale statistics are updated (Fujimoto et al., 27 Jan 2025).

The method is designed to cover both continuous and discrete action spaces. For continuous actions, target policy smoothing uses clipped Gaussian noise and actions are clipped to range. For discrete actions, actions are represented internally as one-hot vectors and the target action is an $$f_\omega: s \rightarrow \mathbf{z}_s, \qquad g_\omega: (s,a) \rightarrow \mathbf{z}_{sa},$$2 after noisy logits. The policy activation is $$f_\omega: s \rightarrow \mathbf{z}_s, \qquad g_\omega: (s,a) \rightarrow \mathbf{z}_{sa},$$3 for continuous actions and Gumbel-Softmax for discrete actions (Fujimoto et al., 27 Jan 2025).

The published implementation profile is notable because it is intentionally fixed across benchmarks. The original paper reports the following shared hyperparameters: $$f_\omega: s \rightarrow \mathbf{z}_s, \qquad g_\omega: (s,a) \rightarrow \mathbf{z}_{sa},$$4, $$f_\omega: s \rightarrow \mathbf{z}_s, \qquad g_\omega: (s,a) \rightarrow \mathbf{z}_{sa},$$5, $$f_\omega: s \rightarrow \mathbf{z}_s, \qquad g_\omega: (s,a) \rightarrow \mathbf{z}_{sa},$$6, $$f_\omega: s \rightarrow \mathbf{z}_s, \qquad g_\omega: (s,a) \rightarrow \mathbf{z}_{sa},$$7, $$f_\omega: s \rightarrow \mathbf{z}_s, \qquad g_\omega: (s,a) \rightarrow \mathbf{z}_{sa},$$8, $$f_\omega: s \rightarrow \mathbf{z}_s, \qquad g_\omega: (s,a) \rightarrow \mathbf{z}_{sa},$$9, target policy noise $Q(s,a) = \mathbf{z}_{sa}^\top \mathbf{w}.$0, target noise clip $Q(s,a) = \mathbf{z}_{sa}^\top \mathbf{w}.$1, LAP smoothing $Q(s,a) = \mathbf{z}_{sa}^\top \mathbf{w}.$2, discount $Q(s,a) = \mathbf{z}_{sa}^\top \mathbf{w}.$3, replay buffer $Q(s,a) = \mathbf{z}_{sa}^\top \mathbf{w}.$4M, batch size $Q(s,a) = \mathbf{z}_{sa}^\top \mathbf{w}.$5, and target update frequency $Q(s,a) = \mathbf{z}_{sa}^\top \mathbf{w}.$6 (Fujimoto et al., 27 Jan 2025). The multitask paper does not reproduce a full hyperparameter table, but it states that MR.Q extends TD3, uses a shared multitask replay buffer, conditions on language embeddings when task information is available, and employs a frozen DINOv2 encoder in the pixel experiments (Obando-Ceron et al., 4 Jun 2026).

5. Empirical performance and representational analysis

The original general-purpose evaluation spans 118 environments from four benchmark suites: 5 Gym locomotion tasks, 28 DMC proprioceptive tasks, 28 DMC visual tasks, and 57 Atari games, with 10 seeds in the main results (Fujimoto et al., 27 Jan 2025). On Gym locomotion, MR.Q attains a TD3-normalized aggregate mean of 1.46, below TD7: 1.57 but above TD-MPC2: 1.04, DreamerV3: 0.76, and PPO: 0.45. On DMC proprioceptive control, it is the best aggregate method at 835.95, exceeding TD-MPC2: 783.57, TD7: 566.97, DreamerV3: 530.33, and PPO: 254.04. On DMC visual control, it again leads with 602.28, ahead of DrQ-v2: 510.81, TD-MPC2: 479.87, DreamerV3: 463.48, and PPO: 110.82. On Atari, MR.Q reaches a human-normalized score of 2.54, below DreamerV3: 3.74 but above Rainbow: 1.08, DQN: 0.25, and PPO: 0 (Fujimoto et al., 27 Jan 2025).

The same paper also emphasizes computational profile. On Gym HalfCheetah-v4, MR.Q uses 4.1M parameters, versus 5.5M for TD-MPC2 and 9.7M for DreamerV3; its training speed is 49 FPS versus 18 and 14, and its evaluation speed is 1.9k FPS versus 236 and 27. On Atari, MR.Q uses 4.4M parameters, whereas DreamerV3 uses 187.3M (Fujimoto et al., 27 Jan 2025). This makes computational efficiency a substantive part of the method’s identity rather than an incidental implementation detail.

Later work re-evaluates MR.Q in the large-scale multitask regime and uses it to argue that representation learning, not planning, is the primary driver of scalability (Obando-Ceron et al., 4 Jun 2026). On the 200-task “soup” benchmark at 2M environment steps, MR.Q obtains a normalized score of 0.11 versus 0.08 for Newt, reported as a 37% relative advantage early in training. The lead remains around 5–8% across training, although both methods converge to similar final performance. In held-out-task transfer after pretraining on 200 tasks for 10M steps and finetuning on 28 held-out tasks for 200k steps, MR.Q records 0.13 versus 0.09 zero-shot, 0.55 versus 0.48 at 100k finetuning steps, and 0.62 versus 0.55 at 200k steps, beating Newt on 17/28 tasks (61%) at the end of finetuning (Obando-Ceron et al., 4 Jun 2026).

The representational analysis in the multitask paper is equally central. Compared with an encoder-free baseline, MR.Q is reported to have higher SRank, higher effective dimensionality, less latent collapse, fewer dormant neurons, and lower value loss. In PCA-based analysis, removing predictive representation learning reduces the $Q(s,a) = \mathbf{z}_{sa}^\top \mathbf{w}.$7-variance dimensionality from 89 to 21 on DMControl-Ext and from 66 to 15 on MuJoCo (Obando-Ceron et al., 4 Jun 2026). These results are used to support the claim that predictive supervision “prevents collapse and preserves expressive capacity” in multitask settings.

Ablations in the original paper identify what matters most. Removing model-based representation learning (“No MR”) hurts across Gym, DMC-prop, DMC-visual, and Atari; removing the target encoder hurts substantially; using 1-step returns instead of multi-step returns degrades performance; and replacing the nonlinear critic with a linear one causes catastrophic drops on every benchmark (Fujimoto et al., 27 Jan 2025). A plausible implication is that MR.Q’s empirical strength is not attributable to any single auxiliary prediction head, but to the joint effect of predictive representation learning, stabilized targets, and a high-capacity nonlinear value function.

6. Relations, extensions, and limitations

MR.Q occupies a specific position in the RL method landscape. It is most closely related, in the original paper’s own account, to TD7, but differs by discarding the original input for downstream actor and critic, adding reward and termination losses, and making implementation choices such as multi-step returns, unrolled dynamics, and categorical reward losses that are aimed at universality (Fujimoto et al., 27 Jan 2025). Relative to model-based systems such as DreamerV3, TD-MPC2, and Newt, its defining omission is planning. The later multitask study sharpens this contrast by writing down Newt’s world-model loss and latent-space planning objective, then stressing that MR.Q never optimizes actions through such a planning problem (Obando-Ceron et al., 4 Jun 2026).

Subsequent work has treated MR.Q as a base representation-learning framework to extend rather than replace. DR.Q, or Debiased model-based Representations for Q-learning, is described explicitly as “MR.Q + InfoNCE + faded prioritized replay,” retaining the model-based representation framework and TD3-style backbone while adding a mutual-information objective and recency-aware prioritization (Lyu et al., 12 May 2026). That paper reports that DR.Q consistently outperforms MR.Q across all benchmarks, including a 26.8% gain over MR.Q on DMC-Visual, and interprets the gap as evidence that latent consistency alone is not sufficient: information preservation and replay debiasing also matter (Lyu et al., 12 May 2026). This suggests that MR.Q established a reusable architectural template whose main open questions concern the quality and bias of the learned representation rather than the viability of the overall approach.

The limitations stated in the literature are specific. The original paper says MR.Q is not equipped for hard exploration and does not address non-Markovian environments (Fujimoto et al., 27 Jan 2025). The multitask study notes that most experiments are on multitask continuous-control benchmarks, that the exact mechanisms by which predictive objectives help are “not fully understood,” and that planning may still help in some settings even if it is “not necessary” to obtain strong scalability there (Obando-Ceron et al., 4 Jun 2026). Both accounts therefore resist the stronger conclusion that model-based control is obsolete. The narrower claim is that, across the reported domains, predictive representation learning may matter more than planning itself (Obando-Ceron et al., 4 Jun 2026).

In that sense, MR.Q is best understood not merely as a particular off-policy algorithm, but as a methodological argument about what should be retained from model-based RL. It keeps latent dynamics, reward, and termination prediction as dense supervisory signals; it rejects planning, search, and imagined rollouts for control; and it argues, through both theory and experiment, that this combination can yield a broadly competitive and computationally lighter agent across diverse benchmarks (Fujimoto et al., 27 Jan 2025, Obando-Ceron et al., 4 Jun 2026).