Reward-Conditioned RL: Methods & Insights

• Reward-conditioned RL is a family of methods that conditions policy actions on explicit reward variables (e.g., return-to-go, advantage) to represent diverse behaviors.
• Different formulations use conditioning variables like normalized reward weights or parameterizations to capture reward sensitivity and adapt without repeated retraining.
• Empirical studies show RCRL enhances sample efficiency and performance across online, offline, multi-task, and bandit settings by embedding reward variability into the training process.

Searching arXiv for recent and foundational papers on reward-conditioned reinforcement learning. {"query":"reward-conditioned reinforcement learning reward-conditioned policies hindsight reward tweaking arXiv", "max_results": 10} {"query":"(Wei et al., 2021) OR (Kumar et al., 2019) OR (Nauman et al., 5 Mar 2026) OR (Ding et al., 2023) OR (Xu et al., 2024)", "max_results": 10} Reward-Conditioned Reinforcement Learning (RCRL) is a family of reinforcement-learning methods in which action selection is conditioned on an explicit reward-related variable, so that a single model can represent a family of reward-specific behaviors. Depending on the formulation, the condition may be a desired return-to-go or advantage, a normalized vector of reward weights, or a reward parameterization used to recompute scalar rewards from stored reward components. Across these variants, the central aim is to amortize adaptation to reward variation into the policy itself, thereby replacing repeated retraining with conditioning at training or deployment time (Kumar et al., 2019, Wei et al., 2021, Nauman et al., 5 Mar 2026).

1. Scope and major formulations

The term RCRL covers several closely related formulations rather than a single canonical algorithm. One lineage treats reinforcement learning as conditional supervised learning over achieved outcomes; another conditions actor–critic methods on parameterized reward functions; a more recent framework learns alternate reward objectives entirely off-policy from a replay buffer gathered under a nominal objective; further extensions address offline RL pathologies and bandit-specific inference-time policy construction.

Formulation	Conditioning variable	Representative setting
"Reward-Conditioned Policies" (Kumar et al., 2019)	$G_t$ or $A_t$	Online and fully offline RL
"Hindsight Reward Tweaking via Conditional Deep Reinforcement Learning" (Wei et al., 2021)	normalized reward condition $c$	Deep continuous control
"Reward-Conditioned Reinforcement Learning" (Nauman et al., 5 Mar 2026)	reward parameterization $\psi$	Single-task, multi-task, and vision-based RL
"Bayesian Reparameterization of Reward-Conditioned Reinforcement Learning with Energy-based Models" (Ding et al., 2023)	desired RTG $R$	Offline RL and Atari
"Improving Reward-Conditioned Policies for Multi-Armed Bandits using Normalized Weight Functions" (Xu et al., 2024)	reward-conditioned policy $\pi_\theta(a \mid c, r)$ with generalized marginalization	MABs and contextual MABs

Historically, "Reward-Conditioned Policies" formalized the idea that suboptimal trajectories can still provide optimal supervision for matching the achieved return or advantage, yielding a policy $\pi_\theta(a \mid s, y)$ trained by conditional maximum likelihood (Kumar et al., 2019). "Hindsight Reward Tweaking via Conditional Deep Reinforcement Learning" shifted the conditioning target from achieved outcomes to environment reward parameters, introducing a hyper-policy $\pi(a \mid s, c)$ that spans a family of policies indexed by a normalized reward condition (Wei et al., 2021). The 2026 paper "Reward-Conditioned Reinforcement Learning" generalized the reward-parameterized view into a replay-based off-policy framework that stores reward components and recomputes scalar rewards for alternate parameterizations during updates (Nauman et al., 5 Mar 2026). Subsequent work then specialized the paradigm: BR-RCRL targeted offline RTG generalization and out-of-distribution RTG queries through Bayesian reparameterization (Ding et al., 2023), while bandit-oriented work studied generalized marginalization with normalized weight functions that may be negative at inference time (Xu et al., 2024).

2. Mathematical structure of reward conditioning

Two mathematical patterns recur throughout the literature. The first conditions on a desired numerical outcome. In "Reward-Conditioned Policies," an infinite-horizon discounted MDP $M = (\mathcal{S}, \mathcal{A}, P, r, \gamma)$ is paired with either total return $R(\tau) = \sum_{t=0}^{T} r_t$, discounted return-to-go

$A_t$0

or estimated advantage

$A_t$1

The policy is written as $A_t$2, where $A_t$3 is either $A_t$4 or $A_t$5, and the basic learning rule is the conditional supervised objective

$A_t$6

A weighted variant uses $A_t$7 (Kumar et al., 2019).

The second pattern conditions on reward specifications themselves. In "Hindsight Reward Tweaking via Conditional Deep Reinforcement Learning," the MDP is $A_t$8 with parametric reward

$A_t$9

where $c$0 are predefined indicator features and $c$1 are reward parameters. One anchor dimension is fixed with constant $c$2, the remaining dimensions are studied in a reduced non-redundant space, and an affine normalization map $c$3 projects reward parameters into a condition space. The conditioned reward is then

$c$4

and the hyper-policy and value functions become

$c$5

Training randomizes reward configurations and optimizes

$c$6

This formulation explicitly models how changes in reward weights alter optimal behavior within a near-optimal region (Wei et al., 2021).

A related but operationally distinct parameterization appears in the 2026 RCRL framework. There, the environment exposes reward components $c$7, and scalar reward is reconstructed as

$c$8

The policy and critic are conditioned on $c$9, and experience is collected under a nominal parameterization $\psi$0 while alternate rewards are learned counterfactually from the same replay data (Nauman et al., 5 Mar 2026).

3. Learning mechanisms and architectural patterns

The supervised RCP family trains on replay tuples $\psi$1 derived from trajectories. "Reward-Conditioned Policies" studies two variants: return-conditioned RCP-R with $\psi$2, and advantage-conditioned RCP-A with $\psi$3, where $\psi$4 is fitted by TD($\psi$5). The policy network is a three-layer fully connected Gaussian policy, and instead of simple concatenation of $\psi$6 to $\psi$7, the architecture uses multiplicative feature-wise modulation with an embedding of $\psi$8 injected into intermediate layers. The training loop collects 2000 environment steps per iteration, stores tuples in a FIFO ring buffer with default size 100k, updates the policy with 1000 gradient steps per iteration using minibatches of size 256, and, for RCP-A, updates the value function for 200 gradient steps per iteration. Target-conditioning is controlled by a Gaussian $\psi$9 over target returns or advantages, updated by exponential tilting toward larger $R$0 via log-sum-exp statistics (Kumar et al., 2019).

Conditional deep RL in the hindsight reward tweaking setting modifies conventional actor–critic algorithms by inserting the reward condition directly into policy and value function inputs. The default mechanism is concatenation, $R$1, though feature-wise modulation or gating is noted as an option for high-dimensional observations. The paper gives conditioned SAC-, TD3-, and PPO/A3C-style losses, all of which include $R$2 in both policy and critic arguments. Its concrete implementations use conditional A3C/PPO with three fully connected hidden layers of 256 units with tanh, and conditional DDPG with two hidden layers of 64 with ReLU and LayerNorm after each layer. The training loop samples reward parameters uniformly from a near-optimal subspace, maps them to $R$3, appends $R$4 to observations, computes $R$5, stores conditioned transitions, and periodically refreshes reward parameters per environment to diversify the data (Wei et al., 2021).

The off-policy 2026 RCRL framework changes the data path rather than the collection policy. All interaction is gathered while conditioning on $R$6. Each replay tuple stores $R$7 together with reward components $R$8, so that scalar rewards can be recomputed for any $R$9 during training. For every transition in a minibatch, the learner samples $\pi_\theta(a \mid c, r)$0 independently from a nominal–auxiliary mixture that effectively means sampling $\pi_\theta(a \mid c, r)$1 with probability $\pi_\theta(a \mid c, r)$2 and $\pi_\theta(a \mid c, r)$3 otherwise; in experiments $\pi_\theta(a \mid c, r)$4. Continuous parameterizations are fed by concatenating $\pi_\theta(a \mid c, r)$5 or $\pi_\theta(a \mid c, r)$6 to the state, while finite auxiliary tasks use a learned embedding table. The paper emphasizes that no importance sampling is required because rewards change but dynamics and behavior data remain the same (Nauman et al., 5 Mar 2026).

These mechanisms suggest a useful taxonomy. Some RCRL methods relabel trajectories with achieved outcomes and train conditionally; others relabel rewards counterfactually by recomputing them from stored components; others train online over randomized reward configurations. The unifying feature is not the source of supervision but the conditioning of policy behavior on reward information.

4. Theoretical interpretations and relations to adjacent paradigms

The strongest theoretical derivation appears in "Reward-Conditioned Policies." For return conditioning, the paper formulates a constrained policy-search problem over the joint trajectory-return distribution,

$\pi_\theta(a \mid c, r)$7

The resulting nonparametric solution is

$\pi_\theta(a \mid c, r)$8

which decomposes into $\pi_\theta(a \mid c, r)$9 and $\pi_\theta(a \mid s, y)$0. That decomposition motivates unweighted conditional maximum likelihood for $\pi_\theta(a \mid s, y)$1, while the target-value distribution is updated by exponential tilting. The advantage-conditioned version uses the policy difference lemma and a trust-region-style approximation to derive an analogous conditional objective over $\pi_\theta(a \mid s, y)$2 rather than $\pi_\theta(a \mid s, y)$3 (Kumar et al., 2019).

This derivation also clarifies RCRL’s relation to earlier weighted-regression methods. RWR, REPS, and AWR learn an unconditioned policy with weights proportional to exponentiated return or advantage. RCP retains the same exponential tilting in the target distribution but replaces weighted regression of an unconditioned policy with conditional regression of $\pi_\theta(a \mid s, y)$4, thereby preserving all samples in the supervised subproblem (Kumar et al., 2019).

BR-RCRL challenges the direct RTG-conditioning template. Its central claim is that vanilla RCRL suffers from "RTG Independence," the tendency to treat different RTG inputs as unrelated prediction problems, and from "Sampling Bias Dominance" when high-RTG slices are data-scarce. The proposed remedy uses the Bayes factorization

$\pi_\theta(a \mid s, y)$5

implemented via an amortized behavior prior $\pi_\theta(a \mid s, y)$6 and an RTG likelihood model $\pi_\theta(a \mid s, y)$7. This induces the energy

$\pi_\theta(a \mid s, y)$8

and the conditional policy becomes an energy-based model. Training combines an InfoNCE-style reconstruction term $\pi_\theta(a \mid s, y)$9 with an RTG-likelihood term $\pi(a \mid s, c)$0, while inference uses an adaptive threshold $\pi(a \mid s, c)$1 to avoid querying out-of-distribution RTGs. The paper further states a KL bound $\pi(a \mid s, c)$2 and a performance improvement result $\pi(a \mid s, c)$3 (Ding et al., 2023).

RCRL is frequently compared with UVFA, goal-conditioned RL, hindsight relabeling, multi-task RL, and meta-RL. The reward-parameterized formulations differ from UVFA and goal-conditioned RL in that they condition on reward specifications rather than target states or goals; they differ from meta-RL because they do not rely on inner-loop adaptation, but instead amortize reward sensitivity into a single forward pass; and they resemble HER in using counterfactual relabeling, although the relabeling variable is reward parameterization rather than goal identity (Wei et al., 2021, Nauman et al., 5 Mar 2026). A common misconception is that reward conditioning is merely goal conditioning under another name. The literature instead treats it as a broader mechanism for controlling dense trade-offs among reward components or desired performance levels.

5. Empirical performance across problem classes

Empirical evaluation spans online continuous control, offline RL, multi-task and vision-based control, and bandits. In the hindsight reward tweaking work, experiments use MuJoCo locomotion tasks—HalfCheetah, Walker2d, and Ant—with Gym’s indicator features and forward reward anchored at $\pi(a \mid s, c)$4. Reward parameters are randomized within bounded ranges, with $\pi(a \mid s, c)$5 for HalfCheetah and Walker2d and $\pi(a \mid s, c)$6 for Ant. The evaluation metric is average forward travel distance over 1000 steps, aggregated over 50 random seeds; outliers from falls are filtered, and runs with more than 10 falls are scored 0. Hindsight evolution heatmaps on Walker2d show multiple high-density regions associated with higher returns, and top-elite RCRL policies during evolution outperform fixed-reward baselines across HalfCheetah, Walker2d, and Ant (Wei et al., 2021).

In the original RCP study, evaluations cover HalfCheetah-v2, Hopper-v2, Walker2d-v2, and LunarLander-v2. RCP-A consistently outperforms RCP-R; against PPO and TRPO, RCP-A often outperforms TRPO and is comparable to PPO, though gaps remain to SAC and DDPG on some tasks. With exponential weights, both RCP variants improve substantially, and RCP-A becomes similar to AWR. A concrete example is LunarLander-v2, where RCP-A reaches $\pi(a \mid s, c)$7, whereas UDRL reports approximately 150. In fully offline learning on static datasets of 1M transitions from partially trained mediocre policies in Hopper-v2 and Walker2d-v2, RCP-A learns policies better than the dataset behavior policy baseline, while RCP-R behaves similarly to behavioral cloning (Kumar et al., 2019).

The 2026 RCRL framework broadens the empirical picture. It is integrated with single-task SimbaV2, multi-task BRC, and vision-based DrQv2; benchmarks include 23 tasks from DeepMind Control Suite, HumanoidBench, and OpenAI Gym, 9 tasks from the DMC Medium benchmark, and three multi-task benchmarks totaling 33 tasks. Under the nominal reward parameterization $\pi(a \mid s, c)$8, reward conditioning improves sample efficiency and final performance; in multi-task settings, the reward-conditioned agent reaches approximately 75% of maximal performance after 150k steps on DMC Dogs and around 80% after approximately 500k steps on HumanoidBench. In HumanoidBench transfers over 8×7 source–target pairs with 10 seeds each, the best pairs achieve up to 40% of optimal performance zero-shot and up to 90% after 250k finetuning steps. Removing conditioning degrades performance by up to 40%, and the best results occur with 30–50% of updates using alternative $\pi(a \mid s, c)$9 (Nauman et al., 5 Mar 2026).

BR-RCRL reports state-of-the-art offline results on D4RL and Atari. On D4RL, average normalized score is 83.2 for BR-RCRL versus 81.8 for Decision Diffuser, 78.9 for Trajectory Transformer, 74.7 for Decision Transformer, 71.7 for RvS, 77.0 for IQL, 77.6 for CQL, 75.3 for TD3+BC, and 51.9 for BC. Representative task-level results include 98.3 on Medium-Replay Hopper, 82.3 on Medium Walker2D, and 95.2 on Medium-Expert HalfCheetah. On Atari with a 10% replay buffer, it reports 1239.2 ± 104.2 on Breakout, 138.0 ± 2.2 on Pong, 117.4 ± 13.5 on Q*bert, and 7.1 ± 3.1 on Seaquest, with up to 11% improvement over vanilla RCRL methods (Ding et al., 2023).

The bandit literature reveals a different empirical pattern. In multi-armed bandits and contextual bandits, standard reward-conditioned policies can underperform classic methods such as UCB and Thompson sampling, but generalized marginalization improves the deployment policy. Across 27 non-contextual configurations with $M = (\mathcal{S}, \mathcal{A}, P, r, \gamma)$0, delays $M = (\mathcal{S}, \mathcal{A}, P, r, \gamma)$1, 5000 observations, and 100 seeds, SubMax is generally the best or competitive RCRL-style policy. In a 10-arm $M = (\mathcal{S}, \mathcal{A}, P, r, \gamma)$2 illustration using 1000 logged observations from a random policy, expected rewards are 0.0816 for random, 0.077 for the $M = (\mathcal{S}, \mathcal{A}, P, r, \gamma)$3-conditioned policy, 0.131 for optimistic conditioning, 0.136 for optimized $M = (\mathcal{S}, \mathcal{A}, P, r, \gamma)$4, and 0.157 for SubMax. Contextual experiments show that SubMax becomes especially competitive in sparse-reward regimes, and CVAE-based reward-conditioned policies remain practical in a 32,768-action combinatorial space (Xu et al., 2024).

6. Limitations, misconceptions, and open research directions

RCRL does not remove the need to specify meaningful reward structure. In the reward-parameterized cDRL setting, if $M = (\mathcal{S}, \mathcal{A}, P, r, \gamma)$5 is misaligned or the designer’s true objectives are not captured by the indicator features, conditioning cannot recover the intended trade-offs. The same work also notes that strong nonlinearity or complex interactions among reward parameters may not be well captured by a linear condition $M = (\mathcal{S}, \mathcal{A}, P, r, \gamma)$6, and that performance can degrade when optimizing $M = (\mathcal{S}, \mathcal{A}, P, r, \gamma)$7 far outside the trained region $M = (\mathcal{S}, \mathcal{A}, P, r, \gamma)$8 (Wei et al., 2021).

Coverage is a second structural limitation. In RCP-style methods, generalization to higher returns requires diverse behavior and at least some trajectories with relatively high $M = (\mathcal{S}, \mathcal{A}, P, r, \gamma)$9 in relevant regions of state space; otherwise, pushing the command variable upward can induce out-of-distribution actions. The paper explicitly notes extrapolation limits, possible failure to use $R(\tau) = \sum_{t=0}^{T} r_t$0 when architecture is weak, and instability from inaccurate value estimates in RCP-A (Kumar et al., 2019). The 2026 replay-based RCRL framework makes the same point in a different form: alternate rewards are learned entirely off-policy from trajectories induced by $R(\tau) = \sum_{t=0}^{T} r_t$1, so reward parameterizations far from the nominal one may be unsupported by replay coverage, and there is no principled method yet to select synergistic $R(\tau) = \sum_{t=0}^{T} r_t$2 sets (Nauman et al., 5 Mar 2026).

A more specific controversy concerns direct RTG conditioning. BR-RCRL argues that vanilla architectures treat RTG values as independent labels and therefore waste low-RTG data when extrapolating to high-RTG queries. Its adaptive inference procedure is an explicit response to the fact that fixed high RTG commands can become out-of-distribution after unfavorable transitions. This suggests that strong deployment performance may depend as much on feasible conditioning and calibration as on the conditional policy class itself (Ding et al., 2023).

Bandit results add another caution. Reward-conditioned policies do not automatically outperform value-based or Bayesian exploration methods, and the generalized-marginalization paper provides no formal regret bounds or convergence rates. Its endpoint optimization in $R(\tau) = \sum_{t=0}^{T} r_t$3 is justified by linearity of the off-policy estimator, but broader optimality claims are left open. Extensions beyond binary rewards and beyond discrete action spaces are also non-trivial (Xu et al., 2024).

Open directions in the literature are correspondingly diverse. Proposed avenues include adversarial or distributionally robust training over reward parameterizations, Bayesian treatments of reward uncertainty, multi-objective scalarization learning, extending conditioning beyond rewards to hyperparameters such as $R(\tau) = \sum_{t=0}^{T} r_t$4 or GAE $R(\tau) = \sum_{t=0}^{T} r_t$5, more principled target-sampling schedules for return-conditioned supervision, sequence-level conditioning that retains the simplicity of per-step supervised objectives, more efficient energy-based inference and calibration for RTG likelihood models, and broader extensions of generalized marginalization to sequential RL (Wei et al., 2021, Kumar et al., 2019, Ding et al., 2023). Taken together, these directions indicate that RCRL is best understood not as a finished algorithmic recipe, but as a general design principle: embed reward variability into the policy interface, and trade repeated optimization over reward design for conditional generalization within the learned model.