Reward-Conditioned Data Modeling

• Reward-conditioned data modeling is a paradigm that integrates reward, preference, and goal signals into the training process to reshape model representations and decision routing.
• It encompasses diverse conditioning regimes such as task routing, goal-conditioned representation learning, generative pretraining, and reference-aware evaluation, each adapting the architecture to specific signals.
• Empirical evaluations show that these methods can mitigate multi-task interference, reduce reward hacking, and accelerate convergence across applications from language models to super-resolution.

Searching arXiv for papers on reward-conditioned modeling and related reward modeling formulations. arxiv_search(query="reward-conditioned data modeling reward-conditioned representations reward-conditioned pretraining reward model", max_results=10, sort_by="relevance") Reward-conditioned data modeling, as the term is suggested by recent work, denotes a family of learning strategies in which reward, preference, goal, or reference signals are injected directly into model training or inference rather than being used only for post-hoc evaluation. Across language reward modeling, text-to-image generation, and super-resolution, these signals appear in several forms: scalar preference labels over response pairs, capability-specific preference labels, future goal-state prototypes, vectors of multiple reward-model scores, and low-resolution inputs treated as semantic anchors. The resulting systems do not merely predict a scalar reward; they restructure representation learning, routing, conditioning, and policy optimization around reward-bearing data (Quan, 2024, Nath et al., 2024, Dufour et al., 29 Oct 2025, Song et al., 25 Mar 2026).

1. Conditioning regimes and problem formulations

A useful synthesis is that recent work instantiates reward conditioning at four distinct levels: task routing, representation learning, generative conditioning, and reference-aware evaluation. In preference-based language reward modeling, the basic object is a dataset of preference pairs $(x,y_w,y_l)$ with a scalar reward model trained by pairwise logistic loss. In goal-conditioned representation learning, this scalar objective is augmented with a contrastive signal over preferred and dispreferred trajectory segments. In text-to-image generation, reward signals are converted into conditioning tokens and supplied to the generator during pretraining. In super-resolution, the reward model itself is conditioned on the low-resolution input, so that a reconstruction is evaluated relative to its source rather than against a global no-reference metric (Quan, 2024, Nath et al., 2024, Dufour et al., 29 Oct 2025, Song et al., 25 Mar 2026).

Paper	Conditioning signal	Principal mechanism
"DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling" (Quan, 2024)	Task categories and capability preference labels	Outer sparse MoE router, inner dense MoE with LoRA experts, MLP aggregation
"Learning Goal-Conditioned Representations for Language Reward Models" (Nath et al., 2024)	Future goal states from preferred trajectories	Joint preference-ranking and contrastive goal-conditioned loss
"MIRO: MultI-Reward cOnditioned pretraining improves T2I quality and efficiency" (Dufour et al., 29 Oct 2025)	Binned scores from multiple reward models	Reward tokens prepended to text tokens in a flow-matching generator
"RefReward-SR: LR-Conditioned Reward Modeling for Preference-Aligned Super-Resolution" (Song et al., 25 Mar 2026)	Low-resolution image and LR-conditioned pairwise rankings	MLLM reward model with LR-HR joint reasoning and GRPO-based SR optimization

This taxonomy suggests that reward-conditioned data modeling is not a single algorithmic template. Rather, it is a design principle: the reward-bearing structure of the data is elevated to a first-class conditioning variable.

2. Preference-pair modeling and conditional decomposition in language reward models

In the standard preference-ranking formulation, a reward model assigns a scalar score $r(y\mid x)\in[0,1]$ and is trained by minimizing

$$\mathcal{L}=-\mathbb{E}_{(x,y_w,y_l)\sim D}\Bigl[\log\,\sigma\bigl(r(y_w\mid x)-r(y_l\mid x)\bigr)\Bigr].$$

"DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling" (Quan, 2024) retains this objective but changes the data model around it. The paper identifies two challenges in reward-model training: multi-task disturbance when one reward model is trained on various categories of data, and substantial annotation noise arising from a human annotation consistency rate of only $60\%$ to $75\%$. Its response is a Double-Layer MoE RM.

The outer layer is a sparse MoE. A frozen top-1 gating network computes

$$g_t(x)=\frac{\exp(u_t(x))}{\sum_{t'}\exp(u_{t'}(x))},\qquad t^*(x)=\arg\max_t g_t(x),$$

and only the corresponding task-specific expert is activated:

$r(y\mid x)={\rm RM}_{t^*(x)}(y\mid x).$

The stated purpose is to avoid multi-task interference by routing each prompt to a task-specific submodel. The inner layer is a dense MoE operating within a task. Here the preferred response is decomposed into multiple capability dimensions, and a LoRA expert is fine-tuned for each capability on the same task-specific data but relabeled for that capability. Each expert produces a hidden representation $z_i(x,y)$ and scalar score

$r_i=\sigma(w_i^\top z_i+b_i).$

The expert embeddings are concatenated and passed through a two-layer MLP,

$$h=\mathrm{ReLU}(W_1 z+b_1),\qquad r=\sigma(W_2 h+b_2),$$

with the explicit rationale that aggregating embeddings rather than only the scalar $r(y\mid x)\in[0,1]$0 allows the model to learn nonlinear correlations among capabilities.

The training pipeline is correspondingly staged. For each task, $r(y\mid x)\in[0,1]$1 of the task-specific preference data is used to full-parameter fine-tune a task-specialist base RM; LoRA capability experts are then trained on API-derived single-capability labels; the remaining $r(y\mid x)\in[0,1]$2 of task data is used to train only the MLP aggregator with all base and LoRA parameters frozen. Capability labels are obtained by calling a public LLM API twice with swapped response order and retaining only pairs where both calls agree, a procedure intended both to reduce left-right bias and to clean noisy pairs.

The reported empirical setting contains 21,680 prompts, 98% Chinese, in five categories: roleplay, chitchat, text creation, obj. QA, and subj. QA. Each prompt has 2–8 responses ranked by two annotators, with 74% inter-annotator agreement. On human-consistency accuracy, DMoERM achieves approximately 71% overall versus 62–63% for the best baseline, while a version without the outer router still reaches approximately 68%. In Best-of-$r(y\mid x)\in[0,1]$3 sampling under equal KL budgets up to approximately 10 nats, the model drives gold-RM scores higher without overoptimization, whereas baselines peak and then degrade. In PPO-based RLHF, at 3,000 PPO steps, DMoERM-scored policies outperform baseline-scored policies in both gold-RM and human judgments and show less reward hacking (Quan, 2024).

A central implication is that reward-conditioned data modeling can be implemented not only by changing the scalar target, but by conditioning the model on structured decompositions of why a sample is preferred.

3. Goal-conditioned representation learning and token-level steerability

"Learning Goal-Conditioned Representations for Language Reward Models" (Nath et al., 2024) modifies the conventional reward-model objective by imposing a representation geometry over preferred trajectories. The baseline loss is again the pairwise logistic objective,

$r(y\mid x)\in[0,1]$4

but it is augmented with a goal-conditioned contrastive term. From each preference pair, the method samples a positive trajectory segment $r(y\mid x)\in[0,1]$5 from the preferred completion and a negative segment $r(y\mid x)\in[0,1]$6 where either the action or the goal comes from the dispreferred completion. Using transformer hidden vectors $r(y\mid x)\in[0,1]$7 and $r(y\mid x)\in[0,1]$8, the critic score is defined as cosine similarity,

$r(y\mid x)\in[0,1]$9

The contrastive loss is

$$\mathcal{L}=-\mathbb{E}_{(x,y_w,y_l)\sim D}\Bigl[\log\,\sigma\bigl(r(y_w\mid x)-r(y_l\mid x)\bigr)\Bigr].$$0

and the joint objective becomes

$$\mathcal{L}=-\mathbb{E}_{(x,y_w,y_l)\sim D}\Bigl[\log\,\sigma\bigl(r(y_w\mid x)-r(y_l\mid x)\bigr)\Bigr].$$1

The architecture consists of a pretrained, instruction-tuned decoder-only transformer as feature extractor $$\mathcal{L}=-\mathbb{E}_{(x,y_w,y_l)\sim D}\Bigl[\log\,\sigma\bigl(r(y_w\mid x)-r(y_l\mid x)\bigr)\Bigr].$$2, together with a reward projection head $$\mathcal{L}=-\mathbb{E}_{(x,y_w,y_l)\sim D}\Bigl[\log\,\sigma\bigl(r(y_w\mid x)-r(y_l\mid x)\bigr)\Bigr].$$3 that maps the final hidden state to a scalar reward. For the contrastive branch, hidden-state cosine similarity serves directly as the goal-conditioned score. The authors characterize this as encouraging the hidden representations to behave like a goal-conditioned $$\mathcal{L}=-\mathbb{E}_{(x,y_w,y_l)\sim D}\Bigl[\log\,\sigma\bigl(r(y_w\mid x)-r(y_l\mid x)\bigr)\Bigr].$$4-function.

The training pipeline is explicit. For mathematical reasoning, OpenMathInstruct-1 is used to construct approximately 257K preferences over approximately 12K problems by matching correct versus incorrect Python-executed solutions. For natural language alignment, the Helpful-Harmless dataset contributes 169K human-labeled preference pairs. Optimization details are given separately for math and Helpful-Harmless; all models are trained on 8 × A100 GPUs with FlashAttention-2 and DeepSpeed.

The reported gains are substantial in the paper’s chosen benchmarks. AUROC improves from 0.805 to 0.852 on GSM8k and from 0.618 to 0.708 on MATH, with average out-of-distribution gains of approximately 0.04 and a maximum gain of +0.08 on svamp. On Helpful-Harmless, a Llama 3-8B baseline reward model improves from 68.2% to 70.5% accuracy. The learned representations also support inference-time control. A prototype goal

$$\mathcal{L}=-\mathbb{E}_{(x,y_w,y_l)\sim D}\Bigl[\log\,\sigma\bigl(r(y_w\mid x)-r(y_l\mid x)\bigr)\Bigr].$$5

defines a token-level score

$$\mathcal{L}=-\mathbb{E}_{(x,y_w,y_l)\sim D}\Bigl[\log\,\sigma\bigl(r(y_w\mid x)-r(y_l\mid x)\bigr)\Bigr].$$6

This quantity is used for token filtering in majority-vote decoding: on GSM8k, Maj.@50 accuracy rises from 84.8% to 86.0% while filtering 35.9% of trajectories and saving 25.5% of tokens; on MATH, accuracy rises from 55.6% to 59.6% while filtering 65.6% of trajectories and saving 55.6% of tokens. In guided decoding on HelpSteer, the same mechanism steers a Llama 3 model toward helpfulness, correctness, coherence, and complexity, with reported win-rate improvements of +9.6%, +12.2%, +16.5%, and +21.6% respectively over a supervised-fine-tuning guided baseline (Nath et al., 2024).

The same paper also states several limitations. Inference-time goal choice currently uses a global mean or small curated set; partial Q-values correlate strongly with scalar reward outputs and may invite reward hacking in an RL loop; and all experiments are off-policy, leaving on-policy retraining and direct integration of the Q-signal into RL as open directions. This is an important qualification: reward-conditioned data modeling can improve steerability without eliminating optimization pathologies by construction.

4. Multi-reward conditioning in generative pretraining

"MIRO: MultI-Reward cOnditioned pretraining improves T2I quality and efficiency" (Dufour et al., 29 Oct 2025) moves reward conditioning from reward-model training into the generative model itself. Its base architecture is a continuous-time flow-matching model with a Transformer-based U-Net, modified with SwiGLU feed-forward layers, RMSNorm instead of LayerNorm, and QK-Norm in self- and cross-attention. Rather than optimizing separate reward losses, the method augments the training dataset with a vector of binned reward scores $$\mathcal{L}=-\mathbb{E}_{(x,y_w,y_l)\sim D}\Bigl[\log\,\sigma\bigl(r(y_w\mid x)-r(y_l\mid x)\bigr)\Bigr].$$7, one discrete channel per reward model.

For each sample $$\mathcal{L}=-\mathbb{E}_{(x,y_w,y_l)\sim D}\Bigl[\log\,\sigma\bigl(r(y_w\mid x)-r(y_l\mid x)\bigr)\Bigr].$$8 and reward model $$\mathcal{L}=-\mathbb{E}_{(x,y_w,y_l)\sim D}\Bigl[\log\,\sigma\bigl(r(y_w\mid x)-r(y_l\mid x)\bigr)\Bigr].$$9, a raw score $60\%$0 is computed and quantile-binned on a held-out calibration subset. Each discrete reward index is embedded by a small sinusoidal embedding and linear projection, producing reward tokens $60\%$1 that are prepended to the text token sequence. All text and reward tokens pass through the same Transformer layers. Reward conditioning is therefore fused by the model’s standard attention mechanisms rather than by bespoke intermediate concatenation.

The training objective remains the standard flow-matching regression loss,

$60\%$2

with $60\%$3, $60\%$4, and $60\%$5. The paper stresses that there is no separate KL or cross-entropy term and that multiple reward models are not combined as weighted losses; they jointly parameterize the conditioning vector.

The reported setup uses $60\%$6 bins, approximately 16M images from CC12M and LAION-Aesthetics6+ at $60\%$7 resolution, batch size 1,024, 500K training steps, and Adam with learning rate $60\%$8, $60\%$9, $75\%$0, $75\%$1, and no weight decay. No multi-stage training is used. The paper attributes stability to RMSNorm, QK-Norm, and SwiGLU, and states that the full spectrum of data quality prevents mode collapse on low-reward modes.

Empirically, MIRO improves a baseline with no reward conditioning from 52 to 57 on GenEval overall, from 0.524 to 1.06 on ImageReward, from 0.247 to 0.29 on HPSv2, and from 5.18 to 6.28 on Aesthetic. With +50% synthetic captions, GenEval reaches 68. The paper states that MIRO outperforms both single-reward models and FLUX-dev while using approximately 370× less compute. On GenEval sub-tasks, compositional scores improve by 12–39%. Training curves show that MIRO reaches the baseline’s final reward values in 3×–19× fewer steps, and test-time scaling shows that it matches the baseline’s best-of-128 performance using only 4–16 samples, up to 32× faster at inference. For sampling, the paper introduces a multi-reward classifier-free guidance form,

$75\%$2

A recurring claim in the paper is that conditioning on the full reward vector avoids the distortions of single-reward optimization. Single-reward models achieve high metric $75\%$3 but degrade on $75\%$4, whereas MIRO trains on every sample with its reward vector rather than discarding low-reward data (Dufour et al., 29 Oct 2025).

5. Reference-aware reward conditioning in super-resolution

"RefReward-SR: LR-Conditioned Reward Modeling for Preference-Aligned Super-Resolution" (Song et al., 25 Mar 2026) addresses a different failure mode: super-resolution metrics that are misaligned with human perception because they either punish semantically plausible detail due to pixel misalignment or reward visually sharp but inconsistent artifacts. The proposed alternative is an LR reference-aware reward model that assesses high-resolution reconstructions conditioned on their low-resolution inputs, with the LR image serving as a semantic anchor.

Formally, for low-resolution input $75\%$5 and super-resolved candidate $75\%$6, the reward model outputs

$75\%$7

Internally, the model combines a format reward for the MLLM output structure with an LR-conditioned pairwise ranking reward. For each LR anchor, annotation groups contain $75\%$8 HR candidates with pairwise preferences $75\%$9. The policy samples $$g_t(x)=\frac{\exp(u_t(x))}{\sum_{t'}\exp(u_{t'}(x))},\qquad t^*(x)=\arg\max_t g_t(x),$$0 quality scores for each candidate, and a Thurstone-style win probability is computed:

$$g_t(x)=\frac{\exp(u_t(x))}{\sum_{t'}\exp(u_{t'}(x))},\qquad t^*(x)=\arg\max_t g_t(x),$$1

The continuous ranking reward for rollout $$g_t(x)=\frac{\exp(u_t(x))}{\sum_{t'}\exp(u_{t'}(x))},\qquad t^*(x)=\arg\max_t g_t(x),$$2 is

$$g_t(x)=\frac{\exp(u_t(x))}{\sum_{t'}\exp(u_{t'}(x))},\qquad t^*(x)=\arg\max_t g_t(x),$$3

At inference time, the total reward is a weighted sum of format correctness and average pairwise reward across rollouts. Architecturally, the reward model is a multimodal LLM such as Qwen3-VL 8B, with a visual encoder for $$g_t(x)=\frac{\exp(u_t(x))}{\sum_{t'}\exp(u_{t'}(x))},\qquad t^*(x)=\arg\max_t g_t(x),$$4 and $$g_t(x)=\frac{\exp(u_t(x))}{\sum_{t'}\exp(u_{t'}(x))},\qquad t^*(x)=\arg\max_t g_t(x),$$5, a decoder transformer with cross-attention over the fused visual tokens, and a scalar head producing quality scores. To capture detail, the framework also introduces global-local crop scoring. Local crops are detected through a RAM → Grounding-DINO → NMS and filtering pipeline, with the top $$g_t(x)=\frac{\exp(u_t(x))}{\sum_{t'}\exp(u_{t'}(x))},\qquad t^*(x)=\arg\max_t g_t(x),$$6 diverse boxes retained, and the final semantic score is area-weighted:

$$g_t(x)=\frac{\exp(u_t(x))}{\sum_{t'}\exp(u_{t'}(x))},\qquad t^*(x)=\arg\max_t g_t(x),$$7

The reward model is then integrated into super-resolution training through Group Relative Policy Optimization. For a group of reconstructions $$g_t(x)=\frac{\exp(u_t(x))}{\sum_{t'}\exp(u_{t'}(x))},\qquad t^*(x)=\arg\max_t g_t(x),$$8 sampled from the SR generator, the objective is

$$g_t(x)=\frac{\exp(u_t(x))}{\sum_{t'}\exp(u_{t'}(x))},\qquad t^*(x)=\arg\max_t g_t(x),$$9

with advantage $r(y\mid x)={\rm RM}_{t^*(x)}(y\mid x).$0. In the integrated SR setting, the total reward becomes

$r(y\mid x)={\rm RM}_{t^*(x)}(y\mid x).$1

The dataset, RefSR-18K, is described as the first large-scale LR-conditioned preference dataset for SR. It begins from 5,130 diverse HQ images from LSDIR; after degradation to LR and generation of candidate pools by 8 state-of-the-art SR methods plus GT, annotators rank groups of four HR candidates by semantic consistency to the LR input and naturalness. After filtering, 4,699 groups and 18,796 samples are retained.

The reported results include agreement of approximately 85% in-domain and approximately 80% out-of-domain, Recall@1 of approximately 84.5%, and Filter@1 of approximately 78%, compared with approximately 84.7% human on the cited metric. Relative to FR and NR baselines and zero-shot MLLMs, the method outperforms on all three alignment metrics. When integrated into SR via GRPO, it yields consistent improvements over the base C-FLUX model and its DPO variant; a user study on 20 images with 15 experts reports a 72.7% win rate versus 67.1% for DPO and 60.2% for the base model (Song et al., 25 Mar 2026).

6. Unifying themes, misconceptions, and open directions

Taken together, these works imply that reward-conditioned data modeling should not be reduced to a single scalar reward regressor. The conditioning object may be a task label, a capability decomposition, a future goal-state embedding, a multi-reward vector, or a source reference such as an LR image. The model component being conditioned also varies: a router in DMoERM, hidden-state geometry in goal-conditioned language RMs, the generator itself in MIRO, and the reward model’s multimodal input in RefReward-SR (Quan, 2024, Nath et al., 2024, Dufour et al., 29 Oct 2025, Song et al., 25 Mar 2026).

One common misconception is that reward conditioning is equivalent to post-hoc rank-and-select. The evidence here runs against that simplification. DMoERM changes the reward model’s architecture and supervision granularity; the goal-conditioned language work modifies the latent representation with a contrastive auxiliary objective; MIRO pushes reward vectors directly into pretraining rather than discarding samples after generation; and RefReward-SR defines a reward function only relative to a semantic anchor and then optimizes the generator with GRPO. This suggests that reward-conditioned data modeling is better understood as a data-design and conditioning paradigm than as a mere selection heuristic.

A second misconception is that adding reward information necessarily collapses diversity or resolves reward hacking automatically. The papers offer a more qualified picture. MIRO argues that full-spectrum data usage and multi-reward conditioning prevent mode collapse on low-reward modes and avoid overfitting to a single metric. DMoERM reports mitigation of overoptimization in BoN and less reward hacking in PPO. By contrast, the goal-conditioned representation paper explicitly notes that partial Q-values can correlate strongly with scalar reward outputs and may invite reward hacking in an RL loop, although an MLP head can partially decouple them. RefReward-SR balances semantic reward with LPIPS and DEQA in its total reward, indicating that preference alignment in perceptual tasks may require composite objectives rather than a single learned score (Dufour et al., 29 Oct 2025, Quan, 2024, Nath et al., 2024, Song et al., 25 Mar 2026).

The open directions named in these works are similarly heterogeneous. Dynamic or more precise goal selection remains open in goal-conditioned language reward models; on-policy retraining of reward and policy is identified as a further step; direct integration of the learned Q-signal into RL is proposed as a next direction. MIRO explicitly recommends treating each reward model as a separate conditioning channel, normalizing reward scales via quantile binning, and applying the same framework to other modalities such as audio, text, and molecule generation. RefReward-SR, for its part, extends reward conditioning toward reasoning-aware multimodal evaluation grounded in paired inputs. A plausible overall implication is that future progress will depend less on inventing a universal reward scalar than on designing conditioning interfaces that preserve the structure of the supervision signal.