Reward Loss Functions

• Reward loss functions are objective functions that assign scalar rewards for desirable actions and penalties for deviations, integrating both human feedback and algorithmic cues.
• They encompass various forms including RP-ε loss, preference-based losses, and adversarial losses, each tailored to optimize performance in regression, reinforcement, and imitation learning tasks.
• Empirical studies demonstrate that proper tuning of these loss functions can lead to significant improvements in metrics like RMSE and segmentation accuracy while addressing challenges in gradient variance and hyperparameter settings.

Reward loss functions are a broad class of objective functions ubiquitous in machine learning, control, and reinforcement learning, where the goal is to shape system outputs or policies according to explicit or inferred reward structures. Such losses appear not only in classical supervised/unsupervised settings, but critically underpin nearly all modern RL, imitation learning, and reward modeling methods, mediating the trade-off between fitting to desirable behaviors (reward) and penalizing discrepancies or undesirable behaviors (loss). The space of reward loss functions includes both handcrafted objectives (e.g., in SVR or segmentation), as well as learned or inferred reward models, sometimes constructed from human feedback, expert demonstrations, or comparisons between generative models.

1. Mathematical Foundations and Representative Forms

At the core, a reward loss function assigns each prediction, action, or trajectory a scalar value, guiding optimization by rewarding desirable outputs and penalizing deviations. Central forms include:

• Regression via Reward-Penalty Loss: The combined reward-cum-penalty loss (\textit{RP-ε loss}) extends the traditional ε-insensitive loss of support vector regression by penalizing residuals outside an ε-tube and assigning a negative (reward) slope for residuals within the tube. The pointwise loss is

$$L_{\tau_1,\tau_2,\epsilon}(u) = \begin{cases} \tau_2 (|u| - \epsilon), & |u| \geq \epsilon \ \tau_1 (|u| - \epsilon), & |u| < \epsilon \end{cases}$$

where $u = y - f(x)$, $\tau_2$ is the penalty outside the tube, and $\tau_1$ is the negative slope (reward) inside the tube (Anand et al., 2019).

• Preference-Based Reward Learning: In reinforcement learning from human feedback, reward learning losses include the Boltzmann cross-entropy for pairwise human comparisons, either parameterized by cumulative return or regret:

$$L_{\rm ret}(\theta) = -\sum_{i=1}^N \left[ y_i \ln P_{\rm ret}(\tau^A_i \succ \tau^B_i | \theta) +(1-y_i) \ln (1-P_{\rm ret}(\tau^A_i \succ \tau^B_i | \theta)) \right]$$

$$L_{\rm reg}(\theta) = -\sum_{i=1}^N \left[ y_i \ln P_{\rm reg}(\tau^A_i \succ \tau^B_i | \theta) +(1-y_i) \ln (1-P_{\rm reg}(\tau^A_i \succ \tau^B_i | \theta)) \right]$$

where $P_{\rm ret}$ is based on segment return, $P_{\rm reg}$ on segment regret (Knox et al., 2022).

• Adversarial/I-Losses: In imitation learning, the reward function is sometimes inferred adversarially from a discriminator. Standard loss forms include the negative log-likelihood of the discriminator's predictions, as well as log-odds ("neutral reward"):

$$R^{\mathrm{neu}}(s,a) = \log D(s,a) - \log(1 - D(s,a))$$

(Jena et al., 2020).

• Diffusion/Optimal Control Fine-Tuning: In stochastic optimal control (SOC), reward loss functions take the form of cost or risk functionals over paths generated by controlled SDEs, with various Monte Carlo estimators and control-theoretic reductions (Domingo-Enrich, 1 Oct 2024), e.g.,

$$\mathcal L_{\rm AdjMatch}(u; X^{\bar u}) = \frac{1}{2} \int_0^T \| u(X^{\bar u}_t, t) + \sigma(t)^\top \tilde{a}(t;X^{\bar u}) \|^2\, dt$$

2. Theoretical Properties and Analysis

Key theoretical properties depend on the construction of the reward loss:

• Convexity: For instance, the RP-ε loss is convex for all $\tau_2 \geq \tau_1 \geq 0$, ensuring convex optimization landscapes when used in SVR (Anand et al., 2019).
• Differentiability and Subgradients: Most losses are piecewise-linear or non-differentiable at regime boundaries (e.g., RP-ε at $u = \pm \epsilon$) but admit subgradients almost everywhere.
• Identifiability: In preference-based reward learning, the regret-based loss is provably identifiable over the optimal policy set, while partial-return based losses can fail to recover true rewards or discount factors in MDPs with variable horizons or stochastic transitions (Knox et al., 2022).
• Robustness: Losses with bounded influence, such as $\epsilon$-insensitive and RP-ε, are robust to outliers. Neural network-based reward models, unless regularized, may be vulnerable to exploitability or overfitting (Xu et al., 2019).
• Gradient Variance: In SOC, multiple loss forms share identical expected gradients but differ in gradient variance, significantly impacting sample efficiency and convergence stability (Domingo-Enrich, 1 Oct 2024).

3. Methodological Advances and Variants

The design and selection of reward loss functions has spawned several methodological innovations:

• Combined or Weighted Losses: Losses combining reward (for desired behavior) and penalty (for errors or undesirable behavior) arise in non-unique segmentation as reward-penalty Dice loss (RPDL), where consensus-labeled pixels are rewarded and outliers penalized, guiding model optimization toward consensus masks (He et al., 2020).
• Gradient Alignment Approaches: For extracting relative reward functions between two diffusion models, losses can be constructed by aligning the gradient of a neural reward network to the difference of score networks from "expert" and "base" models, resulting in reward functions that separate high- and low-quality behaviors (Nuti et al., 2023).
• Positive-Unlabeled (PU) Corrections: To address over- and under-estimation endemic in reward learning, PU risk estimators are introduced, correcting discriminator or reward losses using positive and unlabeled samples, and enforcing non-negativity constraints for stable learning (Xu et al., 2019).

4. Taxonomies and Comparative Insights

Reward loss functions, particularly in SOC and diffusion-based control, are now systematically grouped according to their expected gradients:

Loss Class	Example Loss Functions	Notes
I (RE / Cost-SOCM)	Discrete adjoint, continuous adjoint, PPO, REINFORCE	Moderate variance, standard in RL
II (Adjoint Match)	Adjoint matching, Work-SOCM	Lowest gradient variance, fastest converge
III (SOCM, CE)	SOCM, SOCM–Adjoint, cross-entropy	High variance, not recommended at scale
IV (LogVar, Moment)	Log-variance, Moment	High/unstable variance, for proof-of-concept
V (Variance)	Variance	Excessive gradient variance
VI (UW-SOCM)	Unweighted SOCM	Low variance, extra critical points

All losses in a class share the same optimization landscape; practical differences arise in convergence rate and stability (Domingo-Enrich, 1 Oct 2024).

5. Empirical Performance and Practical Recommendations

Empirical evaluations consistently highlight the importance of loss selection and tuning:

• RP-ε-SVR achieves 5–15% improvements in SSE/SST, RMSE, and MAE over standard ε-SVR, with comparable sparsity, across UCI and synthetic datasets (Anand et al., 2019).
• Regret-based preference learning outperforms partial-return models in both synthetic and real human feedback settings, particularly in reward identifiability and policy alignment (Knox et al., 2022).
• Reward-penalty Dice loss in CNN-based segmentation yields up to 18.4% improvement on surgical datasets where multiple ground-truths exist, substantially outperforming standard Dice loss in high variability regimes (He et al., 2020).
• Neutral rewards in adversarial imitation learning remove the survival and termination biases observed in GAIL, enabling better learning in both single- and multi-terminal environments (Jena et al., 2020).
• PU reward learning eliminates spurious exploitation (overestimation) and collapsed reward signals (underestimation) in RL and imitation, yielding consistently near-expert policy performance in diverse control domains (Xu et al., 2019).
• Adjoint matching losses in SOC reward fine-tuning offer the best trade-off between bias and variance, leading to significantly improved convergence and scaling in diffusion and flow-matching models (Domingo-Enrich, 1 Oct 2024).

6. Design Considerations and Ongoing Challenges

• Hyperparameter Tuning: Most reward loss functions require careful tuning of weights (e.g., $\tau_1$, $\tau_2$ in RP-ε, PU-prior $\eta$, temperature $\beta$ in preference learning). Grid search and cross-validation are typical, but more adaptive strategies remain underexplored (Anand et al., 2019, Xu et al., 2019).
• Gradient Variance Management: In high-dimensional or multimodal spaces, low-variance loss classes (Adjoint Matching, Work-SOCM) are preferable, as variance-driven instability can severely hamper scaling and sample efficiency (Domingo-Enrich, 1 Oct 2024).
• Out-of-Distribution Generalization: Reward models learned from limited or biased feedback can induce "reward hacking." PU-loss and masking approaches substantially mitigate this but rely on appropriate positive/unlabeled data coverage (Xu et al., 2019).
• Axiomatic and Statistical Guarantees: Identifiability and robustness results exist for select loss constructions (notably, regret-based and convex penalty–reward classes), but general theoretical understanding across all settings remains incomplete (Knox et al., 2022).

7. Emerging Directions

Recent research has extended reward loss functions to:

• Extraction and alignment in high-dimensional generative models: Relative reward functions inferred from diffusion model score differences can steer generative and sequential models in vision and robotics (Nuti et al., 2023).
• Unified SOC loss taxonomies: Equivalence-class analysis enables systematic loss selection for diffusion fine-tuning and policy optimization (Domingo-Enrich, 1 Oct 2024).
• Multi-annotator and non-unique ground truth learning: RPDL shows the value—both empirically and theoretically—of integrating consensus and disagreement directly into the loss for challenging structured prediction tasks (He et al., 2020).

Open problems include scaling custom reward loss solvers, principled regularization for high-capacity reward networks, and improved diagnostics and guarantees under feedback or demonstration uncertainty.

---

Reward loss functions form a central axis in learning, control, and generative modeling, shaping solutions not only by specifying which outputs to penalize but also by imbuing learning algorithms with the ability to actively reward fine-grained distinctions among desirable behaviors, outcomes, or predictions. The dynamic interplay between reward specification, loss design, and optimization properties continues to underpin advances across RL, imitation learning, structured prediction, and diffusion-based models.