Multi-Objective Reward Schemes

• Multi-Objective Reward Schemes are frameworks in reinforcement learning that employ vector-valued rewards to balance multiple criteria such as safety, fairness, and efficiency.
• They utilize methods like scalarization, Pareto optimization, and dynamic weighting to effectively negotiate non-convex trade-offs in complex decision-making environments.
• Recent advances include adaptive reward shaping, preference-based learning, and normalization techniques that enhance Pareto frontier coverage and mitigate reward misalignment.

Multi-Objective Reward Schemes

Multi-objective reward schemes are a fundamental class of methodologies in reinforcement learning (RL) and sequential decision making that enable agents to reason about, optimize, and balance multiple often competing objectives. In contrast to standard RL, where a scalar reward encodes the desirability of outcomes, multi-objective settings involve vector-valued rewards, each component corresponding to a separate criterion—such as safety, efficiency, fairness, or user satisfaction. These schemes are central for applications ranging from robotics, recommendation systems, and dialogue systems to LLM alignment and automated trading. The literature spans classical and modern approaches including linear scalarization, nonlinear aggregations, preference-based methods, lexicographic orderings, dynamic and context-aware weighting, reward shaping, and algorithmic solutions to reward hacking and misalignment.

1. Formalism and Problem Definitions

A multi-objective Markov Decision Process (MOMDP) generalizes the classical MDP to a setting where each transition emits a vector reward $r(s,a) \in \mathbb{R}^m$, with $m$ objectives. The agent's policy must be evaluated not by a single scalar, but by the long-term performance across all objectives: $$\mathbf{R}_\pi = \mathbb{E}_{\tau \sim \pi} \left[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\right] \in \mathbb{R}^m.$$ Policies are compared—either via scalarization (mapping $\mathbb{R}^m \to \mathbb{R}$), Pareto dominance, or context-sensitive preference models. Scalarization remains the dominant reduction, most commonly via a weight vector $w \in \Delta^{m-1}$ on the $m$-simplex: $r_w(s,a) = w^\top r(s,a).$ The agent thus optimizes the expected scalar return $$J(w) = \mathbb{E}_\pi[\sum_{t=0}^\infty \gamma^t r_w(s_t,a_t)]$$, with Pareto-optimality defined as the set of policies for which no other policy achieves strictly better return in all objectives and strictly better in at least one.

2. Scalarization and Linear vs. Nonlinear Aggregation

Linear Scalarization and Its Limitations

Linear scalarization collapses vector rewards to a scalar via a fixed weight vector, a design prevalent across MORL algorithms, recommendation systems, and many RL-based controllers (Mu et al., 18 Jul 2025, Jeunen et al., 2024, Friedman et al., 2018). While simple and compatible with gradient-based policy optimization, linear schemes can only recover policies lying on the convex hull of the true Pareto front (Lu et al., 14 Sep 2025). Static weights are thus fundamentally inadequate for representing non-convex trade-offs; important non-linearities between objectives are neglected.

Dynamic and Adaptive Weighting

Dynamic reward weighting addresses this by adaptively tuning $w_t$ during training to steer learning toward objectives with remaining headroom. Notable approaches include:

• Hypervolume-guided adaptation: Boosts rewards when trajectories extend the Pareto frontier hypervolume, propagating a meta-reward signal that directly incentivizes frontier expansion (Lu et al., 14 Sep 2025).
• Gradient-based weight optimization: Treats $w$ as trainable, updating it via mirror descent on a surrogate of total loss, using per-objective gradient influence signals to find directions maximizing joint improvement (Lu et al., 14 Sep 2025).

Table: Linear vs. Dynamic Scalarization

Scheme	Coverage of Pareto	Adaptivity
Fixed linear	Convex hull only	None
Dynamic (hypervolume/grad)	Non-convex, full frontier	Tuned during training

Dynamic approaches consistently achieve Pareto dominance over fixed-weight baselines, especially in tasks with pronounced objective trade-off curvature (Lu et al., 14 Sep 2025).

Nonlinear Welfare Functions

Fairness-motivated scalarizations such as Nash Social Welfare (NSW, $m$0) and proportional fairness ($m$1) favor balance/equality among objectives and are optimized via specialized Q-learning strategies and non-stationary action selection (Fan et al., 2022). These approaches guarantee strong fairness properties, but introduce computational intractability and can necessitate history-dependent (non-stationary) policies.

3. Structural Schemes: Lexicographic and Contextual Approaches

Lexicographic Ordering

Lexicographic schemes impose a strict partial order on objectives: maximize $m$2 first, then $m$3 among $m$4-maximizers, etc. Lexicographic Multi-Objective RL (LMORL) provides value-based and policy-gradient algorithms that converge to such solutions, relying on recursive Bellman optimality per priority level and hierarchical constraint enforcement in policy updates (Skalse et al., 2022). This is particularly effective for strict safety-performance separations and guarantees that lower-priority gains never compromise higher-priority constraints.

Contextual and State-Dependent Orderings

Contextual Lexicographic MDPs (CLMDPs) generalize lexicographic methods by allowing the priority ordering and reward definitions themselves to vary with context, encoded via a state-to-context mapping $m$5 (Rustagi et al., 13 Feb 2025). Bayesian inference techniques learn $m$6 from expert trajectories, and conflict-resolution routines merge local context policies into a globally cycle-free solution. This paradigm is vital for agents operating in multifaceted environments with shifting priorities (e.g., delivery robots balancing safety, speed, and energy in mixed zones).

4. Reward Shaping, Preference-Based, and Data-Driven Schemes

Multi-Objective Reward Shaping

Reward shaping seeks to accelerate learning by augmenting sparse or delayed signals with dense, heuristic objectives. MORSE frames the shaping problem as a bi-level optimization: the policy is trained with a shaped reward, while an outer loop tunes the weights on heuristic components to maximize true task performance. Stochastic exploration in weight space, via novelty-seeking and episodic policy resets, avoids local minima and adapts to high-dimensional, sparse-reward settings (Xie et al., 17 Dec 2025). MORSE matches or outperforms hand-tuned baselines across diverse robotic tasks, illustrating the necessity of reward search in practical multi-objective settings.

Preference-Based MORL

Preference-based frameworks bypass reward engineering by eliciting trajectory-wise or segment-wise rankings from human (or synthetic) teachers. The learned reward model is optimized to reproduce these preferences under arbitrary weightings (Mu et al., 18 Jul 2025). Theoretical guarantees establish that optimizing the inferred reward model's scalarization yields Pareto-optimal policies, and that the full frontier—including non-convex regions—can be recovered from pairwise comparisons. Pb-MORL achieves oracle-level performance on standard benchmarks and outperforms ground-truth–based baselines in energy management and autonomous driving domains.

Data-Centric Multi-Objective Alignment

Reward Consistency Sampling (RCS) targets sample-level alignment conflicts in multi-objective LLM alignment: a pair $m$7 is "reward-consistent" if it is preferred under all reward models. Filtering datasets for such samples ensures gradient contributions align, reducing destructive interference between objective gradients (Xu et al., 15 Apr 2025). RCS is shown to raise aggregate alignment metrics by ~13% in realistic two/three-objective settings compared to baselines relying on mere algorithmic reweighting.

5. Normalization, Robustness, and Reward Hacking Mitigation

Variance-Based Reward Normalization

Variance-based normalization schemes, as instantiated in MO-GRPO, scale each component $m$8 by the inverse of its groupwise standard deviation $m$9 (Ichihara et al., 26 Sep 2025). This guarantees equal gradient contributions from each reward, regardless of relative scale, and preserves preference ordering under all positive-affine transformations. In empirical comparisons, normalization prevents policies from fixating on high-variance objectives, yielding stable Pareto improvements across bandits, control, translation, and instruction-following tasks.

Joint Scalar and Multi-Objective Reward Models

SMORM couples a single-objective Bradley–Terry (BT) loss with multi-objective regression in a shared embedding space (Zhang et al., 10 Jul 2025). Theoretical results link improvement in multi-objective calibration directly to lower pairwise error in BT and attribute MSE, yielding reward models robust to distribution shift (OOD reward hacking) and enabling <10B models to surpass 70B models in reward-benchmarks.

Interpretable Multi-Objective Schemes

Recent work introduces interpretable multi-objective reward models (e.g., ArmoRM+MoE), which decompose scalar feedback into human-labeled axes and aggregate them via context-dependent mixture-of-expert gating networks (Wang et al., 2024). Such decompositions improve both practical RLHF performance and allow humans/analysts to audit which axes influenced agent outputs, closing one class of reward-hacking exploits.

6. Algorithmic Innovations, Optimization, and Empirical Outcomes

Bi-level and Evolutionary Weight Optimization

Automated reward component search is performed by bi-level methods (policy optimization inner loop, outer reward/weight-tuning loop) (Xie et al., 17 Dec 2025, Chu et al., 2023). ERFSL augments this by automating reward-component code generation via LLMs, followed by iterative simulation/log-based Pareto weight optimization, achieving low sample complexity and robust performance even in high-misspecification regimes (Xie et al., 19 May 2026).

Volume-Based and Gradient-Multiobjective Optimization

Discrete prompt and multi-objective tuning tasks benefit from proxying multi-objective reward by hypervolume indicator maximization or expected product of objectives (volume surrogates), yielding better Pareto balance than monotonic-gradient methods (MGDA) in practice (Jafari et al., 2024). Such volume-based criteria incentivize expansion of the frontier, directly aligning the reward signal with desired joint improvement.

Optimization Paradigm	Mechanism	Context	Papers
Bi-level (policy/reward)	Outer reward search	Robotics, CRS, RL design	(Xie et al., 17 Dec 2025, Chu et al., 2023)
Dynamic scalarization (hypervolume, influence)	Adaptive weight tuning	LLM alignment, RL	(Lu et al., 14 Sep 2025)
Variance normalization	Per-objective scaling	Text, control, translation	(Ichihara et al., 26 Sep 2025)
Preference-based, reward model training	Human feedback -> reward model	MORL, RLHF, alignment	(Mu et al., 18 Jul 2025, Xu et al., 15 Apr 2025)
Volume/proxy-based in prompt/sequence design	Pareto surface hypervolume/product	NLP, discrete optimization	(Jafari et al., 2024)

Empirical Best Practices and Outcomes

• Fixed linear scalarization is suboptimal in all constrained, high-dimensional, and conflict-rich regimes (Lu et al., 14 Sep 2025, Xie et al., 17 Dec 2025).
• Dynamic and/or data-driven adaptation of weights, coupled with parity-inducing normalization and bi-level optimization, yields improved Pareto frontier coverage, higher success/retention rates, reduced variance, and robustness to mis-aligned or adversarial reward models across RL, dialogue, and LLM domains.
• Interpretable and context-aware reward decomposition not only matches but often exceeds oracle (ground-truth) RLHF models, while simultaneously improving auditability and preventing opaque failure/hacking (Wang et al., 2024, Zhang et al., 10 Jul 2025).

7. Open Challenges and Research Directions

Despite significant progress, open issues persist:

• Non-Markovian aggregation: When objectives have disparate time preferences (discounts), history-dependent or augmented-state reward schemes are required to ensure dynamic consistency—a result now supported by structural impossibility theorems (Pitis, 2023).
• Scalability and sample complexity: As the number of objectives increases, covering the full Pareto manifold remains costly; efficient sampling, transfer, and neural embeddings for high-$$\mathbf{R}_\pi = \mathbb{E}_{\tau \sim \pi} \left[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\right] \in \mathbb{R}^m.$$0 settings are open fields (Xie et al., 17 Dec 2025, Chen et al., 27 Apr 2026).
• Preference elicitation and uncertainty: Reliance on perfect teachers, consistent labels, and fixed objective sets is a limiting assumption. Richer preference models, online adaptation, and uncertainty quantification are active directions.
• Practical integration: Bridging the gap from theoretical multi-objective guarantees to deployment in environments with nonstationary user cohorts, policy drift, or shifting objective definitions remains a major engineering and scientific challenge.

Multi-objective reward schemes thus constitute a rich methodological and theoretical landscape. Ongoing research integrates mathematical guarantees, human-in-the-loop design, interpretability, and robust optimization to deliver agents capable of optimizing for complex, high-stakes, and evolving real-world multi-criteria objectives.