--- title: 'MORSE: Multi-Objective Reward Shaping' url: https://www.emergentmind.com/topics/multi-objective-reward-shaping-with-exploration-morse type: topic --- # MORSE: Multi-Objective Reward Shaping Multi-Objective Reward Shaping with Exploration (MORSE) refers to a family of principled reinforcement learning (RL) frameworks that combine multi-objective reward shaping with explicit exploration mechanisms. These approaches enable RL agents to efficiently balance and optimize multiple objectives—such as task completion, safety, efficiency, and information acquisition—while systematically structuring both reward signals and exploration incentives. MORSE techniques use bi-level or dynamic optimization of reward aggregation, integrate stochasticity and novelty into reward-space search, and provide provable guarantees under certain Bayesian or potential-based formulations. Recent theoretical and empirical work demonstrates MORSE-based algorithms achieving superior robustness and sample efficiency across continuous control, robotic, and preference-alignment domains [2512.15120][2509.11452][2409.05358]. ## 1. Mathematical Foundations of MORSE In multi-objective RL, the agent operates in a Markov decision process (MDP) or Bayes-Adaptive MDP (BAMDP) characterized by a reward vector $R(s,a) \in \mathbb{R}^K$ for $K$ objectives. Classic scalarization schemes form $r^w_t = \sum_{i=1}^K w_i r^i_t$ for $w \in \Delta^K$. Fixed-weight scalarization cannot capture non-convex Pareto fronts, prompting the need for more adaptive reward weighting and shaping [2509.11452]. A key insight central to MORSE research is the aggregation of extrinsic and intrinsic rewards to drive both exploitation (optimizing for task-specific objectives) and exploration (maximizing information gain or novelty). In particular, the Bayes-Adaptive Potential-Based Shaping Framework (BAMPF) formalizes pseudo-rewards: $$ F((b,s), (b',s')) = \gamma W(b',s') - W(b,s) $$ where $W(\cdot)$ is a multi-objective potential function decomposed as $W(b,s) = \alpha_{\mathrm{phys}} W_{\mathrm{phys}}(b,s) + \alpha_{\mathrm{info}} W_{\mathrm{info}}(b,s)$. The shaped reward at each transition is: $$ R_{\mathrm{shaped}} = R(s,a) + \gamma W(b',s') - W(b,s) $$ This construction provably preserves Bayes-optimality and corrects for learner misestimation in both exploitation and exploration terms [2409.05358]. ## 2. MORSE as Bi-Level and Dynamic Optimization Contemporary MORSE algorithms formulate reward shaping as a bi-level optimization problem. The inner loop trains a policy $\pi_\theta$ using the current shaped reward, while the outer loop adapts the shaping parameters (often the reward weights) to optimize the true task objective. This two-timescale structure is formalized as: - Inner: $\theta^*(\phi) = \arg\max_\theta \mathbb{E}_{\pi_\theta} [\sum_t \gamma^t R_\phi(s_t,a_t)]$ - Outer: $\phi^* = \arg\max_\phi \mathbb{E}_{\pi_{\theta^*(\phi)}} [\sum_t \gamma^t R_{\mathrm{task}}(s_t,a_t)]$ where $R_\phi(s,a) = w_\phi(s)^\top R(s,a)$. Gradients for the outer loop are computed via implicit differentiation or Neumann-series approximations of the Hessian inverse [2512.15120]. To address local minima and improve adaptability, dynamic reward weighting mechanisms are integrated. These can be hypervolume-guided—maximizing the increase in covered Pareto front volume as new policies are discovered—or gradient-based, using meta-objectives to tune weight importance based on gradient influence signals. The latter adjusts $w$ online through entropy-regularized mirror descent, allocating learning effort toward objectives showing maximal potential improvement [2509.11452]. ## 3. Exploration Strategies in MORSE Frameworks MORSE frameworks introduce exploration not solely in the state-action space but also in the reward-weight parameter space. Techniques for reward-space exploration include: - **Stochastic reward perturbation**: Adding zero-mean Gaussian noise to shaped rewards during policy rollouts, with the variance often adaptive to novelty. - **Random Network Distillation (RND) for weight novelty**: Maintaining target and predictor networks over $w$ to score the novelty of candidate reward-weight vectors. New shaping weights are sampled with probability proportional to their predicted novelty, encouraging coverage of under-explored regions [2512.15120]. - **Performance-gated resets**: Triggering re-sampling of shaping parameters when empirical task success shows no improvement over a sliding window, or with probability governed by recent success rates. This approach enables MORSE algorithms to escape poor local optima in non-convex reward landscapes and adaptively discover high-performing weight configurations, which static or periodic exploration fail to achieve. ## 4. Integration of Human Heuristics and Safety Objectives MORSE directly incorporates multiple human-designed heuristic objectives by stacking dense auxiliary signals (e.g., control costs, kinematic penalties) along with sparse task-completion rewards. A state-conditioned linear or non-linear weighting function aggregates these into a unified shaped reward, typically implemented as a small neural network. The approach requires minimal manual tuning and can efficiently adapt weightings to the evolving demands of the learning task [2512.15120]. In settings where safety constraints are critical, related frameworks integrate multi-objective shaping with action-space reshaping. Here, exploration is constrained by learned or hard-coded safety functions, enforcing that policy proposals reside within admissible regions. Such constrained exploration maintains sample efficiency and prevents undesired behaviors without manual penalty balancing [1809.08925]. ## 5. Theoretical Guarantees and Robustness Potential-based shaping in the BAMDP framework ensures that the optimal policy of the original MDP is also optimal under the shaped reward. The BAMDP Potential-Based Shaping Theorem provides that a shaping function $F$ preserves Bayes-optimality if and only if $F$ is a BAMPF. For suboptimal learners, shaping significantly speeds convergence without risk of reward-hacking—convergence to behaviors maximizing only the composite reward at the expense of the true objective. Finite-horizon regret bounds quantify potential increases in suboptimality due to terminal potential misestimation, but these remain dominated by the problem’s intrinsic horizon [2409.05358]. Dynamic reward weighting and exploration in reward space do not formally guarantee global optimality, owing to inherent non-convexity, but empirical results and local convergence statements under standard smoothness and strong convexity conditions are reported [2512.15120][2509.11452]. ## 6. Practical Implementation and Empirical Results MORSE algorithms are compatible with standard on-policy RL algorithms (e.g., PPO, GRPO, REINFORCE, RLOO), requiring minor modifications: tracking per-objective returns, parameterizing the weight update mechanism, and resetting or perturbing reward parameters as exploration dictates. Key hyperparameters include learning rates for policy and weight updates, Neumann approximation order, exploration interval, and novelty metric architecture. Empirical evaluations demonstrate MORSE’s superiority or parity with expert-tuned baselines on synthetic optimization landscapes, continuous-control MuJoCo tasks, and Isaac Sim robotics environments. Specifically: - In MuJoCo “hard” tasks, MORSE achieves success rates $0.90-0.98$ after training, outperforming vanilla bi-level gradient approaches ($0.60-0.75$) and matching oracle baselines ($0.95-0.99$). - On synthetic test functions, RND-guided MORSE outperforms random sampling, CEM, and CMA for finding high-reward regions in weight space under tight sample budgets. - In BAMDP settings, MORSE reduces steps to solution (e.g., Mountain Car solved in $\sim 30$k steps vs. $100$k without shaping), and in Bernoulli bandits, achieves $O(\log T)$ regret with low constants, matching Bayes-optimality [2409.05358][2512.15120]. A summary table of core MORSE algorithmic components is provided: | Component | Mechanism | Role | |----------------------------|--------------------------------------------------------------|----------------------------------| | Reward aggregation | State-conditioned linear or neural $w_\phi$ | Multi-objective shaping | | Exploration in $w$-space | RND, stochastic resets, performance gating | Escape local minima | | Bi-level optimization | Inner RL under $R_\phi$, outer hypergradient on $\phi$ | Task-driven adaptation | | Dynamic weighting | Hypervolume/gradient-based $w_t$ updates | Pareto front coverage | | Theoretical guarantee | BAMDP/BAMPF preserves Bayes-optimality (constant shift) | Robustness | ## 7. Insights, Limitations, and Recommendations Empirical ablations confirm the necessity of gradient-based outer loops, novelty-driven exploration, performance gating, and policy resets. Removing any of these degrades sample efficiency or fails to escape local optima. Sample complexity remains an issue: each reward-weight candidate entails a (partial) policy retraining. In high-dimensional weight spaces, coverage remains a challenge; manifold-based search and parametric priors are suggested as potential improvements. Practical guidelines include ensuring inner-loop convergence before outer updates, slow outer learning rates, moderate exploration intervals, and bounded weight search. For sim-to-real transfer, fixing weights after initial randomization enhances robustness. Successor-feature architectures may further reduce performance drops when policy resets follow weight changes [2512.15120][2509.11452]. MORSE, through structured reward shaping and dynamic exploration in reward and policy space, constitutes a general and extensible paradigm for multi-objective RL across domains that require efficient trade-off management, robustness to heuristic misspecification, and theoretical soundness. Source: https://www.emergentmind.com/topics/multi-objective-reward-shaping-with-exploration-morse