---
title: Mixture of Reward Experts (MoRE)
url: https://www.emergentmind.com/topics/mixture-of-reward-experts-more
type: topic
---

# Mixture of Reward Experts (MoRE)

A Mixture of Reward Experts (MoRE) is a model class in preference-based reinforcement learning and RLHF that replaces the traditional single scalar reward predictor with a composition of several specialized reward “experts.” These experts are combined via a gating network, allowing the system to model heterogeneous, conflicting, multimodal, or personalized reward signals more robustly than single-model approaches. MoRE frameworks have been applied in both robotic and large language model contexts, improving robustness to noisy supervision, adversarial over-optimization, task diversity, and interpretability of reward attributions [2605.00384][2403.01197][2109.12750][2512.00724][2406.12845][2606.04284].

## 1. Problem Motivation and Core Principles

The motivation for MoRE arises from the observation that real-world preference datasets—whether crowdsourced annotations for RL agents, diverse user ratings for LLM outputs, or demonstrations for robotic tasks—are inherently heterogeneous and often internally inconsistent. Single reward models that average across this heterogeneity can overfit spurious regularities, obscure legitimate preference diversity, and degrade in both robustness and downstream learning performance, especially under label noise or out-of-distribution generalization [2605.00384][2403.01197].

MoRE architectures mitigate these failures by learning a set of reward expert models, each specializing in a distinct subspace of preference signals, and then adaptively aggregating their outputs using a (potentially input-dependent) gating or routing network. This design enables explicit modeling of evaluator diversity, task segmentation, and latent reward structure.

## 2. Formal Model Structure and Training Objectives

Let $K$ denote the number of reward experts, with expert $k$ parameterized by $\psi_k$ (neural MoEs) or $\omega_k$ (linear MoEs), and a gating or routing network parameterized by $\phi$. The reward prediction for input $x$ (possibly a trajectory $\tau$ or prompt-response pair $(x,y)$) is computed as:

$$
\bar{r}(x;\phi,\psi) = \sum_{k=1}^K \pi_k(x;\phi) r_k(x;\psi_k)
$$

where $r_k(x;\psi_k)$ is the output of expert $k$ and $\pi_k(x;\phi)$ is the routing weight, typically a softmax over $K$ experts. The preferred training loss, given (possibly noisy) pairwise preference data $(x_0, x_1, y)$, is a Bradley–Terry-type cross-entropy:

$$
\mathcal{L}_{BT}(\phi,\psi) = -\mathbb{E}_{(x_0, x_1, y)}\left[ y\log P[x_1 \succ x_0] + (1-y)\log P[x_0 \succ x_1] \right]
$$

with
$$
P[x_1 \succ x_0] = \frac{\exp(\bar{R}(x_1;\phi,\psi))}{\exp(\bar{R}(x_1;\phi,\psi)) + \exp(\bar{R}(x_0;\phi,\psi))}
$$

A load-balancing term (e.g., $L_\text{bal} = \sum_{k=1}^K (\mathbb{E}_x[\pi_k(x)] - 1/K)^2$) is commonly added to prevent expert collapse, ensuring all experts receive substantial activation [2605.00384]. For sparse MoRE, the routing is sharpened via hard top-$m$ selection or local entropy minimization to encourage interpretable, specialized experts [2606.04284].

## 3. Model Architectures and Specialization Mechanisms

MoRE architectures vary in their backbone and specialization strategies, ranging from linear models over hand-crafted features and simple softmax gates [2109.12750], to deep Transformers with modality-specific and cross-modal encoding [2605.00384], to LLM-based decompositions with per-objective or sparsified linear heads [2406.12845][2606.04284]:

- **Neural MoREs**: Experts are realized as parameter-isolated neural networks or modules (e.g., cross-attention heads, LoRA adapters, FFN branches). For LLM reward attribution, multi-head projection layers yield a vector of expert reward scores, which are contextually weighted via a prompt-aware gating MLP [2406.12845].
- **Task/Capability Decomposition**: DMoERM uses a two-level MoRE structure, where an outer sparse router assigns each input to exactly one task-specific inner MoE. The inner MoE, using LoRA adapters, produces capability-level sub-scores, which are fused by a lightweight MLP. This double-layer approach cleanly factors task granularity and capability decomposition, yielding interpretable intermediate scores [2403.01197].
- **Sparse Routing**: Hard gating or top-$m$ expert selection induces sparse, semantically-coherent routing, facilitating expert interpretability and improved personalization at test time [2606.04284]. Regularizers are introduced to maximize per-sample sparsity, ensure global routing balance, and minimize redundancy among experts.

## 4. Interpretability and Personalization

MoRE frameworks enable interpretable and customizable reward modeling:

- **Interpretability by Design**: Vector-valued reward outputs (one per expert) can be mapped to human-interpretable objectives, such as helpfulness, safety, or task subcategories. The gating network’s weights or routing distributions provide a transparent, instance-wise explanation of which underlying preferences are influencing the reward [2406.12845][2606.04284].
- **Personalization**: Post-training, the gating network can be rapidly adapted (via a few preference feedbacks and gradient steps) to a new user’s latent preference profile, without retraining expert models. This supports efficient test-time personalization in high-variability domains [2606.04284].

## 5. Empirical Performance and Evaluation

MoRE methods have consistently demonstrated superior robustness and sample efficiency across diverse metrics and domains:

- **Robustness to Noisy Preferences**: PrefMoE achieves higher downstream task performance than single-expert baselines across D4RL locomotion and MetaWorld manipulation, retaining an 88% relative performance even under high noise ($\Delta p=0.3$) compared to 74% for single-expert models [2605.00384].
- **Multi-task Generalization**: DMoERM’s double-layer structure surpasses state-of-the-art ensemble and mean/variance strategies in both BoN sampling and RL policy optimization, increases preference prediction consistency with human raters (from ~63% to 71%), and avoids overoptimization plateaus [2403.01197].
- **Efficiency**: Upcycled-and-merged MoE models reach the robustness of specialized ensembles without inference cost inflation, matching best-of-N and avoiding reward hacking in RLHF [2512.00724].
- **Interpretability**: Sparse MoREs yield high-fidelity, specialized experts with routing entropy as low as 0.20 (vs. 1.0 for uniform), expert purity >85%, and attribute steering improvements ($\Delta$ accuracy $+25.81$ points in personalization scenarios) [2606.04284].
- **RewardBench SOTA**: ArmoRM + MoRE ranks competitively (89.0) against the 340B Nemotron-4 (89.3) on RewardBench with interpretable reasoning, outperforming strong dense baselines including GPT-4 judge and standard Bradley–Terry RM [2406.12845].

## 6. Extensions and Open Research Directions

MoRE architectures are being extended in multiple directions:

- **Noise Isolation**: Integration of ensemble-based trust filters within experts to further isolate corrupted labels [2605.00384].
- **Adaptive Expert Management**: Learning to grow/prune experts adaptively based on observed diversity or data volume.
- **Scaling to LLM Reward Models**: Incorporating MoE layers atop Transformer reward heads to handle richly multimodal, high-dimensional preference data in RLHF settings [2512.00724][2406.12845].
- **Online Preference Querying**: Combining MoE with information-theoretic active querying to greedily maximize data informativeness for faster reward structure recovery [2109.12750].
- **Personalized RLHF Pipelines**: Test-time router finetuning for efficient end-user adaptation via a small number of feedback samples [2606.04284].

## 7. Benchmark Datasets and Standard Evaluations

MoRE methods are evaluated across a spectrum of benchmarks capturing task diversity and real-world complexity:

| Domain            | Benchmark(s)                | Key Metrics                                                           |
|-------------------|----------------------------|-----------------------------------------------------------------------|
| Robotics          | D4RL, AntMaze, MetaWorld   | Downstream score, preference accuracy, noise robustness               |
| RLHF/LLM          | RewardBench, AlignBench    | Preference prediction accuracy, human A/B, reward hacking mitigation  |
| Classification    | SHP, RPR, PersonalLLM      | Expert purity, attribute steering, personalization accuracy           |

All reports emphasize ablation over expert count, routing sparsity, and annotation pool diversity, and highlight improvement over single-reward-model and mean/ensemble approaches [2605.00384][2606.04284][2403.01197].

---

MoRE frameworks, through specialist expert modeling and adaptive (often interpretable) routing, advance reward learning’s robustness to label noise, task diversity, annotator heterogeneity, reward hacking, and personalization. The design pattern of explicit mixture, modular expert specialization, and context-sensitive aggregation is now fundamental in both preference-based reinforcement learning and LLM alignment [2605.00384][2606.04284][2406.12845][2403.01197][2109.12750][2512.00724].

Source: https://www.emergentmind.com/topics/mixture-of-reward-experts-more