Mixture of Reward Experts (MoRE)

Updated 3 July 2026

MoRE is a framework that employs multiple specialized reward predictors, aggregated through an adaptive gating network, to address heterogeneous reward signals in RL and LLM training.
The approach improves robustness to noisy supervision and adversarial over-optimization, enhancing performance across tasks like robotics and language model alignment.
MoRE enables interpretable reward attributions and personalized adjustments by mapping outputs from diverse experts to human-understandable objectives.

A Mixture of Reward Experts (MoRE) is a model class in preference-based reinforcement learning and RLHF that replaces the traditional single scalar reward predictor with a composition of several specialized reward “experts.” These experts are combined via a gating network, allowing the system to model heterogeneous, conflicting, multimodal, or personalized reward signals more robustly than single-model approaches. MoRE frameworks have been applied in both robotic and LLM contexts, improving robustness to noisy supervision, adversarial over-optimization, task diversity, and interpretability of reward attributions (Yuan et al., 1 May 2026, Quan, 2024, Myers et al., 2021, Fu, 30 Nov 2025, Wang et al., 2024, Wang et al., 2 Jun 2026).

1. Problem Motivation and Core Principles

The motivation for MoRE arises from the observation that real-world preference datasets—whether crowdsourced annotations for RL agents, diverse user ratings for LLM outputs, or demonstrations for robotic tasks—are inherently heterogeneous and often internally inconsistent. Single reward models that average across this heterogeneity can overfit spurious regularities, obscure legitimate preference diversity, and degrade in both robustness and downstream learning performance, especially under label noise or out-of-distribution generalization (Yuan et al., 1 May 2026, Quan, 2024).

MoRE architectures mitigate these failures by learning a set of reward expert models, each specializing in a distinct subspace of preference signals, and then adaptively aggregating their outputs using a (potentially input-dependent) gating or routing network. This design enables explicit modeling of evaluator diversity, task segmentation, and latent reward structure.

2. Formal Model Structure and Training Objectives

Let $K$ denote the number of reward experts, with expert $k$ parameterized by $\psi_k$ (neural MoEs) or $\omega_k$ (linear MoEs), and a gating or routing network parameterized by $\phi$ . The reward prediction for input $x$ (possibly a trajectory $\tau$ or prompt-response pair $(x,y)$ ) is computed as:

$\bar{r}(x;\phi,\psi) = \sum_{k=1}^K \pi_k(x;\phi) r_k(x;\psi_k)$

where $r_k(x;\psi_k)$ is the output of expert $k$ 0 and $k$ 1 is the routing weight, typically a softmax over $k$ 2 experts. The preferred training loss, given (possibly noisy) pairwise preference data $k$ 3, is a Bradley–Terry-type cross-entropy:

$k$ 4

with

$k$ 5

A load-balancing term (e.g., $k$ 6) is commonly added to prevent expert collapse, ensuring all experts receive substantial activation (Yuan et al., 1 May 2026). For sparse MoRE, the routing is sharpened via hard top- $k$ 7 selection or local entropy minimization to encourage interpretable, specialized experts (Wang et al., 2 Jun 2026).

3. Model Architectures and Specialization Mechanisms

MoRE architectures vary in their backbone and specialization strategies, ranging from linear models over hand-crafted features and simple softmax gates (Myers et al., 2021), to deep Transformers with modality-specific and cross-modal encoding (Yuan et al., 1 May 2026), to LLM-based decompositions with per-objective or sparsified linear heads (Wang et al., 2024, Wang et al., 2 Jun 2026):

Neural MoREs: Experts are realized as parameter-isolated neural networks or modules (e.g., cross-attention heads, LoRA adapters, FFN branches). For LLM reward attribution, multi-head projection layers yield a vector of expert reward scores, which are contextually weighted via a prompt-aware gating MLP (Wang et al., 2024).
Task/Capability Decomposition: DMoERM uses a two-level MoRE structure, where an outer sparse router assigns each input to exactly one task-specific inner MoE. The inner MoE, using LoRA adapters, produces capability-level sub-scores, which are fused by a lightweight MLP. This double-layer approach cleanly factors task granularity and capability decomposition, yielding interpretable intermediate scores (Quan, 2024).
Sparse Routing: Hard gating or top- $k$ 8 expert selection induces sparse, semantically-coherent routing, facilitating expert interpretability and improved personalization at test time (Wang et al., 2 Jun 2026). Regularizers are introduced to maximize per-sample sparsity, ensure global routing balance, and minimize redundancy among experts.

4. Interpretability and Personalization

MoRE frameworks enable interpretable and customizable reward modeling:

Interpretability by Design: Vector-valued reward outputs (one per expert) can be mapped to human-interpretable objectives, such as helpfulness, safety, or task subcategories. The gating network’s weights or routing distributions provide a transparent, instance-wise explanation of which underlying preferences are influencing the reward (Wang et al., 2024, Wang et al., 2 Jun 2026).
Personalization: Post-training, the gating network can be rapidly adapted (via a few preference feedbacks and gradient steps) to a new user’s latent preference profile, without retraining expert models. This supports efficient test-time personalization in high-variability domains (Wang et al., 2 Jun 2026).

5. Empirical Performance and Evaluation

MoRE methods have consistently demonstrated superior robustness and sample efficiency across diverse metrics and domains:

Robustness to Noisy Preferences: PrefMoE achieves higher downstream task performance than single-expert baselines across D4RL locomotion and MetaWorld manipulation, retaining an 88% relative performance even under high noise ( $k$ 9) compared to 74% for single-expert models (Yuan et al., 1 May 2026).
Multi-task Generalization: DMoERM’s double-layer structure surpasses state-of-the-art ensemble and mean/variance strategies in both BoN sampling and RL policy optimization, increases preference prediction consistency with human raters (from ~63% to 71%), and avoids overoptimization plateaus (Quan, 2024).
Efficiency: Upcycled-and-merged MoE models reach the robustness of specialized ensembles without inference cost inflation, matching best-of-N and avoiding reward hacking in RLHF (Fu, 30 Nov 2025).
Interpretability: Sparse MoREs yield high-fidelity, specialized experts with routing entropy as low as 0.20 (vs. 1.0 for uniform), expert purity >85%, and attribute steering improvements ( $\psi_k$ 0 accuracy $\psi_k$ 1 points in personalization scenarios) (Wang et al., 2 Jun 2026).
RewardBench SOTA: ArmoRM + MoRE ranks competitively (89.0) against the 340B Nemotron-4 (89.3) on RewardBench with interpretable reasoning, outperforming strong dense baselines including GPT-4 judge and standard Bradley–Terry RM (Wang et al., 2024).

6. Extensions and Open Research Directions

MoRE architectures are being extended in multiple directions:

Noise Isolation: Integration of ensemble-based trust filters within experts to further isolate corrupted labels (Yuan et al., 1 May 2026).
Adaptive Expert Management: Learning to grow/prune experts adaptively based on observed diversity or data volume.
Scaling to LLM Reward Models: Incorporating MoE layers atop Transformer reward heads to handle richly multimodal, high-dimensional preference data in RLHF settings (Fu, 30 Nov 2025, Wang et al., 2024).
Online Preference Querying: Combining MoE with information-theoretic active querying to greedily maximize data informativeness for faster reward structure recovery (Myers et al., 2021).
Personalized RLHF Pipelines: Test-time router finetuning for efficient end-user adaptation via a small number of feedback samples (Wang et al., 2 Jun 2026).

7. Benchmark Datasets and Standard Evaluations

MoRE methods are evaluated across a spectrum of benchmarks capturing task diversity and real-world complexity:

Domain	Benchmark(s)	Key Metrics
Robotics	D4RL, AntMaze, MetaWorld	Downstream score, preference accuracy, noise robustness
RLHF/LLM	RewardBench, AlignBench	Preference prediction accuracy, human A/B, reward hacking mitigation
Classification	SHP, RPR, PersonalLLM	Expert purity, attribute steering, personalization accuracy

All reports emphasize ablation over expert count, routing sparsity, and annotation pool diversity, and highlight improvement over single-reward-model and mean/ensemble approaches (Yuan et al., 1 May 2026, Wang et al., 2 Jun 2026, Quan, 2024).

MoRE frameworks, through specialist expert modeling and adaptive (often interpretable) routing, advance reward learning’s robustness to label noise, task diversity, annotator heterogeneity, reward hacking, and personalization. The design pattern of explicit mixture, modular expert specialization, and context-sensitive aggregation is now fundamental in both preference-based reinforcement learning and LLM alignment (Yuan et al., 1 May 2026, Wang et al., 2 Jun 2026, Wang et al., 2024, Quan, 2024, Myers et al., 2021, Fu, 30 Nov 2025).