Multi-Residual Mixture of Expert Learning

• MRMEL is a framework that combines residual reinforcement learning with a context-conditioned mixture-of-experts to dynamically select and blend nominal control policies.
• It augments suboptimal nominal autonomous strategies with learned residual corrections, reducing vehicle emissions by 4%-9% in diverse traffic scenarios.
• Validated on real-world data from major US cities, MRMEL improves throughput and eco-driving performance while addressing generalization challenges in heterogeneous traffic.

Searching arXiv for the cited MRMEL and closely related residual/MoE papers to ground the article with up-to-date references. Multi-Residual Mixture of Expert Learning (MRMEL) is a framework for cooperative control in multi-vehicle systems that combines residual reinforcement learning with a mixture-of-experts mechanism over multiple nominal policies. In its defining formulation, a suboptimal nominal autonomous-vehicle control policy is augmented by a learned residual correction, while the controller simultaneously and dynamically selects the most suitable nominal policy from a pool of nominal policies conditioned on traffic scenarios. The framework was introduced for Lagrangian traffic control, where autonomous vehicles act as mobile actuators of traffic flow rather than relying exclusively on fixed-location actuators such as traffic signals. Its principal case study is cooperative eco-driving at signalized intersections, evaluated on real-world data-driven traffic scenarios from Atlanta, Dallas–Fort Worth, and Salt Lake City, where it achieved an additional $4\%-9\%$ reduction in aggregate vehicle emissions relative to the strongest baseline in each setting (Jayawardana et al., 14 Jul 2025).

1. Problem setting and motivation

MRMEL was proposed for Lagrangian traffic control, a setting in which a subset of autonomous vehicles is controlled so that their local actions influence surrounding human drivers and other vehicles through car-following dynamics. The motivating application is cooperative eco-driving at signalized intersections, with objectives that include reducing vehicle emissions, maintaining or improving throughput, avoiding excessive stopping and idling, and respecting physical traffic constraints (Jayawardana et al., 14 Jul 2025).

The target environment is a mixed-autonomy traffic system comprising human-driven vehicles, autonomous vehicles, multi-agent interactions, and non-stationarity induced by human behavior. A central difficulty is generalization across diverse traffic scenarios. Intersections vary in signal timing, lane geometry, speed limits, grades, traffic demand, vehicle mix, and other factors; consequently, a policy that performs well in one regime may perform poorly in another. The framework is therefore explicitly organized around context variation rather than assuming a single stationary control regime (Jayawardana et al., 14 Jul 2025).

The motivating critique of standard approaches has several components. A single nominal policy is not universally effective across traffic regimes; residual reinforcement learning with only one nominal policy inherits the quality limitations of that nominal; and direct multi-task reinforcement learning over all scenario variations is difficult because of the curse of dimensionality and training instability. MRMEL addresses these issues by combining residual learning for local correction with a mixture-of-experts over multiple nominal policies, conditioned on state and context (Jayawardana et al., 14 Jul 2025).

2. Formal definition and policy decomposition

The baseline control problem is first written as a Markov Decision Process

$$M = \left\langle\mathcal{S}, \mathcal{A}, p, r, \rho, \gamma \right\rangle$$

with optimal policy

$$\pi^* = \argmax_{\pi} \mathbb{E} \left[\sum_{t=0}^{H} \gamma^t r(s_t, a_t) \mid s_0 = s, \pi\right].$$

Because traffic scenarios vary, the framework adopts a Contextual Markov Decision Process: $$\mathcal{M} = \left\langle\mathcal{S}, \mathcal{A}, \mathcal{C}, p_c, r_c, \rho_c, \gamma \right\rangle,$$ where $c \in \mathcal{C}$ denotes a traffic scenario and induces a context-specific MDP $\mathcal{M}_c$. The formal objective of learning up to $k$ policies is

$$\Pi_k^{*} = \argmax_{\Pi_k : |\Pi_k| \leq k} \mathbb{E}_{c \sim \rho(c)}\left[ \max_{\pi \in \Pi_k} \mathcal{R}(\pi, \mathcal{M}_c) \right].$$

In the reported experiments, the emphasis is on learning one unified policy per city ($k=1$), motivated by a balance between generalization and deployability on onboard processors (Jayawardana et al., 14 Jul 2025).

The residual-learning substrate follows the standard decomposition

$\pi^*(s) = \pi_n(s) + f_{\theta}(s),$

and its contextual form

$$M = \left\langle\mathcal{S}, \mathcal{A}, p, r, \rho, \gamma \right\rangle$$0

The key extension in MRMEL is to replace the single nominal policy $$M = \left\langle\mathcal{S}, \mathcal{A}, p, r, \rho, \gamma \right\rangle$$1 with a pool of $$M = \left\langle\mathcal{S}, \mathcal{A}, p, r, \rho, \gamma \right\rangle$$2 nominal policies

$$M = \left\langle\mathcal{S}, \mathcal{A}, p, r, \rho, \gamma \right\rangle$$3

together with a gating mechanism $$M = \left\langle\mathcal{S}, \mathcal{A}, p, r, \rho, \gamma \right\rangle$$4. The resulting policy is

$$M = \left\langle\mathcal{S}, \mathcal{A}, p, r, \rho, \gamma \right\rangle$$5

This equation is the defining MRMEL structure: a context-conditioned weighted combination of nominal experts plus a learned residual correction. One nominal expert may be a zero-action policy, allowing the residual to learn the entire action when no nominal prior is useful (Jayawardana et al., 14 Jul 2025).

A common misunderstanding is to treat MRMEL as ordinary residual RL with a slightly richer nominal prior. The defining distinction is stronger: the framework does not assume a single nominal policy and then merely correct it; instead, it learns to select or blend nominal policies contextually and to add a residual on top of that mixture. The paper also notes a practical nuance: although the conceptual description is MoE-style with gating, the final implementation performs residual-action generation and nominal-policy selection with a shared actor rather than a fully separate gating network (Jayawardana et al., 14 Jul 2025).

3. Architecture, control signals, and optimization

MRMEL combines three elements: residual learning, context-conditioning, and mixture-of-experts nominal selection. In the conceptual pipeline, each autonomous vehicle observes its local state and context; the actor outputs a residual acceleration together with a selection over nominal policies; the nominal actions are combined by a weighted sum; the residual is added to the combined nominal action; and the final action is executed (Jayawardana et al., 14 Jul 2025).

Training is described as actor-critic learning with PPO. The implementation uses an MLP actor and critic, each with 4 hidden layers and 256 neurons per layer, and a learning rate of 0.0001. The actor outputs a residual action sampled from a Gaussian and a one-hot encoding for nominal-policy selection. The paper states explicitly that allowing the actor to jointly control both residual actions and nominal policy selection performs better than using a separate gating network for the latter (Jayawardana et al., 14 Jul 2025).

The training setup uses multiple threads that sample traffic scenarios from a predefined distribution, simulate several autonomous vehicles in parallel, and apply one shared policy across those vehicles. This shared-policy design is integral to the multi-agent traffic setting, where generalization is required both across intersections and across local traffic configurations within an intersection (Jayawardana et al., 14 Jul 2025).

The method’s significance lies in how it restructures the role of inductive bias. Instead of committing to one prior controller, it keeps several priors active and delegates context-sensitive reliance among them to the learned policy. A plausible implication is that the residual term is relieved from having to compensate for every mismatch between a single prior and a highly heterogeneous traffic regime; it only needs to correct the contextually selected mixture.

4. Cooperative eco-driving instantiation

The case study objective is to minimize emissions and travel time: $$M = \left\langle\mathcal{S}, \mathcal{A}, p, r, \rho, \gamma \right\rangle$$6 Each vehicle observes its own position and speed, the leading and following vehicles in same and adjacent lanes, the current traffic signal phase, the time remaining until the next phase change, and context features including full signal plan, speed limit, lane length, road grade, vehicle type, engine type, and vehicle age (Jayawardana et al., 14 Jul 2025).

The action is longitudinal acceleration. The instantaneous reward is

$$M = \left\langle\mathcal{S}, \mathcal{A}, p, r, \rho, \gamma \right\rangle$$7

where $$M = \left\langle\mathcal{S}, \mathcal{A}, p, r, \rho, \gamma \right\rangle$$8 is velocity, $$M = \left\langle\mathcal{S}, \mathcal{A}, p, r, \rho, \gamma \right\rangle$$9 is emissions, $$\pi^* = \argmax_{\pi} \mathbb{E} \left[\sum_{t=0}^{H} \gamma^t r(s_t, a_t) \mid s_0 = s, \pi\right].$$0 penalizes stops, and $$\pi^* = \argmax_{\pi} \mathbb{E} \left[\sum_{t=0}^{H} \gamma^t r(s_t, a_t) \mid s_0 = s, \pi\right].$$1 is absolute acceleration, with weights $$\pi^* = \argmax_{\pi} \mathbb{E} \left[\sum_{t=0}^{H} \gamma^t r(s_t, a_t) \mid s_0 = s, \pi\right].$$2, $$\pi^* = \argmax_{\pi} \mathbb{E} \left[\sum_{t=0}^{H} \gamma^t r(s_t, a_t) \mid s_0 = s, \pi\right].$$3, and $$\pi^* = \argmax_{\pi} \mathbb{E} \left[\sum_{t=0}^{H} \gamma^t r(s_t, a_t) \mid s_0 = s, \pi\right].$$4. Cooperative reward assignment is stochastic: with probability $$\pi^* = \argmax_{\pi} \mathbb{E} \left[\sum_{t=0}^{H} \gamma^t r(s_t, a_t) \mid s_0 = s, \pi\right].$$5, the reward is computed using fleet-level metrics and shared across autonomous vehicles; otherwise, each vehicle receives its own reward. The reported value is $$\pi^* = \argmax_{\pi} \mathbb{E} \left[\sum_{t=0}^{H} \gamma^t r(s_t, a_t) \mid s_0 = s, \pi\right].$$6, which is intended to encourage cooperation while preserving an individual learning signal (Jayawardana et al., 14 Jul 2025).

The nominal policy pool contains five controllers:

Nominal policy	Reported role
GLOSA Controller	Green-light optimized speed advisory; uses vehicle dynamics and signal timing; ignores surrounding vehicles
Constant Acceleration	Fixed acceleration of $$\pi^* = \argmax_{\pi} \mathbb{E} \left[\sum_{t=0}^{H} \gamma^t r(s_t, a_t) \mid s_0 = s, \pi\right].$$7
Constant Deceleration	Fixed deceleration of $$\pi^* = \argmax_{\pi} \mathbb{E} \left[\sum_{t=0}^{H} \gamma^t r(s_t, a_t) \mid s_0 = s, \pi\right].$$8
IDM	Intelligent Driver Model; human-like car-following behavior
Zero-Action Policy	Always outputs $$\pi^* = \argmax_{\pi} \mathbb{E} \left[\sum_{t=0}^{H} \gamma^t r(s_t, a_t) \mid s_0 = s, \pi\right].$$9; allows the residual to learn the whole control action

This selection makes the nominal pool deliberately heterogeneous. Some experts encode eco-driving structure, others encode simple motion primitives, and one expert encodes no prior action at all. The paper’s qualitative analysis reports an implicit curriculum in nominal-policy usage: early training uses constant acceleration heavily, while later training shifts toward constant deceleration and zero-action. This suggests that the learned controller first stabilizes motion generation and later refines emission-minimizing behavior (Jayawardana et al., 14 Jul 2025).

5. Evaluation protocol and empirical results

MRMEL is evaluated on IntersectionZoo, described as a real-world, data-driven benchmark for intersection-based eco-driving. The experiments use nearly 5,000 traffic scenarios and 33 key context factors, spanning 1,670 intersections in Dallas–Fort Worth, 621 intersections in Atlanta, and 282 intersections in Salt Lake City. Two autonomous-vehicle penetration rates are considered: 30\% and 100\% (Jayawardana et al., 14 Jul 2025).

The primary baselines are IDM, multi-task reinforcement learning, and GLOSA. The study also includes single-nominal residual baselines without mixture-of-experts selection: RRL (IDM), RRL (Const. acc.), RRL (Const. dec.), and the GLOSA-based residual baseline referred to as Multi-residual Task Learning (MRTL). Evaluation reports percentage improvements relative to IDM in Emissions and Throughput (Jayawardana et al., 14 Jul 2025).

The paper reports that MRMEL is the best-performing method across all settings and yields an additional $$\mathcal{M} = \left\langle\mathcal{S}, \mathcal{A}, \mathcal{C}, p_c, r_c, \rho_c, \gamma \right\rangle,$$0 reduction in aggregate vehicle emissions relative to the strongest baseline in each setting. The reported improvements for MRMEL are:

• Dallas–Fort Worth, 30\% AV penetration: Emissions 5.90\%, Throughput 3.89\%
• Dallas–Fort Worth, 100\%: Emissions 9.98\%, Throughput 11.81\%
• Atlanta, 30\%: Emissions 6.43\%, Throughput 2.68\%
• Atlanta, 100\%: Emissions 14.40\%, Throughput 8.80\%
• Salt Lake City, 30\%: Emissions 6.31\%, Throughput 4.70\%
• Salt Lake City, 100\%: Emissions 10.32\%, Throughput 7.73\% (Jayawardana et al., 14 Jul 2025)

The comparative interpretation is important. The paper notes that single-nominal residual RL can underperform GLOSA in some settings, and that MRTL can be on par with or worse than GLOSA in real-world scenarios, whereas MRMEL remains consistently strong. Qualitative results complement the aggregate metrics: a spatial distribution plot indicates broader intersection-level improvement across the city, and a time-space diagram shows autonomous vehicles gliding toward red signals, reducing idling, and forming platoons that can be influenced more effectively (Jayawardana et al., 14 Jul 2025).

6. Relation to adjacent residual–MoE research and reported limitations

MRMEL belongs to a broader family of methods that combine expert aggregation with residual correction, but its specific formulation is oriented toward context-conditioned control. In universal dexterous grasping, the ResDex framework combines geometry-unaware base policies with a hyper-policy that predicts both residual actions and expert weights, yielding a final action equal to a residual term plus a normalized weighted sum of base-policy actions. The paper explicitly describes ResDex as a close variant of MRMEL, with the main difference lying in the task domain and the use of geometry-unaware base policies for universal dexterous grasping (Huang et al., 2024).

In large-language-model fine-tuning, S’MoRE extends the same broad design direction by arranging low-rank residual experts in a hierarchical residual tree and routing tokens through connected sub-trees rather than a flat expert set. The paper frames S’MoRE as a Structural Mixture of Residual Experts and as a concrete hierarchical refinement of a broader multi-residual mixture-of-experts idea. This suggests that, outside control, the MRMEL pattern can be interpreted as a reusable architectural schema in which residual composition and expert routing are jointly learned, while the structural details vary substantially across domains (Zeng et al., 8 Apr 2025).

A different but related line appears in continual learning for vision-LLMs, where expert usage frequency is tracked and the two most frequently selected experts are merged into a more generalized expert that replaces the least frequently selected one. That method is described as related to MRMEL-like expert merging, but it does not use the residual-control decomposition that defines MRMEL proper. Its significance is mainly conceptual: it shows that MoE research adjacent to MRMEL also treats expert selection statistics as a signal for improving generalization and utilization (Park, 2024).

The limitations reported for MRMEL are specific. The method is limited to continuous control only, like standard residual RL; it learns one policy per city, which is scalable but may not fully exploit multi-policy specialization; it does not provide a discrete-control extension; its empirical validation concerns generalization within the eco-driving domain rather than broader inter-task transfer; and its implementation uses a joint actor for residual action and nominal-policy selection rather than a fully separate gating network (Jayawardana et al., 14 Jul 2025).

Taken together, these characteristics place MRMEL in a precise methodological niche. It is neither merely residual RL nor merely MoE. Its distinctive contribution is the joint use of residual correction and context-dependent nominal-policy mixture for robust performance across heterogeneous scenario spaces.