---
title: 'MoRe-ERL: Residual Motion Refinement'
url: https://www.emergentmind.com/topics/more-erl
type: topic
---

# MoRe-ERL: Residual Motion Refinement

MoRe-ERL, short for **Motion Residuals using Episodic Reinforcement Learning**, is a framework that combines **Episodic Reinforcement Learning (ERL)** and **residual learning** to refine preplanned reference trajectories into safe, feasible, and efficient task-specific trajectories [2508.01409]. Its central assumption is that a robot often already has a usable reference motion from a planner or policy, and that only part of the remaining trajectory must be adapted when the task context changes. Rather than relearning a complete motion from scratch, MoRe-ERL identifies the trajectory segment that requires modification and generates smooth residual corrections with **B-Spline-based movement primitives (BMPs)**, while preserving task-critical maneuvers encoded in the reference [2508.01409].

## 1. Problem setting and residual formulation

MoRe-ERL is designed for **trajectory refinement during execution**. The starting point is a task context \(\bm c_0\) and a reference trajectory
\[
\bm y_{r,0:T} = \{\bm y_{r,0}, \dots, \bm y_{r,T}\}.
\]
At time \(\tau\), the context changes to \(\bm c_\tau\), and the problem becomes the generation of a refinement for the remaining segment \(\bm y_{r,\tau:T}\). The framework encodes the changed context together with the remaining reference trajectory as
\[
\bm s_\tau = s(\bm c_\tau, \bm y_{r,\tau:T}),
\]
so the policy acts on a task-level state that already includes prior motion structure [2508.01409].

The core residual formulation is additive:
\[
\bm{y}_{\tau:T} = \bm{y}_{r,\tau:T} + \Delta \bm{y}_{\tau:T}.
\]
This means that the reference trajectory is not discarded. Instead, the policy learns a correction \(\Delta \bm y_{\tau:T}\), which reduces the search space and preserves useful task knowledge present in the original plan. The paper explicitly motivates this choice by arguing that reference trajectories may already encode hard-to-learn behaviors such as retracting from a shelf, reaching a goal through constrained spaces, or other coordinated maneuvers, and that residual learning can preserve these while adapting only what is necessary [2508.01409].

A defining feature of MoRe-ERL is that the residual need not modify the whole remaining trajectory. The active refinement interval is specified by learned timing variables \(\alpha_s\) and \(\alpha_e\), with
\[
\alpha_s, \alpha_e \in [\tau, T], \qquad \alpha_s \le \alpha_e.
\]
The residual is then applied only on the selected segment:
\[
\Delta \bm{y}_{\tau:T} =
\begin{cases}
\Delta \bm{y}_{k} & \text{for } k \in [\alpha_s, \alpha_e] \\
0 & \text{otherwise}.
\end{cases}
\]
This partial-refinement structure is the mechanism by which the method preserves critical parts of the original motion while intervening only where the changed context requires it [2508.01409].

## 2. Trajectory-level policy and BMP residual parameterization

MoRe-ERL is not a step-wise controller. Its action is a **trajectory-level residual parameterization** consisting of the interval endpoints \((\alpha_s,\alpha_e)\) and the parameters of a residual motion primitive. Conditioned on \(\bm s_\tau\), the policy outputs a correlated Gaussian over these quantities:
\[
[\bm w^\top_{3:N-2}, \alpha_s, \alpha_e]^\top \sim \mathcal N(\bm \mu_{\bm w,\bm \alpha}, \bm \Sigma_{\bm w,\bm \alpha}),
\]
where \(\bm w_{3:N-2}\) are the learnable interior control points of the residual trajectory [2508.01409].

Residuals are represented with **B-Spline-based movement primitives**. For a residual interval \([\alpha_s,\alpha_e]\), with control points \(\bm w_{1:N}\), the paper writes
\[
\Delta \bm{y}_{\alpha_s:\alpha_e} = \bm{\Phi}_{\alpha_s:\alpha_e}^\top \bm{w}_{3:N-2}.
\]
The first two and last two control points are reserved for boundary conditions, while the interior points \(\bm w_{3:N-2} = [w_3,\dots,w_{N-2}]^\top\) determine the shape of the correction. The use of BMPs is motivated by the fact that they preserve the linear movement-primitive structure of ProMPs while supporting explicit boundary conditions, which is essential for entering and exiting the residual interval smoothly [2508.01409].

The spline basis is given as a \(P\)-th order B-spline basis
\[
\bm \Phi^P(u) = [\Phi^P_1(u), \Phi^P_2(u), \dots, \Phi^P_N(u)]^\top,
\]
with recursion
\[
\Phi^p_{n}(u) = j_{n}\, \Phi^{p-1}_{n}(u) + j_{n+1}\, \Phi^{p-1}_{n+1}(u),
\]
where
\[
j_{n} = \frac{u - u_n}{p\,\delta}, \qquad j_{n+1} = \frac{u_{n+p+1} - u}{p\,\delta}.
\]
The paper also emphasizes the derivative property
\[
\bm y^{(i)}(t) = \bm \Phi^{P-i}(u)^\top \bm \omega^{(i)},
\]
with derivative control points
\[
w^{(i)}_n = \frac{P-i}{\delta}\left[w_{n+1}^{(i-1)} - w_n^{(i-1)}\right].
\]
This supports explicit continuity constraints at the residual boundaries [2508.01409].

For the main MoRe-ERL residual formulation, the boundary conditions are set so that **residual position and velocity vanish** at \(\alpha_s\) and \(\alpha_e\). In effect, the refined trajectory leaves the reference smoothly, deviates over the selected interval, and rejoins it smoothly. This is one of the method’s key distinctions from step-based residual RL, where corrections are local and can become jerky [2508.01409].

## 3. Learning objective and the role of episodic reinforcement learning

MoRe-ERL uses ERL in the sense of **trajectory-space policy search** rather than per-step action optimization. Given the encoded state \(\bm s_\tau\), the paper writes the objective as
\[
J = E_{p(\bm s_{\tau}),\pi_{\theta}(\bm w|\bm s_{\tau})}
\left[ R(\bm s_{\tau}, \bm w) - V_{\phi}(\bm s_{\tau}) \right].
\]
In the experiments, the framework adopts **BBRL** as the optimizer, while the paper states that the framework is general enough to incorporate arbitrary ERL methods and motion generators [2508.01409].

The return can be either **Markovian** or **non-Markovian**. The Markovian form is
\[
R_M = \sum_{t=\tau}^{n_e} \gamma^t \left( \beta_c r_{c,t} + \beta_g r_{g,t} + \beta_l r_{l,t} \right),
\]
where \(r_{c,t}\) is a collision penalty, \(r_{g,t}\) is a goal-reaching term, and \(r_{l,t}\) is a joint-limit penalty. The collision reward is
\[
r_{c,t} = -1
\]
upon collision. The non-Markovian form is
\[
R_{NM} = \beta_c r_c + \beta_g r_g + \beta_l \sum_i^{n_e} \gamma^i r_{l,t}.
\]
The paper argues that the non-Markovian formulation better matches whole-trajectory success criteria and avoids reward hacking in tasks where collision and goal attainment are fundamentally episodic properties [2508.01409].

This ERL formulation is central to the method’s identity. The action space is the space of **trajectory residual parameters**, not raw motor torques or one-step joint commands. The paper repeatedly contrasts this with step-based residual RL, arguing that step-level methods tend to require dense Markovian rewards and often produce local corrections that do not naturally preserve trajectory-level smoothness or task structure. A plausible implication is that MoRe-ERL’s main inductive bias is not merely “residual control,” but **residual control at the same temporal scale as the task objective** [2508.01409].

## 4. Algorithmic workflow and refinement variants

The operational pipeline begins with a reference trajectory, often generated in the experiments by **RRT-Connect**. The robot follows this trajectory until time \(\tau\), at which point the context changes. The system then encodes the current context together with the remaining reference, samples the interval variables \((\alpha_s,\alpha_e)\) and BMP parameters \(\bm w\), constructs the residual over the selected interval, and executes the refined trajectory
\[
\bm y_{\tau:T} = \bm y_{r,\tau:T} + \Delta \bm y_{\tau:T}.
\]
After rollout, episodic return is computed and the ERL backend updates the policy [2508.01409].

The paper studies three refinement strategies. The main method is **MoRe-ERL residuals**, in which the residual is active only on the learned interval \([\alpha_s,\alpha_e]\). A second strategy is **full residual**, defined as the special case
\[
\alpha_s=\tau,\qquad \alpha_e=T,
\]
so the entire remaining trajectory is refined. A third strategy is **partial replacement**, in which the selected interval is replaced rather than corrected additively:
\[
\bm{y}_{\tau:T} =
\begin{cases}
\Delta \bm{y}_{k} & \text{for } k \in [\alpha_s, \alpha_e] \\
\bm{y}_{r,k} & \text{otherwise}.
\end{cases}
\]
The paper excludes full replacement because that would amount to learning the remainder of the task from scratch [2508.01409].

These variants are informative because they isolate the role of additive residuals and interval selection. The experiments show that **partial replacement** underperforms, indicating that additive residuals preserve prior task structure better than segment replacement. **Full residual** can be slightly more sample-efficient in the simpler multi-box setting because it avoids learning interval boundaries, but MoRe-ERL gives the best overall performance, especially in the harder dual-arm setting where preserving dexterous task-specific maneuvers matters more [2508.01409].

The framework is explicitly described as modular. The reference can come from a sampling-based planner, optimization-based planner, or other learned motion generator; the ERL backend can in principle vary; and the residual motion generator is implemented with BMPs. This modularity suggests that MoRe-ERL is best understood as a wrapper around trajectory generators and ERL optimizers, rather than as a monolithic new optimizer [2508.01409].

## 5. Empirical evaluation in simulation and hardware

The paper evaluates MoRe-ERL on a **multi-box** MuJoCo task, a **dual-arm** MuJoCo task, and a **real-world dual-arm** setup. In the multi-box task, the robot is a **UR10e** moving in a workspace divided into three regions by two bars, with dynamic obstacles that are adversarially timed so that at least one obstacle would hit the robot if it followed the reference trajectory unchanged; the episode ends at the goal or after **3 seconds**. In the dual-arm task, the active robot is a **KUKA iiwa 14**, the dynamic environment robot is a **UR5**, and the episode horizon is **5 seconds** [2508.01409].

The reported comparison includes **ST-RRT\***, **RRT-Connect**, **ERL + DMP**, **ERL + DMP + Residuals**, **BBRL**, **PPO**, **PPO-Res**, **SAC**, **SAC-Res**, **Full Residual**, **Partial Replacement**, and **MoRe-ERL**. On the multi-box task, MoRe-ERL achieves planning time **0.010 s** and success **0.889**, compared with **0.731** for **BBRL**, **0.881** for **Full Residual**, **0.812** for **Partial Replacement**, **0.587** for **RRT-Connect**, and **0.928** for **ST-RRT\*** with **1.0 s** planning time. On the dual-arm task, MoRe-ERL achieves planning time **0.0098 s** and success **0.767**, compared with **0.133** for **BBRL**, **0.674** for **Full Residual**, **0.299** for **Partial Replacement**, **0.204** for **RRT-Connect**, and **0.392** for **ST-RRT\*** with **5.0 s** planning time [2508.01409].

These results establish two distinct properties. First, relative to ERL trained from scratch, residual learning substantially improves both **task success** and **sample efficiency**. Second, relative to online replanning baselines, MoRe-ERL trades a small drop from **ST-RRT\*** in the easier multi-box case for dramatically lower inference time, while decisively outperforming **ST-RRT\*** in the harder dual-arm case under the reported planning budgets. The paper also notes an approximate upper bound for dual-arm by giving **ST-RRT\*** **120 seconds**, where it reaches **86%** success, which suggests that some cases remain solvable by exhaustive planning but not within realistic online budgets [2508.01409].

The hardware study mirrors the dual-arm setup with a real **UR5** and **KUKA iiwa** in shared workspace. The policy is trained in simulation, where it achieves about **95% success**, and then deployed directly on hardware. The paper reports a **minimal sim-to-real gap** and illustrates behaviors such as collision avoidance and goal reaching without additional online adaptation. This result is consistent with the method’s use of smooth trajectory-level residuals rather than high-frequency reactive corrections, although that interpretation is not explicitly formalized in the paper [2508.01409].

## 6. Interpretation, broader ERL context, and limitations

MoRe-ERL occupies a specific place within the broader ERL literature. In robotics, ERL often denotes **trajectory-level episodic reinforcement learning**, as in "TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning," where policies predict entire action trajectories and often use movement primitives [2410.09536]. In language-model training, however, ERL can denote a different paradigm entirely, as in "Experiential Reinforcement Learning," which inserts an experience-reflection-consolidation loop into RL for language models [2602.13949]. This suggests that MoRe-ERL should be read specifically in the **robot motion generation** sense of ERL: optimization over parameterized trajectories or residual trajectories, not reflection-augmented LM training.

A common misconception is to treat MoRe-ERL as ordinary residual RL. The paper’s evidence does not support that simplification. Its action is not a step-wise residual torque or command; it is a **jointly sampled interval and spline-defined residual trajectory segment**. Another misconception is that any whole-trajectory residual suffices. The ablations show a more nuanced picture: **Full Residual** can be competitive in the simpler multi-box task, but **MoRe-ERL** is best overall, especially in dual-arm, while **Partial Replacement** underperforms in both tasks. The broad implication is that the interval-selection mechanism matters most when task success depends on preserving specific reference maneuvers [2508.01409].

The paper is explicit about several limitations. Performance “highly depends on the quality of reference trajectories.” The method assumes that refinement is triggered at an appropriate time, and the authors state that deciding **when to trigger refinement** is itself crucial and requires a higher-level decision-making layer. They also point to future extensions such as **time compression/stretching** and **better sequence encodings** for reference trajectories. These limitations are structural rather than incidental: MoRe-ERL is a refinement framework, so its performance inherits both the strengths and weaknesses of the reference generator [2508.01409].

Within that scope, MoRe-ERL’s significance lies in its synthesis of three design choices: a reference trajectory is preserved rather than discarded; only a selected interval is modified; and the modification is generated at trajectory level with smooth boundary-constrained BMP residuals. In that sense, it is best understood not as a general replacement for planning or RL from scratch, but as a specific architecture for **fast, smooth, task-preserving adaptation of preplanned motions** [2508.01409].

Source: https://www.emergentmind.com/topics/more-erl