MoRe-ERL: Residual Motion Refinement

• The paper introduces MoRe-ERL, a framework that refines preplanned trajectories by learning smooth, B-Spline-based residual corrections via episodic reinforcement learning.
• It leverages trajectory-level policy search to selectively modify only segments of a reference path, preserving essential maneuvers while adapting to changing task contexts.
• Empirical results demonstrate that MoRe-ERL achieves lower planning times and higher success rates in complex settings, particularly in dual-arm tasks, compared to traditional methods.

MoRe-ERL, short for Motion Residuals using Episodic Reinforcement Learning, is a framework that combines Episodic Reinforcement Learning (ERL) and residual learning to refine preplanned reference trajectories into safe, feasible, and efficient task-specific trajectories (Huang et al., 2 Aug 2025). Its central assumption is that a robot often already has a usable reference motion from a planner or policy, and that only part of the remaining trajectory must be adapted when the task context changes. Rather than relearning a complete motion from scratch, MoRe-ERL identifies the trajectory segment that requires modification and generates smooth residual corrections with B-Spline-based movement primitives (BMPs), while preserving task-critical maneuvers encoded in the reference (Huang et al., 2 Aug 2025).

1. Problem setting and residual formulation

MoRe-ERL is designed for trajectory refinement during execution. The starting point is a task context $\bm c_0$ and a reference trajectory

$$\bm y_{r,0:T} = \{\bm y_{r,0}, \dots, \bm y_{r,T}\}.$$

At time $\tau$, the context changes to $\bm c_\tau$, and the problem becomes the generation of a refinement for the remaining segment $\bm y_{r,\tau:T}$. The framework encodes the changed context together with the remaining reference trajectory as

$\bm s_\tau = s(\bm c_\tau, \bm y_{r,\tau:T}),$

so the policy acts on a task-level state that already includes prior motion structure (Huang et al., 2 Aug 2025).

The core residual formulation is additive: $$\bm{y}_{\tau:T} = \bm{y}_{r,\tau:T} + \Delta \bm{y}_{\tau:T}.$$ This means that the reference trajectory is not discarded. Instead, the policy learns a correction $\Delta \bm y_{\tau:T}$, which reduces the search space and preserves useful task knowledge present in the original plan. The paper explicitly motivates this choice by arguing that reference trajectories may already encode hard-to-learn behaviors such as retracting from a shelf, reaching a goal through constrained spaces, or other coordinated maneuvers, and that residual learning can preserve these while adapting only what is necessary (Huang et al., 2 Aug 2025).

A defining feature of MoRe-ERL is that the residual need not modify the whole remaining trajectory. The active refinement interval is specified by learned timing variables $\alpha_s$ and $\alpha_e$, with

$$\bm y_{r,0:T} = \{\bm y_{r,0}, \dots, \bm y_{r,T}\}.$$0

The residual is then applied only on the selected segment: $$\bm y_{r,0:T} = \{\bm y_{r,0}, \dots, \bm y_{r,T}\}.$$1 This partial-refinement structure is the mechanism by which the method preserves critical parts of the original motion while intervening only where the changed context requires it (Huang et al., 2 Aug 2025).

2. Trajectory-level policy and BMP residual parameterization

MoRe-ERL is not a step-wise controller. Its action is a trajectory-level residual parameterization consisting of the interval endpoints $$\bm y_{r,0:T} = \{\bm y_{r,0}, \dots, \bm y_{r,T}\}.$$2 and the parameters of a residual motion primitive. Conditioned on $$\bm y_{r,0:T} = \{\bm y_{r,0}, \dots, \bm y_{r,T}\}.$$3, the policy outputs a correlated Gaussian over these quantities: $$\bm y_{r,0:T} = \{\bm y_{r,0}, \dots, \bm y_{r,T}\}.$$4 where $$\bm y_{r,0:T} = \{\bm y_{r,0}, \dots, \bm y_{r,T}\}.$$5 are the learnable interior control points of the residual trajectory (Huang et al., 2 Aug 2025).

Residuals are represented with B-Spline-based movement primitives. For a residual interval $$\bm y_{r,0:T} = \{\bm y_{r,0}, \dots, \bm y_{r,T}\}.$$6, with control points $$\bm y_{r,0:T} = \{\bm y_{r,0}, \dots, \bm y_{r,T}\}.$$7, the paper writes

$$\bm y_{r,0:T} = \{\bm y_{r,0}, \dots, \bm y_{r,T}\}.$$8

The first two and last two control points are reserved for boundary conditions, while the interior points $$\bm y_{r,0:T} = \{\bm y_{r,0}, \dots, \bm y_{r,T}\}.$$9 determine the shape of the correction. The use of BMPs is motivated by the fact that they preserve the linear movement-primitive structure of ProMPs while supporting explicit boundary conditions, which is essential for entering and exiting the residual interval smoothly (Huang et al., 2 Aug 2025).

The spline basis is given as a $\tau$0-th order B-spline basis

$\tau$1

with recursion

$\tau$2

where

$\tau$3

The paper also emphasizes the derivative property

$\tau$4

with derivative control points

$\tau$5

This supports explicit continuity constraints at the residual boundaries (Huang et al., 2 Aug 2025).

For the main MoRe-ERL residual formulation, the boundary conditions are set so that residual position and velocity vanish at $\tau$6 and $\tau$7. In effect, the refined trajectory leaves the reference smoothly, deviates over the selected interval, and rejoins it smoothly. This is one of the method’s key distinctions from step-based residual RL, where corrections are local and can become jerky (Huang et al., 2 Aug 2025).

3. Learning objective and the role of episodic reinforcement learning

MoRe-ERL uses ERL in the sense of trajectory-space policy search rather than per-step action optimization. Given the encoded state $\tau$8, the paper writes the objective as

$\tau$9

In the experiments, the framework adopts BBRL as the optimizer, while the paper states that the framework is general enough to incorporate arbitrary ERL methods and motion generators (Huang et al., 2 Aug 2025).

The return can be either Markovian or non-Markovian. The Markovian form is

$\bm c_\tau$0

where $\bm c_\tau$1 is a collision penalty, $\bm c_\tau$2 is a goal-reaching term, and $\bm c_\tau$3 is a joint-limit penalty. The collision reward is

$\bm c_\tau$4

upon collision. The non-Markovian form is

$\bm c_\tau$5

The paper argues that the non-Markovian formulation better matches whole-trajectory success criteria and avoids reward hacking in tasks where collision and goal attainment are fundamentally episodic properties (Huang et al., 2 Aug 2025).

This ERL formulation is central to the method’s identity. The action space is the space of trajectory residual parameters, not raw motor torques or one-step joint commands. The paper repeatedly contrasts this with step-based residual RL, arguing that step-level methods tend to require dense Markovian rewards and often produce local corrections that do not naturally preserve trajectory-level smoothness or task structure. A plausible implication is that MoRe-ERL’s main inductive bias is not merely “residual control,” but residual control at the same temporal scale as the task objective (Huang et al., 2 Aug 2025).

4. Algorithmic workflow and refinement variants

The operational pipeline begins with a reference trajectory, often generated in the experiments by RRT-Connect. The robot follows this trajectory until time $\bm c_\tau$6, at which point the context changes. The system then encodes the current context together with the remaining reference, samples the interval variables $\bm c_\tau$7 and BMP parameters $\bm c_\tau$8, constructs the residual over the selected interval, and executes the refined trajectory

$\bm c_\tau$9

After rollout, episodic return is computed and the ERL backend updates the policy (Huang et al., 2 Aug 2025).

The paper studies three refinement strategies. The main method is MoRe-ERL residuals, in which the residual is active only on the learned interval $\bm y_{r,\tau:T}$0. A second strategy is full residual, defined as the special case

$\bm y_{r,\tau:T}$1

so the entire remaining trajectory is refined. A third strategy is partial replacement, in which the selected interval is replaced rather than corrected additively: $\bm y_{r,\tau:T}$2 The paper excludes full replacement because that would amount to learning the remainder of the task from scratch (Huang et al., 2 Aug 2025).

These variants are informative because they isolate the role of additive residuals and interval selection. The experiments show that partial replacement underperforms, indicating that additive residuals preserve prior task structure better than segment replacement. Full residual can be slightly more sample-efficient in the simpler multi-box setting because it avoids learning interval boundaries, but MoRe-ERL gives the best overall performance, especially in the harder dual-arm setting where preserving dexterous task-specific maneuvers matters more (Huang et al., 2 Aug 2025).

The framework is explicitly described as modular. The reference can come from a sampling-based planner, optimization-based planner, or other learned motion generator; the ERL backend can in principle vary; and the residual motion generator is implemented with BMPs. This modularity suggests that MoRe-ERL is best understood as a wrapper around trajectory generators and ERL optimizers, rather than as a monolithic new optimizer (Huang et al., 2 Aug 2025).

5. Empirical evaluation in simulation and hardware

The paper evaluates MoRe-ERL on a multi-box MuJoCo task, a dual-arm MuJoCo task, and a real-world dual-arm setup. In the multi-box task, the robot is a UR10e moving in a workspace divided into three regions by two bars, with dynamic obstacles that are adversarially timed so that at least one obstacle would hit the robot if it followed the reference trajectory unchanged; the episode ends at the goal or after 3 seconds. In the dual-arm task, the active robot is a KUKA iiwa 14, the dynamic environment robot is a UR5, and the episode horizon is 5 seconds (Huang et al., 2 Aug 2025).

The reported comparison includes ST-RRT*, RRT-Connect, ERL + DMP, ERL + DMP + Residuals, BBRL, PPO, PPO-Res, SAC, SAC-Res, Full Residual, Partial Replacement, and MoRe-ERL. On the multi-box task, MoRe-ERL achieves planning time 0.010 s and success 0.889, compared with 0.731 for BBRL, 0.881 for Full Residual, 0.812 for Partial Replacement, 0.587 for RRT-Connect, and 0.928 for ST-RRT* with 1.0 s planning time. On the dual-arm task, MoRe-ERL achieves planning time 0.0098 s and success 0.767, compared with 0.133 for BBRL, 0.674 for Full Residual, 0.299 for Partial Replacement, 0.204 for RRT-Connect, and 0.392 for ST-RRT* with 5.0 s planning time (Huang et al., 2 Aug 2025).

These results establish two distinct properties. First, relative to ERL trained from scratch, residual learning substantially improves both task success and sample efficiency. Second, relative to online replanning baselines, MoRe-ERL trades a small drop from ST-RRT* in the easier multi-box case for dramatically lower inference time, while decisively outperforming ST-RRT* in the harder dual-arm case under the reported planning budgets. The paper also notes an approximate upper bound for dual-arm by giving ST-RRT* 120 seconds, where it reaches 86% success, which suggests that some cases remain solvable by exhaustive planning but not within realistic online budgets (Huang et al., 2 Aug 2025).

The hardware study mirrors the dual-arm setup with a real UR5 and KUKA iiwa in shared workspace. The policy is trained in simulation, where it achieves about 95% success, and then deployed directly on hardware. The paper reports a minimal sim-to-real gap and illustrates behaviors such as collision avoidance and goal reaching without additional online adaptation. This result is consistent with the method’s use of smooth trajectory-level residuals rather than high-frequency reactive corrections, although that interpretation is not explicitly formalized in the paper (Huang et al., 2 Aug 2025).

6. Interpretation, broader ERL context, and limitations

MoRe-ERL occupies a specific place within the broader ERL literature. In robotics, ERL often denotes trajectory-level episodic reinforcement learning, as in "TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning," where policies predict entire action trajectories and often use movement primitives (Li et al., 2024). In language-model training, however, ERL can denote a different paradigm entirely, as in "Experiential Reinforcement Learning," which inserts an experience-reflection-consolidation loop into RL for LLMs (Shi et al., 15 Feb 2026). This suggests that MoRe-ERL should be read specifically in the robot motion generation sense of ERL: optimization over parameterized trajectories or residual trajectories, not reflection-augmented LM training.

A common misconception is to treat MoRe-ERL as ordinary residual RL. The paper’s evidence does not support that simplification. Its action is not a step-wise residual torque or command; it is a jointly sampled interval and spline-defined residual trajectory segment. Another misconception is that any whole-trajectory residual suffices. The ablations show a more nuanced picture: Full Residual can be competitive in the simpler multi-box task, but MoRe-ERL is best overall, especially in dual-arm, while Partial Replacement underperforms in both tasks. The broad implication is that the interval-selection mechanism matters most when task success depends on preserving specific reference maneuvers (Huang et al., 2 Aug 2025).

The paper is explicit about several limitations. Performance “highly depends on the quality of reference trajectories.” The method assumes that refinement is triggered at an appropriate time, and the authors state that deciding when to trigger refinement is itself crucial and requires a higher-level decision-making layer. They also point to future extensions such as time compression/stretching and better sequence encodings for reference trajectories. These limitations are structural rather than incidental: MoRe-ERL is a refinement framework, so its performance inherits both the strengths and weaknesses of the reference generator (Huang et al., 2 Aug 2025).

Within that scope, MoRe-ERL’s significance lies in its synthesis of three design choices: a reference trajectory is preserved rather than discarded; only a selected interval is modified; and the modification is generated at trajectory level with smooth boundary-constrained BMP residuals. In that sense, it is best understood not as a general replacement for planning or RL from scratch, but as a specific architecture for fast, smooth, task-preserving adaptation of preplanned motions (Huang et al., 2 Aug 2025).