Mobile-Manipulation Reward (MMR)

• Mobile-Manipulation Reward (MMR) is a framework of dense, modular reward functions designed to train reinforcement learning agents for coordinated mobile base and manipulator control.
• It integrates kinematic feasibility, disturbance penalties, and grasping uncertainty to promote sample-efficient and robust policy learning.
• Empirical studies on platforms like PR2, TIAGo, and HSR demonstrate that MMR enhances success rates and facilitates cross-task deployment and sim-to-real transfer.

Mobile-Manipulation Reward (MMR) encompasses a class of reward functions explicitly designed to train reinforcement learning (RL) agents for mobile manipulation tasks, wherein a mobile base and robotic manipulator must be controlled, often jointly, to achieve complex, physically grounded objectives. MMR formulations enable dense, modular, and multi-term feedback to drive sample-efficient policy learning—addressing unique challenges inherent in mobile manipulation, such as kinematic reachability, contact-rich interaction, obstacle/disturbance avoidance, and simultaneous multi-subsystem actuation.

1. Core Formulations and Mathematical Structure

The MMR framework can be instantiated via a variety of reward architectures, each detailed in leading research. Prominent patterns include:

• Dense Kinematic Feasibility Reward: The reward at time $t$ penalizes infeasible end-effector (EE) poses predicted to be unreachable by the arm given the task-level plan and current base command, with an additional quadratic action regularizer:

$$r(s_t, a_t, g) = -\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)} - \lambda \|a_t\|^2$$

where $\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)}$ equals 1 if the next EE pose is kinematically infeasible (as determined by inverse kinematics (IK)), and 0 otherwise; $\lambda$ is a regularizer weighting parameter (Honerkamp et al., 2021).

• Primary Task and Disturbance Penalty Decomposition:

$$r_t = R_{\mathrm{task}}(s_t, a_t) - \lambda \cdot R_{\mathrm{disturb}}(s_t, a_t)$$

$R_{\mathrm{task}}$ captures the traditional ArmPointNav reward (task progress, pick/place bonuses), whereas $R_{\mathrm{disturb}}$ accumulates visible movement of non-target objects, with $\lambda$ tuned empirically (Ni et al., 2021).

• Modular, Multi-term Additive Structure: For whole-body control:

$R(s, a) = \sum_{j=1}^m r_j(s, a)$

with each $r_j$ encoding objectives such as approach, orientation, collision penalties, or manipulability, providing sparse or dense shaping for each sub-goal (Hu et al., 2023).

These structures facilitate dense reward propagation, enable modularity, and permit extension to new robot morphologies or environments.

2. Measurement of Key Reward Terms

• Kinematic Feasibility: Each RL step integrates the desired EE pose using the task-level velocity generator ($$r(s_t, a_t, g) = -\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)} - \lambda \|a_t\|^2$$0). A numerical IK solver attempts to solve for a feasible joint configuration $$r(s_t, a_t, g) = -\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)} - \lambda \|a_t\|^2$$1 that attains the pose. Feasibility is true if a solution exists within joint, workspace, and collision constraints; certain platforms tolerate minor errors during rollout (e.g., up to 10 cm position / $$r(s_t, a_t, g) = -\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)} - \lambda \|a_t\|^2$$2 orientation in HSR), but require zero tolerance at episode termination (Honerkamp et al., 2021).
• Disturbance Penalty: Disturbance is measured as the cumulative Euclidean displacement of all non-target objects from their initial positions:

$$r(s_t, a_t, g) = -\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)} - \lambda \|a_t\|^2$$3

with $$r(s_t, a_t, g) = -\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)} - \lambda \|a_t\|^2$$4; disturbances must exceed a threshold $$r(s_t, a_t, g) = -\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)} - \lambda \|a_t\|^2$$5 to avoid penalizing noise (Ni et al., 2021).

• Grasping Uncertainty as Exploration Signal: An ensemble of grasp predictors estimates

$$r(s_t, a_t, g) = -\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)} - \lambda \|a_t\|^2$$6

where $$r(s_t, a_t, g) = -\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)} - \lambda \|a_t\|^2$$7 is the ensemble mean success probability, $$r(s_t, a_t, g) = -\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)} - \lambda \|a_t\|^2$$8 the standard deviation, $$r(s_t, a_t, g) = -\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)} - \lambda \|a_t\|^2$$9 are hyperparameters. The navigation reward can exploit this estimate as $\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)}$0 (Sun et al., 2021).

3. Integration with Deep RL Algorithms

• Actor-Critic Methods: MMR has been deployed with both off-policy algorithms (e.g., SAC, TD3) and on-policy PPO. In (Honerkamp et al., 2021), SAC was preferred for improved stability under dense feasibility-checking rewards. PPO, coupled with curriculum learning and auxiliary prediction heads, is used for visually driven mobile manipulation with disturbance penalties (Ni et al., 2021).
• Goal Conditioning: The reward may depend on an explicit goal $\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)}$1 in the state, supporting goal-conditioned policies and generalization to unseen tasks.
• Curriculum Learning: For disturbance-sensitive tasks, a two-stage training regime—unconstrained pre-training ($\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)}$2) followed by fine-tuning with disturbance penalty—enables acquisition of both task and disturbance-avoidant behaviors. Auxiliary losses (e.g., disturbance prediction) can further accelerate representation learning (Ni et al., 2021).
• Causal Policy Gradient: By explicitly discovering dependencies between action components and reward terms via conditional mutual information, policy-gradient estimation can be focused on relevant subspaces, reducing variance without bias. The “Causal Matrix” $\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)}$3 encodes which $\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)}$4 causally influence $\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)}$5 and guides policy updates (Hu et al., 2023).

4. Empirical Validation and Performance Comparisons

Ablation and benchmark studies demonstrate the sample efficiency and reliability of MMR variants:

Task/Domain	Baselines (Success %)	MMR Approach (Success %)	Notable Improvements
PR2, TIAGo, HSR (Feasibility) (Honerkamp et al., 2021)	20–70	90–97	Robust generalization across platforms
ManipulaTHOR (SR₍woD₎) (Ni et al., 2021)	35.5 (improved baseline)	47.1 (two-stage MMR)	+11.6% absolute, >30% rel. over safe/constrained RL
iGibson/HSR (Hu et al., 2023)	74 (PPO)	94 (Causal MoMa)	High sim-to-real transferability, dynamic goal/obs.

These results generalize across mobile manipulator types and task morphologies, reinforcing that dense, modular, and causally informed reward shaping substantially improves performance and robustness.

5. Modularity, Extensions, and Limitations

• Modularity and Portability: MMR designs such as (Honerkamp et al., 2021) are agnostic to the specific end-effector motion planner, enabling straightforward adaptation across PR2, TIAGo, HSR, or similar platforms without reward re-design.
• Extension to Multi-Objective and Safety Constraints: Composite rewards can be extended to penalize environment collisions, enforce energy efficiency, or optimize for perceptual objectives. For example, replacing the indicator reward with manipulability indices or adding obstacle-collision penalties enables graded feedback or safety prioritization (Honerkamp et al., 2021, Ni et al., 2021).
• Limitations: Arm-centric kinematic rewards penalize only EE reachability, not environmental or inter-object collisions, which can be critical in cluttered or human-centric environments. Some formulations may exhibit boundary “hugging” if failure is penalized only as a hard indicator; empirical results suggest this is not a dominant failure mode but may warrant future attention. Fast rotations of the mobile base are known to destabilize arm control without low-level coordination (Honerkamp et al., 2021).

6. Hyperparameter Tuning and Recommended Practices

Canonical settings favor moderate regularization (e.g., $\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)}$6 for action penalty), and typically employ early termination upon multiple consecutive infeasibility checks (e.g., 19 IK failures). Standard discount ($\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)}$7) and target smoothing ($\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)}$8) hyperparameters mirror those of modern deep RL algorithms: e.g., $\mathbb{1}_{\neg\mathrm{kin}(s_t, a_t)}$9 for SAC (Honerkamp et al., 2021). Empirical scaling of disturbance penalties (e.g., $\lambda$0 for visible object displacement) is recommended based on validation performance (Ni et al., 2021).

A plausible implication is that cross-task deployment requires only minor hyperparameter grid search, given the reward’s dense and portable structure.

7. Perspectives and Future Directions

Recent MMR developments point toward (i) further integration of perception and physical safety via auxiliary tasks and disturbance prediction (Ni et al., 2021), (ii) automatic discovery of reward-action dependency structure for policy-gradient optimization, eliminating manual decomposition and enhancing scalability (Hu et al., 2023), and (iii) extensions toward multi-modal rewards incorporating tactile, visual, and proprioceptive signals.

Efforts to replace binary feasibility or disturbance indicators with continuous-valued metrics (e.g., manipulability, graded collision risk) may further improve policy smoothness and real-world deployability. MMR principles are now central to state-of-the-art frameworks for learning robust, transferrable mobile manipulation policies with limited supervision and maximal generality.