Reward Fusion Module (RFM) for Multimodal Integration

• Reward Fusion Module (RFM) is a mechanism that nonlinearly fuses manipulation, locomotion, and safety rewards to address dynamic task priorities.
• It employs products, exponentials, clipping, and sigmoid phase gates to enable stage-wise reward prioritization and mitigate conflicting objectives.
• Empirical studies in whole-body loco-manipulation and LLM fusion demonstrate that RFM significantly improves success rates and precision through adaptive reward scheduling.

Searching arXiv for the specified papers to ground the article and citations. arxiv_search.query({"^^^^^^^^7search_query7 OR id:(Jiang et al., 2024)7^^^^^^^^"," arxiv_search.query({"^^^^^^^^7search_query7 Preference Optimization for Implicit Model Fusion\" OR ^^^^^^^^7ti:\7 Whole-Body Loco-Manipulation for Omni-Directional Task Space Pose Tracking with a Wheeled-Quadrupedal-Manipulator\"","^^^^^^^^7start7 OR id:(Jiang et al., 2024)7search_query7^^^^}) Reward Fusion Module (RFM) denotes a reward-construction mechanism that combines multiple task signals through nonlinear or scheduled fusion rather than a static linear weighted sum. In the current arXiv literature, the term is explicitly introduced for whole-body loco-manipulation in a wheeled-quadrupedal-manipulator, where RFM fuses manipulation, locomotion, and basic regularization/safety rewards to train a single policy for omni-directional six-dimensional end-effector pose tracking (&&&^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^&&&). A broader, inferred usage appears in weighted preference optimization for implicit LLM fusion: that paper does not explicitly name an RFM, but its Weighted-Reward Preference Optimization (WRPO) combines source- and target-derived preference rewards into a single weighted internal reward, which functions as a practical reward-fusion design within a DPO-style objective (&&&^^^^7search_query7^^^^&&&).

^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^. Terminological scope and motivating problem

In the wheeled-quadrupedal-manipulator setting, RFM is introduced to address the whole-body loco-manipulation problem in which a floating base and a ^^^^7 OR ti:\7^^^^-DoF arm must be coordinated to achieve direct ^^^^7 OR ti:\7^^^^D end-effector pose tracking in task space. The central difficulty is that direct end-effector tracking requires inherent balance among redundant degrees of freedom in the whole-body motion. The paper argues that linear weighted sums are insufficient because hierarchical priorities change over time, objectives such as end-effector tracking and base stability can conflict, and reward influence must be gated across “locomotion,” “transition,” and “manipulation” phases to avoid jittery switching or deadlock (&&&^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^&&&).

The same general design pressure appears in WRPO for heterogeneous LLM fusion. There, explicit fusion methods such as multi-teacher distillation require vocabulary alignment and merging probability distributions across heterogeneous LLMs, and these steps are described as complex, inefficient, and prone to introducing noise and errors. WRPO instead performs implicit model fusion by combining on-policy and off-policy preference signals through a weighted internal reward and a progressive adaptation schedule. This suggests that “reward fusion” can be understood not only as a robotics reward-engineering module, but also as a more general mechanism for integrating heterogeneous supervisory signals without explicit distribution merging (&&&^^^^7search_query7^^^^&&&).

^^^^7start7^^^^. Task-space formulation in whole-body loco-manipulation

The robotic RFM is built around direct ^^^^7 OR ti:\7^^^^D end-effector pose tracking. The target is supplied in the world frame but represented in the body frame for policy invariance: PRESERVED_PLACEHOLDER_^^^^7search_query7^^^^ The command vector is

PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^

The current end-effector pose is obtained through forward kinematics. With world-frame base pose PRESERVED_PLACEHOLDER_^^^^7start7^^^^ and body-frame arm kinematics PRESERVED_PLACEHOLDER_^^^^7max_results7^^^^, the world-frame end-effector pose is

PRESERVED_PLACEHOLDER_^^^^7search_query7^^^^

Tracking uses an PRESERVED_PLACEHOLDER_^^^^7ti:\7^^^^ log-map metric: PRESERVED_PLACEHOLDER_^^^^7 OR ti:\7^^^^ and the combined tracking error

$$\epsilon_t \;=\; a_1\,d_\theta(t) \;+\; a_2\,d_p(t),\qquad a_1,a_2>0.$$

This formulation matters because the reward is not attached to separate base and arm commands. Instead, the policy receives a task-space end-effector objective and must discover how locomotion and manipulation should be sequenced. That is the specific context in which the RFM’s nonlinear gating and stage-wise fusion are used to encode multi-stage structure (&&&^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^&&&).

^^^^7max_results7^^^^. Nonlinear reward fusion architecture

The RFM organizes rewards into manipulation rewards, locomotion rewards, and basic rewards. Its defining feature is that these terms are not simply summed with fixed coefficients; instead, products, exponentials, clipping, and sigmoid phase gates are used so that some rewards contribute only when others indicate that preconditions are satisfied (&&&^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^&&&).

Within manipulation, the position and orientation tracking terms are

$$r_{ep}(t) = \exp\!\left(-\frac{d_p(t)}{\sigma_p}\right),\quad \sigma_p>0,$$

$$r_{eo}(t) = \exp\!\left(-\frac{d_\theta(t)}{\sigma_o}\right),\quad \sigma_o>0.$$

A micro-enhancement is added: PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query7^^^^ with PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^. A cumulative penalty supplies integral-like shaping: PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7start7^^^^ An all-contact stabilizer is also defined: PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7max_results7^^^^

The prioritization between position and orientation is encoded by

PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query7^^^^

with derivatives

PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7ti:\7^^^^

This construction means that orientation contributes only when position tracking is already good. Manipulation-level fusion is then

PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7 OR ti:\7^^^^

The product gate enforces priority for safe manipulation over tracking: tracking terms contribute only when manipulation regularization is satisfied.

Locomotion is shaped by an PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)77^^^^ reference schedule

PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^8

and a coarse displacement reward

PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^9

The locomotion aggregate is

PRESERVED_PLACEHOLDER_^^^^7start7search_query7^^^^

where PRESERVED_PLACEHOLDER_^^^^7start7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^ penalizes deviations from a stowed arm posture during locomotion.

Stage-wise fusion is controlled by

PRESERVED_PLACEHOLDER_^^^^7start7start7^^^^

with recommended PRESERVED_PLACEHOLDER_^^^^7start7max_results7^^^^, PRESERVED_PLACEHOLDER_^^^^7start7search_query7^^^^. The final reward is

PRESERVED_PLACEHOLDER_^^^^7start7ti:\7^^^^

and the cumulative-penalty weight is scheduled as

PRESERVED_PLACEHOLDER_^^^^7start7 OR ti:\7^^^^

The paper characterizes this construction in terms of boundedness, gradient shaping, and reward-hacking prevention. Exponential terms lie in PRESERVED_PLACEHOLDER_^^^^7start77^^^^, the cumulative penalty is clipped, and phase-dependent product gates suppress gradients from misaligned objectives, such as orientation refinement when position is still poor (&&&^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^&&&).

^^^^7search_query7^^^^. Policy learning, observations, and teacher–student integration

The RFM is embedded in a teacher–student RL pipeline rather than used as a standalone reward evaluator. The observation PRESERVED_PLACEHOLDER_^^^^7start7^^^^8 includes joint positions and velocities, last action, base angular velocity, projected gravity, current end-effector pose, and the PRESERVED_PLACEHOLDER_^^^^7start7^^^^9 reference PRESERVED_PLACEHOLDER_^^^^7max_results7search_query7^^^^. The policy input is PRESERVED_PLACEHOLDER_^^^^7max_results7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^, where PRESERVED_PLACEHOLDER_^^^^7max_results7start7^^^^ is the ^^^^7 OR ti:\7^^^^D end-effector command and PRESERVED_PLACEHOLDER_^^^^7max_results7max_results7^^^^ is a latent estimating privileged dynamics. Actions satisfy PRESERVED_PLACEHOLDER_^^^^7max_results7search_query7^^^^: position targets for non-wheel joints and velocity targets for wheel joints. Torques are applied through PD control: PRESERVED_PLACEHOLDER_^^^^7max_results7ti:\7^^^^

The teacher uses an MLP privileged encoder that maps privileged signals such as contact forces, friction, and end-effector twist to PRESERVED_PLACEHOLDER_^^^^7max_results7 OR ti:\7^^^^. The teacher policy receives PRESERVED_PLACEHOLDER_^^^^7max_results77^^^^, and the encoder and policy are trained jointly with PPO, with GAE used for advantage. The student is warm-^^^^^^^^7start7 from the teacher policy, and an estimator MLP learns PRESERVED_PLACEHOLDER_^^^^7max_results7^^^^8 from a ^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query7^^^^-frame history of proprioception by minimizing

PRESERVED_PLACEHOLDER_^^^^7max_results7^^^^9

after which the student fine-tunes with PPO using PRESERVED_PLACEHOLDER_^^^^7search_query7search_query7^^^^ (&&&^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^&&&).

Training uses Isaac Gym with ^^^^7search_query7search_query7max_results7 OR ti:\7^^^^ environments, curriculum, domain randomization, and decaying entropy; MuJoCo is used for sim-to-sim validation, followed by hardware deployment. The teacher runs for 7^^^^7search_query7search_query7search_query7^^^^ iterations PRESERVED_PLACEHOLDER_^^^^7search_query7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^ ^^^^7start7search_query7^^^^ steps, and the student for ^^^^7search_query7search_query7search_query7search_query7^^^^ iterations, with training on RTX^^^^7max_results7search_query7max_results7search_query7^^^^ taking approximately PRESERVED_PLACEHOLDER_^^^^7search_query7start7^^^^ hours for the teacher and PRESERVED_PLACEHOLDER_^^^^7search_query7max_results7^^^^ hours for the student. The per-step reward PRESERVED_PLACEHOLDER_^^^^7search_query7search_query7^^^^ produced by the RFM feeds both GAE and the TD error for critic training. This placement is important: RFM directly shapes both policy improvement and value estimation, not merely the scalar return (&&&^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^&&&).

^^^^7ti:\7^^^^. Empirical behavior, ablations, and limitations in robotics

Simulation ablations are reported over ^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query7search_query7search_query7^^^^ trials each, with failures excluded in averaged metrics except success. The full RFM obtains Success PRESERVED_PLACEHOLDER_^^^^7search_query7ti:\7^^^^, Avg. Power PRESERVED_PLACEHOLDER_^^^^7search_query7 OR ti:\7^^^^ W, Avg. Accel PRESERVED_PLACEHOLDER_^^^^7search_query77^^^^ radPRESERVED_PLACEHOLDER_^^^^7search_query7^^^^8sPRESERVED_PLACEHOLDER_^^^^7search_query7 end-effector position error PRESERVED_PLACEHOLDER_^^^^7ti:\7search_query7^^^^ m, end-effector orientation error PRESERVED_PLACEHOLDER_^^^^7ti:\7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^ rad, and nominal deviation PRESERVED_PLACEHOLDER_^^^^7ti:\7start7^^^^ m. These results are contrasted against several ablations that isolate the role of the nonlinear fusion structure (&&&^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^&&&).

Replacing Loco–Mani Fusion with a linear sum reduces success to PRESERVED_PLACEHOLDER_^^^^7ti:\7max_results7^^^^, increases power to PRESERVED_PLACEHOLDER_^^^^7ti:\7search_query7^^^^ W and acceleration to PRESERVED_PLACEHOLDER_^^^^7ti:\7ti:\7^^^^ radPRESERVED_PLACEHOLDER_^^^^7ti:\7 OR ti:\7^^^^sPRESERVED_PLACEHOLDER_^^^^7ti:\77^^^^, worsens orientation error to PRESERVED_PLACEHOLDER_^^^^7ti:\7^^^^8 rad, and raises nominal deviation to PRESERVED_PLACEHOLDER_^^^^7ti:\7^^^^9 m; this is associated with jitter and failures due to poor stage coordination. Replacing Reward Prioritization inside the manipulation group with a linear weighted sum yields Success PRESERVED_PLACEHOLDER_^^^^7 OR ti:\7search_query7^^^^, end-effector position error PRESERVED_PLACEHOLDER_^^^^7 OR ti:\7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^ m, and nominal deviation PRESERVED_PLACEHOLDER_^^^^7 OR ti:\7start7^^^^ m, indicating loss of hierarchy and riskier configurations. Removing Enhancement gives Success PRESERVED_PLACEHOLDER_^^^^7 OR ti:\7max_results7^^^^, position error PRESERVED_PLACEHOLDER_^^^^7 OR ti:\7search_query7^^^^ m, and orientation error PRESERVED_PLACEHOLDER_^^^^7 OR ti:\7ti:\7^^^^ rad, indicating reduced local precision. Removing the RFM entirely in favor of a linear weighted-sum baseline yields Success PRESERVED_PLACEHOLDER_^^^^7 OR ti:\7 OR ti:\7^^^^, position error PRESERVED_PLACEHOLDER_^^^^7 OR ti:\77^^^^ m, and nominal deviation PRESERVED_PLACEHOLDER_^^^^7 OR ti:\7^^^^8 m, despite heavy tuning.

Hardware results report fixed-point tracking with position error PRESERVED_PLACEHOLDER_^^^^7 OR ti:\7^^^^9 m and orientation error $$\epsilon_t \;=\; a_1\,d_\theta(t) \;+\; a_2\,d_p(t),\qquad a_1,a_2>0.$$^^^^7search_query7^^^^ rad, and spatial-circle tracking with position error $$\epsilon_t \;=\; a_1\,d_\theta(t) \;+\; a_2\,d_p(t),\qquad a_1,a_2>0.$$^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^ m and orientation error $$\epsilon_t \;=\; a_1\,d_\theta(t) \;+\; a_2\,d_p(t),\qquad a_1,a_2>0.$$^^^^7start7^^^^ rad. The overall real-world performance is summarized as achieving less than $$\epsilon_t \;=\; a_1\,d_\theta(t) \;+\; a_2\,d_p(t),\qquad a_1,a_2>0.$$^^^^7max_results7^^^^ cm position error and less than $$\epsilon_t \;=\; a_1\,d_\theta(t) \;+\; a_2\,d_p(t),\qquad a_1,a_2>0.$$^^^^7search_query7^^^^ rad rotation error. Qualitatively, the paper reports smooth transitions between locomotion and manipulation without manual mode switches, improved efficiency and stability from arm stowing during locomotion, and precise orientation refinement only near the target (&&&^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^&&&).

The stated limitations are parameter sensitivity and task scope. Performance is sensitive to phase parameters $$\epsilon_t \;=\; a_1\,d_\theta(t) \;+\; a_2\,d_p(t),\qquad a_1,a_2>0.$$^^^^7ti:\7^^^^, the micro-enhancement exponent $$\epsilon_t \;=\; a_1\,d_\theta(t) \;+\; a_2\,d_p(t),\qquad a_1,a_2>0.$$^^^^7 OR ti:\7^^^^, and the schedule speed $$\epsilon_t \;=\; a_1\,d_\theta(t) \;+\; a_2\,d_p(t),\qquad a_1,a_2>0.$$7; extreme values may induce over-eager transitions or overly cautious behavior. The cumulative penalty must be clipped because unbounded integral terms can destabilize training. The policy relies on proprioception and world/base state estimation without perception, so tight environments or obstacles would require observation augmentation and perception-informed regularization or rewards.

^^^^7 OR ti:\7^^^^. WRPO as an implicit reward-fusion design in LLM fusion

In WRPO, there is no explicitly named “Reward Fusion Module (RFM).” Nonetheless, the method instantiates reward fusion by combining preference signals from heterogeneous source LLMs and the target LLM into a single weighted internal reward that drives a DPO-style objective, together with a progressive adaptation schedule that mitigates distribution shift (&&&^^^^7search_query7^^^^&&&).

The setup uses source LLMs $$\epsilon_t \;=\; a_1\,d_\theta(t) \;+\; a_2\,d_p(t),\qquad a_1,a_2>0.$$8 and a target LLM $$\epsilon_t \;=\; a_1\,d_\theta(t) \;+\; a_2\,d_p(t),\qquad a_1,a_2>0.$$9, exemplified by LLaMA^^^^7max_results7^^^^-8B-Instruct. For each prompt $$r_{ep}(t) = \exp\!\left(-\frac{d_p(t)}{\sigma_p}\right),\quad \sigma_p>0,$$^^^^7search_query7^^^^, $$r_{ep}(t) = \exp\!\left(-\frac{d_p(t)}{\sigma_p}\right),\quad \sigma_p>0,$$^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^ diverse responses are sampled per source model using top-$$r_{ep}(t) = \exp\!\left(-\frac{d_p(t)}{\sigma_p}\right),\quad \sigma_p>0,$$^^^^7start7^^^^ and temperature $$r_{ep}(t) = \exp\!\left(-\frac{d_p(t)}{\sigma_p}\right),\quad \sigma_p>0,$$^^^^7max_results7^^^^. An external reward model, ArmoRM-LLaMA^^^^7max_results7^^^^-8B-v^^^^7search_query7 OR id:(Jiang et al., 2024)7^^^^, scores the responses, and the single highest-scoring response across all sources is selected as $$r_{ep}(t) = \exp\!\left(-\frac{d_p(t)}{\sigma_p}\right),\quad \sigma_p>0,$$^^^^7search_query7^^^^. From the target model, $$r_{ep}(t) = \exp\!\left(-\frac{d_p(t)}{\sigma_p}\right),\quad \sigma_p>0,$$^^^^7ti:\7^^^^ responses are sampled and scored, the highest-scoring target response is selected as $$r_{ep}(t) = \exp\!\left(-\frac{d_p(t)}{\sigma_p}\right),\quad \sigma_p>0,$$^^^^7 OR ti:\7^^^^, and the lowest-scoring target response as $$r_{ep}(t) = \exp\!\left(-\frac{d_p(t)}{\sigma_p}\right),\quad \sigma_p>0,$$7. The Bradley–Terry model is written as

$$r_{ep}(t) = \exp\!\left(-\frac{d_p(t)}{\sigma_p}\right),\quad \sigma_p>0,$$8

with weighted chosen reward

$$r_{ep}(t) = \exp\!\left(-\frac{d_p(t)}{\sigma_p}\right),\quad \sigma_p>0,$$9

where $$r_{eo}(t) = \exp\!\left(-\frac{d_\theta(t)}{\sigma_o}\right),\quad \sigma_o>0.$$^^^^7search_query7^^^^ balances source-selected preferred responses and target-selected preferred responses.

The internal reward uses the DPO-style reparameterization

$$r_{eo}(t) = \exp\!\left(-\frac{d_\theta(t)}{\sigma_o}\right),\quad \sigma_o>0.$$^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^

The WRPO loss is built from quadruples $$r_{eo}(t) = \exp\!\left(-\frac{d_\theta(t)}{\sigma_o}\right),\quad \sigma_o>0.$$^^^^7start7^^^^, and its reformulation separates a hybrid-policy internal reward margin from an on-policy internal reward margin. The corresponding gradient analysis shows that $$r_{eo}(t) = \exp\!\left(-\frac{d_\theta(t)}{\sigma_o}\right),\quad \sigma_o>0.$$^^^^7max_results7^^^^ controls how much gradient signal is absorbed from source-preferred versus target-preferred samples, while the sigmoid scales update magnitude according to reward estimation error.

The progressive adaptation strategy addresses the fact that directly learning from $$r_{eo}(t) = \exp\!\left(-\frac{d_\theta(t)}{\sigma_o}\right),\quad \sigma_o>0.$$^^^^7search_query7^^^^ is off-policy relative to the target model. In implementation, $$r_{eo}(t) = \exp\!\left(-\frac{d_\theta(t)}{\sigma_o}\right),\quad \sigma_o>0.$$^^^^7ti:\7^^^^ is initialized at $$r_{eo}(t) = \exp\!\left(-\frac{d_\theta(t)}{\sigma_o}\right),\quad \sigma_o>0.$$^^^^7 OR ti:\7^^^^ and increases linearly throughout training until it reaches a predetermined target value; the paper reports greedy search over $$r_{eo}(t) = \exp\!\left(-\frac{d_\theta(t)}{\sigma_o}\right),\quad \sigma_o>0.$$7. Dynamic $$r_{eo}(t) = \exp\!\left(-\frac{d_\theta(t)}{\sigma_o}\right),\quad \sigma_o>0.$$8 generally outperforms static $$r_{eo}(t) = \exp\!\left(-\frac{d_\theta(t)}{\sigma_o}\right),\quad \sigma_o>0.$$9, except for PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query7search_query7^^^^ in a particular setting. The full training pipeline has two stages: Stage ^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^ SFT trains on PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^ using one-third of the data with learning rate PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query7start7^^^^ to produce Target-SFT; Stage ^^^^7start7^^^^ runs WRPO on the remaining data with learning rate PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query7max_results7^^^^, PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query7search_query7^^^^, and PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query7ti:\7^^^^ increased linearly from PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query7 OR ti:\7^^^^ to PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query77^^^^. Both stages use batch size PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query7^^^^8, maximum length PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query7^^^^9, and a single epoch on PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query7^^^^GB NVIDIA A8^^^^7search_query7search_query7^^^^ GPUs. Offline source sampling is one-time, the source models are not active during training, and WRPO’s training runtime overhead is approximately PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^ versus DPO, independent of the number of source LLMs.

The reported outcomes are strong for a single target model. With LLaMA^^^^7max_results7^^^^-8B-Instruct as target, WRPO achieves AlpacaEval-^^^^7start7^^^^ length-controlled win rate PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7start7^^^^ and raw win rate PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7max_results7^^^^, Arena-Hard win rate PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7search_query7^^^^, and MT-Bench overall PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7ti:\7^^^^. On AlpacaEval-^^^^7start7^^^^ length-controlled win rate, baselines reported are DPO PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7 OR ti:\7^^^^, IPO PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)77^^^^, SimPO PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^8, Target-FuseLLM PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^9, and Target-FuseChat PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7start7search_query7^^^^, while WRPO obtains PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7start7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^. The ablations show that removing PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7start7start7^^^^ by setting PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7start7max_results7^^^^ causes a sharp drop of PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7start7search_query7^^^^ length-controlled points on Target, while removing PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7start7ti:\7^^^^ by setting PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7start7 OR ti:\7^^^^ reduces WRPO to DPO and loses PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7start77^^^^–PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7start7^^^^8 length-controlled points versus WRPO. Scaling the number of sources improves performance from PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7start7^^^^9 with PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7max_results7search_query7^^^^ source to PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7max_results7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7^^^^ with PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7max_results7start7^^^^, PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7max_results7max_results7^^^^ with PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7max_results7search_query7^^^^, and PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7max_results7ti:\7^^^^ with PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7max_results7 OR ti:\7^^^^. An extended loss including source dispreferred PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7max_results77^^^^ reduces performance from PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7max_results7^^^^8 to PRESERVED_PLACEHOLDER_^^^^7id:(Yang et al., 2024) OR id:(Jiang et al., 2024)7max_results7^^^^9 and increases compute, so it is reported as not recommended (&&&^^^^7search_query7^^^^&&&).

Taken together, these two usages identify a common technical pattern. In robotics, RFM is an explicitly defined nonlinear reward-composition module aligned with multi-stage control structure. In LLM fusion, WRPO provides a practical instance of reward fusion through weighted internal rewards and scheduling, even though the paper does not name it as RFM. A plausible implication is that reward fusion is best understood as a design principle for coupling heterogeneous supervisory signals to task phase, policy distribution, or both, rather than as a single domain-specific architecture.