MAPL: Multi-Objective Preference Learning for Robot Locomotion

Abstract: Reward design remains a major bottleneck in reinforcement learning for robot locomotion, where successful policies often depend on carefully tuned, task-specific reward functions. Preference-based reinforcement learning offers an alternative, but existing LLM-based methods typically ask for a single overall judgment between behaviors, making it difficult to capture the multiple competing objectives that underlie high-quality locomotion. We present Multi-Objective AI-Informed Preference Learning (MAPL), a framework that learns locomotion rewards from high-level natural language objectives rather than manually engineered reward equations. MAPL prompts a LLM to compare trajectories independently along semantically meaningful criteria, using generic language descriptions that are terrain-invariant and require little domain expertise. These objective-wise preferences are used to train a multi-head preference scoring model, whose outputs are aggregated to form a scalar reward for policy optimization. Across four quadruped locomotion environments, MAPL trains policies using only LLM-generated preferences and achieves performance comparable to or better than expert-designed rewards, while eliminating task-specific reward engineering.

• The paper introduces a MAPL framework that decomposes locomotion into velocity tracking, stability, and smoothness, eliminating manual reward engineering.
• It employs LLM-generated preference signals and a transformer-based reward function to iteratively optimize policies across diverse terrains.
• Experimental results show that MAPL outperforms expert and single-objective methods with improved convergence and minimal velocity tracking error.

MAPL: Multi-Objective Preference Learning for Quadruped Robot Locomotion

Motivation and Problem Statement

Reward engineering remains a fundamental obstacle in RL for robot locomotion, demanding extensive manual intervention and domain expertise to balance trade-offs between conflicting objectives such as velocity tracking, stability, and smoothness. Traditional PbRL approaches and recent RLAIF variants typically collapse these objectives into a single preference, resulting in noisy supervision and suboptimal policy performance. The MAPL framework ("MAPL: Multi-Objective Preference Learning for Robot Locomotion" (2606.25398)) systematically addresses this gap by incorporating multi-objective AI-informed preference learning based solely on high-level natural language prompts, thus eliminating task-specific reward engineering.

MAPL Framework and Training Procedure

MAPL iteratively alternates between scoring model training and policy optimization. In the scoring phase, an LLM (GPT-5-mini) is prompted with trajectory pairs and independently assesses velocity tracking, stability, and smoothness according to semantically grounded, terrain-invariant prompts. The multi-objective preference signals are used to train a transformer-based reward function with individual heads per objective. In policy training, rewards are computed as a potential difference in the aggregate preference score, and PPO-based RL is used for optimization.

Figure 1: MAPL training pipeline, separating scoring model updates and policy rollouts with multi-objective preference integration.

Key methodological innovations include:

• Objective decomposition: Preferences are elicited and modeled separately for each locomotion criterion, minimizing gradient interference and reducing LLM hallucination-induced noise.
• Policy reward aggregation: Per-objective scoring heads, each trained from LLM preference labels, are linearly combined with tunable weights to shape the scalar reward input for RL.
• Potential-difference reward: Aggregate scores operate via a potential difference formulation, providing a smoother and more sample-efficient reward landscape.

Experimental Setup: Environments and Baselines

MAPL is evaluated on the Unitree Go2 quadruped across four terrains using the Isaac Gym simulator: flat plane, uneven hills, stairs, and randomized obstacle fields.

Figure 2: Evaluation environments for quadruped locomotion benchmarking (Plane, Hills, Stairs, Obstacle fields).

Baselines include:

• Expert reward: Manually engineered reward combining velocity, angular tracking, stability, action smoothness, and feasibility terms.
• LLM reward: Reward code directly generated from LLM prompts.
• LAPP: Transformer-based single-preference reward learning.

Performance is measured via velocity tracking rewards, stability, and smoothness criteria under curriculum learning with difficulty-weighted evaluation.

Policy Performance: Numerical Comparisons

MAPL consistently surpasses all baselines—across environments, it achieves final velocity tracking rewards comparable to or exceeding hand-crafted reward policies. LLM reward plateaus at lower returns; single-preference LAPP fails to drive meaningful policy learning without expert reward bootstrapping.

Figure 3: Velocity tracking training curves, showing MAPL convergence and superior reward returns vs. LLM Reward and LAPP.

Velocity tracking error analysis validates MAPL’s generalization: it maintains low residuals as commanded speeds increase across all terrain types, highlighting robust and stable tracking.

Figure 4: MAPL yields consistently minimal velocity tracking error across command magnitudes and terrain complexities.

MAPL also demonstrates improved robot posture and stability over baselines, confirmed via body height, roll, and pitch distributions.

Figure 5: Distribution of body height, roll, and pitch under different reward models—MAPL achieves concentrated stability near the origin.

Ablation and Sensitivity Analyses

Ablation studies elucidate the necessity of multi-objective preference modeling. Policies trained on isolated objectives (velocity-only, stability-only, or smoothness-only) exhibit degraded or unstable behavior. Pairwise combinations improve returns, but the full tri-objective MAPL reward achieves optimal convergence and stability.

Figure 6: Ablation plots for preference components, buffer size, and LLM query count, demonstrating critical contributions and robustness.

Potential-difference reward shaping is indispensable—removing it leads to substantial drops in training speed and convergence; integration is only reliably effective with MAPL due to structured reward aggregation.

Figure 7: Velocity learning curves with/without potential difference; only MAPL robustly benefits from potential shaping.

MAPL is resilient to buffer and LLM query count reductions—single-query per ranking is almost as effective as majority voting, underscoring robustness to LLM hallucinations.

Preference aggregation weight ($\lambda$) sensitivity analysis reveals that higher weights for velocity and stability yield high returns, whereas smoothness plays a minor but non-negligible role, supporting the efficacy of simple linear aggregation.

Figure 8: Evaluation rewards vs. $\lambda$ parameter settings for velocity, stability, and smoothness objectives.

Preference Ranking Accuracy: Multi-Objective Superiority

MAPL’s multi-objective prompts and majority voting outperform single-objective and combined ranking schemes, delivering highest ranking accuracy against ground-truth reward benchmarks, both overall and per-objective.

Implications and Future Perspectives

MAPL demonstrates that RL policies for quadruped locomotion can be trained exclusively from LLM-generated preferences, matching or outperforming manually engineered rewards. The approach is:

• Scalable and task-agnostic: A generic prompt suffices for diverse terrains and environments, eliminating the need for manual reward design and extensive domain expertise.
• Robust to LLM limitations: Objective decomposition and majority voting dampen hallucinations, ranking noise, and balancing trade-offs.
• Efficient and interpretable: Linear aggregation provides straightforward control and tuning, though at the cost of potentially limited expressiveness.

These findings have substantial implications—multi-objective PbRL with AI feedback represents a practical paradigm for robust robot policy learning. Extensions may include nonlinear aggregation schemes, multimodal foundation models (e.g., vision- or tactile-aware), and broader application to loco-manipulation, yielding richer feedback and more complex behavior.

Conclusion

MAPL shows that decomposing natural-language locomotion criteria and leveraging LLM preference rankings are sufficient for training high-performing quadruped policies. Objective-wise reward modeling and potential difference shaping deliver superior learning efficiency and policy robustness. The framework eliminates task-specific reward engineering and enables scalable policy learning with minimal manual intervention. While linear aggregation may constrain expressiveness, future directions in integrating multimodal feedback and nonlinear reward fusion hold promise for expanding MAPL’s applicability to increasingly complex robotic domains.