Reward Ranked Alignment with Expert Exploration

• The paper introduces a unique maximum-margin ordinal regression method to generate robust reward functions using ranked expert demonstrations.
• It leverages both exemplary and suboptimal behaviors to build reward signals that clearly differentiate desirable actions from undesirable ones.
• Applications in urban mobility demonstrate that RAE outperforms traditional grid-based approaches by deriving fine-grained and interpretable reward models.

Reward Ranked Alignment with Expert Exploration (RAE) is a framework for learning reward functions in reinforcement learning and inverse reinforcement learning settings that systematically incorporates ranked expert demonstrations. Unlike classical approaches that focus solely on optimal expert data or try to match expert feature expectations, RAE leverages the full spectrum of behavior—ranging from expert to non-expert—using a principled ordinal regression framework to induce a reward that better delineates both desirable and undesirable behavior. Agents trained under RAE learn not only by imitation of skilled behaviors but also through explicit avoidance of poor strategies as exemplified by low-ranked demonstrators, leading to more robust and nuanced reward models.

1. Core Principles of RAE

RAE is grounded in the observation that learning solely from optimal experts may obscure critical information about suboptimal regions of the policy or reward landscape. By formalizing agent learning within a Markov Decision Process (MDP) where expert demonstrations are available in multiple ranks, RAE replaces the traditional reward specification with a learned function $R(s) = w \cdot \phi(s)$ that is maximally discriminative across these ranks. Feature expectations $\mu_i$ for each demonstrator offer the fundamental data for rank-based learning.

The RAE framework sets up the primary ordinal relationship:

$$\operatorname{rank}(\mu_i) < \operatorname{rank}(\mu_j) \implies w \cdot \mu_i < w \cdot \mu_j$$

This alignment condition ensures that higher-ranked demonstrations dominate lower-ranked ones under the induced reward function, providing informed guidance for both imitation and exploration.

2. Mathematical Formulation and Ordinal Regression Approach

Central to RAE is the use of ordinal regression via a quadratic program (QP) that both enforces rank ordering and maximizes separation (margin) between adjacent ranks. The QP seeks parameters $w, \{a_r\}, \{b_r\}, \{\epsilon\}, \{\zeta\}$ minimizing:

$$\sum_{r=1}^{k-1}(a_r - b_r) + C \sum_{r,i}(\epsilon_{ri} + \zeta_{(r+1)i})$$

Subject to: | Constraint | Mathematical Formulation | Scope | |----------------------------------------|---------------------------------------------------|---------------------| | Rank separation | $a_r \leq b_r$, $b_r \leq a_{r+1}$ | $1 \leq r < k$ | | Feature expectation fit | $w \cdot \mu_i \leq a_r+\epsilon_{ri}$ | $i$ in rank $r$ | | Adjacent rank margin | $b_r - \zeta_{(r+1)i} \leq w \cdot \mu_i$ | $i$ in rank $r+1$ | | Weight norm constraint | $\|w\| \leq 1$ | All $w$ | | Slack non-negativity | $\epsilon_{ri}, \zeta_{(r+1)i} \geq 0$ | All indices |

By maximizing the total margin between ranks, RAE constrains the optimization solution toward reward functions that explain both the good and the “bad”—proving especially powerful in scenarios with a continuum of expert performance.

3. Comparison to Traditional IRL and Reward Learning Approaches

Classical IRL algorithms, such as Abbeel & Ng’s apprenticeship learning and maximum-entropy IRL, focus on matching expert feature expectations, producing potentially degenerate or ambiguous solutions due to limited exposure to suboptimal behaviors. These techniques often require repeated forward MDP solving over sampled policies, leading to variable results dependent on the initial trajectories or solution space topology.

RAE, contrastingly, directly operates on the ranked demonstrator feature expectations, yielding a unique solution for a fixed demonstration set and reducing the risk of reward function degeneracy. This produces a reward that is empirically more discriminative and interpretable, particularly when the demonstrator pool spans a broad spectrum of expertise.

4. Real-World Application: Passenger-Finding Strategies in Urban Mobility

A principal demonstration of RAE is in the derivation of reward functions for complex, real-world domains such as taxi passenger-finding strategies in Hangzhou, China. Here, thousands of GPS trajectories provided by taxi drivers are labeled according to their “unoccupied time ratio”—a proxy for expert performance. The state space is defined by unique road segment and orientation pairs, permitting fine-grained feature representations.

Applying RAE, the city is divided into manageable subregions, each solved independently. The reward function recovered not only promotes high-demand areas (e.g., airports, hospitals) but also actively penalizes problematic zones (mountain roads, poor pickup locations), significantly outperforming grid-based methods lacking such granularity.

5. Implications for Exploration, Alignment, and Policy Robustness

RAE explicitly incorporates exploration signals from lower-ranked behaviors, enabling agents to avoid misleading overgeneralization that arises from pure expert imitation. This systemic integration of both exemplary and anti-exemplary policy regions ensures the learned reward is richer, more robust, and better adapted to heterogeneous expertise distributions.

The concept of Reward Ranked Alignment with Expert Exploration, as instantiated in this framework, addresses critical limitations in IRL: it guides agents not simply to replicate the highest-performing actions but to learn a balanced, well-calibrated reward signal that encompasses the full complexity of behavioral data. This approach generalizes efficiently to other domains where diverse demonstrations are available, including large-scale mobility, multi-agent collaboration, and creative tasks.

6. Limitations and Computational Considerations

Solving the RAE QP is efficient, requiring only one round of optimization for a given demonstrator set. However, computational complexity can scale with the number of ranks and demonstration instances, particularly in high-dimensional feature spaces. The approach presumes that feature representations adequately distinguish state-action pairs at the desired resolution; loss of granularity or poor feature selection may impair reward recovery.

Additionally, practical deployment demands careful preprocessing (such as subregion decomposition) to mitigate scalability concerns in massive, urban-scale environments.

7. Summary and Future Research Directions

Reward Ranked Alignment with Expert Exploration offers a systematic, maximum-margin methodology for learning reward functions from ranked sets of demonstrations in reinforcement learning. By directly enforcing ordinal margins, RAE enables the learned reward to robustly capture both positive and negative behavioral information, yielding interpretable, effective signals for policy optimization.

Open research areas include integration of RAE-style ordinal regression with model-based RL, active learning with adaptive demonstrator ranking, and extension to domains with non-linear or hybrid reward functions. The approach underpins both improved exploration and precise policy alignment, making it a central tool for future scalable and robust learning from demonstration.