Anchor-Changing Regularized NPG (ARNPG)

• The paper introduces ARNPG, which unifies mirror descent, optimism, and extra-gradient methods into a natural policy gradient framework for multi-objective reinforcement learning.
• It optimizes policies in finite MDPs with several reward functions using an anchor-changing strategy and KL regularization to enhance convergence and performance.
• Empirical evaluations demonstrate that ARNPG achieves O(1/T) convergence and outperforms baseline methods in both tabular and deep RL settings.

Anchor-Changing Regularized Natural Policy Gradient (ARNPG) is a multi-objective reinforcement learning (MORL) framework for policy optimization in Markov decision processes (MDPs) with several scalar reward functions. ARNPG unifies and systematically embeds advanced first-order optimization techniques—such as mirror descent, optimism, and extra-gradient methods—into natural policy gradient (NPG) architectures, introducing an anchor-changing strategy and Kullback–Leibler (KL) regularization to achieve theoretically optimal convergence and empirical superiority in complex policy search spaces (Zhou et al., 2022).

1. Multi-Objective Markov Decision Processes

The ARNPG framework operates on a discounted finite MDP defined by:

• finite state space $S$,
• finite action space $A$,
• transition kernel $P: S \times A \rightarrow \Delta(S)$,
• initial distribution $\rho \in \Delta(S)$,
• discount factor $\gamma \in (0,1)$,
• $m$ reward functions $r_i : S \times A \rightarrow [0,1]$ for $i=1,\dots,m$.

For a stationary policy $\pi: S \rightarrow \Delta(A)$, the state-action visitation distribution is: $$d_\rho^\pi(s,a) = (1-\gamma) \mathbb{E}_{s_0 \sim \rho} \left[ \sum_{t \geq 0} \gamma^t \Pr(s_t = s, a_t = a \mid s_0, \pi) \right].$$ Value for reward $A$0 is: $A$1 The achievable multi-reward value set is $A$2 for policy class $A$3, typically parametrized as softmax policies $A$4.

2. Optimization Objectives in ARNPG

ARNPG addresses three principal MORL optimization paradigms:

• Smooth Concave Scalarization (Proportional Fairness):

$A$5

Here $A$6 is $A$7-smooth, $A$8.

• Hard Constraints (Constrained MDP):

$A$9

Using the Lagrangian formulation,

$P: S \times A \rightarrow \Delta(S)$0

• Max–Min Trade-Off:

$P: S \times A \rightarrow \Delta(S)$1

Formulated as a min-max saddle-point problem in the dual space.

3. The ARNPG Meta-Algorithm

The core ARNPG method iterates through macro-steps indexed by anchors $P: S \times A \rightarrow \Delta(S)$2 and ascent directions via advanced regularized subproblems:

1. Macro-Iteration Setup: Anchor policy $P: S \times A \rightarrow \Delta(S)$3, ascent reward $P: S \times A \rightarrow \Delta(S)$4, KL regularization $P: S \times A \rightarrow \Delta(S)$5.
2. Subproblem (“INNER”):

$P: S \times A \rightarrow \Delta(S)$6

where

$P: S \times A \rightarrow \Delta(S)$7

and

$P: S \times A \rightarrow \Delta(S)$8

is the state visitation-weighted KL divergence to the anchor.

1. Natural Policy Gradient Update: Inner loops apply $P: S \times A \rightarrow \Delta(S)$9 steps of natural gradient descent on (INNER):

$\rho \in \Delta(S)$0

with Fisher information matrix $\rho \in \Delta(S)$1.

1. Anchor Update: $\rho \in \Delta(S)$2; switch anchor to next iterate.
2. Dual Updates (for CMDP and max–min): Lagrange multipliers and dual weights updated via optimistic mirror-descent or extra-primal-dual (EPD) mechanisms.

Reward ascent directions $\rho \in \Delta(S)$3 are chosen according to specific paradigm:

• Scalarization: $\rho \in \Delta(S)$4.
• CMDP: $\rho \in \Delta(S)$5.
• Max–Min: $\rho \in \Delta(S)$6, with $\rho \in \Delta(S)$7.

4. Theoretical Guarantees and Convergence

ARNPG-derived algorithms exhibit $\rho \in \Delta(S)$8 global convergence for all three objective classes (macro-steps $\rho \in \Delta(S)$9). Under softmax parametrization, step-size $\gamma \in (0,1)$0, and sufficient inner iterations $\gamma \in (0,1)$1, the anchor update satisfies a fundamental mirror-descent inequality relating anchor KL divergence to value improvement.

Convergence Theorem (Smooth Scalarization Example):

For concave, $\gamma \in (0,1)$2-smooth, $\gamma \in (0,1)$3-Lipschitz scalarizations, choosing regularization $\gamma \in (0,1)$4, inner iterates $\gamma \in (0,1)$5, after $\gamma \in (0,1)$6 macro-steps: $\gamma \in (0,1)$7 Analogous proofs (with dual updates for CMDP and max–min) yield $\gamma \in (0,1)$8 rates via telescoping fundamental inequalities and mirror descent arguments.

5. Connections to First-Order Optimization

ARNPG establishes a formal connection between policy gradient optimization and first-order algorithms:

• Mirror Descent: KL regularization to the anchor policy implements a Bregman divergence ($\gamma \in (0,1)$9), rendering the inner loop a mirror ascent step in the policy space with mirror map $m$0.
• Accelerated and Optimistic Methods: For CMDP, the Extra-Primal-Dual (EPD) update is used to achieve $m$1 rates (consistent with Yu–Neely 2017). For max–min, an Optimistic Mirror-Descent Ascent (OMDA) coupling is applied, enabling simultaneous primal-dual optimization.

By virtue of the anchor/KL design, ARNPG can systematically “plug in” advanced first-order strategies (accelerated mirror descent, optimistic gradient descent, etc.) for use in policy-gradient-based MORL.

6. Empirical Evaluation Across Tabular and Deep RL

Empirical assessments demonstrate ARNPG’s theoretical properties and practical efficacy in both tabular and deep RL domains:

• Tabular CMDP (Exact Gradients): On random MDPs ($m$2, $m$3, $m$4, one constraint), ARNPG-EPD consistently outperforms NPG-Primal-Dual [Ding et al. 2020] and CRPO [Xu et al. 2021] in average reward gap ($m$5) and constraint violation ($m$6), achieving $m$7 convergence versus $m$8 for baselines (log–log slope $m$9).
• Tabular CMDP (Sample-Based): ARNPG-EPD retains fast convergence and constraint satisfaction with sample-based gradient estimation (generative trajectories).
• Deep-RL CMDP (Acrobot-v1, Hopper-v3): Applied with actor-critic neural softmax policies and constraints on link angles/velocities, ARNPG-EPD (1–5 inner loops) achieves higher cumulative reward while satisfying safety constraints than TRPO-based FOCOPS, CRPO, NPG-PD.
• Smooth & Max–Min in Tabular/Deep RL: ARNPG-IMD (scalarization) and ARNPG-OMDA (max–min) demonstrate rapid $r_i : S \times A \rightarrow [0,1]$0 convergence and outperform subgradient-based multi-objective NPG in both exact and sample-driven settings.

Critical implementation practices include:

• Selecting $r_i : S \times A \rightarrow [0,1]$1 to absorb smoothness constants,
• Updating anchor every $r_i : S \times A \rightarrow [0,1]$2 inner steps,
• Tuning EPD step-size $r_i : S \times A \rightarrow [0,1]$3 for CMDP objective-feasibility balancing,
• Employing optimistic mirror descent for dual variables in max–min.

7. Synthesis and Operational Significance

ARNPG delineates a generalizable “anchor-change + KL + NPG” structure, integrating sophisticated first-order update ideas into multi-objective, constrained, and max–min RL. It offers unified global theoretical guarantees ($r_i : S \times A \rightarrow [0,1]$4 convergence rates with exact gradients) and demonstrates robust empirical superiority across tabular and deep RL benchmarks (Zhou et al., 2022). Code and models are accessible at https://github.com/tliu1997/ARNPG-MORL.

A plausible implication is that ARNPG’s anchor-regularized paradigm may serve as an extensible template for future multi-objective policy optimization algorithms needing principled first-order acceleration in RL.