Outcome-Driven Actor-Critic (ODAC)

• Outcome-Driven Actor-Critic (ODAC) is a reinforcement learning framework that unifies actor and critic updates via a joint decision-aware objective to maximize expected returns.
• It introduces tailored surrogate losses based on convex mirror maps and Bregman divergences, ensuring coherent updates and improved policy performance.
• Empirical benchmarks and theoretical guarantees show that ODAC achieves faster convergence and more robust performance compared to standard actor-critic methods.

Outcome-Driven Actor-Critic (ODAC) is a reinforcement learning (RL) framework that introduces a principled joint objective for actor-critic (AC) algorithms, ensuring that policy and critic updates are coherently aligned with the goal of maximizing expected return. Unlike standard AC approaches—where the critic is typically trained via a decorrelated temporal-difference (TD) loss—ODAC couples the actor and critic through a decision-aware objective derived from a lower bound on the true return. The formulation is general and supports arbitrary policy and value function parameterizations, convex mirror maps, and provides theoretical guarantees of monotonic policy improvement under mild conditions. The approach subsumes commonly used surrogate actor losses and yields critic learning rules superior to the standard mean squared error (MSE), particularly under restricted critic capacity (Vaswani et al., 2023).

1. Joint Decision-Aware Objective Formulation

ODAC's central contribution is a joint functional lower bound for expected return $J(\pi)$, evaluated at policy iteration $t$:

$$J(\pi)\;\ge\;J(\pi_t)\;+\;\big\langle \hat g_t,\;\pi-\pi_t\big\rangle \;-\;\Bigl(\frac1\eta+\frac1c\Bigr)\,D_\Phi(\pi,\pi_t) \;-\;\frac1c\,D_{\Phi^*}\Big(\nabla\Phi(\pi_t)-c[\nabla J(\pi_t)-\hat g_t],\nabla\Phi(\pi_t)\Big)$$

where

• $\pi_t$ is the current policy,
• $\hat g_t$ is a parameterized estimator of the policy gradient,
• $\Phi$ is a strictly convex mirror map (e.g., negative entropy or log-sum-exp),
• $D_\Phi$ is the corresponding Bregman divergence,
• $c>0$ controls the actor-critic trade-off,
• $\eta>0$ is a functional step-size.

Actor and critic updates optimize two surrogates derived from this lower bound. The actor surrogate is given by

$$\ell_t(\theta) = \bigl\langle \hat g_t,\;\pi(\theta)-\pi_t\bigr\rangle - \Bigl(\frac1\eta+\frac1c\Bigr)D_\Phi(\pi(\theta),\pi_t)$$

Parameter $t$0 indexes the policy, and maximizing $t$1 subject to $t$2 trust regions provably tightens the lower bound on $t$3.

The critic is fitted via the loss

$t$4

using a parametric critic $t$5. This loss directly enforces fidelity to the actor’s improvement metric.

ODAC supports closed-form instantiations for common mirror maps:

• Negative entropy: yields weighted KL-regularized surrogates for the actor and critic.
• Log-sum-exp: yields log-weighted advantage-based surrogate functions.

2. Algorithmic Structure and Implementation

The ODAC algorithm alternates actor and critic updates, maintaining joint optimization of their respective decision-aware surrogates. Below is the pseudocode as presented:

$\Phi$0

Key hyperparameters include:

• $t$6: actor–critic coupling, optimal in range $t$7 via grid search.
• $t$8: functional step size, independent of parameterization, selected from $t$9.
• $$J(\pi)\;\ge\;J(\pi_t)\;+\;\big\langle \hat g_t,\;\pi-\pi_t\big\rangle \;-\;\Bigl(\frac1\eta+\frac1c\Bigr)\,D_\Phi(\pi,\pi_t) \;-\;\frac1c\,D_{\Phi^*}\Big(\nabla\Phi(\pi_t)-c[\nabla J(\pi_t)-\hat g_t],\nabla\Phi(\pi_t)\Big)$$0, $$J(\pi)\;\ge\;J(\pi_t)\;+\;\big\langle \hat g_t,\;\pi-\pi_t\big\rangle \;-\;\Bigl(\frac1\eta+\frac1c\Bigr)\,D_\Phi(\pi,\pi_t) \;-\;\frac1c\,D_{\Phi^*}\Big(\nabla\Phi(\pi_t)-c[\nabla J(\pi_t)-\hat g_t],\nabla\Phi(\pi_t)\Big)$$1: dynamic step-sizes, set via Armijo line-search.
• $$J(\pi)\;\ge\;J(\pi_t)\;+\;\big\langle \hat g_t,\;\pi-\pi_t\big\rangle \;-\;\Bigl(\frac1\eta+\frac1c\Bigr)\,D_\Phi(\pi,\pi_t) \;-\;\frac1c\,D_{\Phi^*}\Big(\nabla\Phi(\pi_t)-c[\nabla J(\pi_t)-\hat g_t],\nabla\Phi(\pi_t)\Big)$$2, $$J(\pi)\;\ge\;J(\pi_t)\;+\;\big\langle \hat g_t,\;\pi-\pi_t\big\rangle \;-\;\Bigl(\frac1\eta+\frac1c\Bigr)\,D_\Phi(\pi,\pi_t) \;-\;\frac1c\,D_{\Phi^*}\Big(\nabla\Phi(\pi_t)-c[\nabla J(\pi_t)-\hat g_t],\nabla\Phi(\pi_t)\Big)$$3: number of actor and critic gradient steps, balancing computational budget against bias/variance.

3. Theoretical Guarantees

ODAC is equipped with explicit monotonic improvement criteria (Prop 4). Defining $$J(\pi)\;\ge\;J(\pi_t)\;+\;\big\langle \hat g_t,\;\pi-\pi_t\big\rangle \;-\;\Bigl(\frac1\eta+\frac1c\Bigr)\,D_\Phi(\pi,\pi_t) \;-\;\frac1c\,D_{\Phi^*}\Big(\nabla\Phi(\pi_t)-c[\nabla J(\pi_t)-\hat g_t],\nabla\Phi(\pi_t)\Big)$$4, $$J(\pi)\;\ge\;J(\pi_t)\;+\;\big\langle \hat g_t,\;\pi-\pi_t\big\rangle \;-\;\Bigl(\frac1\eta+\frac1c\Bigr)\,D_\Phi(\pi,\pi_t) \;-\;\frac1c\,D_{\Phi^*}\Big(\nabla\Phi(\pi_t)-c[\nabla J(\pi_t)-\hat g_t],\nabla\Phi(\pi_t)\Big)$$5, guaranteed improvement is obtained if:

$$J(\pi)\;\ge\;J(\pi_t)\;+\;\big\langle \hat g_t,\;\pi-\pi_t\big\rangle \;-\;\Bigl(\frac1\eta+\frac1c\Bigr)\,D_\Phi(\pi,\pi_t) \;-\;\frac1c\,D_{\Phi^*}\Big(\nabla\Phi(\pi_t)-c[\nabla J(\pi_t)-\hat g_t],\nabla\Phi(\pi_t)\Big)$$6

In tabular plus Euclidean-KL settings, this reduces to the “relative error < 1” condition of the critic approximation.

ODAC further ensures convergence to a stationary point neighborhood (Prop 5):

$$J(\pi)\;\ge\;J(\pi_t)\;+\;\big\langle \hat g_t,\;\pi-\pi_t\big\rangle \;-\;\Bigl(\frac1\eta+\frac1c\Bigr)\,D_\Phi(\pi,\pi_t) \;-\;\frac1c\,D_{\Phi^*}\Big(\nabla\Phi(\pi_t)-c[\nabla J(\pi_t)-\hat g_t],\nabla\Phi(\pi_t)\Big)$$7

with $$J(\pi)\;\ge\;J(\pi_t)\;+\;\big\langle \hat g_t,\;\pi-\pi_t\big\rangle \;-\;\Bigl(\frac1\eta+\frac1c\Bigr)\,D_\Phi(\pi,\pi_t) \;-\;\frac1c\,D_{\Phi^*}\Big(\nabla\Phi(\pi_t)-c[\nabla J(\pi_t)-\hat g_t],\nabla\Phi(\pi_t)\Big)$$8 denoting critic and projection errors, and $$J(\pi)\;\ge\;J(\pi_t)\;+\;\big\langle \hat g_t,\;\pi-\pi_t\big\rangle \;-\;\Bigl(\frac1\eta+\frac1c\Bigr)\,D_\Phi(\pi,\pi_t) \;-\;\frac1c\,D_{\Phi^*}\Big(\nabla\Phi(\pi_t)-c[\nabla J(\pi_t)-\hat g_t],\nabla\Phi(\pi_t)\Big)$$9 convexity and $\pi_t$0-smoothness assumptions.

4. Comparison with Trust-Region and Proximal Policy Optimization Methods

ODAC’s actor surrogate structurally interpolates standard actor-critic and trust-region style updates, such as TRPO and PPO.

• TRPO maximizes the relative entropy regularized objective:

$\pi_t$1

subject to KL trust-region constraints.

• PPO uses a clipped surrogate:

$\pi_t$2

• ODAC actor surrogate takes the form:

$\pi_t$3

For weighted negative entropy, this reduces to $\pi_t$4, paralleling TRPO but with a data-adaptive $\pi_t$5 coefficient coupling to critic error.

A plausible implication is that ODAC retains the stability of trust-region methods but offers adaptive error-sensitivity through the critic coupling.

5. Empirical Evaluation and Benchmarks

ODAC’s efficacy was benchmarked on:

• Two-armed bandits (deterministic Bernoulli payoff): demonstrated that standard MSE critics may drive policies to sub-optimal arms, while decision-aware critics reliably select optimal actions.
• Grid-world RL tasks, including:
  • Cliff World (Sutton & Barto)
  • Frozen Lake (OpenAI Gym)

Baselines:

• Standard MSE critic (TD loss)
• Adv-MSE critic (squared advantage loss)

Metrics:

• Average return $\pi_t$6 over AC iterations.
• 95% confidence intervals over 5 random seeds.
• Critic expressivity varied by feature dimension ($\pi_t$7).

Key findings:

Critic Expressivity	MSE-Critic	Adv-MSE	Decision-Aware (DA) Critic
High (d ≥ 80)	Optimal return	Optimal return	Optimal return
Moderate (d=40,60)	Sub-optimal, non-monotonic	Nearly always converges	Typically converges faster, higher return

Empirical performance held under both tabular and linear policy parameterizations, for both exact and MC-estimated critic targets. This suggests ODAC’s advantages are robust to policy/critic model class and stochasticity in return estimation.

6. Practical Considerations for Implementation

Key practitioner tips:

• Tuning $\pi_t$8: Moderate values in $\pi_t$9 favor balance; lower $\hat g_t$0 suppresses unreliable critic contributions.
• Step-sizes ($\hat g_t$1, $\hat g_t$2): Use Armijo line-search for both actor and critic losses for adaptive curvature scaling.
• Functional step-size ($\hat g_t$3): Decoupled from model parameters; select from $\hat g_t$4.
• Inner iterations ($\hat g_t$5, $\hat g_t$6): Increase $\hat g_t$7 when critic evaluation is cheap to reduce bias; reduce $\hat g_t$8 when policy evaluation is costly.
• Critic error: Large critic error ($\hat g_t$9) invalidates improvement conditions; remedy by collecting additional data or enriching critic representation.

A plausible implication is that ODAC's design systematically eliminates the actor–critic mismatch present in standard AC, guarantees monotonic improvement under mild oracle error conditions, and demonstrates superior practical performance, especially for resource-constrained critics (Vaswani et al., 2023).