Policy-Extended Value Function Approximators

• PeVFA is a reinforcement learning approach that conditions value estimates on both state and explicit policy embeddings.
• It utilizes methods like probing-state fingerprinting, SPR, and OPR to generate robust policy representations for generalizing value functions.
• Applications include zero-shot policy improvement, sample-efficient evolutionary RL, and enhanced on-policy iterations with demonstrable empirical gains.

Policy-Extended Value Function Approximators (PeVFA) are a class of function approximators in reinforcement learning (RL) that extend conventional value function approximators by explicitly conditioning on both the state (and/or action) and an explicit representation of the policy. This design shift enables simultaneous estimation of value functions for multiple policies, facilitates value generalization across policies, and opens up new algorithmic possibilities such as zero-shot policy improvement, enhanced sample efficiency in evolutionary RL, and more robust policy iteration schemes. By leveraging learned or structured policy embeddings, PeVFAs can interpolate values in policy space, support gradient-based policy optimization without new environment samples, and integrate seamlessly with both policy-gradient and evolutionary methods (Harb et al., 2020, Tang et al., 2020, Hao et al., 2022).

1. Formal Definition and Core Objective

The central objective of a PeVFA is to approximate the value function not just as a mapping from states (and, in Q-function form, actions) to returns, but as a mapping from the joint space of policies and states (and possibly actions) to values. Let $\pi$ denote a policy, $s$ a state, $a$ an action, and $g(\pi)$ or $\phi(\pi)$ a learned fixed-dimensional embedding (or "fingerprint") of policy $\pi$:

• State-value PeVFA: $$\hat v(s; \pi, \theta) = \mathbb{V}_\theta(s, g(\pi)) \approx v^\pi(s)$$
• Action-value PeVFA: $\hat{Q}_\theta(s, a, W) \approx Q^{\pi_W}(s, a)$

In the form leveraged by policy evaluation networks (PVNs), for a policy parameterized by $\psi$, the PeVFA is realized as

$\hat V_\theta(\pi, s) = f_\theta(\phi(\pi), s)$

and, for optimizing start-state return,

$s$0

where $s$1 is a feed-forward neural network with parameters $s$2, and $s$3 is the policy fingerprint (Harb et al., 2020).

2. Policy Representation and Embedding Mechanisms

Effective generalization in PeVFA depends crucially on the construction of the policy representation $s$4 or $s$5. Several methods have been proposed:

• Probing-state fingerprinting: PVNs (Harb et al., 2020) introduce a trainable set of probing states $s$6; the policy’s output on these states is concatenated into a fingerprint vector (dimension $s$7). Both the probing states and network weights are learned via backpropagation.
• Surface Policy Representation (SPR): State–action pairs sampled from policy rollouts are encoded by an MLP followed by mean-pooling to generate the embedding $s$8 (Tang et al., 2020).
• Origin Policy Representation (OPR): Policy network parameters (weights/biases) are encoded per-layer using a small MLP, then pooled and concatenated (Tang et al., 2020).
• Linear policy representation: For linear policies, the policy weight matrix $s$9 itself serves as the embedding, with a learned “policy embedder” mapping $a$0 to a lower-dimensional vector (Hao et al., 2022).

Auxiliary objectives for embedding learning include: end-to-end value fitting, policy-recovery decoders (matching $a$1 from $a$2), and contrastive losses to encourage policy distinction in embedding space (Tang et al., 2020).

3. Training Objectives and Algorithms

Regression and Distributional Losses

• State-value/return prediction: Minimize mean squared error (MSE) between PeVFA predictions and Monte Carlo returns, either on current or historical policies:

$a$3

(Tang et al., 2020, Harb et al., 2020).

• Distributional prediction: Predict the empirical return distribution (policy return histograms) using a categorical output and KL divergence as loss:

$a$4

(Harb et al., 2020).

• Action-value PeVFA: Employs Bellman regression losses, e.g., 1-step TD (target network) updates:

$a$5

(Hao et al., 2022).

Generalized Policy Iteration with PeVFA

GPI with PeVFA cycles through policy evaluation (training PeVFA on all encountered policies with their embeddings and trajectories) and policy improvement (using PeVFA to compute advantages, update via PPO or another objective). The policy encoder and value network are typically updated jointly or on staggered schedules (Tang et al., 2020).

Zero-shot Policy Ascent

By exploiting end-to-end differentiability from the PeVFA’s value output to the policy parameters $a$6, direct gradient ascent in policy space is feasible:

$a$7

This inner-loop optimization requires no environment samples after initial PeVFA pretraining (“zero-shot” improvement) (Harb et al., 2020).

4. Theoretical Foundations and Generalization

PeVFA enables value generalization across policies due to the explicit policy-conditioning. Theoretical results formalize this property under conditions such as L-Lipschitz continuity (in a metric $a$8):

• Two-policy generalization lemma: If PeVFA contracts approximation error on policy $a$9, then, under L-continuity, error on $g(\pi)$0 is bounded by a contraction term and a distance-dependent margin.
• Generalization across policy improvement sequences: Under GPI, the approximation error for the next improved policy accumulates as an exponentially-decaying sum of base error and local margins proportional to the embedding distance between consecutive policies. Locality in embedding space is critical for consistent improvement (Tang et al., 2020).
• Zero-shot improvement guarantee: The PeVFA’s value estimate for a new policy can be provably closer to the true value than using the previous policy, expediting GPI convergence.

A plausible implication is that PeVFA’s effectiveness is tightly linked to the smoothness and informativeness of the policy embedding. Non-stationarity or poor locality in embedding space can degrade generalization.

5. Network Architectures and Practical Implementation

PeVFA architectures typically concatenate learned policy embeddings with states (and optionally actions), feeding the result into a multi-layer perceptron:

• In PVN (Harb et al., 2020), the model uses a 2–3 layer MLP with hidden sizes 50–80 and ReLU activations; fingerprint vectors have dimension $g(\pi)$1.
• In ERL-Re² (Hao et al., 2022), the policy matrix $g(\pi)$2 is embedded via a per-row MLP (301→64→64), averaged to give a 64-dimensional policy vector, concatenated with $g(\pi)$3 and $g(\pi)$4, and passed through two fully-connected layers (400→300→1, leaky ReLU).
• In GPI–PeVFA (Tang et al., 2020), separate encoding streams for state and policy, each with 2 layers of 64 units, are merged by MLPs.

Policy fingerprints based on probing states, policy parameters, or rollouts are treated as learnable parameters and updated by backpropagation jointly with PeVFA network weights.

6. Empirical Findings and Applications

PeVFA demonstrates several benefits across RL settings:

• Zero-shot policy optimization: PVN-trained PeVFAs support direct gradient improvement, achieving convergence to optimal policies from suboptimal starting points in both toy MDPs and environments such as CartPole and MuJoCo Swimmer. Empirical results show that gradient fields computed via PeVFA closely match true policy gradients, and final performance can outperform the policy dataset used for training (Harb et al., 2020).
• Sample-efficient evolutionary RL: In ERL-Re², a PeVFA replaces full MC rollouts for policy fitness estimation. A short rollout is performed and the PeVFA Q-value for the remaining horizon completes the estimate, dramatically improving sample efficiency and accelerating convergence by up to 5×, with 20–30% higher final scores across MuJoCo benchmarks (Hao et al., 2022). Even pure EA + PeVFA outperforms vanilla EA under this surrogate.
• Improved on-policy policy iteration: PPO re-implemented with GPI–PeVFA achieves ∼40% higher returns than vanilla PPO in most MuJoCo environments. Learned policy embeddings (OPR and SPR, especially with contrastive loss) further increase performance (Tang et al., 2020).
• Policy value generalization and lower approximation error: PeVFA consistently achieves lower initial approximation error for new policies along the GPI trajectory, which explains improved value estimation/parsimony over e.g., per-policy critics.

Representative empirical results (max average returns in MuJoCo tasks; means ±0.5 STD over 10 trials) (Tang et al., 2020):

Task	PPO	PPO–PeVFA (SPR-CL/OPR-CL)
HalfCheetah-v1	2621	3725 ± 349
Hopper-v1	1639	2351 ± 231
Walker2d-v1	1505	2038 ± 316
Ant-v1	2835	4019 ± 162

7. Limitations, Open Problems, and Future Directions

Notable limitations include increased memory/storage overhead from tracking and training on historical policies and data, sensitivity of value generalization to the quality and stability of policy representations, and potential non-stationarity when bootstrapping in TD-based methods (Tang et al., 2020, Hao et al., 2022). No formal convergence or optimality guarantees are currently available for off-policy PeVFA training (Hao et al., 2022).

Potential future avenues include:

• Advanced policy representation learning (e.g., manifold embeddings, importance sampling in SPR).
• Rigorous convergence analysis and error bounds incorporating value generalization across policies.
• Extension to value-based RL (e.g., embedding argmax policy for Q-learning).
• Integration with “gradient-through-policy-input” (GTPI) methods for seamless end-to-end RL, beyond current GPI paradigms.

PeVFA represents a theoretically grounded, practical extension to the value function approximator family, equipping RL systems with the capability to reason and interpolate over policy space, thereby improving value estimation, sample efficiency, and opening new lines of algorithmic development (Harb et al., 2020, Tang et al., 2020, Hao et al., 2022).