---
title: Policy-Extended Value Function Approximators
url: https://www.emergentmind.com/topics/policy-extended-value-function-approximators-pevfa
type: topic
---

# Policy-Extended Value Function Approximators

Policy-Extended Value Function Approximators (PeVFA) are a class of function approximators in reinforcement learning (RL) that extend conventional value function approximators by explicitly conditioning on both the state (and/or action) and an explicit representation of the policy. This design shift enables simultaneous estimation of value functions for multiple policies, facilitates value generalization across policies, and opens up new algorithmic possibilities such as zero-shot policy improvement, enhanced sample efficiency in evolutionary RL, and more robust policy iteration schemes. By leveraging learned or structured policy embeddings, PeVFAs can interpolate values in policy space, support gradient-based policy optimization without new environment samples, and integrate seamlessly with both policy-gradient and evolutionary methods [2002.11833][2010.09536][2210.17375].

## 1. Formal Definition and Core Objective

The central objective of a PeVFA is to approximate the value function not just as a mapping from states (and, in Q-function form, actions) to returns, but as a mapping from the joint space of policies and states (and possibly actions) to values. Let $\pi$ denote a policy, $s$ a state, $a$ an action, and $g(\pi)$ or $\phi(\pi)$ a learned fixed-dimensional embedding (or "fingerprint") of policy $\pi$:
- State-value PeVFA: $\hat v(s; \pi, \theta) = \mathbb{V}_\theta(s, g(\pi)) \approx v^\pi(s)$
- Action-value PeVFA: $\hat{Q}_\theta(s, a, W) \approx Q^{\pi_W}(s, a)$

In the form leveraged by policy evaluation networks (PVNs), for a policy parameterized by $\psi$, the PeVFA is realized as
$$
\hat V_\theta(\pi, s) = f_\theta(\phi(\pi), s)
$$
and, for optimizing start-state return,
$$
\hat J_\theta(\pi) = f_\theta(\phi(\pi))
$$
where $f_\theta$ is a feed-forward neural network with parameters $\theta$, and $\phi(\pi)$ is the policy fingerprint [2002.11833].

## 2. Policy Representation and Embedding Mechanisms

Effective generalization in PeVFA depends crucially on the construction of the policy representation $\phi(\pi)$ or $g(\pi)$. Several methods have been proposed:

- **Probing-state fingerprinting**: PVNs [2002.11833] introduce a trainable set of probing states $S_\mathrm{probe} = \{ s_1, \ldots, s_n \}$; the policy’s output on these states is concatenated into a fingerprint vector (dimension $n|\mathcal{A}|$). Both the probing states and network weights are learned via backpropagation.

- **Surface Policy Representation (SPR)**: State–action pairs sampled from policy rollouts are encoded by an MLP followed by mean-pooling to generate the embedding $\chi_\pi = \frac{1}{N} \sum_{j=1}^N \phi(s_j, a_j)$ [2010.09536].

- **Origin Policy Representation (OPR)**: Policy network parameters (weights/biases) are encoded per-layer using a small MLP, then pooled and concatenated [2010.09536].

- **Linear policy representation**: For linear policies, the policy weight matrix $W$ itself serves as the embedding, with a learned “policy embedder” mapping $W$ to a lower-dimensional vector [2210.17375].

Auxiliary objectives for embedding learning include: end-to-end value fitting, policy-recovery decoders (matching $\pi(a|s)$ from $\chi_\pi$), and contrastive losses to encourage policy distinction in embedding space [2010.09536].

## 3. Training Objectives and Algorithms

### Regression and Distributional Losses

- **State-value/return prediction**: Minimize mean squared error (MSE) between PeVFA predictions and Monte Carlo returns, either on current or historical policies:
$$
L_\mathrm{MSE}(\theta) = \mathbb{E}_{(s, \pi)} \left[ \left(\hat v(s; \pi, \theta) - v^\pi(s)\right)^2 \right]
$$
[2010.09536][2002.11833].

- **Distributional prediction**: Predict the empirical return distribution (policy return histograms) using a categorical output and KL divergence as loss:
$$
L(\theta) = \mathbb{E}_{(\pi_i, R_i)} \left[ D_\text{KL}(\hat P_i \| P_\theta(\cdot \mid \phi(\pi_i))) \right]
$$
[2002.11833].

- **Action-value PeVFA**: Employs Bellman regression losses, e.g., 1-step TD (target network) updates:
$$
\mathcal{L}_\mathrm{PeVFA}(\theta) = \mathbb{E}_{(s,a,r,s'),W} \left[ \left(r + \gamma Q_{\theta'}(s', \pi_W(s'), W) - Q_\theta(s,a,W) \right)^2 \right]
$$
[2210.17375].

### Generalized Policy Iteration with PeVFA

GPI with PeVFA cycles through policy evaluation (training PeVFA on all encountered policies with their embeddings and trajectories) and policy improvement (using PeVFA to compute advantages, update via PPO or another objective). The policy encoder and value network are typically updated jointly or on staggered schedules [2010.09536].

### Zero-shot Policy Ascent

By exploiting end-to-end differentiability from the PeVFA’s value output to the policy parameters $\psi$, direct gradient ascent in policy space is feasible:
$$
\psi_{t+1} = \psi_t + \eta_t \nabla_{\psi_t} f_\theta(\phi(\pi_{\psi_t}))
$$
This inner-loop optimization requires no environment samples after initial PeVFA pretraining (“zero-shot” improvement) [2002.11833].

## 4. Theoretical Foundations and Generalization

PeVFA enables value generalization across policies due to the explicit policy-conditioning. Theoretical results formalize this property under conditions such as L-Lipschitz continuity (in a metric $d(\pi,\pi')$):

- **Two-policy generalization lemma**: If PeVFA contracts approximation error on policy $\pi_1$, then, under L-continuity, error on $\pi_2$ is bounded by a contraction term and a distance-dependent margin.
- **Generalization across policy improvement sequences**: Under GPI, the approximation error for the next improved policy accumulates as an exponentially-decaying sum of base error and local margins proportional to the embedding distance between consecutive policies. Locality in embedding space is critical for consistent improvement [2010.09536].
- **Zero-shot improvement guarantee**: The PeVFA’s value estimate for a new policy can be provably closer to the true value than using the previous policy, expediting GPI convergence.

A plausible implication is that PeVFA’s effectiveness is tightly linked to the smoothness and informativeness of the policy embedding. Non-stationarity or poor locality in embedding space can degrade generalization.

## 5. Network Architectures and Practical Implementation

PeVFA architectures typically concatenate learned policy embeddings with states (and optionally actions), feeding the result into a multi-layer perceptron:

- In PVN [2002.11833], the model uses a 2–3 layer MLP with hidden sizes 50–80 and ReLU activations; fingerprint vectors have dimension $n|\mathcal{A}|$.
- In ERL-Re² [2210.17375], the policy matrix $W$ is embedded via a per-row MLP (301→64→64), averaged to give a 64-dimensional policy vector, concatenated with $s$ and $a$, and passed through two fully-connected layers (400→300→1, leaky ReLU).
- In GPI–PeVFA [2010.09536], separate encoding streams for state and policy, each with 2 layers of 64 units, are merged by MLPs.

Policy fingerprints based on probing states, policy parameters, or rollouts are treated as learnable parameters and updated by backpropagation jointly with PeVFA network weights.

## 6. Empirical Findings and Applications

PeVFA demonstrates several benefits across RL settings:

- **Zero-shot policy optimization**: PVN-trained PeVFAs support direct gradient improvement, achieving convergence to optimal policies from suboptimal starting points in both toy MDPs and environments such as CartPole and MuJoCo Swimmer. Empirical results show that gradient fields computed via PeVFA closely match true policy gradients, and final performance can outperform the policy dataset used for training [2002.11833].
- **Sample-efficient evolutionary RL**: In ERL-Re², a PeVFA replaces full MC rollouts for policy fitness estimation. A short rollout is performed and the PeVFA Q-value for the remaining horizon completes the estimate, dramatically improving sample efficiency and accelerating convergence by up to 5×, with 20–30% higher final scores across MuJoCo benchmarks [2210.17375]. Even pure EA + PeVFA outperforms vanilla EA under this surrogate.
- **Improved on-policy policy iteration**: PPO re-implemented with GPI–PeVFA achieves ∼40% higher returns than vanilla PPO in most MuJoCo environments. Learned policy embeddings (OPR and SPR, especially with contrastive loss) further increase performance [2010.09536].
- **Policy value generalization and lower approximation error**: PeVFA consistently achieves lower initial approximation error for new policies along the GPI trajectory, which explains improved value estimation/parsimony over e.g., per-policy critics.

Representative empirical results (max average returns in MuJoCo tasks; means ±0.5 STD over 10 trials) [2010.09536]:

| Task                 | PPO   | PPO–PeVFA (SPR-CL/OPR-CL) |
|----------------------|-------|---------------------------|
| HalfCheetah-v1       | 2621  | 3725 ± 349                |
| Hopper-v1            | 1639  | 2351 ± 231                |
| Walker2d-v1          | 1505  | 2038 ± 316                |
| Ant-v1               | 2835  | 4019 ± 162                |

## 7. Limitations, Open Problems, and Future Directions

Notable limitations include increased memory/storage overhead from tracking and training on historical policies and data, sensitivity of value generalization to the quality and stability of policy representations, and potential non-stationarity when bootstrapping in TD-based methods [2010.09536][2210.17375]. No formal convergence or optimality guarantees are currently available for off-policy PeVFA training [2210.17375].

Potential future avenues include:
- Advanced policy representation learning (e.g., manifold embeddings, importance sampling in SPR).
- Rigorous convergence analysis and error bounds incorporating value generalization across policies.
- Extension to value-based RL (e.g., embedding argmax policy for Q-learning).
- Integration with “gradient-through-policy-input” (GTPI) methods for seamless end-to-end RL, beyond current GPI paradigms.

PeVFA represents a theoretically grounded, practical extension to the value function approximator family, equipping RL systems with the capability to reason and interpolate over policy space, thereby improving value estimation, sample efficiency, and opening new lines of algorithmic development [2002.11833][2010.09536][2210.17375].

Source: https://www.emergentmind.com/topics/policy-extended-value-function-approximators-pevfa