VPES: Value Prediction Error Stability in RL

• VPES is a variance-based measure that quantifies the consistency of TD errors to assess stability in value function learning.
• It employs an exponential moving average to track long-term trends, facilitating dynamic regulation of meta-trust and adaptive scaling of learning rates.
• Empirical results show VPES reduces failure rates and improves tail-risk metrics in RL, particularly under reward corruption scenarios.

Value Prediction Error Stability (VPES) is an internal, variance-based reliability signal for assessing the stability of value function learning in reinforcement learning (RL), particularly in settings with function approximation and corrupted rewards. As introduced in the meta-cognitive RL framework "Meta-Cognitive Reinforcement Learning with Self-Doubt and Recovery" (Zhang et al., 28 Jan 2026), VPES enables an agent to quantify the consistency of its value prediction errors and dynamically regulate its own learning dynamics via meta-trust and fail-safe adaptation mechanisms.

1. Formal Definition and Quantification

Let $s_t\in\mathcal S$, $a_t\in\mathcal A$ denote the state and action at step $t$, and $V_\theta(s)$ a value function with parameters $\theta$. The one-step temporal-difference (TD) error at time $t$ is: $$\delta_t = r_t + \gamma V_\theta(s_{t+1}) - V_\theta(s_t)$$ where $r_t$ is the observed reward, and $\gamma\in[0,1)$ is the discount factor.

For a fixed window length $k$, VPES at time $t$ is defined as the empirical variance of the most recent $k+1$ TD errors: $$\mathrm{VPES}_t = \mathrm{Var}\{\delta_{t-k},\delta_{t-k+1},\dots,\delta_t\} = \frac{1}{k+1}\sum_{i=t-k}^t \delta_i^2 - \left(\frac{1}{k+1}\sum_{i=t-k}^t \delta_i\right)^2$$ This moving-variance measure serves as a direct indicator of dynamic inconsistency in value estimates.

2. Computation and Stability Criteria

At each major training iteration, the procedure for using VPES is as follows:

• Collect a batch of transitions; compute the associated TD errors $\{\delta_i\}$.
• Form the working window $\mathcal W_t = \{\delta_{t-k},\dots,\delta_t\}$.
• Compute VPES:

$$\mathrm{VPES}_t = \frac{1}{|\mathcal W_t|}\sum_{\delta\in\mathcal W_t}\delta^2 - \left(\frac{1}{|\mathcal W_t|}\sum_{\delta\in\mathcal W_t}\delta\right)^2$$

• Maintain an exponential moving average (EMA) to extract longer-term stability information:

$$\bar v_t = (1-\beta_v)\bar v_{t-1} + \beta_v \mathrm{VPES}_t,\quad 0 < \beta_v < 1$$

• Define the stability trend:

$\Delta v_t = \bar v_t - \mathrm{VPES}_t$

• $\Delta v_t > 0$ indicates decreasing variance (stabilizing learning).
• $\Delta v_t < 0$ indicates increasing variance (destabilizing learning).

The trend signal $\Delta v_t$ provides a sensitive, directional cue for meta-cognitive control, often rendering absolute stability thresholds unnecessary.

3. Theoretical Rationale

In approximate RL, the TD error quantifies the deviation of the current value prediction from its expected Bellman backup. High variance in $\delta_t$ over time generally signifies instability—i.e., the value function oscillates or chases spurious feedback, a precursor to catastrophic collapse in late-stage learning. Conversely, persistent reduction in VPES suggests convergence toward a Bellman fixed point, implying stable optimization.

VPES has several notable theoretical properties:

• It is agnostic to the raw quality of observed rewards; thus, it detects instability even under severe external reward corruption.
• It leverages only internal value predictions, separating the learning process's self-doubt from environment uncertainty.
• It can identify instability patterns that evade observation by external reward-focused variance criteria.

4. Meta-Trust and Adaptive Regulation

VPES underpins an asymmetric meta-cognitive regulation mechanism via a meta-trust variable $\tau_t \in [0,1]$. This meta-trust quantifies confidence in the learning process and is updated based on the VPES trend: $\tau_t = \begin{cases} \min\{1,\;\tau_{t-1} + \eta_{\mathrm{up}}\} & \Delta v_t > 0\ (\textrm{stability improving})\[6pt] \max\{0,\;\tau_{t-1} - \eta_{\mathrm{down}}\} & \Delta v_t \le 0\ (\textrm{stability deteriorating}) \end{cases}$ with $0 < \eta_{\mathrm{up}} < \eta_{\mathrm{down}}$, so trust recovers slowly but declines quickly under instability.

A control signal $c_t = f(\tau_t)$—typically $c_t = \tau_t$—is used to scale the base learning rate $\alpha_0$: $\alpha_t = \alpha_0 c_t$ Fail-safe constraints prevent learning rate amplification when trust is low: $$c_t \leq 1 \quad \text{whenever} \quad \tau_t < \tau_{\min}$$ where $\tau_{\min}$ is a small threshold. In effect, the learning rate is only attenuated under low trust and never amplified above the base rate.

5. Algorithmic Integration

The following pseudo-code encapsulates the VPES-driven meta-cognitive RL control loop, specializing Algorithm 1 from (Zhang et al., 28 Jan 2026):

Initialize policy parameters θ Initialize meta-trust τ₀ ∈ (0,1), VPES EMA \bar v₀ ← 0 for t = 1,2, … do Collect rollout under π_θ; compute TD-errors {δ_i} Compute VPES_t = Var( {δ_{i}} ) Update VPES EMA: \bar v_t ← (1-β_v)\bar v_{t-1} + β_v·VPES_t Compute trend: Δv_t ← \bar v_t - VPES_t if Δv_t > 0 then τ_t ← min(1, τ_{t-1} + η_up) else τ_t ← max(0, τ_{t-1} - η_down) end if c_t ← f(τ_t) # e.g. c_t = τ_t if τ_t < τ_min then c_t ← min(c_t, 1.0) # fail-safe: no amplification end if α_t ← α₀ · c_t θ ← PPO_Update(θ; α_t) end for

Here, PPO_Update denotes a single round of policy/value updates using the dynamic learning rate $\alpha_t$. The meta-trust update enacts a rapid response to instability (VPES spikes) and gradual recovery once stability returns.

6. Empirical Evidence and Robustness

Experimental validation was conducted on standard continuous-control tasks such as HalfCheetah-v4, introducing persistent reward corruption: with probability $p=0.5$, the true reward $r_t$ is perturbed by uniform noise in $[-10,10]$.

Key experimental findings:

• The late-stage failure rate (fraction of irrecoverable collapse) is reduced by half:
  • Elastic-PPO baseline: 0.40
  • VPES + meta-trust controller: 0.20
• VPES spikes mark emergent instability; meta-trust $\tau_t$ decays rapidly, and the learning rate $\alpha_t$ is attenuated. As VPES subsides, a slow, controlled recovery restores learning rate and trust.
• Tail-risk (CVaR@20%) is substantially improved:
  • Elastic-PPO: CVaR ≈ −6.8
  • Fail-Safe (no recovery): CVaR ≈ −242
  • Full Meta-Cognitive (VPES + recovery): CVaR ≈ −26.3

These results demonstrate the effectiveness of VPES-driven regulation in reducing both catastrophic failure rates and adverse tail outcomes during RL in corrupted environments (Zhang et al., 28 Jan 2026).

7. Summary and Significance

VPES is a moving variance measure of recent TD errors, providing an internal, reward-agnostic stability signal for value function learning. Its principal roles include:

• Generating a trend signal ($\Delta v_t$) reflecting the dynamic trajectory of value function consistency.
• Driving an asymmetric meta-trust variable ($\tau_t$) that adapts trust in the agent’s learning process in real time.
• Modulating the learning rate with embedded fail-safe constraints, attenuating updates during instability and allowing gradual recovery when stability resumes.

VPES enables robust, self-regulating RL by stabilizing learning in the presence of unreliable feedback and preventing late-stage collapse. Its integration within meta-cognitive frameworks underscores the growing emphasis on internal learning introspection and adaptive control in contemporary reinforcement learning research (Zhang et al., 28 Jan 2026).