Value Function Range (VFR) Overview

• Value Function Range (VFR) is a quantitative measure that defines the span of state-values under a given policy, highlighting performance sensitivity to initial conditions in reinforcement learning.
• In optimization, VFR identifies the range of attainable objective values in mixed integer linear programs, aligning with efficient frontiers and polyhedral representations.
• Within controlled dynamical systems and geometric analyses, VFR characterizes reachable outcome sets and bounds worst-case performance under environmental or adversarial disturbances.

The Value Function Range (VFR) is a mathematical concept that appears in several domains, including reinforcement learning (RL), mixed integer linear optimization, and geometric function theory. VFR quantifies, under varying definitions, the set or span of values attained by a policy's value function over a state space, the attainable outcomes at the endpoint of controlled dynamical systems, or the range of optimal objective values as right-hand sides vary in optimization. Across these contexts, VFR serves as a fundamental tool for understanding robustness, extremal performance, reachable sets, and polyhedral structure.

1. Definition in Reinforcement Learning

Let $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma)$$ denote a Markov Decision Process (MDP) with state space $\mathcal{S}$, action space $\mathcal{A}$, transition kernel $\mathcal{P}$, bounded reward function $\mathcal{R}$, and discount factor $0 \leq \gamma < 1$. For a fixed policy $\pi$, the value function at state $s$ is

$$V_{\mathcal{M},\pi}(s) = \mathbb{E}\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0 = s, \pi, \mathcal{M} \right]$$

The Value Function Range is defined as

$$\hat V_{\mathcal{M},\pi} \triangleq \max_{s \in \mathcal{S}} V_{\mathcal{M},\pi}(s) - \min_{s \in \mathcal{S}} V_{\mathcal{M},\pi}(s)$$

This metric measures the “spread” of the expected returns achievable from all states under $\pi$, reflecting policy sensitivity to initial conditions and serving as a fundamental robustness quantity (Ying et al., 2022).

2. VFR in Robustness and Performance Bounds

The VFR directly controls upper bounds on performance degradation of RL agents under environmental disturbances:

• Transition Disturbance: Suppose the MDP’s transition kernel is perturbed ($\mathcal{P} \to \hat{\mathcal{P}}$). Then,

$$\left| J_{\hat{\mathcal{M}}}(\pi) - J_{\mathcal{M}}(\pi) \right| \leq \frac{2\gamma}{1-\gamma} \max_{s,a} D_{TV}(\mathcal{P}(\cdot|s,a),\hat{\mathcal{P}}(\cdot|s,a))\, \hat V_{\mathcal{M},\pi}$$

where $D_{TV}$ denotes total variation distance, and $J_{\mathcal{M}}(\pi)$ is the normalized return.

• Observation or Adversarial Disturbance: For adversarial remapping $\nu:\mathcal{S} \to \mathcal{S}$,

$$\left| J_{\mathcal{M}}(\pi) - J_{\mathcal{M}}(\hat{\pi}_\nu) \right| \leq \frac{\gamma}{1-\gamma} \epsilon_\pi \hat V_{\mathcal{M},\pi} + \frac{2}{1-\gamma} \epsilon_\pi R_{\max}$$

with $$\epsilon_\pi = \max_s D_{TV}(\pi(\cdot|s),\pi(\cdot|\nu(s)))$$ (Ying et al., 2022).

These theorems show that a smaller VFR provably limits the worst-case loss in return under small perturbations, establishing its centrality for robust and safe RL.

3. Relationship to Classical Risk and Robustness Metrics

The VFR coincides with the span-seminorm of the value function: $\| V \|_{\text{span}} = \max_s V(s) - \min_s V(s)$ It quantifies the maximal difference in expected returns rather than their variance, so it measures state-level dispersion rather than the trajectory-level dispersion captured by variance or Conditional Value-at-Risk (CVaR). The VFR upper-bounds the gap between worst-case and expected returns and is tightly connected with robust MDP literature [(Ying et al., 2022), Howard & Matheson 1972].

4. VFR in Optimization: Mixed Integer Linear Programs

In the context of mixed integer linear optimization, VFR appears as the range of values of a restricted value function (RVF), which for a single-objective MILP is

$$v(b) = \min\{ c^{\top} x : A x = b,\, x \in \mathbb{R}_+^n \cap (\mathbb{Z}^p \times \mathbb{R}^{n-p}) \}$$

for right-hand side $b$ in a domain $C$ of feasible right-hand sides. The epigraph of $v$,

$$\text{epi}(v) = \{ (b, \alpha) \in \mathbb{R}^m \times \mathbb{R} : v(b) \leq \alpha \}$$

contains all attainable (constraint, value) pairs. The value function’s range, in this context, is intimately linked to the efficient frontier (EF) of a multiobjective MILP:

• Every point on the EF corresponds to a point on the boundary of the epigraph of $v$.
• Conversely, every boundary point of $\text{epi}(v)$ is associated with some (possibly weakly) nondominated solution (Fallah et al., 2023).

The mutual expressivity of range and efficient frontier enables algorithmic computation of both via polyhedral cutting-plane procedures. The polyhedral representation of the value function is

$$v(b) = \max_{(u,v) \in E} (b^\top u + \beta^\top v)$$

where $E$ is the set of extreme points of an associated LP relaxation (Fallah et al., 2023).

5. Applications in Reinforcement Learning Algorithms

Direct minimization of VFR is computationally impractical in large-scale or continuous RL problems. Instead, trajectory-level risk constraints such as CVaR are imposed to ensure robustness. The CVaR-Proximal-Policy-Optimization (CPPO) algorithm constrains the risk measure

$-\text{CVaR}_\alpha(-D(\pi_\theta)) \geq \beta$

with

$$\text{CVaR}_\alpha(Z) = \min_{\eta \in \mathbb{R}} \left\{ \eta + \frac{1}{1-\alpha} \mathbb{E}[(Z-\eta)^+] \right\}$$

and applies stochastic gradient methods over policy and Lagrange parameters. This approach effectively keeps the VFR small, as it lower-bounds the worst state-value CVaR and thereby guarantees controlled return degradation under perturbations (Ying et al., 2022).

6. Value Function Range in the Theory of Controlled Dynamical Systems

The concept of value function range also arises in mathematical analysis of dynamical systems, such as the chordal Loewner equation in complex analysis. For the equation

$$\frac{dg(z, t)}{dt} = \frac{2}{g(z, t) - \lambda(t)}, \quad g(z, 0) = z, \quad t \in [0, T]$$

with control $|\lambda(t)| \leq c$, the VFR$(c,T)$ is defined as the set

$$\text{VFR}(c,T) = \{ g(i, T) : \lambda(\cdot) \text{ continuous},\, |\lambda(t)| \leq c \}$$

This set captures all attainable outcomes at time $T$ from initial point $z=i$, subject to the control bound. Analysis proceeds by reducing to a planar control system and employing the Pontryagin maximum principle, yielding a geometric decomposition of the range boundary into analytic arcs corresponding to different control regimes (pure sliding, saturation, and mixed arcs) (Zherdev, 2019).

7. Geometric, Algorithmic, and Practical Implications

• In RL, the VFR quantifies sensitivity to environmental perturbations and underpins safe policy optimization by bounding worst-case return reductions.
• In optimization, the VFR (as the range of an RVF) admits finite polyhedral descriptions that coincide with the efficient frontier structure, enabling exact and approximate algorithmic computations via cutting-plane techniques. At convergence, these algorithms guarantee a complete characterization of the attainable values and efficient points (Fallah et al., 2023).
• In analysis of controlled systems and conformal mappings, VFR provides sharp geometric characterizations of reachable sets, with implications for extremal mapping problems and control under amplitude constraints (Zherdev, 2019).

A plausible implication is that across domains, careful control or characterization of the value function range serves as a unifying principle for ensuring robustness, performance guarantees, and algorithmic tractability. The VFR, whether as a norm, range, or reachable set, bridges optimization theory, learning, and dynamical systems analysis.