Parametric Value Function in Optimization

• Parametric value function is a mapping that assigns optimized outcomes (minimal/maximal values) based on system parameters, central to optimization, control, and game theory.
• It exhibits well-defined regularity and sensitivity properties, including continuity and directional differentiability, derived via Berge’s Maximum Theorem and duality-based methods.
• This function underpins scalable algorithms in reinforcement learning and control, as seen in stochastic dynamic programming, differential games, and bilevel optimization.

A parametric value function is a mapping that assigns to each parameter configuration of a system the value (typically minimal or maximal cost, reward, or objective value) associated with an optimized outcome of a parameter-dependent problem. This concept is foundational in optimization, control, operations research, reinforcement learning, and game theory, providing a formalism for sensitivity analysis, algorithmic differentiation, and scalable value-function learning.

1. Foundational Definition

Given a family of optimization, variational, or equilibrium problems indexed by a parameter $\theta$ (or $p$, $x$, depending on context), the parametric value function is defined as

$$V(\theta) = \inf_{y \in \mathcal{Y}(\theta)} f(\theta, y)$$

where $f$ is a parameter-dependent objective function, and $\mathcal{Y}(\theta)$ is the parameterized feasible set. Variants also include supremum (maximal value functions), as in Stackelberg/multi-level and game-theoretic settings, or value functions over additional domains such as time or state.

In Markov Decision Processes, stochastic optimization, and RL, $V$ may depend on both the current state and a system parameter, e.g., $V(s;\theta)$ or $V(x, t; \theta)$ in differential games. In multiobjective and bilevel optimization, set-valued parametric value functions (frontier maps) characterize Pareto-efficient frontiers as a function of upper-level decisions.

2. Regularity and Sensitivity Properties

Regularity theory concerns continuity, differentiability, and nonsmooth (subdifferential) properties of parametric value functions:

• Continuity is governed by generalizations of Berge's Maximum Theorem, characterizing upper/lower semicontinuity and full continuity in terms of feasible-path-transfer properties (FPTusc/FPTlsc) and inf-compactness, allowing for discontinuous objectives and noncompact feasible sets (Feinberg et al., 2021).
• In convex programming, under convexity and Slater-type conditions, the parametric value function is locally Lipschitz and directionally differentiable, with explicit formulas given in terms of dual solutions and Lagrange multipliers (Luan et al., 2020, Lahiri, 2024).
• In the context of conic programs, polyhedrality of the feasible cone yields exact first-order directional derivatives; for instance, for a conic LP:

$$\phi(b) = \inf \{c^\top x: Ax \geq_K b\}, \quad \phi'(b;d) = \max_{y \in S(D)} y^\top d$$

where $S(D)$ is the dual solution set (Luan et al., 2020).

3. Directional and Second-Order Sensitivity

Parametric value functions in nonsmooth or highly constrained settings require advanced sensitivity tools:

• Directional subdifferentials (limiting, singular, and Fréchet) generalize gradient-based sensitivity and are characterized, under metric subregularity and inf-compactness, by unions over (possibly directional) solution sets, using multiplier-theoretic representations and critical cones (Bai et al., 2022, Bai et al., 2023).
• Second-order analysis involves Mordukhovich generalized Hessians, with inclusions involving coderivatives of the solution and multiplier mappings, facilitating robust Newton-type algorithms for structured programs (including minmax/semi-infinite/bilevel contexts) (Zemkoho, 2017).
• In nonconvex minimax and mathematical programs with equilibrium constraints, Wolfe-duality-based results yield sharp necessary stationarity conditions using only single-valued or union-over-solutions formulas, bypassing the convex hull barrier that plagues classical subdifferential calculus (Guo et al., 2023).

4. Parametric Value Functions in Control, RL, and Games

In optimal control, reinforcement learning, and differential games, parametric value functions are critical for high-dimensional, multi-agent, or safety-constrained settings:

• Parametric RL value functions: For state–action–parameter spaces, low-rank factorized or tensorized models, $Q(s,a;\theta)$, facilitate tractable and data-efficient approximate dynamic programming, with matrix/tensor low-rank SGD and ALS-type updates enabling scalable learning with rigorous convergence properties (Rozada et al., 2022, Rozada et al., 2021).
• Policy-parameterized value functions: In off-policy RL, parameter-based value functions (PBVFs) $V(s,\theta), Q(s,a,\theta)$ support generalization across the space of policies, allow for gradient-based policy search without new environment interaction (zero-shot policy optimization), and yield extended policy-gradient theorems retaining the critical $\nabla_\theta Q$ term (Faccio et al., 2020).
• Parametric Markov chains and reachability: For systems with uncertain or parameterized transitions, statewise reachability functions $V(i; \theta)$ are rational functions in the parameters, and their structure underpins complexity results (coETR-completeness of universal reachability/monotonicity verification), symmetries (same-value equivalence), and model reductions (Engelen et al., 23 Apr 2025).

5. Algorithmic and Learning Methodologies

Parametric value function approximation underlies scalable algorithms in stochastic optimization, RL, and differential games:

• Operator learning methods: For differential games, a hybrid neural operator (HNO) architecture maps player parameterizations and constraints to value functions, combining DeepONet/building-block architectures with PINN regularization and supervised gradient anchoring, enabling real-time, high-dimensional parametric value approximation with safety guarantees (2503.06994).
• Value function gradient learning (VFGL): In multistage stochastic programming, parametric approximators are fit directly to minimize gradient loss (between true and modeled derivatives), using stochastic gradient descent rather than cut accumulation (as in SDDP), providing computational savings while maintaining solution quality (Lee et al., 2022).
• Duality-based differentiation: For convex/non-smooth programs, generalized envelope theorems and convex duality produce subgradients via dual optimizers (e.g., $\nabla_\theta V(\theta) = y^*(\theta)$), expanding applicability beyond traditional envelope formulas and enabling robust, scalable gradient estimation in ML/structured prediction settings (Mehmood et al., 2020, Baotić, 2016).
• Frontier maps in multiobjective bilevel optimization: The set-valued efficient-value mapping $\Phi(x)$ records the Pareto-optimal front at each parameter, and coderivative-based calculus extends first-order necessary conditions and constraint qualifications (GVFCQ) from scalar to vector-valued lower-level problems (Lafhim et al., 2021).

6. Applications and Illustrative Examples

Parametric value functions and their approximations are fundamental in:

Domain	Typical Formulation	Notable Impact
Conic & Linear Programming	$V(b) = \inf \{c^\top x: Ax \geq_K b\}$	Sensitivity, Lipschitz, strong duality
Stochastic Dynamic Programming	$V(x,p)$ via backward recursion	Policy optimization, derivative-based search
RL / MDPs	$Q(s,a;\theta)$ (low-rank/tensor factorized)	Off-policy learning, generalization
Differential Games	$V(x,t;\theta)$ as HJI solution	Multi-agent safety, high-dimensional PINNs
Bilevel & Minimax Programs	$V(x) = \inf_y f(x,y), \sup_y f(x,y)$	Subdifferential-based stationarity, Stackelberg equilibrium

Concrete case studies include production planning, hydrothermal scheduling, portfolio optimization via VFGL (Lee et al., 2022); collision-avoidance in multi-agent games with HNO (2503.06994); and policy search in RL using PBVFs and zero-shot optimization (Faccio et al., 2020).

7. Open Problems and Future Directions

Current research is extending the parametric value function paradigm in several directions:

• Nonconvex and non-smooth extensions: Refining directional and second-order subdifferentials for broader classes of nonconvex or set-valued problems (Bai et al., 2023, Bai et al., 2022).
• Operator learning and neural architectures: Scaling to larger parameter spaces and state dimensions, e.g., epigraphical reformulations to remove reliance on supervised BVP data, and improving robustness of PINNs and operator networks (2503.06994).
• Model reduction and symbolic computation: Further exploiting algebraic symmetries in pMCs to enable reduction, monotonicity verification, and symbolic reachability (Engelen et al., 23 Apr 2025).
• Regularity and sensitivity in stochastic optimization: Developing efficient regularization (e.g., Moreau–Yosida) and high-dimensional sensitivity analysis for multistage programs (Franc et al., 2022).
• Unification across paradigms: Merging variational, duality-based, operator learning, and gradient-based frameworks to enable unified, scalable estimation and differentiation of parametric value functions in high-impact domains.

Parametric value functions thus provide a central framework for the study and algorithmic solution of parameter-dependent optimization and control problems, with applications ranging from large-scale machine learning and RL to multi-agent dynamic games and stochastic programming (2503.06994, Rozada et al., 2022, Mehmood et al., 2020, Franc et al., 2022, Zemkoho, 2017).