Value Function Spaces (VFS)

• Value Function Spaces (VFS) are structured hypothesis classes that organize value functions using geometric, algebraic, or abstract properties to improve learning and planning.
• They encompass formulations such as low-rank matrix/tensor approximations, skill-centric embeddings, reproducing kernel Hilbert spaces, and Wasserstein-based domains, each offering distinct benefits and challenges.
• Empirical results in grid navigation, continuous control, and hierarchical tasks highlight a trade-off between parameter efficiency and approximation accuracy, emphasizing the need for compatible structural assumptions.

Value Function Spaces (VFS) denotes a family of constructions in which value functions are treated not as unstructured tables or generic black-box regressors, but as elements of a deliberately chosen space whose geometry, algebra, or abstraction properties determine what can be represented and how learning or planning proceeds. In current literature, VFS appears in several technically distinct forms: low-rank matrix or tensor classes for discrete $Q$-functions, skill-centric embeddings $Z(s)$ built from pretrained skill values, reproducing kernel Hilbert spaces for recovery of optimal control value functions from verification conditions, and value functions whose domain is the Wasserstein space of probability measures (Rozada et al., 2021, Shah et al., 2021, Ehring et al., 8 Dec 2025, Hynd et al., 2013).

1. Principal formulations

A first formulation arises in discounted finite Markov decision processes after discretization. With discrete state and action sets $\mathcal S$ and $\mathcal A$, the action-value function is represented as a matrix $Q^\pi \in \mathbb R^{D_S\times D_A}$, and the VFS is the structured class

$$\mathcal Q_M=\{Q\in\mathbb R^{D_S\times D_A}:\operatorname{rank}(Q)\le M\},$$

or, operationally, its factorized parameterization $Q\approx LR$ with $L\in\mathbb R^{D_S\times M}$ and $R\in\mathbb R^{M\times D_A}$. In this formulation, the space of admissible value functions is smaller than the full tabular space and enforces a latent bilinear interaction structure between states and actions (Rozada et al., 2021).

A second formulation is explicitly representational. In hierarchical reinforcement learning with a fixed bank of skills $O=\{o_1,\dots,o_k\}$, the state abstraction is

$Z(s)$0

or, equivalently,

$Z(s)$1

Here the VFS is not a hypothesis class of scalar value functions over the original state space, but a skill-indexed embedding space whose coordinates are the values of available skills. The abstraction is therefore defined relative to what the lower-level skills can achieve (Shah et al., 2021, He et al., 2024).

A third formulation treats the value function as an element of a reproducing kernel Hilbert space. For nonlinear infinite-horizon optimal control, the optimal value function $Z(s)$2 is assumed to belong to an RKHS $Z(s)$3, and recovery is posed as a minimal-norm problem in that space subject to Hamilton-Jacobi-Bellman verification equalities and positivity or quadratic inequalities at collocation points. In this setting, the VFS is an ambient function space endowed with a kernel-induced geometry, while the feasible subset is carved out by verification constraints (Ehring et al., 8 Dec 2025).

A fourth formulation places the state itself in an infinite-dimensional space. In the Wasserstein-space theory of deterministic control, the generalized value function

$Z(s)$4

is defined on $Z(s)$5, the $Z(s)$6-Wasserstein space of probability measures. The resulting VFS is therefore a family of value functions whose domain is itself a metric-measure state space rather than $Z(s)$7 (Hynd et al., 2013).

Taken together, these formulations show that VFS is not a single formalism. It can denote a structured hypothesis class, a skill-conditioned abstraction, an RKHS ambient space, or a value-function theory over a non-Euclidean state domain.

2. Skill-centric abstractions and planning

In hierarchical reinforcement learning, VFS was introduced as an answer to the question of which state representation a high-level controller should use when its actions are temporally extended skills. In the semi-Markov setting $Z(s)$8, the proposal is to represent a state by the values of the available skills, so that the abstraction is grounded in skill affordances rather than in generic perceptual compression. The high-level controller then learns either a model-free $Z(s)$9 with Double DQN or a model-based predictor

$\mathcal S$0

on the VFS state, where one abstract transition corresponds to execution of one skill (Shah et al., 2021).

This representation is explicitly action-relative. If a skill has high value in a state, the environment affords that skill; if the value is low, the representation itself encodes inapplicability or low usefulness. The paper’s empirical results support this interpretation. In MiniGrid MultiRoom, VFS attained success rates $\mathcal S$1, $\mathcal S$2, $\mathcal S$3, and $\mathcal S$4 for 2, 4, 6, and 10 rooms, respectively; in KeyCorridor it attained $\mathcal S$5 for 3 rooms and $\mathcal S$6 for 7 rooms. In zero-shot transfer from KeyCorridor to MultiRoom, VFS achieved $\mathcal S$7 on MR4 and $\mathcal S$8 on MR10, compared with a directly trained HRL target policy at $\mathcal S$9 and $\mathcal A$0 (Shah et al., 2021).

The same abstraction was later used as the planning state for Signal Temporal Logic guidance. In that setting, the coordinates of

$\mathcal A$1

are used directly as STL predicates, for example

$\mathcal A$2

Planning then proceeds over a learned abstract dynamics model

$\mathcal A$3

with one VFS transition corresponding to $\mathcal A$4 low-level steps, and robustness is optimized by model predictive control with Monte Carlo Tree Search. The offline dynamics model is trained from $\mathcal A$5 tuples $\mathcal A$6 using a 3-layer MLP with ReLU, mean squared error, and Adam (He et al., 2024).

This VFS-based STL formulation removes dependence on hand-crafted predicates and makes the planning dimension depend on the number of skills rather than on the dimension of the original observation space. In the robustness-representation experiment, the method evaluated 100 random STL formulas in each of three task classes—reach-avoid, chain, and stable—and reported that for all three task classes the lower quartile of VFS robustness scores was above zero. The paper does not claim exact equivalence between VFS robustness and state-space robustness; rather, it claims an approximation supported by the interpretation of skill values as goal-reaching probabilities under the sparse-reward, $\mathcal A$7 setting and by the empirical comparison between VFS and state-space robustness (He et al., 2024).

A recurring limitation in this line of work is that the abstraction is only as informative as the skill library. If the available skills do not span the relevant distinctions or if their critics are inaccurate, the VFS may omit information essential to high-level planning. This suggests that skill coverage is not ancillary but constitutive of the abstraction.

3. Low-rank matrix and tensor spaces

A different VFS tradition restricts the admissible action-value functions themselves. For discounted infinite-horizon problems, the state-action value matrix is approximated by a low-rank factorization $\mathcal A$8, learned online and model-free. This differs from classical linear approximation because the features are not predefined: the state-side and action-side embeddings are jointly learned through the bilinear factorization. It also differs from state aggregation, which imposes hard compression, and from neural approximators, which do not encode the same matrix-structural inductive bias (Rozada et al., 2021, Rozada et al., 2022).

For multidimensional state and action spaces, the matrix view can be replaced by a tensor view. If state and action are Cartesian products of several coordinates, the full family of finite-horizon action-values can be organized as

$\mathcal A$9

where time is an additional tensor mode. The VFS is then the low-CP-rank class

$Q^\pi \in \mathbb R^{D_S\times D_A}$0

which imposes multilinear separability across state dimensions, action dimensions, and time (Rozada et al., 2024).

Representation	VFS definition	Parameter count
Tabular $Q^\pi \in \mathbb R^{D_S\times D_A}$1-space	$Q^\pi \in \mathbb R^{D_S\times D_A}$2 or full value tensor	$Q^\pi \in \mathbb R^{D_S\times D_A}$3 or $Q^\pi \in \mathbb R^{D_S\times D_A}$4
Low-rank matrix	$Q^\pi \in \mathbb R^{D_S\times D_A}$5, $Q^\pi \in \mathbb R^{D_S\times D_A}$6	$Q^\pi \in \mathbb R^{D_S\times D_A}$7
Reshaped low-rank matrix	$Q^\pi \in \mathbb R^{D_S\times D_A}$8, $Q^\pi \in \mathbb R^{D_S\times D_A}$9	$$\mathcal Q_M=\{Q\in\mathbb R^{D_S\times D_A}:\operatorname{rank}(Q)\le M\},$$0
Low-rank PARAFAC tensor	$$\mathcal Q_M=\{Q\in\mathbb R^{D_S\times D_A}:\operatorname{rank}(Q)\le M\},$$1	$$\mathcal Q_M=\{Q\in\mathbb R^{D_S\times D_A}:\operatorname{rank}(Q)\le M\},$$2

The learning rules operate directly in these restricted spaces. In the matrix case, a sampled transition defines a temporal-difference error

$$\mathcal Q_M=\{Q\in\mathbb R^{D_S\times D_A}:\operatorname{rank}(Q)\le M\},$$3

and updates only the latent vector for the visited state and the latent vector for the visited action. In the tensor case, the update touches one active row in each factor matrix and therefore changes a large structured set of tensor entries rather than one scalar table entry. This is the core information-sharing mechanism of the low-rank VFS (Rozada et al., 2021, Rozada et al., 2024).

The empirical evidence is organized around parameter efficiency, approximation quality, and scalability. In FrozenLake-v0, rank-2 low-rank learning uses 40 parameters instead of 64 for the $$\mathcal Q_M=\{Q\in\mathbb R^{D_S\times D_A}:\operatorname{rank}(Q)\le M\},$$4 table and converges to the optimal solution with faster decrease in squared Frobenius error. In Pendulum-v0, an SVD of the tabular $$\mathcal Q_M=\{Q\in\mathbb R^{D_S\times D_A}:\operatorname{rank}(Q)\le M\},$$5-matrix showed that the first three singular values explain more than $$\mathcal Q_M=\{Q\in\mathbb R^{D_S\times D_A}:\operatorname{rank}(Q)\le M\},$$6 of the variance, and the reshaped $$\mathcal Q_M=\{Q\in\mathbb R^{D_S\times D_A}:\operatorname{rank}(Q)\le M\},$$7 rank-10 representation used 5,900 parameters. In Acrobot-v1, a reshaped rank-2 factorization reduced a $$\mathcal Q_M=\{Q\in\mathbb R^{D_S\times D_A}:\operatorname{rank}(Q)\le M\},$$8-matrix with more than $$\mathcal Q_M=\{Q\in\mathbb R^{D_S\times D_A}:\operatorname{rank}(Q)\le M\},$$9 entries to 18,078 learned parameters (Rozada et al., 2021).

The tensor finite-horizon results follow the same pattern. In a $Q\approx LR$0 grid world with horizon $Q\approx LR$1, FHTLR-learning matched FHQ-learning’s return $Q\approx LR$2 with 152 parameters instead of 500. In the wireless communications problem, FHTLR-learning attained return $Q\approx LR$3 with 3,450 parameters, compared with $Q\approx LR$4 and 16,724 parameters for DFHQN, and $Q\approx LR$5 with $Q\approx LR$6 parameters for tabular FHQ-learning (Rozada et al., 2024). In the broader infinite-horizon study, tensor low-rank learning was also the only practical low-rank approach in the high-dimensional highway environment where tabular and matrix methods were infeasible (Rozada et al., 2022).

These methods come with equally explicit limitations. The factorized objectives are nonconvex; rank selection matters; sparse-reward problems can reduce the practical advantage of tensor low rank; and the finite-horizon tensor paper provides no formal convergence theorem for the factorized stochastic method (Rozada et al., 2021, Rozada et al., 2024). The representational trade-off is therefore direct: restricting the VFS reduces estimation and optimization burden, but only insofar as the relevant value function is approximately low rank.

4. RKHS and native-space formulations

In nonlinear optimal control, VFS is often formulated as an RKHS or native space selected by a kernel. One line of work assumes that the unknown optimal value function itself belongs to an RKHS $Q\approx LR$7 and seeks to recover it from HJB verification conditions. The reproducing property

$Q\approx LR$8

and its derivative analogue turn pointwise values and gradients into continuous linear functionals, so the HJB equation and side constraints become algebraic conditions on finitely many functionals of $Q\approx LR$9. The recovery problem is then posed as a minimal-norm search in $L\in\mathbb R^{D_S\times M}$0 subject to nonlinear equalities and inequalities at collocation points (Ehring et al., 8 Dec 2025).

This formulation produces a VFS with two layers. The ambient layer is the RKHS $L\in\mathbb R^{D_S\times M}$1, determined by the kernel $L\in\mathbb R^{D_S\times M}$2. The feasible layer is the subset satisfying verification constraints such as the HJB identity, positivity, or quadratic bounds. Theorem 3.6 gives a representer-style finite-dimensional reduction through the generalized Gramian of the active functionals. The convergence theory then splits into two regimes: if $L\in\mathbb R^{D_S\times M}$3 and $L\in\mathbb R^{D_S\times M}$4, dense collocation yields global convergence

$L\in\mathbb R^{D_S\times M}$5

if only quadratic upper and lower bounds are assumed, convergence is local in $L\in\mathbb R^{D_S\times M}$6. The same framework shows that a Gauss-Newton step for the verification constraints is equivalent to policy evaluation in an RKHS-based policy-iteration scheme (Ehring et al., 8 Dec 2025).

A related but distinct line of work studies the online critic in native spaces. Here the critic target is the policy-evaluation value function $L\in\mathbb R^{D_S\times M}$7, assumed to lie in an RKHS $L\in\mathbb R^{D_S\times M}$8 induced by a Mercer kernel $L\in\mathbb R^{D_S\times M}$9. The finite critic lives in

$R\in\mathbb R^{M\times D_A}$0

and the online evolution is analyzed in function space rather than solely through parameter error. The key quality measures of the approximation space are the power function

$R\in\mathbb R^{M\times D_A}$1

and the fill distance

$R\in\mathbb R^{M\times D_A}$2

which quantify how well the scattered centers $R\in\mathbb R^{M\times D_A}$3 cover the relevant subset of the state space (Niu et al., 2024).

The resulting error theory is explicitly geometric. The critic’s approximation residue satisfies bounds in terms of $R\in\mathbb R^{M\times D_A}$4, and the power function itself can be controlled by kernel-dependent functions of the fill distance. The most concise rates reported in the paper state that, for suitable uncertainty classes, the asymptotic output error behaves as $R\in\mathbb R^{M\times D_A}$5 for Sobolev-Matérn kernels and as $R\in\mathbb R^{M\times D_A}$6 for Wendland kernels; the paper also gives an exponential-type rate for the exponential kernel. Because the power function can be used both a priori and a posteriori, it supports basis selection, basis enrichment, and pruning directly at the level of the VFS rather than only at the level of coefficients (Niu et al., 2024).

The RKHS literature therefore treats VFS not merely as a convenient approximation family, but as a geometric object. Kernel choice determines admissible smoothness, origin behavior, analyticity, and approximation rate; center placement determines the actual finite-dimensional critic space; and verification or evolution equations determine which parts of the ambient space are operationally relevant.

5. Broader domains, topology, and regularity

The notion of a value-function space extends beyond Euclidean state domains. In the Wasserstein-space theory of deterministic control, the value function is defined on $R\in\mathbb R^{M\times D_A}$7, the space of probability measures with finite $R\in\mathbb R^{M\times D_A}$8-th moment, equipped with the $R\in\mathbb R^{M\times D_A}$9-Wasserstein metric. The generalized value function satisfies a dynamic programming principle and solves the Hamilton-Jacobi equation

$O=\{o_1,\dots,o_k\}$0

in a viscosity sense adapted to Wasserstein geometry. In the special case

$O=\{o_1,\dots,o_k\}$1

the paper proves the representation

$O=\{o_1,\dots,o_k\}$2

showing that some measure-space value functions are lifted averages of classical pointwise value functions (Hynd et al., 2013).

At the opposite end of the spectrum, the regularity of value functions can be much worse than standard convex-analysis heuristics suggest. For parameterized conic linear programs, a proper value function may fail to be lower semicontinuous, fail to be subdifferentiable, and even fail to be bounded on any neighborhood of any point of its effective domain. The note proves this by explicit finite-dimensional and Hilbert-space counterexamples, thereby showing that properness plus convex positive homogeneity does not place such value functions in the class of proper lower semicontinuous convex functions (Zalinescu, 2022).

Constructive analysis adds another perspective. On a totally bounded metric space, the class of uniformly bounded and uniformly Lipschitz real-valued functions is itself totally bounded in the supremum metric, and any uniformly continuous functional on that class admits approximate minimizers: $O=\{o_1,\dots,o_k\}$3 In the language of VFS, this means that regularity-constrained spaces of candidate value functions or policies can admit finite approximations at arbitrary precision, even when exact extremizers are not constructively computable (Osinenko et al., 2017).

Adjacent function-space theories broaden the mathematical backdrop. Weighted vector-valued spaces of the form $O=\{o_1,\dots,o_k\}$4 admit linearization

$O=\{o_1,\dots,o_k\}$5

under suitable compatibility and compactness assumptions, while geometric valuation theory studies functionals satisfying

$O=\{o_1,\dots,o_k\}$6

on classes such as Sobolev spaces, convex functions, and $O=\{o_1,\dots,o_k\}$7-spaces (Kruse, 2017, Ludwig, 2021). These theories are not RL-specific, but they underscore that once value functions are treated as elements of structured function spaces, topology, duality, and invariance become intrinsic rather than auxiliary.

6. Recurring themes, misconceptions, and open issues

Across these literatures, VFS is best understood as a deliberate restriction or reparameterization of the value-function domain. In low-rank RL, the restriction is algebraic: admissible $O=\{o_1,\dots,o_k\}$8-functions are low-rank matrices or tensors. In skill-centric HRL, it is representational: admissible abstract states are vectors of skill values. In RKHS recovery, it is geometric: admissible value functions lie in a kernel-induced Hilbert space and often on a verification-defined feasible manifold. In Wasserstein control, it is domain-theoretic: the state itself is a probability measure (Rozada et al., 2021, Shah et al., 2021, Ehring et al., 8 Dec 2025, Hynd et al., 2013).

One common misconception is to treat VFS as interchangeable with fixed-feature linear approximation or with generic neural parameterization. The low-rank matrix papers explicitly separate their approach from classical linear approximation, where $O=\{o_1,\dots,o_k\}$9 is fixed and only a weight vector is learned, and from generic neural approximators, which do not impose the same matrix or tensor structure (Rozada et al., 2021, Rozada et al., 2022). The skill-centric papers likewise distinguish VFS from generic latent representation learning: the embedding is constructed entirely from the values of available skills and is not learned as a separate representation learner (Shah et al., 2021). The RKHS recovery paper makes an analogous point by insisting that the value function is an element of the chosen RKHS, not merely something approximated by a convenient basis (Ehring et al., 8 Dec 2025).

A second recurring theme is the trade-off between approximation error and estimation or optimization burden. Restricting to low rank reduces the parameter count from $Z(s)$00 to $Z(s)$01, or to $Z(s)$02 after reshaping, but introduces bias when the true $Z(s)$03 is not approximately low rank (Rozada et al., 2021). Choosing a kernel-defined RKHS enables sharp approximation and recovery theorems, but a poor kernel may exclude the true value function altogether, and the strongest convergence results may require analyticity (Ehring et al., 8 Dec 2025). Skill-centric VFS removes hand-crafted predicates and often improves long-horizon reasoning, but its adequacy depends on the quality and coverage of the skill library, and the STL paper reports weaker behavior on chain tasks as horizons and trade-offs become harder (He et al., 2024).

The literature also exposes several unresolved technical issues. The stochastic low-rank RL papers do not provide explicit PAC or nonasymptotic convergence rates for the online factorized algorithms, and the tensor methods remain nonconvex (Rozada et al., 2021, Rozada et al., 2024). The verification-based RKHS theory does not cover viscosity solutions and yields only local identification under the weaker quadratic-bounds regime (Ehring et al., 8 Dec 2025). In the Wasserstein setting, the viscosity-solution theory does not yet include a comparison principle (Hynd et al., 2013). In convex-analytic value-function theory, properness alone is insufficient to guarantee lower semicontinuity or subdifferentiability (Zalinescu, 2022).

These limitations do not reduce the importance of VFS; they clarify its role. A value-function space is a design choice that encodes assumptions about structure, smoothness, reachability, or invariance. The effectiveness of that choice depends on how closely those assumptions match the world, the task family, and the learning dynamics.