Kernelized TD Critic in RKHS

• Kernelized TD critic is a policy evaluation method that represents value functions in reproducing kernel Hilbert spaces, enabling Bellman residual minimization.
• It employs iterative functional gradient descent and regularized optimization to compute Q functions from one-step state-action transitions.
• The approach supports online sparse updates and attention mechanisms, extending traditional TD methods with enhanced convergence and efficiency.

Kernelized temporal-difference (TD) critic denotes a class of critic constructions in which the value function or action-value function is represented in a reproducing kernel Hilbert space (RKHS), and Bellman-based policy-evaluation updates are carried out in function space rather than in a fixed finite-dimensional parametric family. In the recent RKHS formulation, the critic estimates $Q^\pi\in\mathcal H(\mathcal S\times\mathcal A)$ from one-step state-action transitions by solving a regularized empirical squared Bellman-residual problem and approximating that solution with kernel gradient descent; related work also develops online sparse kernel critics, attentive kernel features, and projected-Bellman schemes whose linear updates depend only on inner products and therefore admit kernelization by replacing feature inner products with kernel evaluations (Zou et al., 29 Sep 2025).

1. Formal setting and function-space viewpoint

A kernelized TD critic is typically defined on a discounted Markov decision process $(\mathcal S,\mathcal A,P,r,\gamma)$, with either state-value $V^\pi$ or action-value $Q^\pi$ as the critic target. In the RKHS policy-evaluation setting, the Bellman evaluation operator for a fixed policy $\pi$ is

$$[\mathcal T^\pi Q](s,a)=r(s,a)+\gamma\,\mathbb E_{s'\sim P(\cdot\mid s,a),\,a'\sim\pi(\cdot\mid s')}\big[Q(s',a')\big],$$

and $Q^\pi$ is its fixed point. The recent RKHS analysis assumes compact, convex state and action spaces, bounded reward, and a bounded positive-definite kernel $K$ on $(\mathcal S\times\mathcal A)^2$. The critic lives in an RKHS $\mathcal H$, with finite-span subspace

$(\mathcal S,\mathcal A,P,r,\gamma)$0

where $(\mathcal S,\mathcal A,P,r,\gamma)$1 (Zou et al., 29 Sep 2025).

The sampling model used in that formulation is one-step and i.i.d. within each evaluation phase: $(\mathcal S,\mathcal A,P,r,\gamma)$2 This design is central to the modern notion of kernelized TD critic: the critic update uses only one-step transition samples, rather than multi-step returns or generalized advantage estimation. The same paper imposes boundedness assumptions on the stationary and initial state-action distributions and a one-step distribution-shift condition $(\mathcal S,\mathcal A,P,r,\gamma)$3, which enters the statistical rates (Zou et al., 29 Sep 2025).

A broader historical perspective includes state-value critics with linear approximation $(\mathcal S,\mathcal A,P,r,\gamma)$4 and nonlinear critics $(\mathcal S,\mathcal A,P,r,\gamma)$5 in Markov decision process policy evaluation. In that line of work, the critic is the value or Q approximator, and the kernelized case arises by replacing explicit feature maps with RKHS features $(\mathcal S,\mathcal A,P,r,\gamma)$6, so that inner products and prediction differences are expressed entirely through kernel evaluations (Banerjee et al., 2022).

2. Bellman objectives, projection, and RKHS representation

The canonical RKHS estimator is defined by a regularized empirical squared Bellman-residual problem: $(\mathcal S,\mathcal A,P,r,\gamma)$7 This is a temporal-difference version of kernel ridge regression: the target contains the critic’s own next-state prediction, so the estimator is defined by a fixed-point equation rather than by ordinary supervised regression. The same formulation is described as an empirical projected Bellman equation in $(\mathcal S,\mathcal A,P,r,\gamma)$8 (Zou et al., 29 Sep 2025).

By the representer theorem, the solution has the finite kernel expansion

$(\mathcal S,\mathcal A,P,r,\gamma)$9

with

$V^\pi$0

Naively solving this system requires dense $V^\pi$1 inversion, hence $V^\pi$2 cost per policy, which motivates iterative kernel TD procedures (Zou et al., 29 Sep 2025).

A second objective family is organized around the projected Bellman error. In the linear policy-evaluation setting with invariant-distribution weighting $V^\pi$3, the orthogonal projection onto the approximation space is

$V^\pi$4

and the mean squared projected Bellman error is

$V^\pi$5

This differs from the mean squared Bellman error

$V^\pi$6

The distinction is not merely terminological: residual algorithms minimize MSBE, whereas gradient-TD methods such as GTD2 are designed around MSPBE, and the cooperative-networks formulation explicitly aims to track the projected Bellman iterate rather than the raw residual (Banerjee et al., 2022).

The cooperative-networks paper does not formulate its method in RKHS language, but it explicitly notes that its linear updates depend only on feature inner products and prediction differences. In that presentation, replacing $V^\pi$7 by $V^\pi$8 and feature inner products by kernel evaluations yields a kernelized MSPBE-tracking critic. This suggests that kernelized TD critics occupy two closely related objective regimes: regularized Bellman-residual minimization in $V^\pi$9, and projected-Bellman tracking realized through kernelized alternating regression (Banerjee et al., 2022).

3. Optimization mechanisms and gradient structure

The principal iterative RKHS update is functional gradient descent. For a candidate critic $Q^\pi$0, define the empirical TD residual

$Q^\pi$1

and the regularized objective $Q^\pi$2, where

$Q^\pi$3

The kernel TD update is

$Q^\pi$4

In coefficient form, if $Q^\pi$5, then

$Q^\pi$6

The paper interprets this as semi-gradient TD(0)-like in coefficient space, but with RKHS geometry rather than standard Euclidean geometry (Zou et al., 29 Sep 2025).

Optimization and statistics are separated in the corresponding analysis. With $Q^\pi$7, $Q^\pi$8, and $Q^\pi$9, the optimization error decays geometrically and $\pi$0 iterations suffice for the iterative critic $\pi$1 to inherit the same statistical rate as the KRR-TD solution $\pi$2, provided the spectral radius condition is satisfied (Zou et al., 29 Sep 2025).

A common misconception is that all “gradient TD” procedures optimize the same objective. The literature summarized here separates at least four cases. Standard TD(0) is semi-gradient and does not implement true gradient descent on an explicit MSPBE objective. Residual algorithms are true gradients on MSBE. GTD2 minimizes MSPBE through two-time-scale stochastic approximation and an auxiliary weight vector. Cooperative networks instead realize projected-Bellman tracking by alternating two least-squares fits: one approximator fits Bellman targets, the other fits the projection target produced by the first (Banerjee et al., 2022).

That cooperative scheme is written, in the nonlinear case, as two squared losses on each transition $\pi$3: $\pi$4 Because each subproblem is an ordinary differentiable regression step, the same construction is described as compatible with RKHS parameterizations. More recent work on Gradient Iterated TD extends the “true gradient through moving targets” idea to a sequence of critics minimizing a sum of Bellman errors, and explicitly remarks that linear-in-parameters kernel models are especially natural because gradients reduce to feature vectors or kernel evaluations (Vincent et al., 8 Mar 2026).

4. Sparsity, online dictionaries, and attentive kernel critics

A kernelized TD critic need not be batch-based. The fully online line of work represents the value function as a finite dictionary expansion

$\pi$5

and constructs the dictionary online by a Modified Novelty Criterion (MNC). At time $\pi$6, novelty is

$\pi$7

which is computed by the kernel trick as

$\pi$8

For normalized Gaussian kernels this becomes $\pi$9, and a new center is added only when $$[\mathcal T^\pi Q](s,a)=r(s,a)+\gamma\,\mathbb E_{s'\sim P(\cdot\mid s,a),\,a'\sim\pi(\cdot\mid s')}\big[Q(s',a')\big],$$0 (Yang et al., 2022).

The attentive extension replaces raw kernel features with attention-modulated features: $$[\mathcal T^\pi Q](s,a)=r(s,a)+\gamma\,\mathbb E_{s'\sim P(\cdot\mid s,a),\,a'\sim\pi(\cdot\mid s')}\big[Q(s',a')\big],$$1 where the attention weights are

$$[\mathcal T^\pi Q](s,a)=r(s,a)+\gamma\,\mathbb E_{s'\sim P(\cdot\mid s,a),\,a'\sim\pi(\cdot\mid s')}\big[Q(s',a')\big],$$2

The resulting feature vector $$[\mathcal T^\pi Q](s,a)=r(s,a)+\gamma\,\mathbb E_{s'\sim P(\cdot\mid s,a),\,a'\sim\pi(\cdot\mid s')}\big[Q(s',a')\big],$$3 is sparse in an effective, smooth sense: only centers with simultaneously significant kernel value and significant attention weight contribute materially (Yang et al., 2022).

OAKTD combines this representation with a two-time-scale TD scheme. The fast parameter $$[\mathcal T^\pi Q](s,a)=r(s,a)+\gamma\,\mathbb E_{s'\sim P(\cdot\mid s,a),\,a'\sim\pi(\cdot\mid s')}\big[Q(s',a')\big],$$4 is updated by semi-gradient TD on the attentive features,

$$[\mathcal T^\pi Q](s,a)=r(s,a)+\gamma\,\mathbb E_{s'\sim P(\cdot\mid s,a),\,a'\sim\pi(\cdot\mid s')}\big[Q(s',a')\big],$$5

while slow parameters $$[\mathcal T^\pi Q](s,a)=r(s,a)+\gamma\,\mathbb E_{s'\sim P(\cdot\mid s,a),\,a'\sim\pi(\cdot\mid s')}\big[Q(s',a')\big],$$6 and $$[\mathcal T^\pi Q](s,a)=r(s,a)+\gamma\,\mathbb E_{s'\sim P(\cdot\mid s,a),\,a'\sim\pi(\cdot\mid s')}\big[Q(s',a')\big],$$7 are updated by residual-gradient recursions. The step sizes satisfy the standard conditions $$[\mathcal T^\pi Q](s,a)=r(s,a)+\gamma\,\mathbb E_{s'\sim P(\cdot\mid s,a),\,a'\sim\pi(\cdot\mid s')}\big[Q(s',a')\big],$$8, $$[\mathcal T^\pi Q](s,a)=r(s,a)+\gamma\,\mathbb E_{s'\sim P(\cdot\mid s,a),\,a'\sim\pi(\cdot\mid s')}\big[Q(s',a')\big],$$9, $Q^\pi$0, and $Q^\pi$1. Under a Lipschitz Mercer kernel on compact state space, MNC yields a finite stable dictionary $Q^\pi$2; under ergodicity and fixed dictionary, the slow iterates converge almost surely to asymptotically stable equilibria and the fast iterates converge almost surely to a projected Bellman fixed point (Yang et al., 2022).

This online formulation shows that a kernelized TD critic is not restricted to dense kernel-matrix solvers. It can instead be realized as a sparse, adaptive, fully online critic with explicit control of dictionary growth, attention-based locality, and convergence to a projected Bellman solution.

5. Sample complexity, RKHS regimes, and policy-optimization coupling

The recent RKHS analysis gives non-asymptotic rates for the kernelized TD critic that depend on RKHS entropy. If the unit ball $Q^\pi$3 satisfies

$Q^\pi$4

then, with the theorem’s choice of regularization, the empirical $Q^\pi$5 error of the KRR-TD estimator scales as

$Q^\pi$6

up to probabilistic notation. The same framework yields instance-adaptive population rates for several concrete RKHS regimes: essentially $Q^\pi$7 in the tabular case, $Q^\pi$8 for Sobolev kernels, $Q^\pi$9 for the NTK regime considered in the paper, and $K$0 for Gaussian kernels (Zou et al., 29 Sep 2025).

The same work embeds the critic in a KL-regularized proximal policy update. If

$K$1

then after critic evaluation the policy is updated by

$K$2

equivalently $K$3. On sampled states this is the optimizer of an empirical KL-regularized proximal problem, making the kernelized TD critic the evaluation stage in a PPO/TRPO-style loop formulated in function space. With $K$4 and sample sizes chosen so that $K$5, the resulting policy suboptimality attains the optimal stochastic rate $K$6 (Zou et al., 29 Sep 2025).

The experimental instantiation uses a two-layer ReLU MLP in the NTK regime, no GAE, and one-step TD loss

$K$7

The schedule $K$8 is reported as robust on CartPole-v1 and Acrobot-v1, whereas $K$9 becomes unstable and $(\mathcal S\times\mathcal A)^2$0 stagnates. On CartPole, the NPG+TD method processes approximately $(\mathcal S\times\mathcal A)^2$1 state-action pairs per second versus PPO’s $(\mathcal S\times\mathcal A)^2$2, about $(\mathcal S\times\mathcal A)^2$3 more efficient, which the paper attributes to one-step TD requiring neither full trajectories nor GAE recursion (Zou et al., 29 Sep 2025).

6. Related extensions, misconceptions, and limitations

Several adjacent developments refine what a kernelized TD critic can target. Multi-State TD replaces the one-step target by the uniform average of $(\mathcal S\times\mathcal A)^2$4-step targets for $(\mathcal S\times\mathcal A)^2$5: $(\mathcal S\times\mathcal A)^2$6 That work explicitly notes that, in a kernel TD or kernel fitted-Q framework, the one-step Bellman target can be replaced by this multi-state target, with action-loaded and action-generated replay modes furnishing two different ways to define future actions in off-policy control (Wang et al., 2024).

Taylor TD-learning attacks a different deficiency: high-variance TD updates in continuous state-action spaces. It replaces sampled local expectations over state or action noise with first-order Taylor approximations, producing additional gradient-alignment terms such as

$(\mathcal S\times\mathcal A)^2$7

The paper’s synthesis explicitly states that a “kernelized Taylor TD” critic is viable when the kernel is differentiable in its arguments, as with Gaussian RBF or Matérn kernels, because $(\mathcal S\times\mathcal A)^2$8 and $(\mathcal S\times\mathcal A)^2$9 then become analytic sums of kernel derivatives (Garibbo et al., 2023).

Gradient Iterated TD adds yet another axis: learning a sequence $\mathcal H$0 that approximates successive Bellman iterates and minimizes a sum of Bellman errors through auxiliary residual networks $\mathcal H$1. Its derivation is parameterization-agnostic and the paper explicitly states that kernel models are a natural fit because they are linear in parameters and their gradients reduce to feature vectors or kernel evaluations (Vincent et al., 8 Mar 2026).

These developments clarify several recurring misconceptions. A kernelized TD critic is not synonymous with kernel least-squares TD; it may be implemented by direct matrix inversion, by functional gradient descent, by sparse online dictionary updates, or by alternating projected-Bellman regressions. Nor is every such critic optimizing the same error: MSBE, MSPBE, projected Bellman equations, multi-state Bellman averages, and Taylor-expanded local expectations are distinct constructions with different stability and bias-variance profiles.

The limitations are correspondingly varied. Pure kernel implementations have $\mathcal H$2 memory or time costs unless approximations such as Nyström, random features, budgeted kernel methods, or NTK-style neural implementations are used (Zou et al., 29 Sep 2025). OAKTD remains sensitive to $\mathcal H$3, $\mathcal H$4, $\mathcal H$5, and $\mathcal H$6, and its convergence analysis is on-policy and assumes dictionary stabilization before parameter convergence (Yang et al., 2022). Taylor TD requires differentiable reward and transition models and incurs extra derivative computation (Garibbo et al., 2023). The RKHS sample-complexity theory assumes bounded distributions and RKHS realizability conditions that may be strong in complex environments (Zou et al., 29 Sep 2025).

Taken together, the literature defines the kernelized TD critic not as a single algorithm but as a family of Bellman-based critic constructions in RKHSs: some emphasize explicit regularized residual minimization, some projected-Bellman tracking, some online sparsity, and some low-variance or multi-head gradient corrections. The unifying principle is that temporal-difference policy evaluation is lifted from finite-dimensional parameter space to kernel-induced function space, where approximation geometry, sparsity, and statistical complexity become first-class design variables.