---
title: Linear Tile-Coding for Value Function
url: https://www.emergentmind.com/topics/linear-tile-coding-for-value-function
type: topic
---

# Linear Tile-Coding for Value Function

Linear tile-coding for value function approximation is a methodology within reinforcement learning (RL) for representing and estimating the value function of a policy, especially in high-dimensional continuous state spaces. This approach leverages tile coding to create sparse binary feature encodings suitable for linear approximators, and, when combined with left-sided random sketching techniques, yields powerful and computationally efficient solutions for policy evaluation with least-squares temporal difference learning (LSTD) and quasi-Newton accelerated TD methods. Recent theoretical and empirical findings have clarified the bias–variance trade-offs intrinsic to sketching, provided robust approaches for matrix-based learning, and led to practical guidelines for scaling RL with tile coding to thousands of features [1708.01298].

## 1. Standard Linear Value-Function Approximation with Tile Coding

Tile coding transforms each continuous state $s$ into a high-dimensional binary feature vector $\phi(s)\in\mathbb{R}^d$, with $d$ potentially in the thousands, where each feature represents the activation of a tile over the state space. The value function under policy $\pi$ is linearly approximated as $w^\top\phi(s) \approx v_\pi(s)$, with weight vector $w\in\mathbb{R}^d$ to be learned.

The LSTD($\lambda$) algorithm collects samples $(s_t, r_{t+1}, s_{t+1})$ and eligibility traces $e_t = \gamma_t\lambda e_{t-1} + \phi(s_t)$, then forms the normal equations $A w = b$, with
\[
A = \sum_t e_t [\phi(s_t) - \gamma_{t+1}\phi(s_{t+1})]^\top, \qquad b = \sum_t e_t r_{t+1}.
\]
Batch solution $w = A^{-1}b$ has $O(d^3)$ cost and maintaining $A^{-1}$ incrementally by Sherman–Morrison costs $O(d^2)$ per step. Stochastic TD($\lambda$) is $O(d)$ but less sample-efficient and sensitive to stepsize choices.

## 2. Limitations of Two-Sided (Feature) Sketching

A common sketching approach forms a random projection $S\in\mathbb{R}^{k\times d}$ ($k\ll d$) and generates low-dimensional features $\psi(s)=S\phi(s)$. The reduced system $(SAS^\top)\hat{w}=Sb$ is solved for $\hat{w}\in\mathbb{R}^k$. However, for tile-coded features, this introduces substantial bias:
- The projection $S\phi(s)$ unpredictably folds coordinates, mixing tile activations.
- Though $\mathbb{E}[S^\top S]=I$, the projected normal equations minimize the TD error in the sketched space, not the original, so the fixed point shifts.

Empirically, this yields high asymptotic error unless $k$ approaches $d$. The bias $\| \hat{w} - w^\star \|$ only decays when $k$ is nearly as large as $d$, especially for sparse, discontinuous tile-coded features.

## 3. Left-Sided Sketching: Unbiased System Reduction

Left-side sketching applies the sketch $S\in\mathbb{R}^{k\times d}$ only to the constraint side, yielding the system $SAw \approx Sb$. The true solution $w^\star$ to $Aw=b$ also satisfies $SAw^\star=Sb$, so no bias is introduced; all solutions of $Aw=b$ remain valid. In expectation, $S^\top S\approx I$ preserves curvature information with cost for operations on $A$ reduced from $O(d^2)$ to $O(kd)$.

The system $SAw=Sb$ is under-determined (many solutions), so the minimum-norm solution is preferred:
\[
w = \tilde{A}^\top (\tilde{A}\tilde{A}^\top)^{-1}\tilde{b},
\]
where $\tilde{A}=SA$ and $\tilde{b}=Sb$.

## 4. Incremental Sketched-LSTD and Algorithmic Efficiency

The incremental update process maintains $\tilde{A}_t = S A_t \in \mathbb{R}^{k\times d}$ and $\tilde{b}_t = S b_t \in \mathbb{R}^k$. Each sample induces a rank-one update:
\[
\Delta A_t = e_t [\phi(s_t)-\gamma\phi(s_{t+1})]^\top, \quad \Delta b_t = e_t r_{t+1},
\]
yielding the recursive averages:
\[
\tilde{A}_{t+1} = \tilde{A}_t + \frac{1}{t+1}(S\Delta A_t - \tilde{A}_t), \quad
\tilde{b}_{t+1} = \tilde{b}_t + \frac{1}{t+1}(S\Delta b_t - \tilde{b}_t).
\]
The minimum-norm solution employs the SVD $\tilde{A}_t=U\Sigma V^\top$ and is computed as $w_t = V\Sigma^{-1}U^\top \tilde{b}_t = \tilde{A}_t^\top (\tilde{A}_t\tilde{A}_t^\top)^{-1}\tilde{b}_t$, with incremental maintenance of $M_t=\tilde{A}_t\tilde{A}_t^\top$ at $O(k^2)$ per sample and inversion at $O(k^3)$ when needed.

Each sample update costs $O(dk + k^3)$, making left-sided sketched-LSTD feasible for $k\ll d$ (e.g., $k=50$, $d\approx10^3$) and representing a substantial efficiency improvement over unsketched methods.

## 5. Variance Reduction via Quasi-Newton ATD-L Methods

Directly solving $w_t = \tilde{A}_t^\top M_t^{-1}\tilde{b}_t$ is unbiased but susceptible to numerical instability and infrequent inversion of $k \times k$ matrices. The accelerated gradient TD update (ATD-L) leverages the sketched matrix as a preconditioner:
\[
w_{t+1} = w_t + [\alpha \tilde{A}_t^\top M_t^{-1} + \eta I] \delta_t e_t,
\]
where $\delta_t$ is the TD error and $\eta$ is a small regularizer. Under mild conditions (on $A$ and $\eta$), this iteration retains convergence to the unique LSTD solution $w^\star=A^{-1}b$ without requiring full $d \times d$ inversion.

ATD-L inherits the robustness of LSTD with respect to the trace parameter $\lambda$ and regularization $\eta$, further lowering per-step computational overhead.

## 6. Bias–Variance Trade-Offs and Theoretical Guarantees

Two-sided sketching creates nonzero bias—$\|\hat{w} - w^\star\|$—decaying slowly unless $k \sim d$. Left-side sketching is unbiased in expectation, but variance is increased depending on $\|S^\top S - I\|$. The Johnson–Lindenstrauss lemma gives the bound:
\[
P[\|S^\top S - I\|_2 \geq \epsilon] \leq \delta,
\]
if $k = O(\epsilon^{-2}\log(1/\delta))$. Practically, increasing $k$ reduces variance and RMS error but increases computational time, with $k=30\ldots100$ serving as an effective range for tile-coding $d \approx 1,000$.

## 7. Empirical Results and Practical Recommendations

Experiments in Mountain Car, Puddle World, Acrobot, and Energy Allocation domains, with tile coding ($d\approx1,024 \ldots 8,192$) and RBF features, demonstrate:

| Method                      | Sample Efficiency         | Bias (Tile Coding)         | Typical Per-Step Runtime |
|-----------------------------|--------------------------|----------------------------|-------------------------|
| Full LSTD (d×d)             | Best, unaffected by λ    | None                       | $O(d^2)$ ($\sim$150 ms) |
| Two-sided sketch LSTD       | Poor, high error         | High (unless $k\approx d$) | $O(dk+k^2)$             |
| Left-side sketched LSTD     | Excellent                | Unbiased                   | $O(dk)$ ($\sim$1 ms)    |
| ATD-L (Quasi-Newton)        | Excellent, robust to λ   | Unbiased                   | $O(dk)$ ($\sim$0.5 ms)  |

In Mountain Car with $d=1,024,\ k=50$:
- Full LSTD yields RMS error $\approx$0.05 in 2,000 steps at $\sim$150 ms/step.
- Two-sided sketching gives high bias, RMS error $\approx$0.15.
- Left-sided LSTD attains RMS error $\approx$0.05 at $\sim$1 ms/step—a $\sim$150$\times$ speedup.
- ATD-L achieves similar error and $\sim$0.5 ms/step without matrix inversion.

Guidelines for practical implementation include:
- Choice of $k$: $k\approx\sqrt{d}$ or $O(1/\epsilon^2\log d)$ for desired JL distortion; for $d\approx1,000$, $k=30\ldots100$ is typical.
- Sketch type: Gaussian, CountSketch, Subsampled Hadamard behave similarly; Gaussian is simplest.
- Initialization: Set $M_0 = \eta I_k$ with small $\eta$ ($10^{-3}$ to $10^{-1}$), ensuring regularization and invertibility.
- Sensitivity: Left-sided LSTD and ATD-L are robust to $\lambda$ and $\eta$, unlike TD($\lambda$) which requires careful stepsize tuning.
- Sparse tile coding: Increase tilings or mix tiles to expand the effective visit subspace for $S$.

Employing only left-sided sketching within LSTD preserves unbiased estimation of $w^\star$, lowers per-step cost from $O(d^2)$ to $O(dk)$, and, with quasi-Newton ATD updates, provides a sample-efficient and robust policy evaluation framework for large-scale tile-coded representations [1708.01298].

Source: https://www.emergentmind.com/topics/linear-tile-coding-for-value-function