Linear Tile-Coding for Value Function

• Linear tile-coding is a method that transforms continuous states into sparse, high-dimensional binary vectors for efficient value function approximation.
• Left-sided sketching applies dimensionality reduction only to the constraint side, ensuring unbiased estimation while preserving key curvature information.
• The approach significantly reduces computational costs and improves sample efficiency, underpinning robust RL algorithms like LSTD and ATD-L.

Linear tile-coding for value function approximation is a methodology within reinforcement learning (RL) for representing and estimating the value function of a policy, especially in high-dimensional continuous state spaces. This approach leverages tile coding to create sparse binary feature encodings suitable for linear approximators, and, when combined with left-sided random sketching techniques, yields powerful and computationally efficient solutions for policy evaluation with least-squares temporal difference learning (LSTD) and quasi-Newton accelerated TD methods. Recent theoretical and empirical findings have clarified the bias–variance trade-offs intrinsic to sketching, provided robust approaches for matrix-based learning, and led to practical guidelines for scaling RL with tile coding to thousands of features (Pan et al., 2017).

1. Standard Linear Value-Function Approximation with Tile Coding

Tile coding transforms each continuous state $s$ into a high-dimensional binary feature vector $\phi(s)\in\mathbb{R}^d$, with $d$ potentially in the thousands, where each feature represents the activation of a tile over the state space. The value function under policy $\pi$ is linearly approximated as $w^\top\phi(s) \approx v_\pi(s)$, with weight vector $w\in\mathbb{R}^d$ to be learned.

The LSTD($\lambda$) algorithm collects samples $(s_t, r_{t+1}, s_{t+1})$ and eligibility traces $e_t = \gamma_t\lambda e_{t-1} + \phi(s_t)$, then forms the normal equations $A w = b$, with

$\phi(s)\in\mathbb{R}^d$0

Batch solution $\phi(s)\in\mathbb{R}^d$1 has $\phi(s)\in\mathbb{R}^d$2 cost and maintaining $\phi(s)\in\mathbb{R}^d$3 incrementally by Sherman–Morrison costs $\phi(s)\in\mathbb{R}^d$4 per step. Stochastic TD($\phi(s)\in\mathbb{R}^d$5) is $\phi(s)\in\mathbb{R}^d$6 but less sample-efficient and sensitive to stepsize choices.

2. Limitations of Two-Sided (Feature) Sketching

A common sketching approach forms a random projection $\phi(s)\in\mathbb{R}^d$7 ($\phi(s)\in\mathbb{R}^d$8) and generates low-dimensional features $\phi(s)\in\mathbb{R}^d$9. The reduced system $d$0 is solved for $d$1. However, for tile-coded features, this introduces substantial bias:

• The projection $d$2 unpredictably folds coordinates, mixing tile activations.
• Though $d$3, the projected normal equations minimize the TD error in the sketched space, not the original, so the fixed point shifts.

Empirically, this yields high asymptotic error unless $d$4 approaches $d$5. The bias $d$6 only decays when $d$7 is nearly as large as $d$8, especially for sparse, discontinuous tile-coded features.

3. Left-Sided Sketching: Unbiased System Reduction

Left-side sketching applies the sketch $d$9 only to the constraint side, yielding the system $\pi$0. The true solution $\pi$1 to $\pi$2 also satisfies $\pi$3, so no bias is introduced; all solutions of $\pi$4 remain valid. In expectation, $\pi$5 preserves curvature information with cost for operations on $\pi$6 reduced from $\pi$7 to $\pi$8.

The system $\pi$9 is under-determined (many solutions), so the minimum-norm solution is preferred: $w^\top\phi(s) \approx v_\pi(s)$0 where $w^\top\phi(s) \approx v_\pi(s)$1 and $w^\top\phi(s) \approx v_\pi(s)$2.

4. Incremental Sketched-LSTD and Algorithmic Efficiency

The incremental update process maintains $w^\top\phi(s) \approx v_\pi(s)$3 and $w^\top\phi(s) \approx v_\pi(s)$4. Each sample induces a rank-one update: $w^\top\phi(s) \approx v_\pi(s)$5 yielding the recursive averages: $w^\top\phi(s) \approx v_\pi(s)$6 The minimum-norm solution employs the SVD $w^\top\phi(s) \approx v_\pi(s)$7 and is computed as $w^\top\phi(s) \approx v_\pi(s)$8, with incremental maintenance of $w^\top\phi(s) \approx v_\pi(s)$9 at $w\in\mathbb{R}^d$0 per sample and inversion at $w\in\mathbb{R}^d$1 when needed.

Each sample update costs $w\in\mathbb{R}^d$2, making left-sided sketched-LSTD feasible for $w\in\mathbb{R}^d$3 (e.g., $w\in\mathbb{R}^d$4, $w\in\mathbb{R}^d$5) and representing a substantial efficiency improvement over unsketched methods.

5. Variance Reduction via Quasi-Newton ATD-L Methods

Directly solving $w\in\mathbb{R}^d$6 is unbiased but susceptible to numerical instability and infrequent inversion of $w\in\mathbb{R}^d$7 matrices. The accelerated gradient TD update (ATD-L) leverages the sketched matrix as a preconditioner: $w\in\mathbb{R}^d$8 where $w\in\mathbb{R}^d$9 is the TD error and $\lambda$0 is a small regularizer. Under mild conditions (on $\lambda$1 and $\lambda$2), this iteration retains convergence to the unique LSTD solution $\lambda$3 without requiring full $\lambda$4 inversion.

ATD-L inherits the robustness of LSTD with respect to the trace parameter $\lambda$5 and regularization $\lambda$6, further lowering per-step computational overhead.

6. Bias–Variance Trade-Offs and Theoretical Guarantees

Two-sided sketching creates nonzero bias—$\lambda$7—decaying slowly unless $\lambda$8. Left-side sketching is unbiased in expectation, but variance is increased depending on $\lambda$9. The Johnson–Lindenstrauss lemma gives the bound: $(s_t, r_{t+1}, s_{t+1})$0 if $(s_t, r_{t+1}, s_{t+1})$1. Practically, increasing $(s_t, r_{t+1}, s_{t+1})$2 reduces variance and RMS error but increases computational time, with $(s_t, r_{t+1}, s_{t+1})$3 serving as an effective range for tile-coding $(s_t, r_{t+1}, s_{t+1})$4.

7. Empirical Results and Practical Recommendations

Experiments in Mountain Car, Puddle World, Acrobot, and Energy Allocation domains, with tile coding ($(s_t, r_{t+1}, s_{t+1})$5) and RBF features, demonstrate:

Method	Sample Efficiency	Bias (Tile Coding)	Typical Per-Step Runtime
Full LSTD (d×d)	Best, unaffected by λ	None	$(s_t, r_{t+1}, s_{t+1})$6 ($(s_t, r_{t+1}, s_{t+1})$7150 ms)
Two-sided sketch LSTD	Poor, high error	High (unless $(s_t, r_{t+1}, s_{t+1})$8)	$(s_t, r_{t+1}, s_{t+1})$9
Left-side sketched LSTD	Excellent	Unbiased	$e_t = \gamma_t\lambda e_{t-1} + \phi(s_t)$0 ($e_t = \gamma_t\lambda e_{t-1} + \phi(s_t)$11 ms)
ATD-L (Quasi-Newton)	Excellent, robust to λ	Unbiased	$e_t = \gamma_t\lambda e_{t-1} + \phi(s_t)$2 ($e_t = \gamma_t\lambda e_{t-1} + \phi(s_t)$30.5 ms)

In Mountain Car with $e_t = \gamma_t\lambda e_{t-1} + \phi(s_t)$4:

• Full LSTD yields RMS error $e_t = \gamma_t\lambda e_{t-1} + \phi(s_t)$50.05 in 2,000 steps at $e_t = \gamma_t\lambda e_{t-1} + \phi(s_t)$6150 ms/step.
• Two-sided sketching gives high bias, RMS error $e_t = \gamma_t\lambda e_{t-1} + \phi(s_t)$70.15.
• Left-sided LSTD attains RMS error $e_t = \gamma_t\lambda e_{t-1} + \phi(s_t)$80.05 at $e_t = \gamma_t\lambda e_{t-1} + \phi(s_t)$91 ms/step—a $A w = b$0150$A w = b$1 speedup.
• ATD-L achieves similar error and $A w = b$20.5 ms/step without matrix inversion.

Guidelines for practical implementation include:

• Choice of $A w = b$3: $A w = b$4 or $A w = b$5 for desired JL distortion; for $A w = b$6, $A w = b$7 is typical.
• Sketch type: Gaussian, CountSketch, Subsampled Hadamard behave similarly; Gaussian is simplest.
• Initialization: Set $A w = b$8 with small $A w = b$9 ($\phi(s)\in\mathbb{R}^d$00 to $\phi(s)\in\mathbb{R}^d$01), ensuring regularization and invertibility.
• Sensitivity: Left-sided LSTD and ATD-L are robust to $\phi(s)\in\mathbb{R}^d$02 and $\phi(s)\in\mathbb{R}^d$03, unlike TD($\phi(s)\in\mathbb{R}^d$04) which requires careful stepsize tuning.
• Sparse tile coding: Increase tilings or mix tiles to expand the effective visit subspace for $\phi(s)\in\mathbb{R}^d$05.

Employing only left-sided sketching within LSTD preserves unbiased estimation of $\phi(s)\in\mathbb{R}^d$06, lowers per-step cost from $\phi(s)\in\mathbb{R}^d$07 to $\phi(s)\in\mathbb{R}^d$08, and, with quasi-Newton ATD updates, provides a sample-efficient and robust policy evaluation framework for large-scale tile-coded representations (Pan et al., 2017).