Laplacian Keyboard (LK)

• Laplacian Keyboard (LK) is a hierarchical reinforcement learning framework that constructs task-agnostic options from Laplacian eigenvectors for zero-shot optimality within the eigenvector span.
• It builds a library of temporally extended policies using Universal Successor Feature Approximators and a meta-policy that composes these options for efficient policy adaptation.
• LK unifies spectral RL methods with hierarchical option learning, enabling scalable, reusable behavioral primitives validated on domains like the DeepMind Control Suite.

The Laplacian Keyboard (LK) is a hierarchical reinforcement learning (RL) framework that constructs a task-agnostic library of temporally extended options from the eigenvectors of the graph Laplacian induced by environment dynamics. LK guarantees zero-shot optimality for any reward within the linear span of learned Laplacian eigenvectors and enables efficient policy adaptation outside the span through meta-policy composition. This approach unifies spectral RL methods with hierarchical options, providing scalable, structured, and reusable behavioral primitives for offline RL and unsupervised pre-training contexts (Chandrasekar et al., 7 Feb 2026).

1. Foundations and Motivation

Reinforcement learning aims to discover a policy $\pi$ that maximizes the expected discounted return in a Markov Decision Process (MDP), defined as $M = (\mathscr S, \mathscr A, p, r, \gamma)$ with state space $\mathscr S$, action space $\mathscr A$, transition kernel $p(s'|s,a)$, reward $r:\mathscr S\to\mathbb R$, and discount factor $\gamma\in[0,1)$. In unsupervised RL, an agent collects a reward-free dataset $\mathscr D$ of transitions and must later solve downstream tasks specified by new reward functions $r$, leveraging pre-trained representations to reduce sample complexity during adaptation.

Laplacian eigenvectors serve as a powerful basis for approximating smooth reward functions on the transition graph. However, classical spectral RL restricts expressivity to the linear span of these eigenvectors and does not support compositional or hierarchical behavioral synthesis. The Laplacian Keyboard directly addresses these limitations, enabling generalization and rapid policy learning even beyond the linear span.

2. Construction of the Graph Laplacian and Its Span

Given a reversible policy–induced transition matrix,

$P(s,s') = \sum_{a\in\mathscr A}\pi(a|s) p(s'|s,a),$

an undirected graph with states as vertices and edge weights $M = (\mathscr S, \mathscr A, p, r, \gamma)$0 is formed. The degree matrix $M = (\mathscr S, \mathscr A, p, r, \gamma)$1 is defined with $M = (\mathscr S, \mathscr A, p, r, \gamma)$2, while the adjacency matrix $M = (\mathscr S, \mathscr A, p, r, \gamma)$3 has entries $M = (\mathscr S, \mathscr A, p, r, \gamma)$4. The (unnormalized) graph Laplacian is

$M = (\mathscr S, \mathscr A, p, r, \gamma)$5

which, due to Markov chain reversibility, is symmetric positive semidefinite and admits an eigen-decomposition:

$M = (\mathscr S, \mathscr A, p, r, \gamma)$6

with orthonormal eigenvectors $M = (\mathscr S, \mathscr A, p, r, \gamma)$7.

Stacking the first $M = (\mathscr S, \mathscr A, p, r, \gamma)$8 eigenvectors yields $M = (\mathscr S, \mathscr A, p, r, \gamma)$9. Every reward $\mathscr S$0 can be projected into this span via $\mathscr S$1, optimizing the squared error $\mathscr S$2. The graph-norm,

$\mathscr S$3

quantifies smoothness across the transition graph and underpins the regularity induced by the Laplacian eigenbasis.

3. Hierarchical Behavior Library and Options

LK extends the utility of spectral methods by constructing options rather than limiting itself to the linear span at test time. Each option corresponds to a temporally extended policy.

Universal Successor Feature Approximators

The Laplacian representation for a state is $\mathscr S$4. With a parameterized linear reward family,

$\mathscr S$5

LK trains a Universal Successor Feature Approximator (USFA), comprising:

• A policy $\mathscr S$6 conditioned on $\mathscr S$7.
• A successor-feature estimator $\mathscr S$8, trained using off-policy TD updates enforcing Bellman consistency,

$\mathscr S$9

The optimal $\mathscr A$0-value for $\mathscr A$1 is $\mathscr A$2, and $\mathscr A$3 yields the optimal policy for $\mathscr A$4.

Option Construction

Each $\mathscr A$5 defines an option $\mathscr A$6:

• Initiation set $\mathscr A$7 (all states).
• Intra-option policy $\mathscr A$8.
• Termination after a fixed horizon $\mathscr A$9 or according to a learned termination function $p(s'|s,a)$0.

Canonical options ($p(s'|s,a)$1) align with individual eigenvector directions; continuous $p(s'|s,a)$2 allow interpolation in the option-space.

4. Meta-policy for Compositional Policy Synthesis

Adapting to a new reward, possibly outside the Laplacian span, requires the meta-policy $p(s'|s,a)$3. At each decision epoch:

1. State $p(s'|s,a)$4 is observed.
2. $p(s'|s,a)$5 is produced.
3. Option $p(s'|s,a)$6 is executed for $p(s'|s,a)$7 steps.
4. The process repeats until episode termination.

The meta-policy maximizes

$p(s'|s,a)$8

where $p(s'|s,a)$9 is the option's cumulative reward. Off-policy RL (e.g., TD3) is applied at the option-level, with critic $r:\mathscr S\to\mathbb R$0 and actor updates computed via standard policy gradients.

5. Theoretical Guarantees and Approximation Bounds

When the downstream reward $r:\mathscr S\to\mathbb R$1 is exactly within the Laplacian span ($r:\mathscr S\to\mathbb R$2), USFA retrieves the optimal policy immediately in a zero-shot fashion. For general $r:\mathscr S\to\mathbb R$3, projection yields $r:\mathscr S\to\mathbb R$4, with the corresponding optimal values $r:\mathscr S\to\mathbb R$5 and $r:\mathscr S\to\mathbb R$6 satisfying

$r:\mathscr S\to\mathbb R$7

where $r:\mathscr S\to\mathbb R$8 is the $r:\mathscr S\to\mathbb R$9-th smallest Laplacian eigenvalue. This establishes a uniform bound on the value error from projection. LK's meta-policy further composes options to reduce suboptimality, ensuring adaptivity even outside the initial span.

6. Algorithmic Workflow

Pre-Training Phase

1. The first $\gamma\in[0,1)$0 Laplacian eigenvectors are learned using contrastive objectives enforcing smoothness and orthogonality in representation.
2. The USFA (actor and critic) is trained on transitions, with random or goal-based $\gamma\in[0,1)$1 sampled, enforcing temporal-difference consistency of successor features.

Downstream Adaptation

• For a new reward $\gamma\in[0,1)$2, zero-shot initialization estimates $\gamma\in[0,1)$3 via linear regression: $\gamma\in[0,1)$4, yielding a policy $\gamma\in[0,1)$5.
• The meta-policy then iteratively outputs $\gamma\in[0,1)$6, invokes the corresponding option, collects rewards, and updates via off-policy RL at the option-level.

7. Empirical Evaluation and Comparative Analysis

The Laplacian Keyboard is validated on DeepMind Control Suite (DMC) domains (Cheetah, Quadruped, Walker) across multiple tasks (Run, Run-B, Walk, Walk-B) with offline datasets (APS, Proto, RND). Baselines include Forward–Backward (FB) zero-shot, Flat TD3, and Option Keyboard Basis (OKB).

Zero-Shot and Composition Results

With $\gamma\in[0,1)$7, LK achieves parity with FB on zero-shot returns and exceeds 75% of the TD3 oracle on 8/12 tasks:

Domain (Task)	FB	LK	Flat TD3
Cheetah (avg)	552 ± –	450 ± –	786 ± –
...	...	...	...

After 200K interactions, the LK meta-policy uniformly outperforms its zero-shot baseline (e.g., +1296% at $\gamma\in[0,1)$8 on Cheetah), converging rapidly ($\gamma\in[0,1)$92 million steps). Gains decline with larger $\mathscr D$0 but remain positive, indicating that even small Laplacian bases are effectively composable.

Sample Efficiency and Generalization

On Walker–Walk (APS), LK (with $\mathscr D$1) surpasses Flat TD3 in fewer than 100K steps, demonstrating substantial sample efficiency in a high-dimensional option space. In Item-Collector gridworlds with handcrafted OKB reward features, LK nearly matches privileged performance after 500K steps, highlighting the sufficiency of purely dynamics-derived Laplacian behaviors for generalization.

8. Synthesis and Significance

The Laplacian Keyboard offers a behavioral foundation model for RL that is zero-shot optimal within the Laplacian span and capable of hierarchical adaptation beyond it. By combining spectral representation, option learning, and meta-policy composition, LK addresses both sample efficiency and behavioral generalization in offline and unsupervised RL. Its theoretical error bounds and simple off-policy recipe suggest applicability to broad RL domains requiring structured, reusable behaviors (Chandrasekar et al., 7 Feb 2026).