---
title: Eigenoptions in Reinforcement Learning
url: https://www.emergentmind.com/topics/eigenoptions
type: topic
---

# Eigenoptions in Reinforcement Learning

An eigenoption is a temporally extended action in reinforcement learning (RL) derived from the spectral analysis of an environment’s state–transition graph. The core motivation is to automate option discovery by exploiting global connectivity properties of Markov decision processes (MDPs) through proto-value functions (PVFs), the eigenvectors of a graph Laplacian or successor representation (SR). Each eigenoption consists of an intrinsic reward function that promotes exploration along a principal “diffusion mode” of the state space, independent of extrinsic rewards. Eigenoptions are particularly efficacious in sparse-reward and high-dimensional domains, capturing environment geometry and enabling hierarchical abstraction, representation learning, and scalable exploration.

## 1. Construction of Eigenoptions via Graph Spectral Methods

Let $G=(\mathcal{S},\mathcal{E})$ be the undirected state–transition graph of an MDP, with adjacency matrix $A$ (possibly weighted) and degree matrix $D$. The (combinatorial) Laplacian is $L = D - A$, or its symmetric normalization $L_{\mathrm{norm}}=I - D^{-1/2} A D^{-1/2}$ is used for improved numerical properties [1703.00956][2403.10855][1712.04065][2507.09127]. The eigenvalue problem
$$
L\,\phi_i = \lambda_i\,D\,\phi_i\,,
$$
defines a sequence of smooth PVFs $\{\phi_i\}$ over $\mathcal{S}$, ordered by increasing $\lambda_i$. These eigenvectors capture large-scale structural features of the state space.

Each PVF $\phi_i$ generates an intrinsic reward ("eigenpurpose"):
$$
r_i(s,s') = \phi_i(s') - \phi_i(s) \,.
$$
This reward is maximized by policies that move the agent “uphill” along diffusion mode $i$.

The corresponding eigenoption $\omega_i$ is defined as a triple $\langle I_i, \pi_i, \beta_i \rangle$ where:
- $I_i = \{ s \in \mathcal{S} \mid \exists a\in \mathcal{A}: q^*_i(s,a) > 0 \}$ (states where action upwards in $\phi_i$ is possible)
- $\pi_i(a\mid s) = \arg\max_{a\in \mathcal{A}\cup\{\bot\}} q^*_i(s,a)$ (intra-option policy maximizing cumulative $r_i$)
- $\beta_i(s) = 1$ iff $q^*_i(s,a) \le 0$ for all $a$ (termination at local maxima of $\phi_i$)

Option-value functions $q^*_i$ are computed using standard RL techniques on the SMDP induced by $r_i$ [1703.00956][2403.10855][1712.04065].

## 2. The Successor Representation Perspective and Equivalence

The successor representation under policy $\pi$ is $M^\pi = (I-\gamma P_\pi)^{-1}$, where $P_\pi$ is the state-transition matrix and $\gamma$ the discount factor [2110.05740][1710.11089][2507.09127]. Its spectral decomposition
$$
M^\pi e_k = \lambda_k e_k
$$
connects SR eigenvectors $e_k$ to PVFs. Under mild conditions (e.g., uniform policy, reversibility), the eigenvectors of $L$ and $M^\pi$ coincide up to scaling and ordering. This allows eigenoptions to be derived either from the graph Laplacian or from the SR, and facilitates sample-based or function-approximate discovery in large or continuous state spaces [2110.05740][1710.11089].

In feature-based or deep learning settings, the intrinsic reward generalizes as
$$
r^e(s,s')= e^\top (\phi(s') - \phi(s))
$$
where $e$ is an SR eigenvector and $\phi$ a learned representation [1710.11089].

## 3. Algorithmic Frameworks and Discovery Procedures

### Tabular/Offline Discovery

1. Estimate the adjacency or transition matrix from data.
2. Construct $L$ or $M^\pi$ and extract the top $k$ nontrivial eigenvectors (excluding the trivial constant mode).
3. For each $\phi_i$, define $r_i(s,s')$ as above.
4. For each $i$, learn the optimal intra-option policy $\pi_i$ (e.g., by Q-learning on $r_i$), determine initiation and termination sets from the optimal $Q$-function.
5. Aggregate eigenoptions as macro-actions in the agent’s action set [1703.00956][1712.04065][2110.05740][2507.09127].

### Sample-Based and Deep RL Approaches

- For large or continuous domains, transition samples are collected to build an incidence or feature-difference matrix. The top $k$ right singular vectors (via SVD) yield approximate PVFs.
- In the deep SR approach, a neural network encoder learns $\psi(\phi(s))$ (estimated SF) via temporal difference learning, possibly from pixel observations [1710.11089].
- Online methods alternate between updating SR (or Laplacian estimation), extracting new eigenvectors, defining new options, and learning their policies in a “representation–option discovery” (ROD) loop [2507.09127].

### Eigenoption-Critic (EOC)

EOC interleaves eigenoption discovery and control within the Option-Critic framework, blending intrinsic and extrinsic rewards:
$$
r(s,a,o) = \alpha\,r_{\rm in}^{(o)}(s,s') + (1-\alpha)\,r_{ex}(s,a)
$$
with online updates of intra-option policies, termination conditions, and Nyström extensions for continuous spaces [1712.04065].

## 4. Properties, Theoretical Guarantees, and Empirical Effects

### Geometric and Exploratory Properties

- **Principal direction skills**: Each eigenoption $\omega_i$ pushes the agent along a principal axis of the state space, as defined by PVF $\phi_i$. Lower-frequency (smooth) eigenvectors traverse wide regions and bottlenecks.
- **Temporal scales**: The spectral index (or eigenvalue $\lambda_i$) determines the typical option duration: lower $\lambda_i$ yields longer, smoother behaviors.
- **Task-independence**: Eigenoptions' definition is decoupled from environment rewards, promoting uninformed exploration especially in sparse/reward-agnostic settings [1703.00956][2110.05740][2507.09127].

### Theoretical Guarantees

- Eigenoptions capture global graph connectivity and are guaranteed to terminate (since their reward is potential-based) in any ergodic finite MDP [2110.05740].
- Spectral invariance: The option set is invariant under graph automorphisms, reducing complexity in symmetric environments and supporting broad transfer [2403.10855].

### Empirical Observations

- In gridworlds such as Four Rooms, augmenting an agent’s primitive action set with several eigenoptions reduces the episodes needed to reach all parts of the state space by approximately 50% [1703.00956][2403.10855].
- As credit-assignment propagators, value-aware eigenoptions improve learning speed and reward propagation relative to handcrafted or bottleneck-based options [2507.09127].
- With deep option-value heads (DQN extensions), modest but present gains are observed in large pixel-based mazes and environments, with termination mechanism design being crucial [2507.09127][1710.11089].

## 5. Extensions, Limitations, and Open Challenges

### Adaptations

- **Sample-based Laplacian/SR**: Approximate spectral methods use transition samples for SVD or Nyström approximations, bypassing $O(|S|^3)$ cost of full eigendecomposition [1703.00956][1712.04065].
- **Function Approximation**: Deep RL methods infer eigenpurposes over latent feature spaces, enabling eigenoption discovery from high-dimensional (e.g., pixel) inputs [1710.11089].
- **Generalization via Option Keyboard**: Option-value functions can be linearly composed to support policy evaluation/improvement for any linear combination of eigenpurposes, inducing a combinatorially rich option space at no additional environmental cost [2110.05740].

### Known Limitations

- Scalability and spectral computation: Full spectral decomposition becomes intractable with large $|\mathcal{S}|$. Sampling and approximation trade precision for memory and computational efficiency [1703.00956][2403.10855][1712.04065].
- Alignment: Discovered eigenoptions are task-independent and may not optimize for extrinsic goals unless explicit reward mixing or task-driven selection is applied [1712.04065][2507.09127][1710.11089].
- Termination design: Under function approximation, threshold-based termination can result in unstable or extremely short/long options; alternative designs (fixed-horizon, soft termination) are active research topics [2507.09127][1710.11089].
- Online discovery may bias experience and sometimes hinder exploration by reinforcing suboptimal option values if not managed carefully [2507.09127].

### Open Directions

- Online, scalable, and adaptive selection of eigenoptions in conjunction with representation learning.
- Improved algorithms for smooth, differentiable, and robust option termination.
- Integration with reward-aware Laplacians (Default Representation) to better align temporal abstractions with downstream control tasks.
- Hierarchical or compositional assembly to build higher-order, value-aligned options automatically [2507.09127][1710.11089].

## 6. Domains of Application and Notable Empirical Examples

- **Gridworlds**: The second PVF in Four Rooms environments splits the rooms, yielding options that traverse doors or principal directions [1703.00956][2403.10855][2110.05740][2507.09127].
- **Atari**: In Montezuma’s Revenge and Freeway, sample-based or deep eigenoptions enable agents to reach bottlenecks/subgoals that are rarely explored by primitives alone. Option policies derived via Q-learning on eigenpurposes show spatial coherence in bottleneck/hallway traversal [1703.00956][1710.11089].
- **Deep RL**: Architectures incorporating eigenoption heads have demonstrated improved exploration and value propagation in pixel-based, high-dimensional tasks, though advances in option termination and selection remain ongoing [1710.11089][2507.09127].

## 7. Synthesis and Practical Recommendations

Eigenoptions provide a theoretically principled and empirically validated means of discovering temporal abstractions in RL, leveraging global structure via spectral analysis of either the transition graph or the successor representation. Their task-independent intrinsic rewards facilitate broad exploration, representation learning, and compositionality. For effective use, careful selection of eigenvectors, robust option-termination mechanisms, and alignment with extrinsic objectives remain central. Future research is focusing on scalable online discovery, adaptive termination, and task-aware reward integration to further realize the full potential of these diffusive temporal abstractions in deep and hierarchical RL settings [1703.00956][2403.10855][1712.04065][2110.05740][1710.11089][2507.09127].

Source: https://www.emergentmind.com/topics/eigenoptions