Eigenoptions in Reinforcement Learning

• Eigenoptions are temporally extended actions created by applying spectral methods to state-transition graphs, revealing principal diffusion modes for exploration.
• They leverage proto-value functions from the graph Laplacian or successor representation to generate intrinsic rewards that drive hierarchical learning.
• Employing eigenoptions in RL can reduce exploration time, improve learning efficiency in gridworlds, and facilitate representation learning in high-dimensional domains.

An eigenoption is a temporally extended action in reinforcement learning (RL) derived from the spectral analysis of an environment’s state–transition graph. The core motivation is to automate option discovery by exploiting global connectivity properties of Markov decision processes (MDPs) through proto-value functions (PVFs), the eigenvectors of a graph Laplacian or successor representation (SR). Each eigenoption consists of an intrinsic reward function that promotes exploration along a principal “diffusion mode” of the state space, independent of extrinsic rewards. Eigenoptions are particularly efficacious in sparse-reward and high-dimensional domains, capturing environment geometry and enabling hierarchical abstraction, representation learning, and scalable exploration.

1. Construction of Eigenoptions via Graph Spectral Methods

Let $G=(\mathcal{S},\mathcal{E})$ be the undirected state–transition graph of an MDP, with adjacency matrix $A$ (possibly weighted) and degree matrix $D$. The (combinatorial) Laplacian is $L = D - A$, or its symmetric normalization $L_{\mathrm{norm}}=I - D^{-1/2} A D^{-1/2}$ is used for improved numerical properties (Machado et al., 2017, Ghriss et al., 2024, Liu et al., 2017, Kotamreddy et al., 12 Jul 2025). The eigenvalue problem

$L\,\phi_i = \lambda_i\,D\,\phi_i\,,$

defines a sequence of smooth PVFs $\{\phi_i\}$ over $\mathcal{S}$, ordered by increasing $\lambda_i$. These eigenvectors capture large-scale structural features of the state space.

Each PVF $\phi_i$ generates an intrinsic reward ("eigenpurpose"):

$A$0

This reward is maximized by policies that move the agent “uphill” along diffusion mode $A$1.

The corresponding eigenoption $A$2 is defined as a triple $A$3 where:

• $A$4 (states where action upwards in $A$5 is possible)
• $A$6 (intra-option policy maximizing cumulative $A$7)
• $A$8 iff $A$9 for all $D$0 (termination at local maxima of $D$1)

Option-value functions $D$2 are computed using standard RL techniques on the SMDP induced by $D$3 (Machado et al., 2017, Ghriss et al., 2024, Liu et al., 2017).

2. The Successor Representation Perspective and Equivalence

The successor representation under policy $D$4 is $D$5, where $D$6 is the state-transition matrix and $D$7 the discount factor (Machado et al., 2021, Machado et al., 2017, Kotamreddy et al., 12 Jul 2025). Its spectral decomposition

$D$8

connects SR eigenvectors $D$9 to PVFs. Under mild conditions (e.g., uniform policy, reversibility), the eigenvectors of $L = D - A$0 and $L = D - A$1 coincide up to scaling and ordering. This allows eigenoptions to be derived either from the graph Laplacian or from the SR, and facilitates sample-based or function-approximate discovery in large or continuous state spaces (Machado et al., 2021, Machado et al., 2017).

In feature-based or deep learning settings, the intrinsic reward generalizes as

$L = D - A$2

where $L = D - A$3 is an SR eigenvector and $L = D - A$4 a learned representation (Machado et al., 2017).

3. Algorithmic Frameworks and Discovery Procedures

Tabular/Offline Discovery

1. Estimate the adjacency or transition matrix from data.
2. Construct $L = D - A$5 or $L = D - A$6 and extract the top $L = D - A$7 nontrivial eigenvectors (excluding the trivial constant mode).
3. For each $L = D - A$8, define $L = D - A$9 as above.
4. For each $L_{\mathrm{norm}}=I - D^{-1/2} A D^{-1/2}$0, learn the optimal intra-option policy $L_{\mathrm{norm}}=I - D^{-1/2} A D^{-1/2}$1 (e.g., by Q-learning on $L_{\mathrm{norm}}=I - D^{-1/2} A D^{-1/2}$2), determine initiation and termination sets from the optimal $L_{\mathrm{norm}}=I - D^{-1/2} A D^{-1/2}$3-function.
5. Aggregate eigenoptions as macro-actions in the agent’s action set (Machado et al., 2017, Liu et al., 2017, Machado et al., 2021, Kotamreddy et al., 12 Jul 2025).

Sample-Based and Deep RL Approaches

• For large or continuous domains, transition samples are collected to build an incidence or feature-difference matrix. The top $L_{\mathrm{norm}}=I - D^{-1/2} A D^{-1/2}$4 right singular vectors (via SVD) yield approximate PVFs.
• In the deep SR approach, a neural network encoder learns $L_{\mathrm{norm}}=I - D^{-1/2} A D^{-1/2}$5 (estimated SF) via temporal difference learning, possibly from pixel observations (Machado et al., 2017).
• Online methods alternate between updating SR (or Laplacian estimation), extracting new eigenvectors, defining new options, and learning their policies in a “representation–option discovery” (ROD) loop (Kotamreddy et al., 12 Jul 2025).

Eigenoption-Critic (EOC)

EOC interleaves eigenoption discovery and control within the Option-Critic framework, blending intrinsic and extrinsic rewards:

$L_{\mathrm{norm}}=I - D^{-1/2} A D^{-1/2}$6

with online updates of intra-option policies, termination conditions, and Nyström extensions for continuous spaces (Liu et al., 2017).

4. Properties, Theoretical Guarantees, and Empirical Effects

Geometric and Exploratory Properties

• Principal direction skills: Each eigenoption $L_{\mathrm{norm}}=I - D^{-1/2} A D^{-1/2}$7 pushes the agent along a principal axis of the state space, as defined by PVF $L_{\mathrm{norm}}=I - D^{-1/2} A D^{-1/2}$8. Lower-frequency (smooth) eigenvectors traverse wide regions and bottlenecks.
• Temporal scales: The spectral index (or eigenvalue $L_{\mathrm{norm}}=I - D^{-1/2} A D^{-1/2}$9) determines the typical option duration: lower $L\,\phi_i = \lambda_i\,D\,\phi_i\,,$0 yields longer, smoother behaviors.
• Task-independence: Eigenoptions' definition is decoupled from environment rewards, promoting uninformed exploration especially in sparse/reward-agnostic settings (Machado et al., 2017, Machado et al., 2021, Kotamreddy et al., 12 Jul 2025).

Theoretical Guarantees

• Eigenoptions capture global graph connectivity and are guaranteed to terminate (since their reward is potential-based) in any ergodic finite MDP (Machado et al., 2021).
• Spectral invariance: The option set is invariant under graph automorphisms, reducing complexity in symmetric environments and supporting broad transfer (Ghriss et al., 2024).

Empirical Observations

• In gridworlds such as Four Rooms, augmenting an agent’s primitive action set with several eigenoptions reduces the episodes needed to reach all parts of the state space by approximately 50% (Machado et al., 2017, Ghriss et al., 2024).
• As credit-assignment propagators, value-aware eigenoptions improve learning speed and reward propagation relative to handcrafted or bottleneck-based options (Kotamreddy et al., 12 Jul 2025).
• With deep option-value heads (DQN extensions), modest but present gains are observed in large pixel-based mazes and environments, with termination mechanism design being crucial (Kotamreddy et al., 12 Jul 2025, Machado et al., 2017).

5. Extensions, Limitations, and Open Challenges

Adaptations

• Sample-based Laplacian/SR: Approximate spectral methods use transition samples for SVD or Nyström approximations, bypassing $L\,\phi_i = \lambda_i\,D\,\phi_i\,,$1 cost of full eigendecomposition (Machado et al., 2017, Liu et al., 2017).
• Function Approximation: Deep RL methods infer eigenpurposes over latent feature spaces, enabling eigenoption discovery from high-dimensional (e.g., pixel) inputs (Machado et al., 2017).
• Generalization via Option Keyboard: Option-value functions can be linearly composed to support policy evaluation/improvement for any linear combination of eigenpurposes, inducing a combinatorially rich option space at no additional environmental cost (Machado et al., 2021).

Known Limitations

• Scalability and spectral computation: Full spectral decomposition becomes intractable with large $L\,\phi_i = \lambda_i\,D\,\phi_i\,,$2. Sampling and approximation trade precision for memory and computational efficiency (Machado et al., 2017, Ghriss et al., 2024, Liu et al., 2017).
• Alignment: Discovered eigenoptions are task-independent and may not optimize for extrinsic goals unless explicit reward mixing or task-driven selection is applied (Liu et al., 2017, Kotamreddy et al., 12 Jul 2025, Machado et al., 2017).
• Termination design: Under function approximation, threshold-based termination can result in unstable or extremely short/long options; alternative designs (fixed-horizon, soft termination) are active research topics (Kotamreddy et al., 12 Jul 2025, Machado et al., 2017).
• Online discovery may bias experience and sometimes hinder exploration by reinforcing suboptimal option values if not managed carefully (Kotamreddy et al., 12 Jul 2025).

Open Directions

• Online, scalable, and adaptive selection of eigenoptions in conjunction with representation learning.
• Improved algorithms for smooth, differentiable, and robust option termination.
• Integration with reward-aware Laplacians (Default Representation) to better align temporal abstractions with downstream control tasks.
• Hierarchical or compositional assembly to build higher-order, value-aligned options automatically (Kotamreddy et al., 12 Jul 2025, Machado et al., 2017).

6. Domains of Application and Notable Empirical Examples

• Gridworlds: The second PVF in Four Rooms environments splits the rooms, yielding options that traverse doors or principal directions (Machado et al., 2017, Ghriss et al., 2024, Machado et al., 2021, Kotamreddy et al., 12 Jul 2025).
• Atari: In Montezuma’s Revenge and Freeway, sample-based or deep eigenoptions enable agents to reach bottlenecks/subgoals that are rarely explored by primitives alone. Option policies derived via Q-learning on eigenpurposes show spatial coherence in bottleneck/hallway traversal (Machado et al., 2017, Machado et al., 2017).
• Deep RL: Architectures incorporating eigenoption heads have demonstrated improved exploration and value propagation in pixel-based, high-dimensional tasks, though advances in option termination and selection remain ongoing (Machado et al., 2017, Kotamreddy et al., 12 Jul 2025).

7. Synthesis and Practical Recommendations

Eigenoptions provide a theoretically principled and empirically validated means of discovering temporal abstractions in RL, leveraging global structure via spectral analysis of either the transition graph or the successor representation. Their task-independent intrinsic rewards facilitate broad exploration, representation learning, and compositionality. For effective use, careful selection of eigenvectors, robust option-termination mechanisms, and alignment with extrinsic objectives remain central. Future research is focusing on scalable online discovery, adaptive termination, and task-aware reward integration to further realize the full potential of these diffusive temporal abstractions in deep and hierarchical RL settings (Machado et al., 2017, Ghriss et al., 2024, Liu et al., 2017, Machado et al., 2021, Machado et al., 2017, Kotamreddy et al., 12 Jul 2025).