---
title: 'EARL: Energy-Aware Reinforcement Learning for LSMs'
url: https://www.emergentmind.com/topics/expa-reinforcement-learning-earl
type: topic
---

# EARL: Energy-Aware Reinforcement Learning for LSMs

EARL denotes **Energy-Aware Reinforcement Learning**, a hybrid hyperparameter optimization framework for Liquid State Machines (LSMs) that jointly optimizes classification accuracy and energy consumption under the tight trial budgets and computational constraints characteristic of pervasive, resource-constrained AI [2601.05205]. In the paper that introduces it, the term **“ExpA”** is not explicitly defined or used; however, the implemented method is an adaptive exploration–exploitation policy driven by reinforcement learning on top of a Bayesian optimization core, so the label “ExpA Reinforcement Learning” is a plausible interpretation only insofar as it refers to that exploration–exploitation adaptation mechanism [2601.05205].

## 1. Problem setting and conceptual scope

EARL is motivated by the deployment difficulty of LSMs in low-power temporal processing systems. The central difficulty is not merely model training, but hyperparameter optimization under a search landscape that is described as **highly non-smooth and discrete**, **expensive to evaluate**, and **multi-objective**, because each candidate configuration must be assessed in terms of both predictive performance and energy consumption [2601.05205].

The framework targets reservoir hyperparameters such as leak rate, spectral radius, reservoir size, and connectivity. These parameters materially affect both accuracy and energy because they alter the reservoir’s stability regime, the number of active neurons and synapses, and the resulting compute cost. Traditional search procedures such as grid search, random search, and vanilla Bayesian optimization are characterized in the source as methods that often assume smoother objectives, optimize a single metric, and do not explicitly model energy [2601.05205].

EARL addresses this by combining four elements into a single HPO loop: **Sobol initialization**, **Gaussian Process–based Bayesian Optimization**, an **RL-based candidate selection policy**, and **adaptive early termination**. Its conceptual decomposition is explicit: Bayesian optimization functions as a global explorer and surrogate model; reinforcement learning acts as a local trial-selection policy; the optimization target is an energy-aware scalar reward rather than accuracy alone [2601.05205].

A recurring misconception is to read EARL as a hard-constrained energy optimization method. It is not formulated that way. The framework uses a **soft trade-off via scalarization**, not a rule of the form “energy must be below \(X\).” Another common confusion is to equate its stopping rule with early stopping methods such as Hyperband. EARL’s termination mechanism does **not** terminate individual training runs; it terminates the **entire optimization process** when combined reward and energy improvements plateau [2601.05205].

## 2. Objective function and optimization architecture

For a hyperparameter vector \(x\), EARL evaluates classification accuracy \(f_1(x)\), energy consumption \(f_2(x)\), and a scalarized reward
\[
r(x) = f_1(x) - \alpha \cdot f_2(x).
\]
Here, \(f_1(x)\) is validation accuracy, \(f_2(x)\) is energy consumption measured in \( \mathrm{pJ/sample} \) in the experiments, and \(\alpha\) controls the accuracy–energy trade-off [2601.05205].

All observations are stored as
\[
\mathcal{D}_t = \{(x_i, f_1(x_i), f_2(x_i), r(x_i))\}_{i=1}^{t}.
\]
The best-so-far reward and best energy are tracked during search. Search begins with **Sobol initialization**, after which EARL fits a Gaussian Process surrogate over the scalar reward,
\[
r(x) \sim \mathcal{GP}(\mu(x), k(x,x')),
\]
using a **Matérn kernel** to capture non-smooth structure [2601.05205].

Candidate generation uses **Expected Improvement** over the current best reward \(r^\ast\):
\[
\text{EI}(x) = \mathbb{E}[\max(r(x)-r^\ast,0)].
\]
The paper gives the standard closed form,
\[
\text{EI}(x)=
\begin{cases}
(\mu(x)-r^\ast)\Phi(Z)+\sigma(x)\phi(Z), & \sigma(x)>0,\\
0, & \sigma(x)=0,
\end{cases}
\]
with
\[
Z=\frac{\mu(x)-r^\ast}{\sigma(x)}.
\]
EARL generates a **batch of \(K\) candidates** by maximizing the EI acquisition, then uses diversity heuristics to avoid duplicates. For each candidate \(x_{t+1}^{(j)}\), the GP supplies a predictive Gaussian reward distribution
\[
r(x_{t+1}^{(j)}) \sim \mathcal{N}(\mu_j,\sigma_j^2),
\]
and these \((\mu_j,\sigma_j^2)\) pairs become the state seen by the RL selector [2601.05205].

Although the search itself is scalarized, the final analysis is explicitly multi-objective. After optimization, the framework computes a **Pareto frontier** over accuracy and energy, so the operational search criterion is scalar while the reported outcome space remains bi-objective [2601.05205]. This suggests a pragmatic compromise: scalarization simplifies online decision-making, whereas Pareto analysis preserves the post hoc trade-off structure.

## 3. Reinforcement-learning policy for adaptive exploration–exploitation

The RL component is the part of EARL that most closely matches the intuition behind an “ExpA” reading. At each BO iteration, the surrogate proposes \(K\) candidates, each summarized by predicted reward mean and variance. The RL state is the concatenated, min–max normalized vector
\[
s_t = [\tilde{\mu}_1,\tilde{\sigma}_1^2,\dots,\tilde{\mu}_K,\tilde{\sigma}_K^2].
\]
This state encodes both predicted utility and uncertainty across the candidate batch [2601.05205].

The action space is discrete:
\[
a_t \in \{1,\dots,K\},
\]
meaning that the agent chooses **which one** of the \(K\) BO-suggested configurations is actually evaluated. The remaining \(K-1\) candidates are discarded for that iteration [2601.05205].

Action selection uses an \(\epsilon\)-greedy policy,
\[
a_t=
\begin{cases}
\text{uniform}(1,\dots,K), & \text{with probability } \epsilon,\\
\arg\max_j Q(s_t,j), & \text{otherwise},
\end{cases}
\]
with exponentially decaying exploration
\[
\epsilon_t = \max(\kappa \epsilon_{t-1}, \epsilon_{\min}).
\]
The \(Q\)-function is approximated by a neural network. A FIFO replay buffer of capacity \(C\) stores transitions \((s_t,a_t,r_t,s_{t+1})\), and the Q-network is updated every \(F\) iterations using one-step temporal-difference learning:
\[
Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \eta \big[r_t + \gamma \max_{a'} \bar{Q}(s_{t+1},a') - Q(s_t,a_t)\big].
\]
Here \(\bar{Q}\) is a target network updated every \(F\) steps, and the RL reward is exactly the scalarized HPO reward,
\[
r_t = r(x_t)=f_1(x_t)-\alpha f_2(x_t).
\]
The paper characterizes this mechanism as essentially a **DQN-style contextual bandit**: GP predictions provide the context, the \(K\) candidates are the arms, and the learned selector decides which uncertainty/mean profile is worth evaluating next [2601.05205].

This RL layer is the key distinction from a pure Bayesian optimization loop. The source does not provide an ablation that removes RL entirely, so the separate contribution of the selector is not isolated experimentally. Nevertheless, the reported interpretation is that the RL component “effectively direct[s] the search toward high-potential regions of the hyperparameter space and minimiz[es] unnecessary evaluations” [2601.05205]. A cautious reading is therefore that the selector is architecturally central, but its isolated marginal effect is not quantified in the paper.

## 4. Liquid State Machine instantiation and energy-aware search space

The underlying model is an **LSM with Leaky Integrate-and-Fire neurons and a GRU readout**. The LIF membrane dynamics are given as
\[
\tau_m \frac{dV_i}{dt} = -(V_i - V_{\text{rest}}) + I_i(t),
\]
with threshold-triggered spiking and reset. The reservoir state is also described in a standard reservoir-computing form,
\[
x_t = (1-\alpha)x_{t-1} + \alpha \tanh(Wx_{t-1} + W_{\text{in}}u_t),
\]
where \(u_t\) is the input, \(W_{\text{in}}\) the input weight matrix, \(W\) the recurrent weight matrix scaled by spectral radius, and \(\alpha\) the leak rate [2601.05205].

The text emphasizes the dynamical role of the leak rate: low \(\alpha\) values, such as \(<0.1\), are described as very stable and long-memory but potentially sluggish, whereas high \(\alpha\) values, such as \(>0.3\), can become more chaotic and degrade temporal coherence. The reservoir itself is **fixed**, with no backpropagation through \(W\) or \(W_{\text{in}}\), which is described as standard in reservoir computing and beneficial for energy [2601.05205].

The GRU readout evolves according to
\[
h_t = z_t \odot h_{t-1} + (1-z_t)\odot \tilde{h}_t,
\]
and only the GRU and output layer parameters are trained. In the experiments, GRU training uses **AdamW**, **100 epochs**, and **batch size 64** [2601.05205].

The optimized hyperparameter space is
\[
\mathcal{X} = \mathcal{X}_{\text{size}} \times \mathcal{X}_{\text{conn}} \times \mathcal{X}_{\text{spectral}} \times \mathcal{X}_{\beta},
\]
with the following bounds: leak rate \([0.1,0.4]\), spectral radius \([0.6,1.1]\), reservoir size \([100,1000]\) as an integer, and connectivity \([0.2,0.7]\) [2601.05205]. The full HPO budget is **50 trials**, decomposed into **20 Sobol initialization trials** and **30 BO/RL steps**, with fixed seed **42** [2601.05205].

Energy is not modeled analytically. Instead, it is treated as an **empirical measurement** taken per configuration in \( \mathrm{pJ/sample} \) for the full LSM+GRU model on an **NVIDIA Tesla T4 GPU with 16 GB VRAM**. This is an important implementation detail: the energy-aware objective is grounded in observed hardware measurements, not a symbolic proxy [2601.05205]. At the same time, the architecture and motivation are described as relevant to neuromorphic deployment, including **Loihi** and **memristive crossbars**, because of the event-driven character of spike-based dynamics [2601.05205].

## 5. Experimental protocol and quantitative performance

The evaluation uses three benchmark datasets: **FSDD (Free Spoken Digit Dataset)** with approximately 3,000 spoken-digit audio samples processed as MFCC features, **Occupancy Detection** with approximately 10,000 environmental sensor readings for binary occupancy classification, and **UCI HAR** with approximately 15,000 multivariate accelerometer/gyroscope sequences and six activity classes [2601.05205]. All datasets are normalized to zero mean and unit variance, with **80/20 stratified train/validation splits**.

The baselines are **Optuna with NSGA-II multi-objective sampling** and **Ray Tune with asynchronous parallel sampling**. All methods share the same search space, training pipeline, dataset splits, and total trial budget. Reported metrics are **validation accuracy**, **energy**, **total training time**, and **total optimization time**. Each experiment is repeated **100 times**, and results are reported as means with **95% confidence intervals** [2601.05205].

On **FSDD**, EARL reaches \(95.39 \pm 0.44\%\) accuracy, \(0.2089 \pm 0.0129\) pJ/sample, 10.31 minutes of train time, and 10.57 minutes of optimization time. Optuna records \(82.25 \pm 2.75\%\) accuracy and \(0.3644 \pm 0.0091\) pJ/sample, while Ray reaches \(88.15 \pm 2.18\%\) and \(0.4178 \pm 0.0042\) pJ/sample, both with optimization times above 100 minutes [2601.05205].

On **HAR**, EARL reports \(96.99 \pm 0.34\%\) accuracy, \(0.20796 \pm 0.02362\) pJ/sample, 18.73 minutes of train time, and 19.53 minutes of optimization time. Optuna yields \(90.52 \pm 1.16\%\) and \(0.6265 \pm 0.0193\) pJ/sample; Ray yields \(94.41 \pm 0.67\%\) and \(0.43737 \pm 0.00512\) pJ/sample, again with substantially longer runtimes [2601.05205].

On **Occupancy**, EARL reaches \(98.47 \pm 0.00008\%\) accuracy, \(0.0278 \pm 0.0052\) pJ/sample, 14.42 minutes of train time, and 15.28 minutes of optimization time. Optuna reports \(97.44 \pm 0.27\%\) and \(0.1599 \pm 0.0092\) pJ/sample, whereas Ray matches the \(98.47\%\) accuracy level but at \(0.1624 \pm 0.0030\) pJ/sample and with much longer optimization time [2601.05205].

Across the three benchmarks, the paper summarizes the aggregate effect as **6–15% higher accuracy**, **60–80% lower energy consumption**, and **up to an order of magnitude reduction in optimization time** relative to the compared frameworks [2601.05205]. The trajectory plots reportedly show convergence in **30–45 trials**, while the baselines continue exploring longer with worse trade-offs [2601.05205]. A plausible implication is that the combination of scalarized energy-aware reward, RL-based candidate prioritization, and whole-run early termination improves sample efficiency at the HPO level rather than only at the model-training level.

## 6. Scalability, limitations, and nomenclatural ambiguity

EARL is explicitly described as a framework for **moderate-dimensional HPO**, and the concrete instantiation optimizes **four primary hyperparameters**. The Gaussian Process core is acknowledged to scale poorly in very high-dimensional spaces or at very large trial counts, although the method partially mitigates this with Sobol initialization, RL-assisted candidate choice, and early termination [2601.05205]. The paper further notes that alternative surrogates such as TPE, random forests, or neural surrogates could, in principle, replace the GP [2601.05205].

Several limitations are stated or implied. GP-EI may still become trapped in local basins in extremely noisy or adversarial landscapes; RL training itself consumes time and may overfit noisy reward signals; the scalarization parameter \(\alpha\) must be chosen appropriately because large \(\alpha\) values bias strongly toward low-energy configurations and small \(\alpha\) values shift the search toward accuracy; and the reported energy measurements are **GPU-based rather than neuromorphic hardware-in-the-loop** [2601.05205]. The paper also does **not** present a detailed ablation isolating BO-only, RL-only, or early-termination-only variants, so the individual contribution of each component remains analytically separable but experimentally entangled [2601.05205].

The architecture is nevertheless described as **model-agnostic** in structure: it requires only a search space \(\mathcal{X}\), a predictive metric \(f_1(x)\), an energy metric \(f_2(x)\), and the reward definition \(r(x)=f_1(x)-\alpha f_2(x)\). The authors explicitly state that the same BO+RL+early-termination scheme can be applied to **general spiking neural networks**, **conventional deep nets**, or other ML models, provided that practical energy measurement methods exist [2601.05205]. This suggests that the LSM instantiation is specific, but the optimization template is broader.

A final source of confusion is the acronym itself. In the LSM paper, **EARL** means **Energy-Aware Reinforcement Learning**, whereas the literature also uses **EARL** for **Environments for Autonomous Reinforcement Learning** [2112.09605], **Efficient Agentic Reinforcement Learning Systems for Large Language Models** [2510.05943], and, in a different line of work, **ExpA Reinforcement Learning** over an **Expanded Action space** for LLMs [2510.07581]. The phrase “ExpA Reinforcement Learning (EARL)” is therefore not a stable field-wide designation. In the specific context of LSM hyperparameter optimization, the precise referent is the energy-aware BO+RL framework of [2601.05205], and “ExpA” is best treated as an interpretive shorthand for its adaptive exploration–exploitation policy rather than as the paper’s formal terminology.

Source: https://www.emergentmind.com/topics/expa-reinforcement-learning-earl