EARL: Energy-Aware Reinforcement Learning for LSMs

• The paper introduces EARL, a hybrid hyperparameter optimization framework that combines Sobol initialization, Gaussian Process-based Bayesian Optimization, and an RL policy to balance accuracy and energy in LSMs.
• The framework employs a scalarized reward function, r(x)=f1(x)-α·f2(x), and adaptive early termination to efficiently navigate a discrete, multi-objective search landscape.
• Empirical results on benchmark datasets demonstrate that EARL achieves higher accuracy, significantly lower energy consumption, and faster optimization compared to conventional methods.

EARL denotes Energy-Aware Reinforcement Learning, a hybrid hyperparameter optimization framework for Liquid State Machines (LSMs) that jointly optimizes classification accuracy and energy consumption under the tight trial budgets and computational constraints characteristic of pervasive, resource-constrained AI (Iqbal et al., 8 Jan 2026). In the paper that introduces it, the term “ExpA” is not explicitly defined or used; however, the implemented method is an adaptive exploration–exploitation policy driven by reinforcement learning on top of a Bayesian optimization core, so the label “ExpA Reinforcement Learning” is a plausible interpretation only insofar as it refers to that exploration–exploitation adaptation mechanism (Iqbal et al., 8 Jan 2026).

1. Problem setting and conceptual scope

EARL is motivated by the deployment difficulty of LSMs in low-power temporal processing systems. The central difficulty is not merely model training, but hyperparameter optimization under a search landscape that is described as highly non-smooth and discrete, expensive to evaluate, and multi-objective, because each candidate configuration must be assessed in terms of both predictive performance and energy consumption (Iqbal et al., 8 Jan 2026).

The framework targets reservoir hyperparameters such as leak rate, spectral radius, reservoir size, and connectivity. These parameters materially affect both accuracy and energy because they alter the reservoir’s stability regime, the number of active neurons and synapses, and the resulting compute cost. Traditional search procedures such as grid search, random search, and vanilla Bayesian optimization are characterized in the source as methods that often assume smoother objectives, optimize a single metric, and do not explicitly model energy (Iqbal et al., 8 Jan 2026).

EARL addresses this by combining four elements into a single HPO loop: Sobol initialization, Gaussian Process–based Bayesian Optimization, an RL-based candidate selection policy, and adaptive early termination. Its conceptual decomposition is explicit: Bayesian optimization functions as a global explorer and surrogate model; reinforcement learning acts as a local trial-selection policy; the optimization target is an energy-aware scalar reward rather than accuracy alone (Iqbal et al., 8 Jan 2026).

A recurring misconception is to read EARL as a hard-constrained energy optimization method. It is not formulated that way. The framework uses a soft trade-off via scalarization, not a rule of the form “energy must be below $X$.” Another common confusion is to equate its stopping rule with early stopping methods such as Hyperband. EARL’s termination mechanism does not terminate individual training runs; it terminates the entire optimization process when combined reward and energy improvements plateau (Iqbal et al., 8 Jan 2026).

2. Objective function and optimization architecture

For a hyperparameter vector $x$, EARL evaluates classification accuracy $f_1(x)$, energy consumption $f_2(x)$, and a scalarized reward

$r(x) = f_1(x) - \alpha \cdot f_2(x).$

Here, $f_1(x)$ is validation accuracy, $f_2(x)$ is energy consumption measured in $\mathrm{pJ/sample}$ in the experiments, and $\alpha$ controls the accuracy–energy trade-off (Iqbal et al., 8 Jan 2026).

All observations are stored as

$$\mathcal{D}_t = \{(x_i, f_1(x_i), f_2(x_i), r(x_i))\}_{i=1}^{t}.$$

The best-so-far reward and best energy are tracked during search. Search begins with Sobol initialization, after which EARL fits a Gaussian Process surrogate over the scalar reward,

$x$0

using a Matérn kernel to capture non-smooth structure (Iqbal et al., 8 Jan 2026).

Candidate generation uses Expected Improvement over the current best reward $x$1: $x$2 The paper gives the standard closed form,

$x$3

with

$x$4

EARL generates a batch of $x$5 candidates by maximizing the EI acquisition, then uses diversity heuristics to avoid duplicates. For each candidate $x$6, the GP supplies a predictive Gaussian reward distribution

$x$7

and these $x$8 pairs become the state seen by the RL selector (Iqbal et al., 8 Jan 2026).

Although the search itself is scalarized, the final analysis is explicitly multi-objective. After optimization, the framework computes a Pareto frontier over accuracy and energy, so the operational search criterion is scalar while the reported outcome space remains bi-objective (Iqbal et al., 8 Jan 2026). This suggests a pragmatic compromise: scalarization simplifies online decision-making, whereas Pareto analysis preserves the post hoc trade-off structure.

3. Reinforcement-learning policy for adaptive exploration–exploitation

The RL component is the part of EARL that most closely matches the intuition behind an “ExpA” reading. At each BO iteration, the surrogate proposes $x$9 candidates, each summarized by predicted reward mean and variance. The RL state is the concatenated, min–max normalized vector

$f_1(x)$0

This state encodes both predicted utility and uncertainty across the candidate batch (Iqbal et al., 8 Jan 2026).

The action space is discrete: $f_1(x)$1 meaning that the agent chooses which one of the $f_1(x)$2 BO-suggested configurations is actually evaluated. The remaining $f_1(x)$3 candidates are discarded for that iteration (Iqbal et al., 8 Jan 2026).

Action selection uses an $f_1(x)$4-greedy policy,

$f_1(x)$5

with exponentially decaying exploration

$f_1(x)$6

The $f_1(x)$7-function is approximated by a neural network. A FIFO replay buffer of capacity $f_1(x)$8 stores transitions $f_1(x)$9, and the Q-network is updated every $f_2(x)$0 iterations using one-step temporal-difference learning: $f_2(x)$1 Here $f_2(x)$2 is a target network updated every $f_2(x)$3 steps, and the RL reward is exactly the scalarized HPO reward,

$f_2(x)$4

The paper characterizes this mechanism as essentially a DQN-style contextual bandit: GP predictions provide the context, the $f_2(x)$5 candidates are the arms, and the learned selector decides which uncertainty/mean profile is worth evaluating next (Iqbal et al., 8 Jan 2026).

This RL layer is the key distinction from a pure Bayesian optimization loop. The source does not provide an ablation that removes RL entirely, so the separate contribution of the selector is not isolated experimentally. Nevertheless, the reported interpretation is that the RL component “effectively direct[s] the search toward high-potential regions of the hyperparameter space and minimiz[es] unnecessary evaluations” (Iqbal et al., 8 Jan 2026). A cautious reading is therefore that the selector is architecturally central, but its isolated marginal effect is not quantified in the paper.

4. Liquid State Machine instantiation and energy-aware search space

The underlying model is an LSM with Leaky Integrate-and-Fire neurons and a GRU readout. The LIF membrane dynamics are given as

$f_2(x)$6

with threshold-triggered spiking and reset. The reservoir state is also described in a standard reservoir-computing form,

$f_2(x)$7

where $f_2(x)$8 is the input, $f_2(x)$9 the input weight matrix, $r(x) = f_1(x) - \alpha \cdot f_2(x).$0 the recurrent weight matrix scaled by spectral radius, and $r(x) = f_1(x) - \alpha \cdot f_2(x).$1 the leak rate (Iqbal et al., 8 Jan 2026).

The text emphasizes the dynamical role of the leak rate: low $r(x) = f_1(x) - \alpha \cdot f_2(x).$2 values, such as $r(x) = f_1(x) - \alpha \cdot f_2(x).$3, are described as very stable and long-memory but potentially sluggish, whereas high $r(x) = f_1(x) - \alpha \cdot f_2(x).$4 values, such as $r(x) = f_1(x) - \alpha \cdot f_2(x).$5, can become more chaotic and degrade temporal coherence. The reservoir itself is fixed, with no backpropagation through $r(x) = f_1(x) - \alpha \cdot f_2(x).$6 or $r(x) = f_1(x) - \alpha \cdot f_2(x).$7, which is described as standard in reservoir computing and beneficial for energy (Iqbal et al., 8 Jan 2026).

The GRU readout evolves according to

$r(x) = f_1(x) - \alpha \cdot f_2(x).$8

and only the GRU and output layer parameters are trained. In the experiments, GRU training uses AdamW, 100 epochs, and batch size 64 (Iqbal et al., 8 Jan 2026).

The optimized hyperparameter space is

$r(x) = f_1(x) - \alpha \cdot f_2(x).$9

with the following bounds: leak rate $f_1(x)$0, spectral radius $f_1(x)$1, reservoir size $f_1(x)$2 as an integer, and connectivity $f_1(x)$3 (Iqbal et al., 8 Jan 2026). The full HPO budget is 50 trials, decomposed into 20 Sobol initialization trials and 30 BO/RL steps, with fixed seed 42 (Iqbal et al., 8 Jan 2026).

Energy is not modeled analytically. Instead, it is treated as an empirical measurement taken per configuration in $f_1(x)$4 for the full LSM+GRU model on an NVIDIA Tesla T4 GPU with 16 GB VRAM. This is an important implementation detail: the energy-aware objective is grounded in observed hardware measurements, not a symbolic proxy (Iqbal et al., 8 Jan 2026). At the same time, the architecture and motivation are described as relevant to neuromorphic deployment, including Loihi and memristive crossbars, because of the event-driven character of spike-based dynamics (Iqbal et al., 8 Jan 2026).

5. Experimental protocol and quantitative performance

The evaluation uses three benchmark datasets: FSDD (Free Spoken Digit Dataset) with approximately 3,000 spoken-digit audio samples processed as MFCC features, Occupancy Detection with approximately 10,000 environmental sensor readings for binary occupancy classification, and UCI HAR with approximately 15,000 multivariate accelerometer/gyroscope sequences and six activity classes (Iqbal et al., 8 Jan 2026). All datasets are normalized to zero mean and unit variance, with 80/20 stratified train/validation splits.

The baselines are Optuna with NSGA-II multi-objective sampling and Ray Tune with asynchronous parallel sampling. All methods share the same search space, training pipeline, dataset splits, and total trial budget. Reported metrics are validation accuracy, energy, total training time, and total optimization time. Each experiment is repeated 100 times, and results are reported as means with 95% confidence intervals (Iqbal et al., 8 Jan 2026).

On FSDD, EARL reaches $f_1(x)$5 accuracy, $f_1(x)$6 pJ/sample, 10.31 minutes of train time, and 10.57 minutes of optimization time. Optuna records $f_1(x)$7 accuracy and $f_1(x)$8 pJ/sample, while Ray reaches $f_1(x)$9 and $f_2(x)$0 pJ/sample, both with optimization times above 100 minutes (Iqbal et al., 8 Jan 2026).

On HAR, EARL reports $f_2(x)$1 accuracy, $f_2(x)$2 pJ/sample, 18.73 minutes of train time, and 19.53 minutes of optimization time. Optuna yields $f_2(x)$3 and $f_2(x)$4 pJ/sample; Ray yields $f_2(x)$5 and $f_2(x)$6 pJ/sample, again with substantially longer runtimes (Iqbal et al., 8 Jan 2026).

On Occupancy, EARL reaches $f_2(x)$7 accuracy, $f_2(x)$8 pJ/sample, 14.42 minutes of train time, and 15.28 minutes of optimization time. Optuna reports $f_2(x)$9 and $\mathrm{pJ/sample}$0 pJ/sample, whereas Ray matches the $\mathrm{pJ/sample}$1 accuracy level but at $\mathrm{pJ/sample}$2 pJ/sample and with much longer optimization time (Iqbal et al., 8 Jan 2026).

Across the three benchmarks, the paper summarizes the aggregate effect as 6–15% higher accuracy, 60–80% lower energy consumption, and up to an order of magnitude reduction in optimization time relative to the compared frameworks (Iqbal et al., 8 Jan 2026). The trajectory plots reportedly show convergence in 30–45 trials, while the baselines continue exploring longer with worse trade-offs (Iqbal et al., 8 Jan 2026). A plausible implication is that the combination of scalarized energy-aware reward, RL-based candidate prioritization, and whole-run early termination improves sample efficiency at the HPO level rather than only at the model-training level.

6. Scalability, limitations, and nomenclatural ambiguity

EARL is explicitly described as a framework for moderate-dimensional HPO, and the concrete instantiation optimizes four primary hyperparameters. The Gaussian Process core is acknowledged to scale poorly in very high-dimensional spaces or at very large trial counts, although the method partially mitigates this with Sobol initialization, RL-assisted candidate choice, and early termination (Iqbal et al., 8 Jan 2026). The paper further notes that alternative surrogates such as TPE, random forests, or neural surrogates could, in principle, replace the GP (Iqbal et al., 8 Jan 2026).

Several limitations are stated or implied. GP-EI may still become trapped in local basins in extremely noisy or adversarial landscapes; RL training itself consumes time and may overfit noisy reward signals; the scalarization parameter $\mathrm{pJ/sample}$3 must be chosen appropriately because large $\mathrm{pJ/sample}$4 values bias strongly toward low-energy configurations and small $\mathrm{pJ/sample}$5 values shift the search toward accuracy; and the reported energy measurements are GPU-based rather than neuromorphic hardware-in-the-loop (Iqbal et al., 8 Jan 2026). The paper also does not present a detailed ablation isolating BO-only, RL-only, or early-termination-only variants, so the individual contribution of each component remains analytically separable but experimentally entangled (Iqbal et al., 8 Jan 2026).

The architecture is nevertheless described as model-agnostic in structure: it requires only a search space $\mathrm{pJ/sample}$6, a predictive metric $\mathrm{pJ/sample}$7, an energy metric $\mathrm{pJ/sample}$8, and the reward definition $\mathrm{pJ/sample}$9. The authors explicitly state that the same BO+RL+early-termination scheme can be applied to general spiking neural networks, conventional deep nets, or other ML models, provided that practical energy measurement methods exist (Iqbal et al., 8 Jan 2026). This suggests that the LSM instantiation is specific, but the optimization template is broader.

A final source of confusion is the acronym itself. In the LSM paper, EARL means Energy-Aware Reinforcement Learning, whereas the literature also uses EARL for Environments for Autonomous Reinforcement Learning (Sharma et al., 2021), Efficient Agentic Reinforcement Learning Systems for LLMs (Tan et al., 7 Oct 2025), and, in a different line of work, ExpA Reinforcement Learning over an Expanded Action space for LLMs (Yue et al., 8 Oct 2025). The phrase “ExpA Reinforcement Learning (EARL)” is therefore not a stable field-wide designation. In the specific context of LSM hyperparameter optimization, the precise referent is the energy-aware BO+RL framework of (Iqbal et al., 8 Jan 2026), and “ExpA” is best treated as an interpretive shorthand for its adaptive exploration–exploitation policy rather than as the paper’s formal terminology.