Energy-Optimal On-Device AI

• Energy-optimal on-device AI is a field that employs algorithmic, architectural, and system-level techniques to reduce energy consumption on resource-constrained devices while maintaining accuracy.
• It leverages methods such as quantization, structured pruning, and hardware-aware neural architecture search to achieve significant reductions in energy usage and computational overhead.
• Adaptive inference, dynamic scheduling, and neuromorphic designs further enable devices to operate within strict energy budgets, often achieving energy savings of over 60% in practical scenarios.

Energy-optimal on-device AI encompasses algorithmic, architectural, and system-level techniques that enable learning and inference on resource-constrained devices under strict energy budgets, while ensuring performance and accuracy requirements. This objective spans the full stack, from predictive energy modeling and device-specific optimization through novel hardware, quantization/pruning methods, adaptive runtime control, and job scheduling. The following sections synthesize core principles, technical methodologies, empirical results, and open questions, as developed across the contemporary literature.

1. Foundational Principles and Modeling Paradigms

Recent research establishes that the total energy cost of on-device AI, especially for deep neural networks (DNNs), can be robustly decomposed and estimated at a fine granularity. The THOR framework identifies an additive layer-wise energy property: for a given $n$-layer DNN, total training energy $E_{\rm total}$ is modeled as

$$E_{\rm total} = E_{\rm input}(C_1) + \sum_{i=2}^{n-1} E_{\rm hidden}(C_{i-1}, C_i) + E_{\rm output}(C_{n-1}),$$

where $C_i$ is the channel width or hidden dimension at the $i$th layer, and energy per layer depends parametrically on input/output cardinality and other hyperparameters (e.g., kernel size, batch size) (Zhang et al., 27 Jan 2025). This insight enables parameter sharing across structurally identical layers, reduces measurement overhead, and reveals that DNN training energy can be predicted for arbitrary architectures from a small number of per-layer measurements.

Energy prediction is most often operationalized using Gaussian Process (GP) regression, both in THOR and in broader system profiling efforts such as EPAM, where DNN structure, runtime configuration, and device features feed into a GP whose posterior mean and variance yield robust energy estimates (Mallik et al., 2023). Surrogate modeling with GPs and random forests is also central to systems such as EnerInfer for LLM workloads, which disaggregate power and throughput prediction by accelerator configuration, thus enabling fast, generalizable energy estimation as a function of model structure and hardware state (Zou et al., 22 Jun 2026). Empirical mean absolute percentage error (MAPE) for these surrogate models is typically 2–15% for inference and training.

2. Compression, Quantization, and Model-Topology Optimization

Multiple energy-aware approaches reduce the per-inference or per-training energy by compressing models through quantization, pruning, low-rank approximation, and knowledge distillation (Millar et al., 18 May 2025). Fixed-point and mixed-precision quantization (down to 4–8 bits for weights and activations) can reduce dynamic arithmetic power by up to $8\times$, and decrease DRAM bandwidth requirements (as lower bit tensors shrink transfer cost per operand) (Millar et al., 18 May 2025, Lokhande et al., 10 Jun 2025). Layerwise mixed-precision assignments, as in POLARON, are chosen following quantization-aware sensitivity profiling, ensuring per-layer bit-widths meet global accuracy-loss budgets (Lokhande et al., 10 Jun 2025).

Structured pruning techniques exploit both weight and channel sparsity. Energy-aware pruning leverages per-layer energy gradients $\partial E/\partial C$ (as in THOR), guiding the iterative removal of units/filters that yield maximum energy reduction per infinitesimal accuracy drop, sometimes achieving $>50\%$ total energy savings (Zhang et al., 27 Jan 2025). Adaptive multi-exit and early-exit designs insert classifiers at intermediate layers; confident predictions at shallow depths allow immediate halting, saving up to $42\%$ of energy over always-full-depth execution (Merkel, 2020).

Hardware-aware neural architecture search (NAS) methods incorporate offline or surrogate energy models into the search process, optimizing directly for both accuracy and energy or energy-delay-product (EDP). Multi-objective frameworks (e.g., NSGA-II-based as in (Tundo et al., 2023), Bayesian optimization with RL as in EARL (Iqbal et al., 8 Jan 2026)) guide candidate generation in both model topology and runtime parameters, efficiently locating Pareto-optimal tradeoffs between energy, accuracy, and latency.

3. Hardware-Software Co-Design and Microarchitectural Techniques

Energy-optimality mandates cross-layer co-design, integrating model properties with the capabilities and constraints of the target hardware. Modern on-device accelerators and co-processors (GAP9 NE16, POLARON PARV-CE, AI-integrated RISC-V SoC, Loihi 2, FPGA/ASIC SNN platforms) support a range of features enabling aggressive energy savings:

• Unified, runtime-reconfigurable SIMD datapaths support multiple precisions (4/8/16-bit fixed/floating/posit), allowing on-the-fly adaptation to layerwise sensitivity without dark-silicon overhead (Lokhande et al., 10 Jun 2025).
• Dedicated accelerator primitives (e.g., block circulant or FFT-mapped multiply for DNN FC layers, event-driven SNN synaptic integration) reduce arithmetic cost by orders of magnitude (Islam et al., 2021, Dobrita et al., 2024, Imanov et al., 2 Feb 2026).
• On-chip high-density eDRAM or SRAM with application-driven refresh or checkpoint strategies allows sustaining both working set and gradient/activation storage with minimal leakage and off-chip access, as in CAMEL and EON-1 (Zhang et al., 2023, Dobrita et al., 2024).
• Power-cycling, clock gating, on-chip memory-compute co-location, and fine-grained duty-cycling further minimize dynamic and idle energy, especially for always-on or intermittent workloads (Moosmann et al., 2023, Islam et al., 2021).

Table: Empirical Energy Efficiency of Recent On-Device Platforms

Platform	Energy/inference (mJ)	Throughput	Key Feature
POLARON (ASIC, 28nm)	5.7pJ/MAC @ 4-bit	>10 fps (DNNs)	Unified multi-precision SIMD
TinyissimoYOLO/GAP9	1.27	18 fps (object det)	MCU w/ 8-core accel
CAMEL (eDRAM, 16nm)	$2.8\times$ less than SRAM	2.5x speedup	Refresh-free reversible DNN
EON-1 (SNN ASIC, 22nm)	<0.2	60 fps (UHD video)	Binary STDP, 1% learning ovh
NeuEdge (Loihi2)	1.21	2–4 ms latency	SNN, adaptive coding

(Lokhande et al., 10 Jun 2025, Moosmann et al., 2023, Zhang et al., 2023, Dobrita et al., 2024, Imanov et al., 2 Feb 2026)

4. Dynamic Scheduling, Adaptive Inference, and Feedback Control

Device-aware scheduling and runtime adaptation are critical to realizing energy-optimality under battery, thermal, and workload constraints. Several approaches address system-level coordination:

• Per-job energy estimation (e.g., via THOR's $E_{\rm total}$0) enables admission control and batch scheduling: jobs are only admitted if cumulative predicted energy will not exceed budget (Zhang et al., 27 Jan 2025).
• The MNN-AECS system dynamically selects core affinity for different LLM decode phases; using a reduced-core, energy-efficient profile for the memory-bound autoregressive decode yields $E_{\rm total}$1–$E_{\rm total}$2 energy reduction without user-perceptible slowdown (Huang et al., 24 Jun 2025).
• EnerInfer manages NPU and memory (DDR) frequencies based on model-aware throughput/power prediction, selecting Pareto-optimal configurations that maintain QoE while minimizing energy. Thermal controllers use MPC-driven predictions to avoid shell temperature violations, dynamically throttling runtime if necessary. Gains of up to $E_{\rm total}$3 energy reduction and strict adherence to throughput constraints are demonstrated (Zou et al., 22 Jun 2026).

Multi-exit, multi-model, and early exit controllers can adapt energy per input based on momentary energy storage state, prediction confidence, and ambient energy harvesting. Theoretical analysis via MDPs demonstrates that confidence-aware and incremental (exit-by-exit) control yield up to $E_{\rm total}$4 higher effective accuracy under energy-harvesting constraints (Bullo et al., 2024).

5. Neuromorphic and SNN-Based Energy-Optimal Designs

Event-driven, sparsity-exploiting computing as in spiking neural networks (SNNs) delivers state-of-the-art energy efficiency for temporal and edge AI workloads. NeuEdge introduces hybrid temporal-rate coding and adaptive neuron thresholding, cutting spike counts $E_{\rm total}$5 and reducing energy up to $E_{\rm total}$6 (drone case study) compared to DNN baselines, without accuracy loss (Imanov et al., 2 Feb 2026). On-chip SNN processors (EON-1) achieve sub-1% learning overhead, high throughput on HD/UHD video, and runtime adaptation for continuous on-device learning under low power (Dobrita et al., 2024).

Liquid state machine optimization under joint energy and accuracy via Bayesian surrogate modeling and RL-based candidate selection (EARL) demonstrates $E_{\rm total}$7–$E_{\rm total}$8 energy reduction and $E_{\rm total}$9–$$E_{\rm total} = E_{\rm input}(C_1) + \sum_{i=2}^{n-1} E_{\rm hidden}(C_{i-1}, C_i) + E_{\rm output}(C_{n-1}),$$0 accuracy gain over hyperparameter search baselines, underscoring the value of energy modeled as a primary objective (Iqbal et al., 8 Jan 2026).

6. Application Domains and Empirical Guidelines

Case studies and benchmarking frameworks extend energy-optimal methodologies to diverse domains:

• On-device LLMs: Counterintuitive findings show that memory-bound decode phases dominate battery drain; quantization alone has negligible energy benefit unless model memory use is coupled with reduced active compute. MoE LLMs achieve superlinear energy efficiency per quality by activating only a subset of experts per token (Ehsani et al., 27 Mar 2026).
• Embedded vision in smart glasses: With power-cycling hardware and aggressive quantization/tiny YOLO, system-level energy drops to $$E_{\rm total} = E_{\rm input}(C_1) + \sum_{i=2}^{n-1} E_{\rm hidden}(C_{i-1}, C_i) + E_{\rm output}(C_{n-1}),$$1 mJ/inference, enabling $$E_{\rm total} = E_{\rm input}(C_1) + \sum_{i=2}^{n-1} E_{\rm hidden}(C_{i-1}, C_i) + E_{\rm output}(C_{n-1}),$$2 hours continuous operation on miniature batteries (Moosmann et al., 2023).
• Edge speech enhancement: End-to-end pipeline co-design with quantization-aware training, dual-path time-frequency networks, and state caching yields $$E_{\rm total} = E_{\rm input}(C_1) + \sum_{i=2}^{n-1} E_{\rm hidden}(C_{i-1}, C_i) + E_{\rm output}(C_{n-1}),$$3 mJ/inference on a sub-100 mW wearable, retaining high-quality speech enhancement (Itani et al., 24 Mar 2025).
• Microcontrollers: Pareto front analysis of energy and accuracy across pruned/quantized models and processor cores (Cortex-M0+/M4/M7) enables precise mapping from application duty-cycle to the optimal hardware-model pairing (Jain et al., 19 Feb 2026).

Across domains, multi-objective search methods (e.g., genetic, Bayesian, RL-driven) together with prediction-driven profiling produce practical operation points that optimally trade off energy, accuracy, and latency for the device and task class.

7. Open Challenges and Research Directions

Several technical frontiers remain:

• Universal, structure-based predictive energy models that generalize across hardware, DNN types, and configuration spaces are needed to automate NAS and inference scheduling at scale. Representative approaches include surrogate models using graph/AST encodings, but cross-platform generalization is not solved (Millar et al., 18 May 2025, Mallik et al., 2023).
• Achieving robust energy-optimal operation under intermittent-power (via energy harvesting or power failures) requires further advances in refresh-free memory, reversible computation, and minimal checkpointing (Islam et al., 2021, Zhang et al., 2023).
• Efficient on-device training under severe memory/energy limits (microjoule regime) demands reversible architectures, ultra-low-leakage memory subsystems, integer/fixed-point arithmetic, and selective parameter updates guided by fine-grained energy models (Zhang et al., 2023, Heppe et al., 2020).
• Energy-aware multi-modal and transformer-based networks, particularly for vision transformers and RL agents, remain less explored, especially on tiny MCU/ASIC targets (Millar et al., 18 May 2025).
• Adaptive runtime algorithms that combine dynamic DVFS, active/inactive core management, early-exit, and energy-aware batching to maximize utility under fluctuating energy or workload regimes are still an active area (Zou et al., 22 Jun 2026, Huang et al., 24 Jun 2025).

Energy-optimal on-device AI is characterized by a convergence of model- and device-aware optimization, predictive modeling, cross-layer co-design, and real-time adaptive control. The field continues to progress towards principled, practical frameworks that automatically discover and deploy solutions maximizing energy efficiency for a given accuracy and latency profile on any device and workload class. (Zhang et al., 27 Jan 2025, Imanov et al., 2 Feb 2026, Iqbal et al., 8 Jan 2026, Ehsani et al., 27 Mar 2026, Millar et al., 18 May 2025)