LLM Inference Energy Management

• LLM inference energy management frameworks are system-level methods that balance energy, latency, thermal performance, and sustainability targets for LLM deployments.
• They integrate hardware controls—such as DVFS, instance scaling, and batch size tuning—with runtime scheduling across on-device, cluster, and geo-distributed settings.
• They leverage phase- and workload-aware models combining offline energy prediction with online adjustment to achieve significant energy savings and improved efficiency.

An LLM inference energy management framework is a system-level method for minimizing or trading off inference energy against latency, quality of experience (QoE), thermal comfort, carbon emissions, water usage, or service-level objectives (SLOs) by observing runtime state and actuating serving or hardware knobs. Recent work spans on-device controllers that jointly manage NPU and memory frequencies and shell temperature, node-level GPU DVFS controllers, cluster-level reconfiguration across instance counts and tensor parallelism, geo-distributed request routing for sustainability, and offline energy-modeling layers used to choose serving configurations before deployment (Zou et al., 22 Jun 2026, Kakolyris et al., 2024, Stojkovic et al., 2024, Moore et al., 13 May 2026, Palladino et al., 11 May 2026).

1. Conceptual scope and system boundaries

The term covers several different classes of systems. Some papers describe full runtime frameworks. EnerInfer is presented as “the first on-device LLM inference framework that jointly manages energy efficiency, throughput, and thermal comfort for LLM workloads,” with control over NPU frequency, DDR frequency, and a runtime switch between energy-aware and thermal-aware modes (Zou et al., 22 Jun 2026). throttLL'eM is an “SLO-aware energy-management framework for LLM inference serving” that combines iteration-level GPU frequency scaling with coarser-grained instance autoscaling (Kakolyris et al., 2024). DynamoLLM is described as “the first energy-management framework for LLM inference environments” at cluster scope, dynamically reconfiguring number of instances, tensor parallelism, request-to-pool routing, and GPU frequency under TTFT and TBT SLOs (Stojkovic et al., 2024). MARLIN operates one level higher, as a geo-distributed cloud meta-scheduler that co-optimizes TTFT, carbon emissions, water usage, and energy cost by routing requests across datacenters (Moore et al., 13 May 2026).

Other papers provide partial framework components rather than a complete runtime controller. EnergyLens is an interpretable, closed-form energy modeling layer for configuration selection rather than an online scheduler (Palladino et al., 11 May 2026). “Offline Energy-Optimal LLM Serving” formulates offline workload assignment across multiple hosted LLMs using token-length-aware energy and runtime predictors, but assumes full prior knowledge of the workload, including output tokens (Wilkins et al., 2024). “ENERGY STAR” studies LLM-enabled software engineering tools through an offline benchmarking framework centered on RAG, prompt augmentation, and CodeCarbon-based measurement, and explicitly “does not present a runtime controller that dynamically routes requests, an online scheduler, admission control, autoscaling logic, carbon-aware serving, or multi-tenant batching management” (Thakur et al., 27 Jan 2026).

Framework	Scope	Reported outcome
EnerInfer (Zou et al., 22 Jun 2026)	On-device NPU/DDR/thermal control	Up to 65% efficiency gain on phones, 12% on a laptop, 24% on a development board
throttLL'eM (Kakolyris et al., 2024)	SLO-aware GPU frequency scaling and autoscaling	Up to 43.8% lower energy and 1.71×–1.78× higher energy efficiency
DynamoLLM (Stojkovic et al., 2024)	Cluster-level reconfiguration	53% energy reduction, 38% operational carbon reduction, 61% cost reduction
AGFT (Ye et al., 3 Aug 2025)	Single-node GPU frequency tuning	44.3% GPU energy savings, under 10% latency overhead, up to 40.3% EDP optimization
Camel (Xu et al., 7 Aug 2025)	Edge GPU frequency and batch-size tuning	12.4%–29.9% EDP reduction
MARLIN (Moore et al., 13 May 2026)	Geo-distributed cloud routing	At least 18% TTFT, 33% carbon, 43% water, and 11% energy-cost reduction

This range of scopes matters because the phrase does not denote a single canonical architecture. In current literature it can refer to online hardware control, serving-layer scheduling, cluster orchestration, geo-routing, or offline predictive planning.

2. Phase structure, workload characterization, and why inference energy is not monolithic

A recurring result is that LLM inference energy depends strongly on phase structure. Multiple papers distinguish prefill from decode. “Towards Greener LLMs” states that prefill is more compute-intensive while decode is more memory-intensive, and that longer inputs increase GPU parallelism demand and prolong prefill, whereas longer outputs increase decode iterations and queueing delay (Stojkovic et al., 2024). EnerInfer makes the same distinction on devices: prefill mainly determines TTFT, decoding dominates long-run power and thermal behavior, and prefill is “typically 15–25× faster than decoding,” so EnerInfer keeps prefill at peak hardware frequency and focuses optimization on decode (Zou et al., 22 Jun 2026).

Phase-aware energy analysis becomes more explicit in “Towards Green AI: Decoding the Energy of LLM Inference in Software Development.” That study reports that decoding dominates total inference energy in four out of five workloads, while long-context, short-output code understanding can make prefill dominant. It also reports that increases in prefill cost amplify the energy cost per token during decoding by 1.3% to 51.8%, depending on the model, and identifies “babbling behavior” in three out of ten models, with babbling suppression yielding 44% to 89% energy savings without affecting generation accuracy (Solovyeva et al., 5 Feb 2026).

Workload characterization is therefore central. DynamoLLM buckets requests by input and output length into nine request types—SS, SM, SL, MS, MM, ML, LS, LM, LL—and derives request-type-specific TTFT and TBT SLOs (Stojkovic et al., 2024). “Towards Greener LLMs” uses nine workload classes formed by small, medium, and large input lengths crossed with small, medium, and large output lengths, showing that the best frequency, batch size, and parallelism depend on workload type (Stojkovic et al., 2024). On the edge, Camel models request latency as the sum of batch-formation waiting time and batch execution time, making arrival rate part of the optimization problem rather than treating batching as a pure throughput mechanism (Xu et al., 7 Aug 2025).

This suggests that an energy-management framework must be phase-aware and workload-aware at the same time. Input length, output length, decode duration, and prompt-context inflation do not affect energy through a single uniform pathway.

3. Objectives and control surfaces

The literature uses several objective formulations, but all depart from pure speed maximization. EnerInfer defines energy efficiency for a configuration $u$ as

$\frac{\mathcal{R}(u)}{P(u)},$

constructs a 2D Pareto frontier over throughput and efficiency, and then chooses the most energy-efficient point whose predicted throughput exceeds the QoE target $r_{QoE}$ (Zou et al., 22 Jun 2026). Its explicit QoE dimensions include TTFT, decoding speed in tokens/s, and thermal comfort, with a shell-temperature threshold such as $42^\circ\mathrm{C}$.

Cloud and edge frameworks often encode the trade-off more directly. Camel formulates

$$\min_{f,b} \sum_{t=1}^{T} \left\{ \alpha E(t) + (1-\alpha)L(t) \right\},$$

where $f$ is GPU frequency, $b$ is batch size, $E(t)$ is average energy per request, and $L(t)$ is average latency per request; it also evaluates energy-delay product, $EDP = E \times D$ (Xu et al., 7 Aug 2025). A simpler heterogeneous-routing formulation appears in “Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloads,” which defines

$\frac{\mathcal{R}(u)}{P(u)},$0

for input tokens $\frac{\mathcal{R}(u)}{P(u)},$1, output tokens $\frac{\mathcal{R}(u)}{P(u)},$2, and system $\frac{\mathcal{R}(u)}{P(u)},$3, then partitions requests across hardware types (Wilkins et al., 2024).

At larger scales, objectives expand. throttLL'eM minimizes energy under KV-cache capacity, TBT, and E2E constraints, using throughput in iterations per second and the identity

$\frac{\mathcal{R}(u)}{P(u)},$4

to test compliance (Kakolyris et al., 2024). MARLIN minimizes a weighted sum of total latency, carbon emissions, water usage, and energy cost across epochs, subject to latency and memory constraints (Moore et al., 13 May 2026). Token-level hybrid frameworks go finer still: “Energy-Efficient Wireless LLM Inference via Uncertainty and Importance-Aware Speculative Decoding” makes a binary upload decision $\frac{\mathcal{R}(u)}{P(u)},$5 at each token, uploading only when a token is both uncertain and important (Park et al., 18 Aug 2025).

The control surface is correspondingly broad. Current papers expose NPU and DDR frequencies, GPU core frequency, batch size, tensor parallelism, pipeline parallelism, number of active instances, request routing, RAG enablement, prompt augmentation, speculative offloading, and decoding-time stopping behavior (Zou et al., 22 Jun 2026, Kakolyris et al., 2024, Stojkovic et al., 2024, Thakur et al., 27 Jan 2026, Park et al., 18 Aug 2025). A practical implication is that “LLM inference energy management framework” denotes a joint hardware–runtime control problem, not a single DVFS heuristic.

4. Modeling, observability, and prediction

A central challenge is limited observability. EnerInfer emphasizes that commercial devices often lack component-level power sensors, NPU kernel visibility, and rich performance counters, so it replaces per-model profiling with a disaggregated, model-structure-aware prediction pipeline. It uses six decoder-structure hyperparameters—number of layers, hidden size, number of attention heads, number of key-value heads, intermediate size, and vocabulary size—plus total parameter count for peak-throughput prediction, and reports combined throughput MAPEs of 4.2%, 2.3%, and 2.4% on laptop, phone, and board respectively; power-prediction MAPE is 1.5%, 2.2%, and 1.7% (Zou et al., 22 Jun 2026).

throttLL'eM addresses observability differently. It projects future KV-cache usage and batch size from active-request metadata, then uses an XGBoost model to predict throughput in iterations per second from engine size, batch size, KV-cache usage, and GPU frequency. Across evaluated engines, it reports $\frac{\mathcal{R}(u)}{P(u)},$6 from 0.97 to 0.99, MAPE from 2.8% to 5.8%, and MAE below 1 IPS, with model inference latency of about 3 ms on CPU (Kakolyris et al., 2024). AGFT removes prompt-content inspection entirely and instead uses a 7-dimensional context vector based on queue presence, prefill throughput, decode throughput, packing efficiency, concurrency, GPU cache usage, and cache hit rate, feeding a LinUCB contextual bandit that adapts GPU core frequency online (Ye et al., 3 Aug 2025).

Offline predictors cover a complementary regime. “Offline Energy-Optimal LLM Serving” fits bilinear per-model energy and runtime predictors of the form $\frac{\mathcal{R}(u)}{P(u)},$7 and $\frac{\mathcal{R}(u)}{P(u)},$8, obtaining $\frac{\mathcal{R}(u)}{P(u)},$9 for all energy and runtime fits across seven open-source LLMs on a heterogeneous CPU-GPU node (Wilkins et al., 2024). EnergyLens uses symbolic regression to derive a single twelve-parameter closed-form model that separates prefill from decode and decouples tensor from pipeline parallelism; fitted from as few as 50 profiling measurements, it achieves 88.2% Top-1 configuration selection accuracy across 940 evaluation scenarios and matches ensemble ML methods with 10× fewer profiling samples (Palladino et al., 11 May 2026).

These approaches differ in instrumentation burden, but they converge on the same design principle: direct energy prediction is more useful than treating latency as an energy proxy. EnergyLens explicitly states that latency and energy optima diverge in over 20% of tested configurations, and EnerInfer shows that the most energy-efficient point is not always the lowest frequency that meets throughput (Palladino et al., 11 May 2026, Zou et al., 22 Jun 2026).

5. Runtime architectures and representative control loops

The most fully developed frameworks combine offline models with online correction. EnerInfer builds throughput and efficiency tables once at initialization for an unseen LLM, chooses a Pareto-frontier configuration for decoding, monitors inter-token throughput, and escalates monotonically along the predicted ranking if runtime speed falls below $r_{QoE}$0. When shell temperature is predicted to cross a threshold within a short horizon, it switches into a thermal-aware mode inspired by model predictive control (Zou et al., 22 Jun 2026).

throttLL'eM uses a similar prediction–control split at cloud-node scale. On each request admission, it virtually inserts the request into a Scoreboard, projects future KV and batch trajectories assuming no additional arrivals, checks feasibility at maximum frequency, then performs binary search over supported GPU frequencies to find the minimum SLO-satisfying value. A slower autoscaler adjusts tensor-parallel engine size every 10 seconds, using shadow instancing because new-engine startup exceeds 20 seconds (Kakolyris et al., 2024).

DynamoLLM pushes the hierarchy upward. It has a Cluster Manager, Pool Manager, and Instance Manager operating at different time scales. The Cluster Manager predicts peak load for request-type pools and allocates nodes; the Pool Manager chooses TP composition and re-sharding; the Instance Manager adjusts GPU frequency every few seconds. It also includes emergency mechanisms—queue-based urgency reordering, max-frequency fallback, and rerouting of queued requests—to handle mispredictions or bursts (Stojkovic et al., 2024).

Edge frameworks are usually smaller but not simpler conceptually. Camel defines 49 arms from 7 GPU frequencies and 7 batch sizes, then uses a Thompson-sampling-style Gaussian bandit to choose the minimum-cost configuration online. It reports that after 49 search rounds Camel reduces cumulative regret relative to grid search by factors of 3.8 and 2.3 for Llama3.2-1B and Qwen2.5-3B, respectively (Xu et al., 7 Aug 2025). AGFT similarly treats GPU frequency as the action space, but couples LinUCB exploration with action-space pruning and maturity-based local refinement around a promising frequency anchor (Ye et al., 3 Aug 2025).

Hybrid and application-specific architectures broaden the notion of “control loop.” In wireless HLM inference, the loop is token-level: SLM draft token, uncertainty and importance estimation, binary upload decision, optional cloud verification, then continuation (Park et al., 18 Aug 2025). In RAG-enabled software-engineering tools, the loop is prompt-time rather than hardware-time: embed query, retrieve similar examples, augment prompt, run generation, and measure total workflow energy and inference time (Thakur et al., 27 Jan 2026).

6. Misconceptions, limitations, and open directions

A common misconception is that fastest is always best. EnerInfer explicitly argues that for on-device decoding “fastest” is often not “best,” shows a representative phone case where default on-device inference consumed more than 138% of the power of cloud offloading, and reports a configuration that reduced NPU+memory power by 44% while keeping throughput above 10 tokens/s (Zou et al., 22 Jun 2026). Another misconception is that the minimal configuration satisfying a deadline is automatically efficient. EnerInfer shows that the best energy-efficient point is not always the lowest frequency that meets throughput, and “Towards Greener LLMs” likewise reports that optimizing power, optimizing total energy, and optimizing performance can yield different preferred frequencies (Zou et al., 22 Jun 2026, Stojkovic et al., 2024).

Latency is also an unreliable energy proxy. EnergyLens states that latency and energy optima diverge in over 20% of tested configurations, and its comparison against MaverIQ shows large gaps in Top-1 configuration selection accuracy (Palladino et al., 11 May 2026). RAG is not uniformly energy-saving either: “ENERGY STAR” reports energy reductions with RAG for GPT-2 and CodeLlama, but increased energy for DeepSeek Coder and Qwen, while only CodeLlama was 25% faster with RAG (Thakur et al., 27 Jan 2026). Quantization and parallelism are similarly non-monotone: EnergyLens notes that 4-bit quantization can increase energy because of dequantization overhead and irregular access, while “Towards Greener LLMs” shows that more tensor parallelism is not automatically better in energy terms (Palladino et al., 11 May 2026, Stojkovic et al., 2024).

Current frameworks also expose clear limitations. EnerInfer is platform-specific, trains separate models for different quantization levels, assumes DVFS-capable accelerators, and does not natively support speculative decoding or dynamic sparsity because their throughput and power depend on history-dependent hit rates (Zou et al., 22 Jun 2026). throttLL'eM does not optimize TTFT directly and is less effective when the engine is already near the SLO boundary or KV capacity is small (Kakolyris et al., 2024). Camel does not include thermal-aware control, battery-state-aware policies, or hard QoS constraints (Xu et al., 7 Aug 2025). DynamoLLM focuses on tensor parallelism within a node and does not make batching a primary optimization variable (Stojkovic et al., 2024). MARLIN is simulator-evaluated and operates over a relatively narrow action space dominated by request-to-datacenter assignment (Moore et al., 13 May 2026).

These limitations suggest that the field is converging toward a layered architecture rather than a single universal controller. A plausible implication is that future LLM inference energy management frameworks will combine offline interpretable energy models, workload- and phase-aware online prediction, local hardware control, and cluster- or geo-level routing in one stack. The papers already identify several recurring requirements: phase-aware optimization, ranking fidelity rather than just absolute prediction accuracy, proactive thermal or sustainability control, and avoidance of assumptions such as component-level sensors, vendor-internal counters, or exhaustive per-model profiling (Zou et al., 22 Jun 2026, Palladino et al., 11 May 2026).