---
title: 'GreenLLM: Sustainable LLM Serving'
url: https://www.emergentmind.com/topics/greenllm-framework
type: topic
---

# GreenLLM: Sustainable LLM Serving

GreenLLM refers to a class of serving frameworks for large language models (LLMs), each designed to reduce energy consumption or carbon emissions during inference while meeting service-level objectives (SLOs) for latency and throughput. The most prominent instantiations are: (1) disaggregation across heterogeneous GPUs for carbon minimization [2412.20322], and (2) SLO-aware dynamic frequency scaling for GPU energy reduction [2508.16449]. Both frameworks target the computational and environmental inefficiencies inherent to contemporary LLM deployment at scale, leveraging phase-aware resource management and data-driven scheduling.

## 1. System Architectures and Design Principles

### Disaggregation-Based GreenLLM

The GreenLLM architecture for carbon minimization comprises three principal components [2412.20322]:

- **Disaggregated Execution Layer**: Employs two optimizers—Disg-Pref-Decode and Disg-Spec-Decode—that disaggregate computation between a new, high-performance GPU (e.g., NVIDIA A100) and an older, lower-performance GPU (e.g., T4 or V100).
- **Profiler**: A lightweight module that exhaustively measures per-phase latency (TTFT: time-to-first-token, TPOT: time-per-output-token), per-phase energy draw, and per-request carbon breakdown across a grid of request parameters.
- **SLO-Aware Scheduler**: A runtime engine that, for each incoming request characterized by input length and QPS, and given a target latency SLO, selects the configuration that minimizes total carbon emissions while satisfying SLO constraints via lookup tables and collaborative filtering for missing profiles.

A disaggregated cluster typically involves at least one node with a new GPU and one with an old GPU, connected by ≥10 Gbps network bandwidth, with orchestration and inter-GPU data transfer managed by the disaggregators.

### DVFS-Based GreenLLM

The energy-focused GreenLLM variant [2508.16449] integrates as a thin control layer in the serving stack (e.g., NVIDIA Dynamo + TensorRT-LLM):

- **Ingress and Length-Based Routing**: Requests are classified based on prompt length, enabling queueing to separate short from long prompts to mitigate head-of-line blocking.
- **Prefill and Decode Pools**: Separate worker pools for compute-bound prefill and memory-bound decode phases, each with phase-specific resource management.
- **Telemetry and Profiling**: Real-time collection of performance and energy metrics.
- **Control Plane**: Issues streaming-multiprocessor (SM) frequency updates to prefill and decode worker pools via NVML app-clocks, informed by latency, queue, and energy modeling.

## 2. Disaggregation Use Cases and Phase Optimization

### Phase-Splitting (Disg-Pref-Decode)

- **Prefill Phase** ($L_{\mathrm{in}}$): Processes the input prompt and builds the key-value (KV) cache, running on the new GPU $A$ due to high TFLOPs demand.
- **Decoding Phase**: Autoregressive generation is memory-bound; runs on the old GPU $B$. At the prefill/decoding boundary, the entire KV cache is serialized and DMA-transferred, requiring ≥10 Gbps for 7B-parameter models at 1 QPS.

### Speculative-Splitting (Disg-Spec-Decode)

- **Draft Model on Old GPU ($B$)**: Emits speculative tokens and draft probabilities.
- **Target Model on New GPU ($A$)**: Only token IDs are initially transferred; the validation and final output generation occur on the high-performance node.
- **Bandwidth Efficiency**: Communication consists only of token IDs and draft probabilities, reducing inter-GPU bandwidth demand by $65\times$ – $434\times$ compared to phase split.

### Phase-Specific DVFS

In the dynamic frequency scaling variant [2508.16449]:

- **Prefill**: Latency-power models (quadratic in prompt length) determine the energy-optimal clock frequency for a batch of prefill jobs, within SLO constraints.
- **Decode**: A dual-loop controller uses token-per-second (TPS) measurements and tail TBT to dynamically adjust GPU frequency via a combination of coarse (200 ms, TPS-bucketed) and fine (20 ms, TBT-based) feedback loops.

## 3. SLO-Aware Scheduling and Control Algorithms

Both GreenLLM variants operationalize SLO-aware optimization:

- **Scheduler for Disaggregated GreenLLM**: Maintains matrices $C[i,j]$ (carbon per token) and $\mathrm{SLO}_{\mathrm{att}}[i,j]$ (fraction SLO compliance) for each configuration $i$ and workload bin $j$. At runtime:
  1. Loads/completes the metrics using collaborative filtering.
  2. Identifies feasible configurations with $\mathrm{SLO}_{\mathrm{att}}[i,j] \geq \mathrm{SLO}_{\mathrm{target}}$.
  3. Selects $i^* = \arg\min_{i \in \mathrm{Feasible}_j} C[i,j]$, falling back to maximal SLO compliance otherwise.

- **DVFS-Selection for Energy-GreenLLM**: For prefill, an explicit optimization is solved per queue to minimize
  $$
  E_{\mathrm{total}}(f)
  $$
  subject to busy time constraint
  $$
  \frac{f_{\mathrm{ref}}}{f} T_{\mathrm{ref}} \leq D.
  $$
  For decode, dual nested loops maintain TBT within SLO by discrete frequency nudges.

## 4. Theoretical Analysis of Carbon and Energy Savings

- **Carbon Model for Disaggregated Serving** ([2412.20322]):
  - **Operational Carbon** $O_X = N_X \alpha$ for GPU $X$ (energy $N_X$, carbon intensity $\alpha$),
  - **Embodied Carbon** $E_X = \frac{t_X}{L_X} \mathcal{E}_X$ (manufacturing carbon $\mathcal{E}_X$ amortized over lifetime $L_X$ and usage time $t_X$).
  - **Total Carbon per Request**:
    $$
    C = \underbrace{\frac{t}{L} \mathcal{E}_{\text{embodied}}}_{\text{embodied}} + \underbrace{N \alpha}_{\text{operational}}
    $$
  - **Savings Condition**: Disaggregation strictly reduces carbon if
    $$
    N_A > N_A' + N_B
    $$
    where primed variables are those for the new GPU under disaggregation.

  - **Savings Scale with Grid Intensity and Lifetime**: Savings increase with higher grid carbon ($\alpha$), older GPU reuse ($L_B$ up), and shorter new GPU amortization ($L_A$ down).

- **Energy Model for DVFS** ([2508.16449]):
  - For compute-bound prefill,
    $$
    t_k(f) = t_k^{\mathrm{ref}} \frac{f_{\mathrm{ref}}}{f}
    $$
    $$
    P(f) = k_3 f^3 + k_2 f^2 + k_1 f + k_0; \quad E_{\mathrm{total}}(f) = P(f) \cdot \mathrm{busy}(f) + P_{\mathrm{idle}} [D - \mathrm{busy}(f)]
    $$
    The optimal $f$ is the minimal-energy frequency meeting all pending jobs' deadlines.

- *This suggests that programmer-level and operational control at the request and phase levels can be systematically leveraged for resource, energy, and carbon optimization in LLM serving.*

## 5. Implementation

- **Disaggregation (Carbon-Focused) GreenLLM**:
  - Built atop vLLM [2412.20322], refactored to enable phase-split and spec-split plugin execution models.
  - Profiler (Python + Shell) polls GPU power via pynvml at 200 ms intervals; utilizes REST for aggregation.
  - Heterogeneous processing with Ray actors, NCCL and PyTorch nonblocking communication for overlap of compute and transfer.
  - Scheduler in Python (~200 LOC) with collaborative filtering for profile completion (Paragon-style).

- **DVFS (Energy-Focused) GreenLLM**:
  - Integrated as a control plane over commodity GPU serving stacks (e.g., TensorRT-LLM).
  - Implements length-based routing, per-phase SM frequency updates via NVML.
  - Real-time phase-specific energy and latency profiling, convex modeling, enumerative frequency selection, and dual-level feedback control.

## 6. Empirical Evaluation and Reported Effectiveness

### Carbon Reduction (Disaggregation [2412.20322])

- Up to 40.6% total carbon reduction (ShareGPT app, under ≥90% SLO compliance).
- Savings up to 27.9% at lowest regional grid carbon intensity (NCSW, 17 gCO$_2$/kWh).
- Bandwidth ≥8 Gbps required for DPD mode at appreciable QPS; at lower bandwidths or higher QPS, SpecDecode and DSD dominate.
- Carbon savings scale with increasing old-GPU reuse (lifetime up to 10 years) and decrease for shortened new-GPU amortization (2 years).

### Energy Reduction (DVFS [2508.16449])

- Up to 34% total energy reduction for Qwen3-14B on Alibaba and Azure workloads, with <3.5% additional SLO violations.
- At low/mid QPS and TPS, energy-optimal frequencies reduce prefill energy by 10–30% and decode energy by 8–25%.
- TTFT/TBT SLOs are met for ≥88–94% of requests, up to saturation.

### General Observations

- Both frameworks maintain model-agnosticity, requiring no changes to LLM internals.
- Efficacy increases as operators can exploit request/phase heterogeneity, grid carbon knowledge, and hardware lifecycles.
- All results hold across several representative LLMs (Qwen3-14B, Llama-7B), workloads (chatbot, code gen, summarization), and system regimes.

## 7. Significance, Scope, and Extensions

GreenLLM frameworks demonstrate that SLO-aware, phase-specific resource management—either across heterogeneous hardware (disaggregation) or via dynamic device-level control (DVFS)—can realize substantial operational efficiency gains for LLM inference workloads. Their architectures present validated paths for carbon or energy abatement at scale without sacrificing service-level objectives. The analytical models provide quantitative bounds and inform principled scheduling policy. A plausible implication is that these approaches generalize to broader distributed DNN serving contexts, subject to extensions for emerging accelerator and networking substrates. The frameworks’ reliance on data-driven modeling, collaborative profile completion, and lightweight runtime controls positions them as practical solutions for sustainable, production-grade LLM deployment [2412.20322][2508.16449].

Source: https://www.emergentmind.com/topics/greenllm-framework