---
title: Cray Power Management Overview
url: https://www.emergentmind.com/topics/cray-power-management
type: topic
---

# Cray Power Management Overview

In published work on Cray XC-30 and HPE Cray EX systems, Cray power management denotes a stack of measurement, accounting, and control mechanisms that spans embedded power sensing, job-attributed energy reporting, node- and device-level operating-point control, and workload-aware optimization of energy to solution rather than only time to solution. The literature is anchored by Cray’s Power Management Database (PMDB) on XC-30/Piz Daint and by telemetry-driven GPU power-management studies on Frontier, while related work on MPI slack reclamation, checkpointing, and processor-local closed-loop control clarifies how Cray-like machines can couple application behavior, runtime policy, and hardware control under strict power budgets [1408.2657][2408.01552].

## 1. Measurement and observability on Cray XC-30

Cray introduced PMDB on XC-30 as a **PostgreSQL database** available with the **Cray SMW** that **collects power measurements from hardware sensors throughout the system** and exposes **power, energy, current, and voltage** for **racks, blades, and nodes**. On Piz Daint, PMDB collected readings at **1 Hz** for **every node**, **every GPU**, **every blower**, and **every network chip**. This built-in sensing infrastructure was intended to make power “readily available” in a way analogous to performance counters, and to avoid intrusive instrumentation or cumbersome external metering campaigns [1408.2657].

The XC-30 measurement model distinguished several scopes. Per-node readings covered **total node power consumption** and, on hybrid nodes, **GPU power consumption**. Separate sensors monitored the **Aries network chip**, and PMDB also recorded **blower power**. The distinction is operationally important because node- and job-level views isolate application energy, whereas cabinet- and facility-level views include additional subsystems. The literature therefore treats “power” not as a single number but as a boundary-dependent quantity.

| Interface | What it provides | Main caveat |
|---|---|---|
| **PMDB** | 1 Hz time series for nodes, GPUs, Aries chips, blowers | Hosted on the SMW and not directly available to normal users |
| **RUR** | Total energy consumed by each application run on a compute node | Post-run only; cannot isolate a code region |
| **sysfs `pm_counters`** | `power`, `energy`, `generation`, `startup`, `freshness`, `version`, `power_cap` | Polling causes a system interrupt |

Cray also exposed two user-visible paths outside the database. The **Resource Utilization Report (RUR)** provided total energy consumed by each application run on a compute node after job completion. The sysfs **pm counters** provided point-in-time power and accumulated energy during execution, with files updated at **10 Hz**. The warning attached to the sysfs path is explicit: polling those counters causes a **system interrupt**, so they should **not** be used for high-frequency performance monitoring. Their intended use is coarse-grain instrumentation, such as reading cumulative energy before and after a long timestep loop.

The XC-30 studies also showed that the **Aries network has almost constant power consumption**, reported as **100 W per rack**, or about **25 W per node**, and **largely independent of network load**. Blower power, by contrast, varied with cooling demand, approximately **4400–5600 W per blower**. This separation between relatively stable network power and load-dependent cooling power is central to Cray energy accounting because it determines which components can be treated as quasi-constant corrections and which must be measured dynamically.

## 2. Power and energy accounting boundaries

A defining feature of Cray power management on XC-30 is the explicit treatment of accounting boundaries. PMDB sensor values were taken on the **DC side** and did **not include AC/DC conversion losses**. The CSCS validation study reported that, for the tested applications, an assumed constant **95% efficiency** matched external measurements well. On that basis, the total job energy excluding blowers was estimated as

$$
\text{total energy} = \frac{E_n + N/4\times100\times\tau}{0.95},
$$

where \(E_n\) is nodal energy summed over the \(N\) nodes and \(N/4 \times 100 \times \tau\) is the estimated Aries network energy. The corresponding expression including blowers was written as

$$
\text{total energy} = \frac{E_n + N/4\times100\times\tau + B \times 4440 }{0.95}.
$$

These formulas formalized a practical distinction that recurs throughout Cray power work: node- or job-level energy is not the same as wall energy unless network, blower, and conversion terms are added [1408.2657].

The same study used **HPL** as a prototypical optimization target and framed energy-to-solution through

$$
\mbox{ETS}=\int_0^{T} \bar{W} \, dt = \bar{W}\times\mbox{TTS}.
$$

For fixed problem size, minimizing ETS is therefore equivalent to maximizing performance per watt. On the hybrid XC-30, this mattered because the reported energy efficiency of the **NVIDIA K20X GPU** was about **5.83 GFLOPS/W**, whereas the **Sandy Bridge E5-2670 CPU** was about **1.44 GFLOPS/W**. The PMDB-guided tuning workflow therefore favored pushing **DGEMM** work onto the GPU and reducing CPU frequency.

The operational value of PMDB was demonstrated on small pilot runs and then on a full **5256-node** Green500 HPL configuration with process grid **\(72 \times 73\)**, matrix size **\(N = 3612672\)**, CPU frequency **1.9 GHz**, and all DGEMM on GPU. The available Sandy Bridge frequencies ranged from **1.2 GHz up to 2.6 GHz**, plus **2.601 GHz** corresponding to turbo, exposed through `aprun --p-state`. PMDB-guided experiments on only **8 nodes** indicated that **turbo boost (2.601 GHz)** was very energy inefficient and that the best **GFLOPS/W** was achieved at about **1.9 GHz** with **DGEMM split ratio = 1**. The final Green500 run used **1.9 GHz**.

The measured average powers for that run were **1526 kW** from RUR, **1536 kW** from PMDB job-level accounting, **1600 kW** from PMDB cabinet-level accounting, and **1635 kW** from facility measurement. The interpretation was not that one measurement path was wrong, but that each path included a different electrical boundary. The study concluded that the differences between any two measurements were **within 1%** once network, blower, and AC/DC corrections were handled consistently.

A common misconception in this area is that external meters are automatically ground truth. The XC-30 validation contradicted that assumption in one COSMO experiment, where a **single-cabinet external meter disagreed by about 5%** with PMDB and was later found to need recalibration. Within the limits of the reported experiments, PMDB functioned not only as an observability tool but also as a validation reference for operational metering.

## 3. Runtime power control for Cray-like MPI workloads

At the runtime layer, power management research for Cray-like systems has focused on tightly coupled MPI applications in which energy is wasted during synchronization slack. A prominent example is **COUNTDOWN Slack**, a user-space MPI-interposition runtime designed to reduce node energy consumption **without requiring source-code changes** and with **negligible overhead**. Although evaluated on an **IBM NeXtScale cluster** rather than a Cray platform, the study is explicitly framed as directly relevant to **Cray-like large-scale HPC systems** because it targets dual-socket x86 nodes, distributed-memory MPI execution, and node-local DVFS under production constraints [1909.12684].

The method rests on the decomposition

$$
T_{comm} = T_{slack} + T_{copy},
$$

with the central claim that only \(T_{slack}\) should be targeted for low-power operation if the objective is performance-neutral energy saving. Prior approaches were criticized for relying too heavily on prediction of future phase lengths and for lowering frequency across all MPI time, including copy time. COUNTDOWN Slack instead used a **reactive** policy: it isolated slack with inserted synchronization, then applied a **timeout algorithm** so that low-power states were used only when the slack interval was long enough to overcome hardware DVFS latency.

For collective MPI calls, the runtime intercepted the call, inserted an **`MPI_Barrier` on the same communicator**, applied the timeout-based low-frequency policy during that barrier, restored maximum frequency when the barrier completed, and then invoked the original collective. For point-to-point calls, it inserted artificial **`MPI_Isend`/`MPI_Irecv`** operations plus **`MPI_Wait`** to construct a point-to-point “barrier” without deadlock. The implementation was a **shared library in C**, injected via **`LD_PRELOAD`**, wrapping **all MPI v3 PMPI symbols**, and running entirely in **user space**.

The control threshold was **\(500 \mu s\)**, reflecting measured Intel DVFS actuation behavior. On entry to an isolated slack region, the runtime armed a timer at **\(500 \mu s\)**; if the callback fired, it set the lowest P-state; on exit, it restored the highest P-state. The system used Intel MSRs through **MSR_SAFE**, and energy results referred to **CPU package + DRAM consumption** measured with **RAPL**.

Quantitatively, the reported averages across benchmarks and the **1056-core OMEN** run were **0.79%** execution-time overhead, **9.96%** energy saving, and **10.73%** power saving. For **OMEN_1056p**, the runtime reported **0.38%** overhead, **22.11%** energy saving, and **22.92%** power saving. The robustness claim was equally important: worst-case overhead was **3.02%**, and worst-case energy outcome was still positive, with minimum saving **1.04%**. This directly challenged predictive approaches such as Fermata, Andante, and Adagio, which in the reported comparisons could incur much larger overheads or even negative energy outcomes.

For Cray power management, the main implication is that MPI-runtime power control is most effective when it is **reactive, slack-only, and latency-aware**. A plausible implication is that system-level power caps and runtime-level slack reclamation should be treated as complementary rather than competing layers.

## 4. Checkpointing, recovery, and phase-dependent energy behavior

Power management on large Cray-style systems is inseparable from resilience overheads, particularly checkpoint/restart. An empirical study on **Mira**, an **IBM Blue Gene/Q** machine, analyzed how **application-level checkpointing and fault recovery affect power, execution time, and energy** on a large HPC system and stated explicitly that the findings are directly relevant to **Cray-style large-scale machines** because the same design tensions exist there: large node counts, strict power budgets, checkpoint/restart for resilience, shared interconnect and storage pressure, and the need for scheduler/runtime cooperation [2109.01943].

The study was measurement-driven rather than model-driven. It used **MonEQ**, a **user-level profiling library** that collects power information at the **node card level**, reporting both aggregate node-card power and a breakdown into six hardware power domains: **DRAM voltage**, **link chip voltage**, **SRAM voltage**, **optics voltage**, **PCIExpress voltage**, and **link chip core voltage**. The core metrics were **average power**, **execution time**, and **energy**, and the study repeatedly emphasized that **execution time dominates energy differences** more than average power does.

The strongest general result was that adding checkpoints usually does **not** increase average power much and often slightly decreases it, but it significantly increases execution time and therefore energy. Across the presented NPB checkpoint-only cases, reported average-power changes ranged roughly from **\(-3.58\%\)** to **\(+0.98\%\)**, while checkpoint-only time overheads ranged from **6.41% to 38.85%** and energy overheads from **4.17% to 27.95%**. This directly contradicts the common assumption that checkpoint phases are necessarily power peaks. The interpretation offered in the study was that checkpointing is **not computation-intensive**, so chip-core activity falls during checkpoint phases and average node power may remain flat or decrease slightly.

The resilience path was implemented with **FTI (Fault Tolerance Interface)** for NPB and STREAM. Because **there is no local disk on nodes**, FTI saves **level 1, 2, and 3 checkpoints in memory**. That detail became especially important for **STREAM**, which was selected as a **memory-bandwidth-intensive** benchmark. With checkpoints added, STREAM showed **33.15%** execution-time increase and **31.99%** energy increase, and the study explicitly attributed this to competition for memory resources and to **higher DRAM power** than in the other benchmarks.

Recovery overheads were even more asymmetrical than checkpoint overheads. For **LU** at **8192 nodes**, injected faults relative to the no-FTI baseline produced **+121.78%** execution time and **+129.38%** energy. Even relative to the checkpointed run, one fault pattern caused **+90.69%** time and **+91.60%** energy. The study identified recovery from **level 4 checkpoint** as the **most time-consuming recovery process** and stated that, as the number of nodes increases, recovery takes more time. At the same time, it also concluded that, **if cases with fatal error are ignored, FTI recovery is very efficient**, with nonfatal incremental overheads of **0.30% to 6.27%** in execution time and **0.31% to 4.79%** in energy.

For Cray power management, the practical lesson is that checkpointing is primarily an **energy problem through runtime extension**, while fault recovery is also a **power stability** problem. The study reported more outliers and more spikes after injecting faults, with larger fluctuations due to alternating retrieval and re-computation phases. This suggests that scheduler and runtime policies should be checkpoint-aware, memory-domain-aware, and especially sensitive to **tail recovery cost** rather than only to mean checkpoint overhead.

## 5. System-scale GPU power management on HPE Cray EX

On **Frontier**, an exascale **HPE Cray EX** system at OLCF, power management research has moved toward telemetry-driven, system-scale estimation of what selective GPU **DVFS/frequency capping** or **power capping** could achieve. The system was described as having **9408 compute nodes**, **1.9 EF** peak performance, and **29 MW** peak power. Each compute node had **4 AMD MI250X GPUs**, each GPU with **2 GCDs**, **64 GB HBM2E per GCD**, **560 W** max GCD power, and **1700 MHz** max GCD frequency. The study focused on GPU controls because, on a fully utilized node, other components were said to be **less than 20%** and therefore “dwarfed” by GPU energy use [2408.01552].

The telemetry stack combined out-of-band node telemetry, **SLURM** scheduler logs, per-node scheduler data, and **ROCm SMI**. Power telemetry was captured at **2-second intervals** and aggregated to **15-second intervals**. The resulting job-level view provided GPU power utilization at a **15-second frequency from each GPU on every node** used by the workload. Over the reported **3 months** of Frontier telemetry, total GPU energy use was **16820 MWh**.

A benchmark-informed classification then mapped telemetry into four **power modes**:

| Power mode | GPU power range | Share of GPU hours |
|---|---:|---:|
| Latency, Network or I/O bound | `\(\leq 200\)` W | **29.8%** |
| Memory intensive (M.I.) | `200-420` W | **49.5%** |
| Compute intensive (C.I.) | `420-560` W | **19.5%** |
| Boosted frequency | `\(\ge 560\)` W | **1.1%** |

This classification was explicitly benchmark-driven. The paper treated **memory-intensive** execution as the main opportunity class because lowering GPU frequency often left runtime nearly unchanged while reducing power. By contrast, **latency / network / I/O-bound jobs** were stated to be poor candidates under the tested controls because reducing power or frequency proportionally slowed execution.

System-scale projections compared frequency caps from **1700 MHz to 700 MHz** and power caps centered on **500 W, 400 W, 300 W, 200 W**. For frequency capping, the most aggressive total projected saving was at **900 MHz**, with **1493.9 MWh** total savings, **8.8%** energy savings, and **11.2%** time increase. The practically important column was the study’s **\(\Delta T = 0\)** view, where **900 MHz** corresponded to **8.5%** energy savings **without performance loss**, equivalent to roughly **1438 MWh**. At that setting, the savings were overwhelmingly attributed to the memory-intensive region: **1438.3 MWh** from M.I. versus **55.6 MWh** from C.I.

Power capping produced smaller near-zero-slowdown projections. At **500 W**, the study projected **558.83 MWh** total savings, **3.32%** energy savings, and **0.1%** time increase, with **\(\Delta T = 0\)** savings of **3.2%**. At **200 W**, savings increased to **5.7%**, but time increase rose to **16.53%** and compute-intensive savings became negative. The reported conclusion was therefore that **frequency capping is more promising than power capping** on Frontier for the observed workload mix.

The study also stratified jobs by scheduling class and science domain. Frontier job classes were defined as **A**: **5645–9408 nodes**, **B**: **1882–5644 nodes**, **C**: **184–1881 nodes**, **D**: **92–183 nodes**, and **E**: **1–91 nodes**, with maximum walltime **12 h** for A/B/C, **6 h** for D, and **2 h** for E. Heatmap analysis indicated that most energy use, and therefore most potential savings, came from **large jobs**, especially classes **A** and **B**, and to some extent **C**. The study was careful to frame all results as an **upper ceiling** or **upper limit**, not as a deployed closed-loop controller inside Frontier’s production software stack. That distinction matters: the work established best-case projections grounded in real telemetry, not a production HPE Cray EX control plane.

## 6. Node-local control layers and joint policy design

Beyond deployed Cray telemetry and runtime mechanisms, the research literature points to two lower-layer design principles that are directly relevant to future Cray-like nodes: joint optimization of speed and idle-state behavior, and fast processor-local closed-loop control.

The first principle is represented by **SleepScale**, which analyzed runtime joint speed scaling and sleep-state management in a single-server setting with **DVFS** and multiple low-power states. The study modeled active power as \(P_0 f^3\), characterized each low-power state by a three-tuple \((P_i, \tau_i, w_i)\), and showed that the optimum is often neither “always fastest” nor “always slowest.” It explicitly rejected a universal race-to-halt rule and reported that, in one low-utilization DNS-like case, the global optimum used **\(C6S3\)** at **\(f=0.42\)** with about **70 W**, while race-to-halt could consume **50% more power**. Although this work was not about Cray systems, it was presented as highly relevant because the same node-level knobs exist in analogous form on HPC platforms. This suggests that Cray node software should not treat DVFS and idle-state selection as independent knobs, particularly when idle intervals vary between short synchronization waits and long scheduler-visible gaps [1404.5121].

The second principle is represented by **ControlPULP**, an open-source, **RISC-V**-based on-chip **Power Controller System (PCS)** targeted at a **72-core HPC processor**. The platform combined a **CV32E40P** manager core, an **8-worker** cluster accelerator with **64 KiB L1 RAM**, **512 KiB L2 RAM**, DMA engines, and **FreeRTOS**, and implemented a **Periodic Frequency Control Task** at **2 kHz** (**500 µs**) and a **Periodic Voltage Control Task** at **8 kHz** (**125 µs**). The reported role of the PCS was node-local: receive requests from OS and BMC, read sensors and workload information, estimate power, enforce power and thermal caps, and dispatch per-core frequency plus coarse-grained voltage-domain settings [2306.09501].

The quantitative results matter because they define the timescale at which future Cray-like nodes can react locally. For the full control action, the multi-core cluster achieved **5.5×** speedup over single-core execution for **72 controlled cores** and **6.5×** at **296 controlled cores**. For the overall PFCT step, the reported acceleration was **4.9×**, reducing the control step from **134.66 µs** to **27.282 µs** at **500 MHz**. In the accelerated case, the **500 µs** hyper-period retained about **95%** slack, leaving room for more sophisticated policies. In FPGA-based hardware-in-the-loop evaluation, the controller achieved **DVFS tracking with a mean deviation within 3% of system TDP**, with the emulated **TDP = 120 W** and per-core thermal limit **85 °C**.

For Cray power management as a whole, these studies imply a layered control hierarchy. Built-in telemetry and accounting characterize energy and establish trustworthy boundaries; runtime mechanisms exploit safe low-power intervals in applications; checkpoint-aware policies avoid converting resilience into avoidable energy cost; system-scale analytics identify which job classes merit selective capping; and processor-local controllers enforce sub-millisecond power and thermal constraints beneath OS and scheduler policy. A plausible implication is that future Cray systems will benefit most when these layers are coordinated rather than optimized in isolation.

Source: https://www.emergentmind.com/topics/cray-power-management