Cray Power Management Overview

• Cray Power Management is a suite of measurement, accounting, and control techniques that integrate embedded sensor data with workload-aware energy optimization.
• It leverages tools like PMDB and sysfs counters to provide high-resolution, component-specific power and energy readings critical for precise accounting.
• The approach also incorporates reactive runtime control, including MPI slack reclamation and GPU DVFS/frequency capping, to improve energy efficiency without significant performance loss.

In published work on Cray XC-30 and HPE Cray EX systems, Cray power management denotes a stack of measurement, accounting, and control mechanisms that spans embedded power sensing, job-attributed energy reporting, node- and device-level operating-point control, and workload-aware optimization of energy to solution rather than only time to solution. The literature is anchored by Cray’s Power Management Database (PMDB) on XC-30/Piz Daint and by telemetry-driven GPU power-management studies on Frontier, while related work on MPI slack reclamation, checkpointing, and processor-local closed-loop control clarifies how Cray-like machines can couple application behavior, runtime policy, and hardware control under strict power budgets (Fourestey et al., 2014, Karimi et al., 2024).

1. Measurement and observability on Cray XC-30

Cray introduced PMDB on XC-30 as a PostgreSQL database available with the Cray SMW that collects power measurements from hardware sensors throughout the system and exposes power, energy, current, and voltage for racks, blades, and nodes. On Piz Daint, PMDB collected readings at 1 Hz for every node, every GPU, every blower, and every network chip. This built-in sensing infrastructure was intended to make power “readily available” in a way analogous to performance counters, and to avoid intrusive instrumentation or cumbersome external metering campaigns (Fourestey et al., 2014).

The XC-30 measurement model distinguished several scopes. Per-node readings covered total node power consumption and, on hybrid nodes, GPU power consumption. Separate sensors monitored the Aries network chip, and PMDB also recorded blower power. The distinction is operationally important because node- and job-level views isolate application energy, whereas cabinet- and facility-level views include additional subsystems. The literature therefore treats “power” not as a single number but as a boundary-dependent quantity.

Interface	What it provides	Main caveat
PMDB	1 Hz time series for nodes, GPUs, Aries chips, blowers	Hosted on the SMW and not directly available to normal users
RUR	Total energy consumed by each application run on a compute node	Post-run only; cannot isolate a code region
sysfs pm_counters	power, energy, generation, startup, freshness, version, power_cap	Polling causes a system interrupt

Cray also exposed two user-visible paths outside the database. The Resource Utilization Report (RUR) provided total energy consumed by each application run on a compute node after job completion. The sysfs pm counters provided point-in-time power and accumulated energy during execution, with files updated at 10 Hz. The warning attached to the sysfs path is explicit: polling those counters causes a system interrupt, so they should not be used for high-frequency performance monitoring. Their intended use is coarse-grain instrumentation, such as reading cumulative energy before and after a long timestep loop.

The XC-30 studies also showed that the Aries network has almost constant power consumption, reported as 100 W per rack, or about 25 W per node, and largely independent of network load. Blower power, by contrast, varied with cooling demand, approximately 4400–5600 W per blower. This separation between relatively stable network power and load-dependent cooling power is central to Cray energy accounting because it determines which components can be treated as quasi-constant corrections and which must be measured dynamically.

2. Power and energy accounting boundaries

A defining feature of Cray power management on XC-30 is the explicit treatment of accounting boundaries. PMDB sensor values were taken on the DC side and did not include AC/DC conversion losses. The CSCS validation study reported that, for the tested applications, an assumed constant 95% efficiency matched external measurements well. On that basis, the total job energy excluding blowers was estimated as

$$\text{total energy} = \frac{E_n + N/4\times100\times\tau}{0.95},$$

where $E_n$ is nodal energy summed over the $N$ nodes and $N/4 \times 100 \times \tau$ is the estimated Aries network energy. The corresponding expression including blowers was written as

$$\text{total energy} = \frac{E_n + N/4\times100\times\tau + B \times 4440 }{0.95}.$$

These formulas formalized a practical distinction that recurs throughout Cray power work: node- or job-level energy is not the same as wall energy unless network, blower, and conversion terms are added (Fourestey et al., 2014).

The same study used HPL as a prototypical optimization target and framed energy-to-solution through

$\mbox{ETS}=\int_0^{T} \bar{W} \, dt = \bar{W}\times\mbox{TTS}.$

For fixed problem size, minimizing ETS is therefore equivalent to maximizing performance per watt. On the hybrid XC-30, this mattered because the reported energy efficiency of the NVIDIA K20X GPU was about 5.83 GFLOPS/W, whereas the Sandy Bridge E5-2670 CPU was about 1.44 GFLOPS/W. The PMDB-guided tuning workflow therefore favored pushing DGEMM work onto the GPU and reducing CPU frequency.

The operational value of PMDB was demonstrated on small pilot runs and then on a full 5256-node Green500 HPL configuration with process grid $72 \times 73$, matrix size $N = 3612672$, CPU frequency 1.9 GHz, and all DGEMM on GPU. The available Sandy Bridge frequencies ranged from 1.2 GHz up to 2.6 GHz, plus 2.601 GHz corresponding to turbo, exposed through aprun --p-state. PMDB-guided experiments on only 8 nodes indicated that turbo boost (2.601 GHz) was very energy inefficient and that the best GFLOPS/W was achieved at about 1.9 GHz with DGEMM split ratio = 1. The final Green500 run used 1.9 GHz.

The measured average powers for that run were 1526 kW from RUR, 1536 kW from PMDB job-level accounting, 1600 kW from PMDB cabinet-level accounting, and 1635 kW from facility measurement. The interpretation was not that one measurement path was wrong, but that each path included a different electrical boundary. The study concluded that the differences between any two measurements were within 1% once network, blower, and AC/DC corrections were handled consistently.

A common misconception in this area is that external meters are automatically ground truth. The XC-30 validation contradicted that assumption in one COSMO experiment, where a single-cabinet external meter disagreed by about 5% with PMDB and was later found to need recalibration. Within the limits of the reported experiments, PMDB functioned not only as an observability tool but also as a validation reference for operational metering.

3. Runtime power control for Cray-like MPI workloads

At the runtime layer, power management research for Cray-like systems has focused on tightly coupled MPI applications in which energy is wasted during synchronization slack. A prominent example is COUNTDOWN Slack, a user-space MPI-interposition runtime designed to reduce node energy consumption without requiring source-code changes and with negligible overhead. Although evaluated on an IBM NeXtScale cluster rather than a Cray platform, the study is explicitly framed as directly relevant to Cray-like large-scale HPC systems because it targets dual-socket x86 nodes, distributed-memory MPI execution, and node-local DVFS under production constraints (Cesarini et al., 2019).

The method rests on the decomposition

$T_{comm} = T_{slack} + T_{copy},$

with the central claim that only $T_{slack}$ should be targeted for low-power operation if the objective is performance-neutral energy saving. Prior approaches were criticized for relying too heavily on prediction of future phase lengths and for lowering frequency across all MPI time, including copy time. COUNTDOWN Slack instead used a reactive policy: it isolated slack with inserted synchronization, then applied a timeout algorithm so that low-power states were used only when the slack interval was long enough to overcome hardware DVFS latency.

For collective MPI calls, the runtime intercepted the call, inserted an MPI_Barrier on the same communicator, applied the timeout-based low-frequency policy during that barrier, restored maximum frequency when the barrier completed, and then invoked the original collective. For point-to-point calls, it inserted artificial MPI_Isend/MPI_Irecv operations plus MPI_Wait to construct a point-to-point “barrier” without deadlock. The implementation was a shared library in C, injected via LD_PRELOAD, wrapping all MPI v3 PMPI symbols, and running entirely in user space.

The control threshold was $E_n$0, reflecting measured Intel DVFS actuation behavior. On entry to an isolated slack region, the runtime armed a timer at $E_n$1; if the callback fired, it set the lowest P-state; on exit, it restored the highest P-state. The system used Intel MSRs through MSR_SAFE, and energy results referred to CPU package + DRAM consumption measured with RAPL.

Quantitatively, the reported averages across benchmarks and the 1056-core OMEN run were 0.79% execution-time overhead, 9.96% energy saving, and 10.73% power saving. For OMEN_1056p, the runtime reported 0.38% overhead, 22.11% energy saving, and 22.92% power saving. The robustness claim was equally important: worst-case overhead was 3.02%, and worst-case energy outcome was still positive, with minimum saving 1.04%. This directly challenged predictive approaches such as Fermata, Andante, and Adagio, which in the reported comparisons could incur much larger overheads or even negative energy outcomes.

For Cray power management, the main implication is that MPI-runtime power control is most effective when it is reactive, slack-only, and latency-aware. A plausible implication is that system-level power caps and runtime-level slack reclamation should be treated as complementary rather than competing layers.

4. Checkpointing, recovery, and phase-dependent energy behavior

Power management on large Cray-style systems is inseparable from resilience overheads, particularly checkpoint/restart. An empirical study on Mira, an IBM Blue Gene/Q machine, analyzed how application-level checkpointing and fault recovery affect power, execution time, and energy on a large HPC system and stated explicitly that the findings are directly relevant to Cray-style large-scale machines because the same design tensions exist there: large node counts, strict power budgets, checkpoint/restart for resilience, shared interconnect and storage pressure, and the need for scheduler/runtime cooperation (Fan, 2021).

The study was measurement-driven rather than model-driven. It used MonEQ, a user-level profiling library that collects power information at the node card level, reporting both aggregate node-card power and a breakdown into six hardware power domains: DRAM voltage, link chip voltage, SRAM voltage, optics voltage, PCIExpress voltage, and link chip core voltage. The core metrics were average power, execution time, and energy, and the study repeatedly emphasized that execution time dominates energy differences more than average power does.

The strongest general result was that adding checkpoints usually does not increase average power much and often slightly decreases it, but it significantly increases execution time and therefore energy. Across the presented NPB checkpoint-only cases, reported average-power changes ranged roughly from $E_n$2 to $E_n$3, while checkpoint-only time overheads ranged from 6.41% to 38.85% and energy overheads from 4.17% to 27.95%. This directly contradicts the common assumption that checkpoint phases are necessarily power peaks. The interpretation offered in the study was that checkpointing is not computation-intensive, so chip-core activity falls during checkpoint phases and average node power may remain flat or decrease slightly.

The resilience path was implemented with FTI (Fault Tolerance Interface) for NPB and STREAM. Because there is no local disk on nodes, FTI saves level 1, 2, and 3 checkpoints in memory. That detail became especially important for STREAM, which was selected as a memory-bandwidth-intensive benchmark. With checkpoints added, STREAM showed 33.15% execution-time increase and 31.99% energy increase, and the study explicitly attributed this to competition for memory resources and to higher DRAM power than in the other benchmarks.

Recovery overheads were even more asymmetrical than checkpoint overheads. For LU at 8192 nodes, injected faults relative to the no-FTI baseline produced +121.78% execution time and +129.38% energy. Even relative to the checkpointed run, one fault pattern caused +90.69% time and +91.60% energy. The study identified recovery from level 4 checkpoint as the most time-consuming recovery process and stated that, as the number of nodes increases, recovery takes more time. At the same time, it also concluded that, if cases with fatal error are ignored, FTI recovery is very efficient, with nonfatal incremental overheads of 0.30% to 6.27% in execution time and 0.31% to 4.79% in energy.

For Cray power management, the practical lesson is that checkpointing is primarily an energy problem through runtime extension, while fault recovery is also a power stability problem. The study reported more outliers and more spikes after injecting faults, with larger fluctuations due to alternating retrieval and re-computation phases. This suggests that scheduler and runtime policies should be checkpoint-aware, memory-domain-aware, and especially sensitive to tail recovery cost rather than only to mean checkpoint overhead.

5. System-scale GPU power management on HPE Cray EX

On Frontier, an exascale HPE Cray EX system at OLCF, power management research has moved toward telemetry-driven, system-scale estimation of what selective GPU DVFS/frequency capping or power capping could achieve. The system was described as having 9408 compute nodes, 1.9 EF peak performance, and 29 MW peak power. Each compute node had 4 AMD MI250X GPUs, each GPU with 2 GCDs, 64 GB HBM2E per GCD, 560 W max GCD power, and 1700 MHz max GCD frequency. The study focused on GPU controls because, on a fully utilized node, other components were said to be less than 20% and therefore “dwarfed” by GPU energy use (Karimi et al., 2024).

The telemetry stack combined out-of-band node telemetry, SLURM scheduler logs, per-node scheduler data, and ROCm SMI. Power telemetry was captured at 2-second intervals and aggregated to 15-second intervals. The resulting job-level view provided GPU power utilization at a 15-second frequency from each GPU on every node used by the workload. Over the reported 3 months of Frontier telemetry, total GPU energy use was 16820 MWh.

A benchmark-informed classification then mapped telemetry into four power modes:

Power mode	GPU power range	Share of GPU hours
Latency, Network or I/O bound	$E_n$4 W	29.8%
Memory intensive (M.I.)	200-420 W	49.5%
Compute intensive (C.I.)	420-560 W	19.5%
Boosted frequency	$E_n$5 W	1.1%

This classification was explicitly benchmark-driven. The paper treated memory-intensive execution as the main opportunity class because lowering GPU frequency often left runtime nearly unchanged while reducing power. By contrast, latency / network / I/O-bound jobs were stated to be poor candidates under the tested controls because reducing power or frequency proportionally slowed execution.

System-scale projections compared frequency caps from 1700 MHz to 700 MHz and power caps centered on 500 W, 400 W, 300 W, 200 W. For frequency capping, the most aggressive total projected saving was at 900 MHz, with 1493.9 MWh total savings, 8.8% energy savings, and 11.2% time increase. The practically important column was the study’s $E_n$6 view, where 900 MHz corresponded to 8.5% energy savings without performance loss, equivalent to roughly 1438 MWh. At that setting, the savings were overwhelmingly attributed to the memory-intensive region: 1438.3 MWh from M.I. versus 55.6 MWh from C.I.

Power capping produced smaller near-zero-slowdown projections. At 500 W, the study projected 558.83 MWh total savings, 3.32% energy savings, and 0.1% time increase, with $E_n$7 savings of 3.2%. At 200 W, savings increased to 5.7%, but time increase rose to 16.53% and compute-intensive savings became negative. The reported conclusion was therefore that frequency capping is more promising than power capping on Frontier for the observed workload mix.

The study also stratified jobs by scheduling class and science domain. Frontier job classes were defined as A: 5645–9408 nodes, B: 1882–5644 nodes, C: 184–1881 nodes, D: 92–183 nodes, and E: 1–91 nodes, with maximum walltime 12 h for A/B/C, 6 h for D, and 2 h for E. Heatmap analysis indicated that most energy use, and therefore most potential savings, came from large jobs, especially classes A and B, and to some extent C. The study was careful to frame all results as an upper ceiling or upper limit, not as a deployed closed-loop controller inside Frontier’s production software stack. That distinction matters: the work established best-case projections grounded in real telemetry, not a production HPE Cray EX control plane.

6. Node-local control layers and joint policy design

Beyond deployed Cray telemetry and runtime mechanisms, the research literature points to two lower-layer design principles that are directly relevant to future Cray-like nodes: joint optimization of speed and idle-state behavior, and fast processor-local closed-loop control.

The first principle is represented by SleepScale, which analyzed runtime joint speed scaling and sleep-state management in a single-server setting with DVFS and multiple low-power states. The study modeled active power as $E_n$8, characterized each low-power state by a three-tuple $E_n$9, and showed that the optimum is often neither “always fastest” nor “always slowest.” It explicitly rejected a universal race-to-halt rule and reported that, in one low-utilization DNS-like case, the global optimum used $N$0 at $N$1 with about 70 W, while race-to-halt could consume 50% more power. Although this work was not about Cray systems, it was presented as highly relevant because the same node-level knobs exist in analogous form on HPC platforms. This suggests that Cray node software should not treat DVFS and idle-state selection as independent knobs, particularly when idle intervals vary between short synchronization waits and long scheduler-visible gaps (Liu et al., 2014).

The second principle is represented by ControlPULP, an open-source, RISC-V-based on-chip Power Controller System (PCS) targeted at a 72-core HPC processor. The platform combined a CV32E40P manager core, an 8-worker cluster accelerator with 64 KiB L1 RAM, 512 KiB L2 RAM, DMA engines, and FreeRTOS, and implemented a Periodic Frequency Control Task at 2 kHz (500 µs) and a Periodic Voltage Control Task at 8 kHz (125 µs). The reported role of the PCS was node-local: receive requests from OS and BMC, read sensors and workload information, estimate power, enforce power and thermal caps, and dispatch per-core frequency plus coarse-grained voltage-domain settings (Ottaviano et al., 2023).

The quantitative results matter because they define the timescale at which future Cray-like nodes can react locally. For the full control action, the multi-core cluster achieved 5.5× speedup over single-core execution for 72 controlled cores and 6.5× at 296 controlled cores. For the overall PFCT step, the reported acceleration was 4.9×, reducing the control step from 134.66 µs to 27.282 µs at 500 MHz. In the accelerated case, the 500 µs hyper-period retained about 95% slack, leaving room for more sophisticated policies. In FPGA-based hardware-in-the-loop evaluation, the controller achieved DVFS tracking with a mean deviation within 3% of system TDP, with the emulated TDP = 120 W and per-core thermal limit 85 °C.

For Cray power management as a whole, these studies imply a layered control hierarchy. Built-in telemetry and accounting characterize energy and establish trustworthy boundaries; runtime mechanisms exploit safe low-power intervals in applications; checkpoint-aware policies avoid converting resilience into avoidable energy cost; system-scale analytics identify which job classes merit selective capping; and processor-local controllers enforce sub-millisecond power and thermal constraints beneath OS and scheduler policy. A plausible implication is that future Cray systems will benefit most when these layers are coordinated rather than optimized in isolation.