Digital Twin-Based Cooling System Optimization for Data Center

Abstract: Data center cooling systems consume significant auxiliary energy, yet optimization studies rarely quantify the gap between theoretically optimal and operationally deployable control strategies. This paper develops a digital twin of the liquid cooling infrastructure at the Frontier exascale supercomputer, in which a hot-temperature water system comprises three parallel subloops, each serving dedicated coolant distribution unit clusters through plate heat exchangers and variable-speed pumps. The surrogate model is built based on Modelica and validated through one full calendar year of 10-minute operational data following ASHRAE Guideline 14. The model achieves a subloop coefficient of variation of the root mean square error below 2.7% and a normalized mean bias error within 2.5%. Using this validated surrogate model, a layered optimization framework evaluates three progressively constrained strategies: an analytical flow-only optimization achieves 20.4% total energy saving, unconstrained joint optimization of flow rate and supply temperature demonstrates 30.1% total energy saving, and ramp-constrained optimization of flow rate and supply temperature, enforcing actuator rate limits, can reach total energy saving of 27.8%. The analysis reveals that the baseline system operates at 2.9 times the minimum thermally safe flow rate, and the co-optimizing supply temperature with flow rate nearly doubles the savings achievable by flow reduction alone.

• The paper develops and validates a physics-based digital twin of Frontier’s liquid cooling system, achieving 1.96–2.67% CV-RMSE across 47,186 operating records.
• The optimization reduces total cooling energy by 20.4% with flow control, 30.1% with unconstrained flow–temperature co-optimization, and 27.8% with practical ramp limits.
• The results show cooling tower fans consume 73% of baseline cooling energy, while the 92.4% recovery ratio demonstrates that most theoretical savings remain deployable despite actuator constraints.

Motivation and scope

Data center cooling accounts for an estimated 30–40% of facility electricity consumption, yet the global average Power Usage Effectiveness (PUE) has remained between 1.55 and 1.59 since 2020, indicating that incremental efficiency gains have plateaued. This paper addresses the problem at the extreme end of the demand spectrum: the liquid cooling infrastructure of the Frontier exascale supercomputer at Oak Ridge National Laboratory, a facility operating 100% direct liquid cooling with waste heat loads of 8–28 MW. The authors develop a physics-based digital twin of Frontier's hot-temperature water (HTW) loop, validate it against a full year of operational data, and use it to evaluate three progressively constrained optimization strategies. The central question is one that the literature has largely ignored: how much of the theoretically optimal cooling energy savings survives when practical actuator constraints—specifically pump and setpoint ramp-rate limits—are imposed? To formalize this, the paper introduces an implementability gap metric and its complement, the recovery ratio.

System architecture and digital twin

The Frontier cooling system is modeled as three thermally coupled loops. The tertiary loop circulates a 50/50 ethylene glycol–water mixture through 25 coolant distribution units (CDUs) serving 74 compute cabinets; plate heat exchangers transfer heat to the HTW secondary loop; and a primary cooling tower water (CTW) loop rejects heat via mechanical-draft towers. The HTW system comprises three active parallel subloops with flow fractions of approximately 24.6%, 26.0%, and 49.5%. Critically, subloop 3 carries roughly 63% of the total heat load through only 49.5% of the flow, making it the binding thermal branch at nearly every operating point.

The digital twin is implemented in OpenModelica using the Modelica Buildings Library, with $\varepsilon$-NTU heat exchanger models, affinity-law pump models, and a fixed-approach cooling tower model. A notable methodological decision concerns the specific heat capacity: the published operational dataset embeds an internal conversion constant of $c_p = 3{,}709$ J/(kg·K) in its derived heat-load columns. Using that constant to validate return temperature predictions would constitute circular validation. The authors instead adopt $c_p = 3{,}500$ J/(kg·K), deliberately accepting a 6% discrepancy so that the validation constitutes a genuine predictive test rather than a numerical artefact. This is a careful and honest treatment of a subtle data provenance issue.

Validation

Validation follows ASHRAE Guideline 14-2014 against 47,186 filtered 10-minute operating records from calendar year 2023, with stricter internal targets (CV-RMSE ≤ 5%, |NMBE| ≤ 5%) than the guideline's hourly thresholds. Per-subloop return temperature prediction achieves CV-RMSE of 1.96–2.67% and NMBE of +1.69% to +2.43%, with $R^2$ of 0.9933 and 0.9927 for subloops 1 and 2 but only 0.9088 for subloop 3, where the larger temperature differential amplifies sensitivity to the assumed $c_p$. The consistent positive bias of 0.5–0.9 °C is attributed to the deliberate $c_p$ offset. These are strong results for a single-calibration-parameter model, and they provide the evidentiary basis for trusting the downstream optimization. The validation is nonetheless steady-state in nature; transient behavior during rapid load ramps is not independently verified, a point the authors concede.

Baseline diagnosis: systematic over-pumping

Comparing measured baseline flow against the analytical minimum thermally safe flow reveals a median over-pumping ratio of 1.5×, with the abstract reporting the baseline operates at 2.9× the minimum flow. The facility runs pumps at fixed setpoints—approximately 200 kg/s in winter, stepping abruptly to 350–400 kg/s in May through October—rather than modulating flow with thermal demand. Because pump power scales as $\dot{m}^n$ with $n \in [2,3]$, this fixed-speed operation is the dominant source of waste. The diagnosis is unambiguous and well supported by the full-year data.

Layered optimization results

The optimization enforces a return temperature constraint of 42 °C per subloop, providing a 3 °C buffer below the 45 °C equipment limit. Three strategies are compared:

Metric	Baseline	A (flow only)	B (unconstrained co-opt)	C (ramp-constrained co-opt)
Pump energy (kWh)	488,857	118,772	180,174	193,033
CT fan energy (kWh)	1,324,797	1,324,797	1,087,613	1,116,024
Total energy (kWh)	1,813,654	1,443,569	1,267,787	1,309,057
Total savings (%)	—	20.4	30.1	27.8

Strategy A admits a closed-form analytical solution for the minimum feasible flow; Strategy B adds supply temperature setpoint as a decision variable and is solved with SLSQP per timestep; Strategy C adds ramp-rate limits of ±50 kg/s and ±1 °C per 10-minute step, solved sequentially with bound tightening in an MPC-like fashion, plus a safety override that prioritizes thermal safety over ramp smoothness.

The most consequential finding is structural: cooling tower fan energy constitutes 73% of baseline cooling consumption (1,325 MWh of 1,814 MWh), a 2.7:1 ratio over pump energy. This inverts the conventional emphasis on pump affinity laws. The co-optimized Strategy B actually uses more pump energy than the flow-only Strategy A (180,174 vs. 118,772 kWh) yet delivers greater total savings, because it raises mean supply temperature by 2.7 °C, cutting CT fan energy by 17.9%. Component-level optimization is demonstrably suboptimal here, and the implication for operators is that supply temperature reset—not flow reduction alone—is where the larger savings reside. Savings are seasonal: 33% peak in July versus 15–20% in winter, with the summer quarter contributing 43% of annual savings in 25% of the year.

The implementability result is the paper's headline claim: ramp constraints cost only 2.3 percentage points of savings, yielding a recovery ratio of 92.4% (27.8% vs. 30.1%). Strategy B's unconstrained trajectories would demand flow changes up to 189 kg/s and temperature changes up to 4.4 °C per step, both operationally unacceptable; Strategy C respects the limits by construction. The nine thermal exceedances under Strategy C (0.02% of operating points) occur exclusively at maximum pump capacity of 420 kg/s during extreme summer peaks, peaking at 42.9 °C—still below the 45 °C limit—indicating a system capacity boundary rather than an optimizer failure.

Robustness

Because Frontier's actual pump curves are not publicly available, all strategies are evaluated across exponents $n \in \{2.0, 2.5, 3.0\}$. Strategy C savings range from 21.3% ($n=2$) to 27.8% ($c_p = 3{,}709$0), and the relative advantage of co-optimization over flow-only optimization grows at lower exponents (21.3% vs. 10.9% at $c_p = 3{,}709$1), since CT fan savings dominate when pump power is less flow-sensitive. This robustness is practically significant given that installed pump curves are frequently unavailable. At typical utility rates, the authors estimate $c_p = 3{,}709$250,000 in annual cost reduction.

Limitations

The paper is candid about several constraints on its claims. The cooling tower fan model uses a simplified power–approach relationship rather than a manufacturer-specific model, so CT fan savings should be read as directionally correct rather than exact—this matters because CT savings are the paper's central mechanism. Supply temperature setpoint independence from tower operation is assumed but may require control system modifications. The ramp-constrained strategy is a static, point-by-point optimization with retrospective ramp enforcement rather than a full receding-horizon MPC with look-ahead, which could matter during rapid load transients. Validation is steady-state only, with the 6% $c_p = 3{,}709$3 discrepancy acknowledged; independent calorimetric measurement would strengthen absolute predictions. Finally, the ramp limits themselves (±50 kg/s, ±1 °C per step) are conservative values informed by general ASHRAE guidance rather than measured Frontier actuator specifications.

Conclusion

This paper delivers a validated, single-calibration-parameter Modelica digital twin of an exascale liquid cooling system and uses it to establish a quantified hierarchy of savings: 20.4% from flow-only optimization, 30.1% from unconstrained flow–temperature co-optimization, and 27.8% under realistic ramp constraints, for a 92.4% recovery ratio. Its two durable contributions are the demonstration that cooling tower fan energy dominates the HPC cooling budget and therefore that supply temperature reset is the primary lever, and the implementability gap metric, which gives the field a reproducible way to report how much of a theoretical optimum is deployable. The open questions the paper leaves—transient validation, receding-horizon formulation, and pilot deployment on physical hardware—are precisely the steps required before the reported savings can be confirmed in operation.