---
title: 'ExaDigiT: Supercomputer Digital Twin Framework'
url: https://www.emergentmind.com/topics/exadigit
type: topic
---

# ExaDigiT: Supercomputer Digital Twin Framework

Searching arXiv for the ExaDigiT supercomputing digital twin papers and related MIT SuperCloud extension.
Searching arXiv for ExaDigiT.
ExaDigiT is an open-source digital twin framework for liquid-cooled supercomputers and, in its later extension, for supercomputers and data centers more broadly. It couples workload replay, power modeling, cooling modeling, scheduling, and visual analytics so that researchers and operators can study system behavior not only in terms of performance, but also in terms of power draw, energy conversion losses, cooling demand, carbon footprint, and scheduling efficiency within a single simulation environment. In the Frontier case study, ExaDigiT models the supercomputer and the central energy/cooling plant together as one coupled system; in the MIT SuperCloud TX-GAIA extension, it adds heterogeneity, multi-tenancy, cloud-scale workloads, and trace replay/rescheduling for reinforcement-learning-based experimentation with sustainability policies [2410.05133] [2509.16513].

## 1. Definition, motivation, and evolution

ExaDigiT is presented as a digital twin in the strong operational sense: a virtual construct that mimics the structure and behavior of a physical asset, remains updated by physical telemetry, and informs decisions. Its stated purposes include verification and validation against real telemetry, what-if analysis for future operational changes, system optimization across power and cooling, virtual prototyping of future machines, and eventually more intelligent, autonomous control [2410.05133].

The framework was originally motivated by the need to model supercomputing facilities so that one can reason about power, thermal behavior, and operational policies before deploying them on real infrastructure. That motivation becomes sharper in two settings described in the literature. For liquid-cooled exascale systems such as Frontier, future gains require careful analysis of small losses, transient cooling dynamics, and cross-system interactions because the machine is already highly efficient. For AI supercomputing systems, the challenge is different but related: next-generation GPU datacenters require hundreds of megawatts and can produce very large and very fast power swings, making utility-side prediction and grid coordination difficult. The MIT SuperCloud paper connects this need to current pressure from hyperscale AI builds and to initiatives such as DCFlex, which aims to help utilities handle datacenter flexibility [2410.05133] [2509.16513].

The later ExaDigiT formulation also broadens the framework’s institutional and technical scope. It emphasizes that digital twins have already been built for systems such as Frontier, Fugaku, LUMI, Adastra, Setonix, Marconi100, and Lassen, and describes ExaDigiT as a global open-source community with more than 150 members, ten supercomputing centers, and industrial partners including HPE and NVIDIA. This suggests that ExaDigiT functions not merely as a single-site modeling stack, but as a reusable framework for cross-site supercomputing and datacenter studies [2509.16513].

## 2. Coupled system architecture

ExaDigiT is organized around three main modules that integrate telemetry, simulation, and visualization into a coupled digital twin workflow [2410.05133] [2509.16513].

| Module | Role | Noted implementation or interface |
|---|---|---|
| Resource Allocator and Power Simulator (RAPS) | Workload replay, scheduling, power simulation, statistics | Python; later extended with an OpenAI Gym-compatible interface |
| Transient thermo-fluidic cooling model | Dynamic liquid-cooling and control simulation | Modelica; exported as an FMU via FMI |
| Visual analytics layer | 3D, AR/VR, replay, dashboard interaction | Unreal Engine 5, Microsoft HoloLens 2, web dashboard |

The architecture is explicitly coupled. Telemetry or synthetic workloads enter RAPS, which simulates job arrival, scheduling, resource allocation, and node/rack/system power. RAPS then passes heat or power inputs to the cooling model. The cooling model computes transient thermal-fluid behavior, control response, staged equipment, and PUE. Outputs from both models are exposed through the visual analytics layer, which can display live or replayed states and support what-if experiments. In the Frontier configuration, the cooling model is called every 15 seconds; in the MIT SuperCloud extension, the trace-replay workflow emphasizes one-second scheduling and power updates with telemetry averaged over a trace quantum appropriate to the data source [2410.05133] [2509.16513].

A central characteristic of the framework is that it does not isolate compute, facility, and operator-facing analytics into separate studies. Instead, it treats system behavior, facility cooling behavior, and human interpretation as parts of a single modeling pipeline. This is the basis for the claim that ExaDigiT is more than a visualization or power-estimation tool: it is a virtual cloud system in which policies can be tested, workloads can be replayed, and scheduling strategies can be optimized [2509.16513].

## 3. RAPS: workload replay, scheduling, and power accounting

RAPS, expanded as the “resource allocator and power simulator,” is the operational core of ExaDigiT. It was originally designed to replay job traces to predict total system power, but it has since been modified to support multiple scheduling algorithms and backfill policies, with behavior aligned to production Slurm scheduling capabilities. Across the two papers, RAPS is described as supporting replay of real telemetry workloads, synthetic workloads derived from telemetry statistics or performance modeling tools such as Calculon or PerfVec, job rescheduling under different policies, and prediction of system power over time [2410.05133] [2509.16513].

The Frontier paper characterizes RAPS as a job-centric simulation rather than a fine-grained per-cycle hardware model. Each job can include number of nodes required, wall time, CPU utilization trace, and GPU utilization trace. Telemetry is sampled at 15-second quanta, while the simulation advances every second for power calculations. Jobs can be generated from telemetry or synthetically and submitted via a Poisson process with interarrival time
$$
\tau = -\frac{\ln(1-U)}{\lambda},
$$
where $\lambda = 1/t_{avg}$ and $t_{avg}$ is the mean time between job arrivals from telemetry. Jobs can be scheduled using policies such as SJF or FCFS [2410.05133].

The power model aggregates major component powers. For a Frontier node,
$$
P_{node} = P_{CPU} + 4P_{GPU} + 4P_{NIC} + P_{RAM} + 2P_{NVMe},
$$
reflecting 1 CPU, 4 GPUs, 4 NICs, RAM, and 2 NVMe devices per node. Rack power is then modeled as
$$
P_{rack} = \sum_i^{N=128} P_{node}^i + 32P_{switch},
$$
with 128 nodes per rack and fixed networking overhead from 32 switches per rack. After rack power is computed, the model adds rectification and DC-DC conversion losses and CDU pump power [2410.05133].

The energy-conversion chain is explicitly represented. In Frontier’s rack-level architecture, AC power enters active rectifiers, which produce 380 V DC, and SIVOC DC-DC converters step this down to 48 V DC at the blade or node level. Total conversion efficiency is represented as
$$
\eta_{system} = \eta_R \eta_S,
$$
where $\eta_R$ is rectifier efficiency and $\eta_S$ is SIVOC efficiency. The paper reports typical values $\eta_R = 0.96$ and $\eta_S = 0.98$, so total system efficiency is about 0.94. The significance is explicitly practical: even a few percent loss at exascale translates into megawatts [2410.05133].

RAPS also serves as the statistics engine. The Frontier study reports outputs including completed jobs, throughput, average power, total energy, conversion losses, CO\(_2\) emissions, and energy cost, with emissions estimated from average system power and an emissions factor,
$$
E_f = E_I \times 1\,\text{metric ton} / 2204.6\,\text{lbs} \times 1/\eta_{system}.
$$
The MIT SuperCloud extension states that RAPS records total system power, energy conversion efficiency, throughput, system FLOPS, utilization, carbon footprint, and related statistics, and can estimate power conversion efficiency, power usage effectiveness (PUE), and energy efficiency in terms of GFlops/W. It also notes non-scheduling studies such as replacing hardware with smart load-sharing rectifiers, moving to medium-voltage DC power, and modeling network congestion [2410.05133] [2509.16513].

## 4. Cooling dynamics and visual analytics

The cooling module is a transient thermo-fluidic model built in Modelica using open-source libraries such as `Modelica.Fluid`, `TRANSFORM`, and the Modelica Buildings Library. The design choice is explicit: rather than using expensive proprietary CFD-heavy tools, the framework adopts a system-level dynamic model that is open source, extensible, and suitable for transient simulation [2410.05133].

For Frontier, the cooling model represents three linked loops: a cooling tower loop with 5 cooling towers and 20 cells total; a primary pump loop moving water through heat exchangers and high-temperature pumps; and a CDU-rack loop in which each CDU serves 3 racks. The paper states that each CDU therefore cools 192 blades and 384 nodes. The model takes as inputs wet-bulb temperature and heat extracted for each of the 25 CDUs, where extracted heat is computed as
$$
H = \rho \cdot Q \cdot \Delta T \cdot c.
$$
Outputs include CDU flow rates, supply and return temperatures, pressures, CDU pump speeds and pump power, how many pumps, heat exchangers, and cooling towers are staged, total facility energy, and PUE. PUE is computed as facility energy divided by IT power, with auxiliary systems such as CDU pumps, HTWPs, CTWPs, and cooling tower fans included [2410.05133].

The cooling model also includes control logic rather than only heat-transfer equations. The CDU loop uses PID control to regulate pump speed and supply temperature. The primary loop uses PID control to regulate high-temperature water pumps, with heat exchangers staged based on tower activity. The cooling tower loop controls pump speed and tower staging from pressure and temperature gradients. The Frontier study emphasizes nonlinear interactions and delays between the loops, especially between primary and cooling-tower behavior [2410.05133].

The visual analytics layer is the user-facing representation of the twin. In the Frontier paper it is an augmented reality model of the supercomputer and central energy plant implemented in Unreal Engine 5 and available in desktop mode or through a Microsoft HoloLens 2 AR headset. In the later ExaDigiT description it is generalized to a visual analytics layer with augmented/virtual reality and a web dashboard. The layer is not treated as decorative visualization: it queries backend models and telemetry to display job activity, power, temperatures, pressures, cooling staging, and facility-wide state, and to support replay of operational history and what-if experiments [2410.05133] [2509.16513].

## 5. Frontier demonstration, verification, and what-if studies

The Frontier study is presented as the first comprehensive analysis of a liquid-cooled exascale supercomputer of its kind. It combines verification, validation, and long-horizon telemetry replay. Verification of RAPS is reported for three operating points: idle power, HPL core phase, and peak power. The reported results are 7.4 MW telemetry versus 7.24 MW RAPS with 2.1% error for idle, 21.3 MW telemetry versus 22.3 MW RAPS with 4.7% error for HPL core, and 27.4 MW telemetry versus 28.2 MW RAPS with 3.1% error for peak. The cooling model is also verified using a synthetic benchmark scenario including HPL and OpenMxP [2410.05133].

Validation against real telemetry uses about 24 hours of data on 2024-04-07. The comparison covers CDU flow rates, temperatures, pressures, and PUE. The study reports that most parameters match well, that RMSE and MAE are within reasonable bounds, and that PUE prediction is within 1.4% of telemetry-based PUE. One discrepancy is specifically identified: the model’s CDU secondary supply temperatures fluctuate more than the physical system, which the paper interprets as an area for future calibration [2410.05133].

A larger replay experiment covers 183 days of Frontier telemetry, from 2023-09-06 to 2024-03-18. This replay is used to evaluate workload behavior, power prediction, losses, energy use, emissions, and performance under real operational conditions. The reported runtime is about 9 minutes per replayed day with cooling and about 3 minutes without cooling, and the six-month study can be run in about an hour when days are parallelized on a single Frontier node. This indicates that the framework is intended as an operational analysis platform rather than only as a high-fidelity but impractical simulator [2410.05133].

The Frontier paper also uses the twin for what-if analysis. Average conversion losses are reported as 1.14 MW, about 6.74%, with values up to 1.84 MW on some days, and a yearly cost impact around \$900k/year. A smart load-sharing rectifier study yields only 0.1% efficiency gain but about \$120k/year savings. A direct 380V DC power scenario improves system efficiency from 93.3% to 97.3%, with potential savings of \$542k/year and carbon footprint reduction of 8.2%. These examples establish the framework’s role in evaluating design or control changes that would be expensive or risky to test directly on the production machine [2410.05133].

## 6. MIT SuperCloud TX-GAIA extension and reinforcement learning

The MIT SuperCloud TX-GAIA work extends ExaDigiT beyond traditional HPC-style replay toward cloud-style and hyperscale-style simulation. The key extensions are support for heterogeneity, multi-tenancy, cloud-scale workloads, and trace replay/rescheduling. The paper reports four early integration outcomes: development of an MIT SuperCloud dataloader for RAPS, extension of RAPS to support heterogeneity and multi-tenancy, enablement of replay and rescheduling of jobs, and integration of an OpenAI Gym-compatible interface for reinforcement learning [2509.16513].

The workflow is a trace-driven replay and rescheduling pipeline. The simulator takes system telemetry traces as input, or alternatively synthetic workloads generated from Calculon or PerfVec. MIT SuperCloud traces are averaged over a trace quantum, usually on the order of seconds, with examples of 10 s for MIT SuperCloud CPU telemetry and 100 ms for GPU telemetry. These traces are then fed into RAPS, which advances time in 1 s increments, dispatches jobs from the queue according to the selected policy, and computes power and performance statistics at each step. The framework can replay the system telemetry “as scheduled” or reschedule it using alternative scheduling and backfill policies. This makes it suitable for both reconstruction of observed behavior and counterfactual policy evaluation [2509.16513].

The reinforcement-learning formulation is deliberately operational rather than formal. ExaDigiT/RAPS exposes state, accepts actions, and returns reward through an OpenAI Gym-compatible interface. The reported implementation uses Stable Baselines 3 with Proximal Policy Optimization (PPO). The state contains the current queue, running jobs, available resources, and system-level metrics such as power consumption and throughput. The action space corresponds to scheduling decisions: which job to dispatch, when to backfill, and where to place jobs on nodes or partitions. The reward function combines energy consumption, carbon footprint, and job throughput, so that the learned policy targets energy-aware scheduling rather than pure throughput alone. The paper also cites prior work defining job slowdown as “wait time + run time divided by job run time,” while emphasizing that the ExaDigiT environment additionally tracks power, carbon, and congestion [2509.16513].

The reported results are preliminary but specific. Early PPO runs show that episodic reward improves over time, the runtime statistics track job throughput and energy consumption simultaneously, and another panel shows the system power trace under RL-driven scheduling. The authors interpret these outcomes as evidence of feasibility rather than final optimality: RAPS can function as an RL environment, can simulate a cloud-like supercomputer, can replay realistic traces, and can support energy-aware policy learning. At the same time, the paper explicitly states that validation of the RL agent and the multi-tenant scheduler remains to be completed [2509.16513].

## 7. Limitations, applicability, and research significance

The literature is explicit that ExaDigiT is still developing. The Frontier paper identifies future directions such as better application fingerprinting and tighter integration of machine-learned workload models, reinforcement-learning control agents, and the physical system. The MIT SuperCloud paper states that the work is “in progress,” that the RL agent and the multi-tenant scheduler have not yet been fully validated, and that the current environment is built from trace replay rather than fully endogenous workload generation. It also implies modeling assumptions through telemetry averaged over a chosen quantum and simulation that advances at a fixed step size [2410.05133] [2509.16513].

These limitations do not reduce the framework to passive observation. Both papers describe ExaDigiT as a platform for studying policies before deployment on real infrastructure. In Frontier, this includes system optimization across compute, power conversion, cooling, and facility operations. In the MIT SuperCloud extension, it includes scheduling strategies, incentive structures, virtual prototypes of hardware and software, and speculative system designs. A plausible implication is that ExaDigiT’s main research value lies in providing a common experimental substrate in which workload behavior, electrical losses, facility dynamics, and control policies can be evaluated together rather than as disconnected subproblems [2410.05133] [2509.16513].

The broader significance assigned to ExaDigiT is therefore methodological as much as architectural. It is described as an extensible digital twin platform whose RAPS module provides the replay, scheduling, and power-accounting core; whose cooling model captures transient thermo-fluidic and control behavior; and whose visualization layer supports interpretation of high-volume, high-velocity, and high-variety telemetry. In that form, ExaDigiT is positioned as an instrument for sustainable, energy-efficient supercomputing and for cloud-like supercomputing studies in which throughput, efficiency, and sustainability must be optimized jointly rather than sequentially [2410.05133] [2509.16513].

Source: https://www.emergentmind.com/topics/exadigit