MIT Supercloud Dataset Overview

• MIT Supercloud Dataset is an anonymized, multi-modal telemetry corpus from the TX‑Gaia system that supports research in HPC operations and AI/ML workloads.
• It integrates scheduler logs, CPU/GPU time series, filesystem events, and physical facility data to enable workload classification, anomaly detection, and simulation studies.
• The dataset’s fine-grained measurements—from 10-second CPU intervals to 100ms GPU traces—and labeled subsets drive innovations in energy-aware scheduling and system sustainability.

Searching arXiv for the MIT Supercloud Dataset and closely related papers to ground the article in the current literature. The MIT Supercloud Dataset is a public, anonymized, multi-layer telemetry and workload trace collected from the TX‑Gaia production system at the MIT Lincoln Laboratory Supercomputing Center. It was introduced to support research on AI/ML methods for datacenter and HPC operations, including workload characterization, scheduling, energy and power analysis, anomaly detection, and failure prediction (Samsi et al., 2021). The dataset combines scheduler logs, CPU and GPU time series, memory usage, file system logs, node-level health metrics, and physical monitoring data from a heterogeneous HPC cluster, and later work has added labeled subsets for workload classification and used the traces as the basis for digital-twin simulation and reinforcement-learning experiments (Samsi et al., 2021, Tang et al., 2022, Brewer et al., 20 Sep 2025).

1. Definition, scope, and system context

The dataset originates from the MIT Supercloud TX‑Gaia cluster, described as a petascale heterogeneous system designed for interactive supercomputing, data analysis, and AI workloads (Samsi et al., 2021). TX‑Gaia has two main partitions. The GPU‑accelerated partition has 224 nodes, each with 2 × Intel Xeon Gold 6248, 384 GB RAM, and 2 × NVIDIA Volta V100 GPUs with 32 GB RAM per V100. The CPU‑only partition has 480 nodes, each with 2 × Intel Xeon Platinum 8260, giving 48 CPU cores per node, and 192 GB RAM per node (Samsi et al., 2021).

The storage and facility context are integral to the dataset’s interpretation. TX‑Gaia uses a Lustre high-performance parallel filesystem backed by a 3‑PB Cray L300 parallel storage array with 1 metadata server + 4 data servers. The network uses an Arista Ethernet core switch with 25‑GbE links to login, compute, and service nodes, and four channel‑bonded 100‑GbE links per file-system data server (Samsi et al., 2021). The cluster resides in a modular EcoPOD 240a datacenter with 44 racks of IT equipment, extensive IT and HVAC power sensing, and environmental measurements near rack rows (Samsi et al., 2021). This cross-layer instrumentation is one of the dataset’s defining properties.

The initial dataset release covered 6 months and was described as ongoing, with > 1 million jobs in the CPU time-series portion and approximately 90,000 GPU job traces (Samsi et al., 2021). Later challenge-oriented work described the broader corpus as ~2.1 TB and a workload-classification labeled subset as ≈2 GB (Tang et al., 2022). The combination of production provenance, heterogeneity, and multi-layer telemetry distinguishes the dataset from scheduler-only traces and from datasets lacking fine-grained GPU or facility sensing (Samsi et al., 2021).

2. Data modalities and measurement granularity

The dataset is organized around several telemetry layers collected from the production system. The principal categories are scheduler data, CPU time-series data, GPU time-series data, node-level health and utilization data, Lustre filesystem logs, and physical monitoring data (Samsi et al., 2021).

CPU time series are collected via Slurm’s acct_gather_profile/hdf5 plugin with a sampling interval of 10 seconds (Samsi et al., 2021). The monitored fields include CPUFrequency, CPUTime, CPUUtilization, RSS, VMSize, Pages, ReadMB, and WriteMB, together with job and node identifiers and step information (Samsi et al., 2021). Later challenge documentation described the CPU feature schema in similar terms, referring to CPUUtilization, CPUFrequency, CPUTime, power_draw_W, RSS, VMSize, Pages, ReadMB, and WriteMB (Tang et al., 2022).

GPU monitoring is performed via nvidia-smi at a sampling interval of 100 ms (Samsi et al., 2021). The GPU fields include utilization_gpu_pct, utilization_memory_pct, memory_free_MiB, memory_used_MiB, temperature_gpu, temperature_memory, power_draw_W, and pcie_link_width_current, tied to anonymized node IDs, job IDs, timestamps, and GPU indices (Samsi et al., 2021). This same seven-sensor GPU schema became the basis for the workload-classification challenge datasets (Tang et al., 2022).

Node-level monitoring runs independently of Slurm at 5-minute intervals and records LoadAvg, MemoryFreeInactiveKB, LustreRPCTotals, FSlatency, and UserPIDCount (Samsi et al., 2021). The filesystem component includes parsed Lustre metadata-server logs containing operations such as open, close, link, unlink, mkdir, rmdir, rename, getattr, setattr, statfs, and sync (Samsi et al., 2021).

Physical monitoring data includes total power, line voltage and current, power supply alarms, outside air temperature and humidity, cold/hot aisle temperatures, cold/hot aisle pressures, exhaust fan speeds, DX cooling enable, heater enable, humidifier status, and safety indicators such as fire/smoke detection and water detection (Samsi et al., 2021). This facility layer is unusually rich for a public HPC trace and directly motivated later power-aware scheduling and sustainability studies (Brewer et al., 20 Sep 2025).

3. Collection methodology, preprocessing, and anonymization

The collection pipeline was designed to minimize interference with production jobs. CPU time series are started in job prolog and stopped in epilog; GPU monitoring is likewise started and stopped using Slurm prolog/epilog scripts (Samsi et al., 2021). During runtime, telemetry is written to local disk, then moved to central storage after job completion (Samsi et al., 2021). This local-write strategy reduces pressure on shared storage during execution.

The raw data were not released directly. A preprocessing pipeline performed data cleaning, outlier removal, format conversion, and metadata enrichment (Samsi et al., 2021). Slurm HDF5 time series were extracted and converted to CSV, GPU traces were linked to jobs by adding id_job and node identifiers during preprocessing, and logs from distinct sources were transformed into structured, analysis-oriented files (Samsi et al., 2021). Consistent linkage across scheduler, CPU, GPU, node, filesystem, and physical data was explicitly preserved (Samsi et al., 2021).

Privacy preservation is central to the public release. Sensitive fields such as job names, working directories, MAC addresses, kernel versions, and other site-specific constants were removed (Samsi et al., 2021). Key identifiers such as job IDs, user IDs, and node names were anonymized using SHA‑256 with a secret salt, with the same identifier mapped consistently across all files (Samsi et al., 2021). For node names, rack and node components were hashed and recombined into forms such as r[hash]-n[hash] (Samsi et al., 2021). This preserves relational structure while preventing direct reidentification.

A recurrent limitation follows from this design. Because semantic identifiers such as job names and working directories are removed, application-level interpretation must rely on utilization signatures, resource patterns, and the manually labeled subsets rather than on human-readable metadata (Samsi et al., 2021). This constraint shaped subsequent work on workload classification (Tang et al., 2022, Weiss et al., 2022).

4. Labeled subsets and workload-classification research

A major derivative of the dataset is the MIT Supercloud Workload Classification Challenge, which introduced a labeled subset for supervised learning (Tang et al., 2022). The challenge uses monitoring logs from TX‑Gaia and focuses on GPU-accelerated jobs. The labeled subset contains 3,430 unique jobs with labels and 26 distinct classes corresponding to deep learning architectures and sub-architectures (Tang et al., 2022). The workload families include VGG, ResNet, Inception, U‑Net variants, BERT, DistilBERT, and graph neural network models such as DimeNet, SchNet, PNA, and NNConv (Tang et al., 2022).

Each challenge sample is a 60-second window of GPU-only telemetry resampled to a fixed shape (N_trials, 540, 7), where the seven channels are the GPU sensors from the main dataset (Tang et al., 2022). Windows are extracted from the first 60 seconds, the middle 60 seconds, or a random 60-second segment, yielding datasets such as 60-start-1, 60-middle-1, and 60-random-* (Tang et al., 2022). An 80/20 train/test split is used, and .npz files contain X_train, y_train, model_train, X_test, y_test, and model_test (Tang et al., 2022).

The labeling methodology is explicit. The labeled jobs were generated by running known benchmark models on TX‑Gaia in controlled experiments, then manually assigning labels at both the family and variant levels and propagating those labels to all associated GPU time series for each job (Tang et al., 2022). This means that the labeled subset is not an inferred taxonomy from production logs, but a curated ground-truth corpus.

Workload-classification baselines established that the telemetry is discriminative at the architecture level. Using covariance-based features extracted from each $540 \times 7$ trial, the challenge paper reports Random Forest accuracy up to 93.02% on 60-middle-1, XGBoost at 88.47%, and Bidirectional LSTM at approximately 90.81% on 60-random-1 (Tang et al., 2022). The feature-importance analysis highlighted covariance between GPU % utilization and CPU % utilization, variance of GPU % utilization, and variance of power draw as highly predictive (Tang et al., 2022).

Subsequent work on time-series preprocessing addressed the alignment problem for variable-length multichannel traces derived from the dataset (Weiss et al., 2022). Using a labeled GPU subset of 19,481 distinct GPU time series from 3,430 jobs and evaluating low-overhead preprocessing methods, that study reported classification accuracy greater than 95%, stated as 5% above previous approaches (Weiss et al., 2022). The strongest results came from summary statistics over windows and sampling Fourier coefficients, rather than fixed contiguous windows (Weiss et al., 2022). This suggests that the dataset contains stable workload signatures distributed over the full job trajectory, not only in local temporal segments.

5. Power, utilization, and sustainability uses

The dataset was designed not only for workload identification but also for operational studies of resource use, energy, and datacenter behavior (Samsi et al., 2021). The paper introducing the dataset explicitly lists goals including improved scheduling policies, identifying inefficiencies in resource utilization, optimizing energy and power consumption, predicting failures, and detecting policy violations such as crypto-currency mining and password cracking (Samsi et al., 2021).

Several descriptive statistics in the original dataset paper illustrate the operational character of the traces. GPU jobs span runtimes from minutes to hundreds of minutes; the median runtime for “any GPU” and single-GPU jobs is approximately 4 minutes, while 85% of GPU jobs use only one GPU (Samsi et al., 2021). Single-GPU jobs have median SM utilization ≈ 23%, whereas 2-GPU and ≥4-GPU jobs show a mixture of near-zero and high-utilization behavior, interpreted there as a combination of debugging/exploratory workloads and mature production jobs (Samsi et al., 2021). Strong positive correlations were observed among power consumption, SM utilization, memory utilization, GPU temperature, and memory temperature, while pcie_link_width_current showed little variation and weak correlation with the other metrics (Samsi et al., 2021).

Later work turned these traces into a simulation environment for sustainability research. “Trace Replay Simulation of MIT SuperCloud for Studying Optimal Sustainability Policies” uses the MIT SuperCloud dataset as a historical telemetry and workload trace for the TX‑GAIA system (Brewer et al., 20 Sep 2025). In that work, the dataset supplies CPU telemetry, GPU telemetry, power data at 100 ms intervals, and CPU frequency information, together with job scheduling and workload traces (Brewer et al., 20 Sep 2025). CPU telemetry is averaged into 10 s bins, GPU telemetry remains at 100 ms, and the simulator advances in 1 s steps (Brewer et al., 20 Sep 2025).

Inside that simulation workflow, the traces are replayed or rescheduled in ExaDigiT/RAPS, which aggregates total system power, energy conversion efficiency, throughput, system FLOPS, and system utilization, and estimates rectification and voltage conversion losses and PUE (Brewer et al., 20 Sep 2025). An OpenAI Gym–compatible interface and early PPO experiments use reward functions combining energy consumption, carbon footprint, and job throughput (Brewer et al., 20 Sep 2025). Within that framing, the dataset becomes a real-world substrate for experimentation with energy-aware scheduling, demand response, and load shaping rather than a static archive alone.

This later use clarifies an important property of the dataset: it contains both fine-grained device telemetry and job-level temporal structure rich enough to support rescheduling counterfactuals. A plausible implication is that the dataset occupies an unusual middle ground between system trace archives and controllable simulation benchmarks.

6. Position within the broader MIT SuperCloud literature

The term “MIT SuperCloud dataset” has been used in more than one sense in the literature, and distinguishing these senses prevents confusion. The 2021 dataset paper defines the phrase as the TX‑Gaia operational monitoring corpus of scheduler, CPU, GPU, filesystem, node-health, and physical datacenter traces (Samsi et al., 2021). The workload classification challenge uses the same dataset name for the public datacenter-trace release and its labeled subsets (Tang et al., 2022). The trace replay simulation paper uses “MIT SuperCloud dataset” specifically as a telemetry and workload trace source for TX‑GAIA (Brewer et al., 20 Sep 2025).

Other MIT SuperCloud papers use the platform as an execution environment rather than introducing this dataset. For example, D4M and GraphBLAS studies benchmark streaming updates of synthetic power-law graphs on MIT SuperCloud, achieving sustained rates from 1.9 billion to over 200,000,000,000 updates per second across many nodes, but they do not define a named MIT SuperCloud dataset in the same sense (Kepner et al., 2019, Kepner et al., 2019, Kepner et al., 2020, Kepner et al., 2021). Likewise, work on internet-traffic analysis used MAWI “Day in the Life” traces processed on MIT SuperCloud rather than a native SuperCloud monitoring dataset (Gadepally et al., 2018). Other papers describe deployable census analytics (Samuel et al., 2021), portal-based web access (Prout et al., 2017), or database management infrastructure (Prout et al., 2015), again referring to datasets hosted on or produced by the platform, not to the TX‑Gaia telemetry release.

This distinction matters because some readers may assume that every “MIT SuperCloud” paper concerns the same data artifact. It does not. In current usage, the canonical research meaning of MIT Supercloud Dataset refers to the TX‑Gaia monitoring corpus introduced in 2021 and extended through public challenge datasets and trace-driven simulation studies (Samsi et al., 2021, Tang et al., 2022, Brewer et al., 20 Sep 2025).

7. Limitations, access, and research significance

The dataset’s strengths are balanced by clear limitations documented in the source papers. The CPU time series are sampled at 10 s, node metrics at 5 minutes, and only GPU telemetry has 100 ms resolution, so some transient CPU or system-level phenomena are necessarily unobserved (Samsi et al., 2021). The labeled subset is relatively small compared with the full corpus: ≈2 GB labeled versus ~2.1 TB total, with 3,430 labeled jobs across 26 classes, and the labels focus heavily on AI/ML workloads rather than a full cross-section of scientific HPC applications (Tang et al., 2022). Challenge results also showed lower accuracy for windows taken from the start of jobs, suggesting that early phases are often dominated by generic setup or data-loading behavior rather than architecture-specific signatures (Tang et al., 2022).

Cluster specificity is another constraint. The telemetry reflects Volta V100 GPUs, the TX‑Gaia scheduler and partitions, and the particular facility configuration of the EcoPOD datacenter (Samsi et al., 2021, Tang et al., 2022). This means that operational conclusions and learned representations may not transfer directly to clusters with different accelerators, software stacks, or cooling infrastructures. The trace-replay work likewise notes that validation of RL agents and multi-tenant schedulers remains pending, and that the detailed schema must be obtained from the original dataset documentation rather than from the simulation paper alone (Brewer et al., 20 Sep 2025).

Access is provided through the MIT Datacenter Challenge website, https://dcc.mit.edu, and later papers also state that the data are available through the AWS Open Data Registry, with download instructions on the DCC site (Samsi et al., 2021, Weiss et al., 2022). The original dataset paper specifies that researchers must accept a data use agreement before download and that licensing details are posted on the challenge website (Samsi et al., 2021).

The dataset’s research significance lies in its combination of properties rather than any single field. It is simultaneously a job-linked scheduler trace, a multichannel CPU/GPU time-series corpus, a filesystem and node-health log, and a physical infrastructure telemetry archive (Samsi et al., 2021). It has supported architecture-level workload classification (Tang et al., 2022), low-overhead time-series alignment research (Weiss et al., 2022), and sustainability-oriented digital-twin scheduling experiments (Brewer et al., 20 Sep 2025). This suggests that the MIT Supercloud Dataset functions less as a single benchmark narrowly tied to one task and more as a shared systems-observability substrate for studies spanning HPC operations, ML for systems, and energy-aware datacenter management.