---
title: MIT Supercloud Dataset Overview
url: https://www.emergentmind.com/topics/mit-supercloud-dataset
type: topic
---

# MIT Supercloud Dataset Overview

Searching arXiv for the MIT Supercloud Dataset and closely related papers to ground the article in the current literature.
The **MIT Supercloud Dataset** is a public, anonymized, multi-layer telemetry and workload trace collected from the **TX‑Gaia** production system at the MIT Lincoln Laboratory Supercomputing Center. It was introduced to support research on AI/ML methods for datacenter and HPC operations, including workload characterization, scheduling, energy and power analysis, anomaly detection, and failure prediction [2108.02037]. The dataset combines **scheduler logs**, **CPU and GPU time series**, **memory usage**, **file system logs**, **node-level health metrics**, and **physical monitoring data** from a heterogeneous HPC cluster, and later work has added labeled subsets for workload classification and used the traces as the basis for digital-twin simulation and reinforcement-learning experiments [2108.02037], [2204.05839], [2509.16513].

## 1. Definition, scope, and system context

The dataset originates from the **MIT Supercloud TX‑Gaia cluster**, described as a petascale heterogeneous system designed for interactive supercomputing, data analysis, and AI workloads [2108.02037]. TX‑Gaia has two main partitions. The **GPU‑accelerated partition** has **224 nodes**, each with **2 × Intel Xeon Gold 6248**, **384 GB RAM**, and **2 × NVIDIA Volta V100** GPUs with **32 GB RAM per V100**. The **CPU‑only partition** has **480 nodes**, each with **2 × Intel Xeon Platinum 8260**, giving **48 CPU cores per node**, and **192 GB RAM per node** [2108.02037].

The storage and facility context are integral to the dataset’s interpretation. TX‑Gaia uses a **Lustre high-performance parallel filesystem** backed by a **3‑PB Cray L300 parallel storage array** with **1 metadata server + 4 data servers**. The network uses an **Arista Ethernet core switch** with **25‑GbE** links to login, compute, and service nodes, and four channel‑bonded **100‑GbE** links per file-system data server [2108.02037]. The cluster resides in a modular **EcoPOD 240a** datacenter with **44 racks of IT equipment**, extensive IT and HVAC power sensing, and environmental measurements near rack rows [2108.02037]. This cross-layer instrumentation is one of the dataset’s defining properties.

The initial dataset release covered **6 months** and was described as **ongoing**, with **> 1 million** jobs in the CPU time-series portion and approximately **90,000** GPU job traces [2108.02037]. Later challenge-oriented work described the broader corpus as **~2.1 TB** and a workload-classification labeled subset as **≈2 GB** [2204.05839]. The combination of production provenance, heterogeneity, and multi-layer telemetry distinguishes the dataset from scheduler-only traces and from datasets lacking fine-grained GPU or facility sensing [2108.02037].

## 2. Data modalities and measurement granularity

The dataset is organized around several telemetry layers collected from the production system. The principal categories are **scheduler data**, **CPU time-series data**, **GPU time-series data**, **node-level health and utilization data**, **Lustre filesystem logs**, and **physical monitoring data** [2108.02037].

CPU time series are collected via Slurm’s `acct_gather_profile/hdf5` plugin with a **sampling interval of 10 seconds** [2108.02037]. The monitored fields include `CPUFrequency`, `CPUTime`, `CPUUtilization`, `RSS`, `VMSize`, `Pages`, `ReadMB`, and `WriteMB`, together with job and node identifiers and step information [2108.02037]. Later challenge documentation described the CPU feature schema in similar terms, referring to `CPUUtilization`, `CPUFrequency`, `CPUTime`, `power_draw_W`, `RSS`, `VMSize`, `Pages`, `ReadMB`, and `WriteMB` [2204.05839].

GPU monitoring is performed via `nvidia-smi` at a **sampling interval of 100 ms** [2108.02037]. The GPU fields include `utilization_gpu_pct`, `utilization_memory_pct`, `memory_free_MiB`, `memory_used_MiB`, `temperature_gpu`, `temperature_memory`, `power_draw_W`, and `pcie_link_width_current`, tied to anonymized node IDs, job IDs, timestamps, and GPU indices [2108.02037]. This same seven-sensor GPU schema became the basis for the workload-classification challenge datasets [2204.05839].

Node-level monitoring runs independently of Slurm at **5-minute intervals** and records `LoadAvg`, `MemoryFreeInactiveKB`, `LustreRPCTotals`, `FSlatency`, and `UserPIDCount` [2108.02037]. The filesystem component includes parsed Lustre metadata-server logs containing operations such as `open`, `close`, `link`, `unlink`, `mkdir`, `rmdir`, `rename`, `getattr`, `setattr`, `statfs`, and `sync` [2108.02037].

Physical monitoring data includes **total power**, **line voltage and current**, **power supply alarms**, **outside air temperature and humidity**, **cold/hot aisle temperatures**, **cold/hot aisle pressures**, **exhaust fan speeds**, **DX cooling enable**, **heater enable**, **humidifier status**, and safety indicators such as **fire/smoke detection** and **water detection** [2108.02037]. This facility layer is unusually rich for a public HPC trace and directly motivated later power-aware scheduling and sustainability studies [2509.16513].

## 3. Collection methodology, preprocessing, and anonymization

The collection pipeline was designed to minimize interference with production jobs. CPU time series are started in job prolog and stopped in epilog; GPU monitoring is likewise started and stopped using Slurm prolog/epilog scripts [2108.02037]. During runtime, telemetry is written to **local disk**, then moved to central storage after job completion [2108.02037]. This local-write strategy reduces pressure on shared storage during execution.

The raw data were not released directly. A preprocessing pipeline performed **data cleaning**, **outlier removal**, **format conversion**, and **metadata enrichment** [2108.02037]. Slurm HDF5 time series were extracted and converted to **CSV**, GPU traces were linked to jobs by adding `id_job` and node identifiers during preprocessing, and logs from distinct sources were transformed into structured, analysis-oriented files [2108.02037]. Consistent linkage across scheduler, CPU, GPU, node, filesystem, and physical data was explicitly preserved [2108.02037].

Privacy preservation is central to the public release. Sensitive fields such as **job names**, **working directories**, **MAC addresses**, **kernel versions**, and other site-specific constants were removed [2108.02037]. Key identifiers such as job IDs, user IDs, and node names were anonymized using **SHA‑256 with a secret salt**, with the same identifier mapped consistently across all files [2108.02037]. For node names, rack and node components were hashed and recombined into forms such as `r[hash]-n[hash]` [2108.02037]. This preserves relational structure while preventing direct reidentification.

A recurrent limitation follows from this design. Because semantic identifiers such as job names and working directories are removed, application-level interpretation must rely on utilization signatures, resource patterns, and the manually labeled subsets rather than on human-readable metadata [2108.02037]. This constraint shaped subsequent work on workload classification [2204.05839], [2209.05300].

## 4. Labeled subsets and workload-classification research

A major derivative of the dataset is the **MIT Supercloud Workload Classification Challenge**, which introduced a labeled subset for supervised learning [2204.05839]. The challenge uses monitoring logs from TX‑Gaia and focuses on GPU-accelerated jobs. The labeled subset contains **3,430 unique jobs** with labels and **26 distinct classes** corresponding to deep learning architectures and sub-architectures [2204.05839]. The workload families include **VGG**, **ResNet**, **Inception**, **U‑Net variants**, **BERT**, **DistilBERT**, and graph neural network models such as **DimeNet**, **SchNet**, **PNA**, and **NNConv** [2204.05839].

Each challenge sample is a **60-second window** of GPU-only telemetry resampled to a fixed shape **`(N_trials, 540, 7)`**, where the seven channels are the GPU sensors from the main dataset [2204.05839]. Windows are extracted from the **first 60 seconds**, the **middle 60 seconds**, or a **random 60-second segment**, yielding datasets such as `60-start-1`, `60-middle-1`, and `60-random-*` [2204.05839]. An **80/20 train/test split** is used, and `.npz` files contain `X_train`, `y_train`, `model_train`, `X_test`, `y_test`, and `model_test` [2204.05839].

The labeling methodology is explicit. The labeled jobs were generated by running known benchmark models on TX‑Gaia in controlled experiments, then manually assigning labels at both the family and variant levels and propagating those labels to all associated GPU time series for each job [2204.05839]. This means that the labeled subset is not an inferred taxonomy from production logs, but a curated ground-truth corpus.

Workload-classification baselines established that the telemetry is discriminative at the architecture level. Using covariance-based features extracted from each **\(540 \times 7\)** trial, the challenge paper reports **Random Forest** accuracy up to **93.02%** on `60-middle-1`, **XGBoost** at **88.47%**, and **Bidirectional LSTM** at approximately **90.81%** on `60-random-1` [2204.05839]. The feature-importance analysis highlighted covariance between **GPU % utilization** and **CPU % utilization**, variance of **GPU % utilization**, and variance of **power draw** as highly predictive [2204.05839].

Subsequent work on time-series preprocessing addressed the alignment problem for variable-length multichannel traces derived from the dataset [2209.05300]. Using a labeled GPU subset of **19,481** distinct GPU time series from **3,430** jobs and evaluating low-overhead preprocessing methods, that study reported **classification accuracy greater than 95%**, stated as **5%** above previous approaches [2209.05300]. The strongest results came from **summary statistics over windows** and **sampling Fourier coefficients**, rather than fixed contiguous windows [2209.05300]. This suggests that the dataset contains stable workload signatures distributed over the full job trajectory, not only in local temporal segments.

## 5. Power, utilization, and sustainability uses

The dataset was designed not only for workload identification but also for operational studies of resource use, energy, and datacenter behavior [2108.02037]. The paper introducing the dataset explicitly lists goals including improved scheduling policies, identifying inefficiencies in resource utilization, optimizing **energy and power consumption**, predicting failures, and detecting policy violations such as **crypto-currency mining** and **password cracking** [2108.02037].

Several descriptive statistics in the original dataset paper illustrate the operational character of the traces. GPU jobs span runtimes from minutes to hundreds of minutes; the **median runtime** for “any GPU” and single-GPU jobs is approximately **4 minutes**, while **85% of GPU jobs use only one GPU** [2108.02037]. Single-GPU jobs have median **SM utilization ≈ 23%**, whereas 2-GPU and ≥4-GPU jobs show a mixture of near-zero and high-utilization behavior, interpreted there as a combination of debugging/exploratory workloads and mature production jobs [2108.02037]. Strong positive correlations were observed among **power consumption**, **SM utilization**, **memory utilization**, **GPU temperature**, and **memory temperature**, while `pcie_link_width_current` showed little variation and weak correlation with the other metrics [2108.02037].

Later work turned these traces into a simulation environment for sustainability research. “Trace Replay Simulation of MIT SuperCloud for Studying Optimal Sustainability Policies” uses the **MIT SuperCloud dataset** as a **historical telemetry and workload trace** for the **TX‑GAIA** system [2509.16513]. In that work, the dataset supplies **CPU telemetry**, **GPU telemetry**, **power data at 100 ms intervals**, and **CPU frequency information**, together with job scheduling and workload traces [2509.16513]. CPU telemetry is averaged into **10 s** bins, GPU telemetry remains at **100 ms**, and the simulator advances in **1 s** steps [2509.16513].

Inside that simulation workflow, the traces are replayed or rescheduled in **ExaDigiT/RAPS**, which aggregates **total system power**, **energy conversion efficiency**, **throughput**, **system FLOPS**, and **system utilization**, and estimates **rectification and voltage conversion losses** and **PUE** [2509.16513]. An **OpenAI Gym–compatible interface** and early **PPO** experiments use reward functions combining **energy consumption**, **carbon footprint**, and **job throughput** [2509.16513]. Within that framing, the dataset becomes a real-world substrate for experimentation with **energy-aware scheduling**, **demand response**, and **load shaping** rather than a static archive alone.

This later use clarifies an important property of the dataset: it contains both fine-grained device telemetry and job-level temporal structure rich enough to support rescheduling counterfactuals. A plausible implication is that the dataset occupies an unusual middle ground between system trace archives and controllable simulation benchmarks.

## 6. Position within the broader MIT SuperCloud literature

The term “MIT SuperCloud dataset” has been used in more than one sense in the literature, and distinguishing these senses prevents confusion. The **2021 dataset paper** defines the phrase as the TX‑Gaia operational monitoring corpus of scheduler, CPU, GPU, filesystem, node-health, and physical datacenter traces [2108.02037]. The **workload classification challenge** uses the same dataset name for the public datacenter-trace release and its labeled subsets [2204.05839]. The **trace replay simulation** paper uses “MIT SuperCloud dataset” specifically as a telemetry and workload trace source for TX‑GAIA [2509.16513].

Other MIT SuperCloud papers use the platform as an execution environment rather than introducing this dataset. For example, D4M and GraphBLAS studies benchmark streaming updates of synthetic power-law graphs on MIT SuperCloud, achieving sustained rates from **1.9 billion** to **over 200,000,000,000 updates per second** across many nodes, but they do **not** define a named MIT SuperCloud dataset in the same sense [1902.00846], [1907.04217], [2001.06935], [2108.06650]. Likewise, work on internet-traffic analysis used **MAWI “Day in the Life”** traces processed on MIT SuperCloud rather than a native SuperCloud monitoring dataset [1808.08353]. Other papers describe deployable census analytics [2108.11525], portal-based web access [1707.05900], or database management infrastructure [1506.08506], again referring to datasets hosted on or produced by the platform, not to the TX‑Gaia telemetry release.

This distinction matters because some readers may assume that every “MIT SuperCloud” paper concerns the same data artifact. It does not. In current usage, the canonical research meaning of **MIT Supercloud Dataset** refers to the TX‑Gaia monitoring corpus introduced in 2021 and extended through public challenge datasets and trace-driven simulation studies [2108.02037], [2204.05839], [2509.16513].

## 7. Limitations, access, and research significance

The dataset’s strengths are balanced by clear limitations documented in the source papers. The CPU time series are sampled at **10 s**, node metrics at **5 minutes**, and only GPU telemetry has **100 ms** resolution, so some transient CPU or system-level phenomena are necessarily unobserved [2108.02037]. The labeled subset is relatively small compared with the full corpus: **≈2 GB** labeled versus **~2.1 TB** total, with **3,430 labeled jobs** across **26 classes**, and the labels focus heavily on **AI/ML workloads** rather than a full cross-section of scientific HPC applications [2204.05839]. Challenge results also showed lower accuracy for windows taken from the **start** of jobs, suggesting that early phases are often dominated by generic setup or data-loading behavior rather than architecture-specific signatures [2204.05839].

Cluster specificity is another constraint. The telemetry reflects **Volta V100** GPUs, the TX‑Gaia scheduler and partitions, and the particular facility configuration of the EcoPOD datacenter [2108.02037], [2204.05839]. This means that operational conclusions and learned representations may not transfer directly to clusters with different accelerators, software stacks, or cooling infrastructures. The trace-replay work likewise notes that validation of RL agents and multi-tenant schedulers remains pending, and that the detailed schema must be obtained from the original dataset documentation rather than from the simulation paper alone [2509.16513].

Access is provided through the **MIT Datacenter Challenge** website, **https://dcc.mit.edu**, and later papers also state that the data are available through the **AWS Open Data Registry**, with download instructions on the DCC site [2108.02037], [2209.05300]. The original dataset paper specifies that researchers must accept a **data use agreement** before download and that licensing details are posted on the challenge website [2108.02037].

The dataset’s research significance lies in its combination of properties rather than any single field. It is simultaneously a **job-linked scheduler trace**, a **multichannel CPU/GPU time-series corpus**, a **filesystem and node-health log**, and a **physical infrastructure telemetry archive** [2108.02037]. It has supported architecture-level workload classification [2204.05839], low-overhead time-series alignment research [2209.05300], and sustainability-oriented digital-twin scheduling experiments [2509.16513]. This suggests that the MIT Supercloud Dataset functions less as a single benchmark narrowly tied to one task and more as a shared systems-observability substrate for studies spanning HPC operations, ML for systems, and energy-aware datacenter management.

Source: https://www.emergentmind.com/topics/mit-supercloud-dataset