Google Cluster Data (GCD) Overview

• GCD is a public benchmark dataset capturing anonymized workload traces, machine events, and placement constraints from Google clusters.
• It includes detailed records from 2011 and 2019 releases, offering granular insights into resource usage, job lifecycle, and failure dynamics.
• GCD supports research in workload classification, failure prediction, and constraint-aware scheduling to enhance large-scale cloud management.

Searching arXiv for relevant papers on Google Cluster Data / Google Cluster Trace. Google Cluster Data (GCD) denotes the public Google cluster workload traces that have become benchmark datasets for studying large-scale production cloud behavior, scheduling, resource utilization, and failure dynamics. In the literature, the term is anchored primarily in the 2011 “ClusterData2011_1” release, which captured approximately 29 days of activity from a cluster of approximately 11,000 machines, and has later been extended in practice to include the 2019 Google Cluster Trace dataset, which comprises roughly 2.4 TiB of workload traces from eight geographically distributed Borg clusters collected over May 2019. Across these releases, GCD exposes anonymized machine events, job and task lifecycle events, placement constraints, and periodic resource-usage measurements, enabling both descriptive workload analysis and predictive modeling (Alam et al., 2015, Bappy et al., 2023).

1. Scope, releases, and archival structure

GCD is not a single immutable corpus. The 2011 release and the 2019 release differ substantially in scale, organization, and analytical emphasis. The 2011 trace was published as a trace of a cluster of 11k machines, referred to as Google Cluster Trace, and contains cell information of about 29 days. The 2019 Google Cluster Trace dataset is much larger, with traces gathered from eight different clusters around the world over a 30 day period and totaling roughly 2.4 TiB of raw event logs (Alam et al., 2015, Bappy et al., 2023).

Release	Scale and duration	Main organization
2011 “ClusterData2011_1”	Approximately 11,000 machines; 29 days; May 2011	Six CSV-style tables totaling ~40 GB compressed (~300 GB decompressed)
2019 Google Cluster Trace	Roughly 2.4 TiB; eight geographically distributed Borg clusters; month of May 2019	Four main record types: Machine Events, Instance Events, Collection Events, Resource Usage Records

The 2011 archive is commonly described as six CSV files, including MachineEvents.csv, MachineAttributes.csv, JobEvents.csv, TaskEvents.csv, TaskConstraints.csv, and TaskUsage.csv. Analyses of the same corpus also describe a normalized logical schema with tables such as task_events, job_events, machine_events, resource_usage, and task_constraints, reflecting downstream ingestion pipelines rather than a different underlying dataset (Alam et al., 2015, Balliu et al., 2014).

A recurrent source of confusion is to treat GCD as only a workload log. In fact, the trace family combines workload, placement, and infrastructure state. Machine additions, removals, and capacity updates coexist with task lifecycle transitions such as submit, schedule, evict, fail, kill, finish, lost, and update-related events. This breadth is the reason GCD has been used not only for scheduler analysis, but also for simulation, failure characterization, and constraint-aware placement research (Zhu et al., 2015, Sliwko, 22 Sep 2025).

2. Event semantics, schema, and resource observability

The 2011 trace is highly anonymized; each job has one or more tasks, and each task runs on one machine. Machine-level events include [ADD](https://www.emergentmind.com/topics/adversarial-diffusion-distillation-add), REMOVE, and UPDATE, with recorded capacity fields for CPU and RAM. In one description of the trace, CPU is recorded in cores, most machines have 0.5 core, and RAM appears in four discrete levels: 0.25, 0.5, 0.75, and 1.0. Job events include submit, schedule, evict, fail, kill, and finish. Task events add task-specific timestamps, event type, priority class, requested CPU and RAM, and machine-placement constraints (Alam et al., 2015).

Task constraints are expressed as Boolean or inequality predicates on machine attributes, such as kernel version. In the 2011 archive, task_constraints.csv records constraintID, attributeName, operator, and value; in later constraint-oriented analyses of clusterdata-2011, the same logical content is described as per-task constraint rows with attribute_name, operator_code, and attribute_value, where operator_code is mapped as 0 → Equal, 1 → Not-Equal, 2 → Greater-Than, and 3 → Less-Than. The same 2025 analysis reports 67 distinct attribute names in machine_attributes.csv (Sliwko, 22 Sep 2025).

Resource usage is sampled at five-minute intervals. For the 2011 trace, task resource-usage logs are described as being internally sampled each second and exposed every five minutes, with fields including CPU rate, memory used, memory allocated, page-cache usage, local disk I/O time, cycles, and other low-level counters. The 2019 trace preserves the five-minute reporting cadence and records per-machine and per-task usage of CPU, memory, page cache, and disk I/O, with key fields such as cpu_usage_avg, memory_usage_avg, page_cache_usage, and disk_io (Alam et al., 2015, Bappy et al., 2023).

The 2019 schema introduces explicit multi-cluster identifiers and richer scheduling metadata. Key fields include timestamp as Unix-epoch time in microseconds, cluster_id, machine_id, event_type / event_subtype, job_id, task_index, scheduling_class, priority, resource_request_cpu, resource_request_memory, resource_request_disk, and runtime. Scheduling class is defined as one of four scheduling tiers, and priority as five distinct priority levels tied to different price or SLO tiers within Borg (Bappy et al., 2023).

3. Workload structure, job taxonomy, and resource-demand regimes

A central line of work on the 2011 trace profiles jobs by CPU and memory consumption. Across all jobs, CPU usage is highly skewed, with mean CPU fraction approximately 0.05 cores, median approximately 0.01, and 90th percentile approximately 0.20; the variance is large, with a long tail up to approximately 0.4 cores. Memory usage is similarly skewed, with mean approximately 0.10 as a fraction of machine RAM, median approximately 0.02, 90th percentile approximately 0.30, and a small fraction of jobs consuming more than 0.5 RAM. The per-job peak percentile table reports CPU peaks of 0.5%, 1.0%, 5.0%, 20%, and 40% at the 25th, 50th, 75th, 90th, and maximum observed levels respectively; the corresponding RAM peaks are 1.0%, 2.0%, 10.0%, 30%, and 80% (Alam et al., 2015).

The same analysis describes a CPU histogram with a strong spike near zero and a long tail extending to approximately 0.4 cores. The memory usage CDF indicates that 70% of jobs stay below 0.1 RAM and 95% stay below 0.4. These shapes motivated a feature representation in which each job is a point $x = (x_1, x_2)$ in $\mathbb{R}^2$, with $x_1$ equal to peak CPU usage and $x_2$ equal to peak memory usage. Clustering was then carried out with $k$-means by minimizing

$$J = \sum_{i=1}^{k} \sum_{x \in C_i} \|x - \mu_i\|^2,$$

using the Euclidean norm in CPU–RAM space, random centroids, and values $k=3$ and $k=5$ (Alam et al., 2015).

The coarse $k=3$ solution yields a trimodal classification: Short, Medium, and Long. The reported centroids are (0.01, 0.02), (0.10, 0.08), and (0.30, 0.25) respectively, with fractions of jobs 70%, 20%, and 10%. Within-cluster variance is lowest for the short cluster and largest for the long cluster. A second-level $k=5$ clustering subdivides the workload into Very short, Approaching-mid, Mid, Receding-long, and Very long, with centroids shifting gradually from (0.005,0.01) up to (0.4,0.3) (Alam et al., 2015).

The claim that production jobs are naturally trimodal is significant because a common simplification is to describe cluster workloads as purely bimodal. The 2015 workload analysis explicitly argues that trimodal classification more faithfully describes production behavior. In the same account, short jobs have lifetimes of seconds to a few minutes and low resource demands, medium jobs run for tens of minutes and are moderately RAM-intensive, and long jobs run for hours, are the most resource-hungry, and often request more resources than they strictly use. Aggregate resource consumption is correspondingly concentrated: short jobs account for 70% of jobs but only 15% of aggregated CPU time and 10% of aggregated RAM usage, while long jobs constitute 10% of jobs yet account for 65% of aggregated CPU time and 75% of aggregated RAM usage (Alam et al., 2015).

Another notable result is the symmetry observed in long-job tasks. Tasks within the same long job exhibit very similar resource-usage curves, with high correlation. The stated implication is that the scheduler can predict per-task demand from a few sampled tasks and co-locate or pack them more efficiently. This suggests that GCD supports not only static workload classification but also intra-job demand inference relevant to scheduler design (Alam et al., 2015).

4. Temporal dynamics, analytical toolchains, and simulation

Several analyses of GCD focus on event-stream dynamics rather than only per-job summaries. A 2015 study of the Google cluster trace describes a preprocessing workflow consisting of data cleaning and filtering, de-duplication of submit events, state-transition normalization that collapses update_pending and update_running so that each task appears to move among {pending, running, dead}, joins by <job ID, task index> and by timestamp, and time-series aggregation into per-timestamp counters for submissions, schedules, completions, evictions, and related events. The same study uses cumulative distribution functions, moving averages, queue-length time series, and cluster-level utilization measures such as

$\mathbb{R}^2$0

task density

$\mathbb{R}^2$1

and resource-allocation efficiency

$\mathbb{R}^2$2

(Zhu et al., 2015)

Within that temporal view, several regularities are reported. The CDF of first-time submit events is nearly a straight line, indicating a nearly constant submission rate, and the finish-event CDF is almost identical, implying completion throughput approximately matches arrival throughput. At the same time, weighted CDF comparisons show backlog growth, queue length grows linearly with a modest slope, and the number of running tasks increases especially in the final third of the month. Despite this, moving-average execution time and scheduling latency show no upward trend. Task durations are broad: only 2 completed tasks are shorter than 1 s, 80% complete within 30 minutes, and the longest approach 9 days (Zhu et al., 2015).

BiDAl operationalized GCD analysis as a reusable framework. It is described as a prototype log-data analysis framework written in Java with a modular and extensible architecture. Supported storage backends are HDFS and SQLite, and supported analysis languages are SQL, R, and Hadoop MapReduce. On the 2011 trace, importing 17 GB of task_events.csv into SQLite took approximately 11 minutes on a 16 GB-RAM, quad-core laptop, and a first GROUP BY query for distinct job-task pairs took approximately 4 minutes on SQLite and approximately 6 minutes on HDFS+MapReduce using a 4-node mini-cluster; R-based statistical fits on reduced data took less than 30 s per distribution (Balliu et al., 2014).

BiDAl’s statistical analyses identified further structure in GCD. Completed task durations follow a long-tailed distribution well-fitted by a mixture of Weibull components, while killed task durations exhibit a different tail. CPU-request and RAM-request distributions were fitted by cubic splines and a lognormal body with heavy tail, respectively. The time between machine update events is approximately exponential with rate $\mathbb{R}^2$3, corresponding to a mean of approximately 3300 s. Pearson correlation between CPU and RAM requests is reported as $\mathbb{R}^2$4\rho < 0.1. Burstiness in job arrivals is quantified by a [coefficient of variation](https://www.emergentmind.com/topics/coefficient-of-variation-cv) of approximately1.2, and [FFT](https://www.emergentmind.com/topics/implicit-padding-in-spectral-convolution-fft) reveals daily cycles with peaks at24 h` and harmonics (Balliu et al., 2014).

The same framework was used to build a discrete-event simulator in Omnet++/C++ with passive entities Job and Task, and active entities JobArrival, MachineArrival, Scheduler, Machine, and Network. Job arrivals are modeled as a non-stationary Poisson process with time-varying rate $\mathbb{R}^2$5 estimated by kernel-smoothed counts, service times follow empirical distributions fitted by splines or mixtures of Weibulls, and validation against the trace on four smoothed metrics—running tasks, completed tasks, waiting queue size, and evicted tasks—achieved root-mean-square error below 5% for all four metrics (Balliu et al., 2014).

5. Failure characteristics and user-level heterogeneity in the 2019 trace

The 2019 Google Cluster Trace has been used to study failure characteristics at a substantially larger scale. The dataset contains traces from eight geographically distributed Borg clusters, each managing on the order of 12 000 heterogeneous machines, and during the 30 day period the clusters processed on the order of 5 million distinct jobs composed of tens of millions of tasks, generating over one hundred million collection events in total. Analyses compare finished, killed, and failed tasks using resource usage, job priority, scheduling class, job duration, and the number of task resubmissions (Bappy et al., 2023).

Several quantitative hallmarks of failure are reported. CPU usage of failed jobs is, on average, up to 13× higher than that of successful jobs, and CPU usage for failing tasks ramps up sharply after roughly the 60th runtime percentile. Memory usage diverges in the opposite direction: failed tasks often consume less total RAM and almost no page cache compared with successfully completed tasks. Approximately 40 % of failed jobs run longer than the average successful job. Resubmission is strongly associated with failure: failed collections are re-submitted an order of magnitude more often than successful ones, and the worst offender was re-submitted 7 947 555 times (Bappy et al., 2023).

Failure and kill rates vary across clusters, but the same study reports a typical failure range of 10 – 30 % of all submitted jobs, with killed tasks accounting for an additional ~5 – 10 % of events and roughly 60 – 75 % of tasks finishing normally. High-priority, low-latency scheduling classes exhibit lower failure rates, whereas best-effort batch jobs show the highest incidence of both outright failures and preemptions. These results position GCD as a substrate for studying how resource pressure, priority, and job persistence interact in production cloud environments (Bappy et al., 2023).

A distinctive contribution of the 2019 analysis is its treatment of user behavior. By linking submissions to obfuscated user identifiers, it reports that the single largest user in one cluster generated 55.53 % of all collection events and was responsible for 75.50 % of all KILL events on that cluster. In many cases, a single user accounted for over 20 % of that cluster’s failures, and 24.9 % of failed jobs resided within a single collection. Across the top five users, vertical scaling usage patterns ranged from fully-automated autoscaling to entirely constrained or disabled scaling, yet all five were concentrated in either production or best-effort batch classes (Bappy et al., 2023).

These findings have a direct scheduling interpretation. The analysis contends that CPU ramp-up slopes, low RAM utilization, long runtime, and high resubmission rates could serve as predictive features for early failure prediction, while tenant-level skew motivates constrained scaling modes or soft throttling for users who dominate collection events or failures. This suggests that GCD is not merely a passive benchmarking resource; it also supports policy design for dynamic rescheduling and failure-aware capacity management (Bappy et al., 2023).

6. Constraint-aware scheduling and machine-learning extensions

A more recent research direction treats GCD as a source of structured placement constraints rather than only as a workload trace. Using clusterdata-2011 and the AGOCS framework, one study extracts node attributes from machine_events.csv and machine_attributes.csv, task requirements from task_events.csv, and placement constraints from task_constraints.csv. Each machine is represented with capacities and an attribute map, and each task is associated with decoded logical operators in {Equal, NotEqual, GreaterThan, LessThan} (Sliwko, 22 Sep 2025).

The core methodological problem is that a single task can have multiple constraint rows for the same attribute. The reported compaction strategy groups constraints by attribute_name, keeps the single maximum bound when only lower-bound predicates are present, replaces compatible lower-bound and upper-bound predicates by a Between constraint, collects multiple NotEqual exclusions into NotEqualArray, and lets Equal override any exclusions on the same attribute. Conflicting constraints are rare and lead to the task being dropped. After compaction, each task has at most one constraint per attribute in one of four classes: Equal(attr, v), NotEqualArray(attr, {v_1,\ldots,v_k}), GreaterEqual(attr, v_min), or Between(attr, v_min, v_max) (Sliwko, 22 Sep 2025).

Eligibility is then defined by evaluating each node’s attribute map and capacity against the compacted constraint set. The paper expresses this as

$\mathbb{R}^2$6

and

$\mathbb{R}^2$7

Tasks are grouped into Group A when $N_{\mathrm{eligible}} = 1<code>,</code>Group C<code>when</code>1 &lt; N_{\mathrm{eligible}} \le 1000<code>, and</code>Group Z` otherwise (<a href="/papers/2509.17695" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Sliwko, 22 Sep 2025</a>).</p> <p>The resulting statistics show how sparse highly constrained placement can be. The study reports <code>12 500</code> total cluster nodes, <code>48.4 million</code> tasks in <code>30 days</code>, and <code>23.29% ≈ 935 900</code> tasks with at least one constraint operator. After grouping by <code>(job_id, task_index)</code> and removing duplicate constraint-memory-CPU patterns, the dataset contains <code>27 700</code> unique data points. The distribution by eligible-node count is <code>15 400</code> tasks in Group A, <code>3 200</code> tasks in Group C, and <code>910 200</code> tasks in Group Z, with an average of approximately <code>3.2</code> constraints per constrained task (<a href="/papers/2509.17695" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Sliwko, 22 Sep 2025</a>).</p> <p>For machine learning, compacted constraints are one-hot encoded. Although the theoretical space is up to <code>67 attributes × 4 operator-classes = up to 268 possible &quot;attribute + operator&quot; categories</code>, the practical vocabulary after compaction is reported as <code>4 404</code> distinct token-values. Each task becomes a binary vector in $\mathbb{R}^2$8, augmented with two min–max normalized numeric features for CPU and memory requests, yielding a feature vector in $\mathbb{R}^{4404+2}`. The final 27 700 × 4406 matrix is split using train_test_split with test_size=0.25, random_state=42, and shuffling enabled, producing approximately 20 775 training points and 6 925 test points. The final ensemble voting classifier achieved 98% accuracy and a 1.5-1.8% misclassification rate for tasks with a single suitable node (Sliwko, 22 Sep 2025).

The broader significance is that GCD’s placement metadata is sufficiently rich to support predictive allocation, not only retrospective workload analysis. Earlier GCD studies emphasized utilization, backlog, and task classes; the constraint-oriented work shows that the same public trace also supports supervised learning on rare-node affinity patterns. This suggests a convergence between trace analysis, scheduler design, and data-driven placement control within a single benchmark family (Sliwko, 22 Sep 2025).