---
title: Google Cluster Data (GCD) Overview
url: https://www.emergentmind.com/topics/google-cluster-data-gcd
type: topic
---

# Google Cluster Data (GCD) Overview

Searching arXiv for relevant papers on Google Cluster Data / Google Cluster Trace.
Google Cluster Data (GCD) denotes the public Google cluster workload traces that have become benchmark datasets for studying large-scale production cloud behavior, scheduling, resource utilization, and failure dynamics. In the literature, the term is anchored primarily in the 2011 “ClusterData2011_1” release, which captured approximately 29 days of activity from a cluster of approximately 11,000 machines, and has later been extended in practice to include the 2019 Google Cluster Trace dataset, which comprises roughly 2.4 TiB of workload traces from eight geographically distributed Borg clusters collected over May 2019. Across these releases, GCD exposes anonymized machine events, job and task lifecycle events, placement constraints, and periodic resource-usage measurements, enabling both descriptive workload analysis and predictive modeling [1501.01426] [2308.02358].

## 1. Scope, releases, and archival structure

GCD is not a single immutable corpus. The 2011 release and the 2019 release differ substantially in scale, organization, and analytical emphasis. The 2011 trace was published as a trace of a cluster of 11k machines, referred to as Google Cluster Trace, and contains cell information of about 29 days. The 2019 Google Cluster Trace dataset is much larger, with traces gathered from eight different clusters around the world over a 30 day period and totaling roughly 2.4 TiB of raw event logs [1501.01426] [2308.02358].

| Release | Scale and duration | Main organization |
|---|---|---|
| 2011 “ClusterData2011_1” | Approximately 11,000 machines; 29 days; May 2011 | Six CSV-style tables totaling ~40 GB compressed (~300 GB decompressed) |
| 2019 Google Cluster Trace | Roughly 2.4 TiB; eight geographically distributed Borg clusters; month of May 2019 | Four main record types: Machine Events, Instance Events, Collection Events, Resource Usage Records |

The 2011 archive is commonly described as six CSV files, including `MachineEvents.csv`, `MachineAttributes.csv`, `JobEvents.csv`, `TaskEvents.csv`, `TaskConstraints.csv`, and `TaskUsage.csv`. Analyses of the same corpus also describe a normalized logical schema with tables such as `task_events`, `job_events`, `machine_events`, `resource_usage`, and `task_constraints`, reflecting downstream ingestion pipelines rather than a different underlying dataset [1501.01426] [1410.1309].

A recurrent source of confusion is to treat GCD as only a workload log. In fact, the trace family combines workload, placement, and infrastructure state. Machine additions, removals, and capacity updates coexist with task lifecycle transitions such as submit, schedule, evict, fail, kill, finish, lost, and update-related events. This breadth is the reason GCD has been used not only for scheduler analysis, but also for simulation, failure characterization, and constraint-aware placement research [1508.02111] [2509.17695].

## 2. Event semantics, schema, and resource observability

The 2011 trace is highly anonymized; each job has one or more tasks, and each task runs on one machine. Machine-level events include `ADD`, `REMOVE`, and `UPDATE`, with recorded capacity fields for CPU and RAM. In one description of the trace, CPU is recorded in cores, most machines have `0.5` core, and RAM appears in four discrete levels: `0.25`, `0.5`, `0.75`, and `1.0`. Job events include submit, schedule, evict, fail, kill, and finish. Task events add task-specific timestamps, event type, priority class, requested CPU and RAM, and machine-placement constraints [1501.01426].

Task constraints are expressed as Boolean or inequality predicates on machine attributes, such as kernel version. In the 2011 archive, `task_constraints.csv` records `constraintID`, `attributeName`, `operator`, and `value`; in later constraint-oriented analyses of `clusterdata-2011`, the same logical content is described as per-task constraint rows with `attribute_name`, `operator_code`, and `attribute_value`, where `operator_code` is mapped as `0 → Equal`, `1 → Not-Equal`, `2 → Greater-Than`, and `3 → Less-Than`. The same 2025 analysis reports `67 distinct attribute names` in `machine_attributes.csv` [2509.17695].

Resource usage is sampled at five-minute intervals. For the 2011 trace, task resource-usage logs are described as being internally sampled each second and exposed every five minutes, with fields including CPU rate, memory used, memory allocated, page-cache usage, local disk I/O time, cycles, and other low-level counters. The 2019 trace preserves the five-minute reporting cadence and records per-machine and per-task usage of CPU, memory, page cache, and disk I/O, with key fields such as `cpu_usage_avg`, `memory_usage_avg`, `page_cache_usage`, and `disk_io` [1501.01426] [2308.02358].

The 2019 schema introduces explicit multi-cluster identifiers and richer scheduling metadata. Key fields include `timestamp` as Unix-epoch time in microseconds, `cluster_id`, `machine_id`, `event_type / event_subtype`, `job_id`, `task_index`, `scheduling_class`, `priority`, `resource_request_cpu`, `resource_request_memory`, `resource_request_disk`, and `runtime`. Scheduling class is defined as one of four scheduling tiers, and priority as five distinct priority levels tied to different price or SLO tiers within Borg [2308.02358].

## 3. Workload structure, job taxonomy, and resource-demand regimes

A central line of work on the 2011 trace profiles jobs by CPU and memory consumption. Across all jobs, CPU usage is highly skewed, with mean CPU fraction approximately `0.05` cores, median approximately `0.01`, and 90th percentile approximately `0.20`; the variance is large, with a long tail up to approximately `0.4` cores. Memory usage is similarly skewed, with mean approximately `0.10` as a fraction of machine RAM, median approximately `0.02`, 90th percentile approximately `0.30`, and a small fraction of jobs consuming more than `0.5` RAM. The per-job peak percentile table reports CPU peaks of `0.5%`, `1.0%`, `5.0%`, `20%`, and `40%` at the 25th, 50th, 75th, 90th, and maximum observed levels respectively; the corresponding RAM peaks are `1.0%`, `2.0%`, `10.0%`, `30%`, and `80%` [1501.01426].

The same analysis describes a CPU histogram with a strong spike near zero and a long tail extending to approximately `0.4` cores. The memory usage CDF indicates that `70%` of jobs stay below `0.1` RAM and `95%` stay below `0.4`. These shapes motivated a feature representation in which each job is a point $x = (x_1, x_2)$ in $\mathbb{R}^2$, with $x_1$ equal to peak CPU usage and $x_2$ equal to peak memory usage. Clustering was then carried out with $k$-means by minimizing
$$
J = \sum_{i=1}^{k} \sum_{x \in C_i} \|x - \mu_i\|^2,
$$
using the Euclidean norm in CPU–RAM space, random centroids, and values $k=3$ and $k=5$ [1501.01426].

The coarse $k=3$ solution yields a trimodal classification: `Short`, `Medium`, and `Long`. The reported centroids are `(0.01, 0.02)`, `(0.10, 0.08)`, and `(0.30, 0.25)` respectively, with fractions of jobs `70%`, `20%`, and `10%`. Within-cluster variance is lowest for the short cluster and largest for the long cluster. A second-level $k=5$ clustering subdivides the workload into `Very short`, `Approaching-mid`, `Mid`, `Receding-long`, and `Very long`, with centroids shifting gradually from `(0.005,0.01)` up to `(0.4,0.3)` [1501.01426].

The claim that production jobs are naturally trimodal is significant because a common simplification is to describe cluster workloads as purely bimodal. The 2015 workload analysis explicitly argues that trimodal classification more faithfully describes production behavior. In the same account, short jobs have lifetimes of seconds to a few minutes and low resource demands, medium jobs run for tens of minutes and are moderately RAM-intensive, and long jobs run for hours, are the most resource-hungry, and often request more resources than they strictly use. Aggregate resource consumption is correspondingly concentrated: short jobs account for `70%` of jobs but only `15%` of aggregated CPU time and `10%` of aggregated RAM usage, while long jobs constitute `10%` of jobs yet account for `65%` of aggregated CPU time and `75%` of aggregated RAM usage [1501.01426].

Another notable result is the symmetry observed in long-job tasks. Tasks within the same long job exhibit very similar resource-usage curves, with high correlation. The stated implication is that the scheduler can predict per-task demand from a few sampled tasks and co-locate or pack them more efficiently. This suggests that GCD supports not only static workload classification but also intra-job demand inference relevant to scheduler design [1501.01426].

## 4. Temporal dynamics, analytical toolchains, and simulation

Several analyses of GCD focus on event-stream dynamics rather than only per-job summaries. A 2015 study of the Google cluster trace describes a preprocessing workflow consisting of data cleaning and filtering, de-duplication of submit events, state-transition normalization that collapses `update_pending` and `update_running` so that each task appears to move among `{pending, running, dead}`, joins by `<job ID, task index>` and by timestamp, and time-series aggregation into per-timestamp counters for submissions, schedules, completions, evictions, and related events. The same study uses cumulative distribution functions, moving averages, queue-length time series, and cluster-level utilization measures such as
$$
U_{\mathrm{cpu}}(t) = \frac{\sum_m cpu_m(t)}{M},
$$
task density
$$
\rho(t) = \frac{N_{\mathrm{run}}(t)}{M},
$$
and resource-allocation efficiency
$$
\eta(t) = \frac{\sum_i u_i(t)}{\sum_i r_i}.
$$
[1508.02111]

Within that temporal view, several regularities are reported. The CDF of first-time submit events is nearly a straight line, indicating a nearly constant submission rate, and the finish-event CDF is almost identical, implying completion throughput approximately matches arrival throughput. At the same time, weighted CDF comparisons show backlog growth, queue length grows linearly with a modest slope, and the number of running tasks increases especially in the final third of the month. Despite this, moving-average execution time and scheduling latency show no upward trend. Task durations are broad: only `2` completed tasks are shorter than `1 s`, `80%` complete within `30 minutes`, and the longest approach `9 days` [1508.02111].

BiDAl operationalized GCD analysis as a reusable framework. It is described as a prototype log-data analysis framework written in Java with a modular and extensible architecture. Supported storage backends are `HDFS` and `SQLite`, and supported analysis languages are `SQL`, `R`, and `Hadoop MapReduce`. On the 2011 trace, importing `17 GB` of `task_events.csv` into SQLite took approximately `11 minutes` on a `16 GB`-RAM, quad-core laptop, and a first `GROUP BY` query for distinct job-task pairs took approximately `4 minutes` on SQLite and approximately `6 minutes` on `HDFS+MapReduce` using a `4-node mini-cluster`; R-based statistical fits on reduced data took less than `30 s` per distribution [1410.1309].

BiDAl’s statistical analyses identified further structure in GCD. Completed task durations follow a long-tailed distribution well-fitted by a mixture of Weibull components, while killed task durations exhibit a different tail. CPU-request and RAM-request distributions were fitted by cubic splines and a lognormal body with heavy tail, respectively. The time between machine update events is approximately exponential with rate $\lambda_u \approx 0.0003\,\mathrm{s}^{-1}$, corresponding to a mean of approximately `3300 s`. Pearson correlation between CPU and RAM requests is reported as $\rho \approx 0.65`, indicating moderate positive correlation, whereas disk I/O versus priority shows low correlation with $\rho < 0.1`. Burstiness in job arrivals is quantified by a coefficient of variation of approximately `1.2`, and FFT reveals daily cycles with peaks at `24 h` and harmonics [1410.1309].

The same framework was used to build a discrete-event simulator in `Omnet++/C++` with passive entities `Job` and `Task`, and active entities `JobArrival`, `MachineArrival`, `Scheduler`, `Machine`, and `Network`. Job arrivals are modeled as a non-stationary Poisson process with time-varying rate $\lambda(t)$ estimated by kernel-smoothed counts, service times follow empirical distributions fitted by splines or mixtures of Weibulls, and validation against the trace on four smoothed metrics—running tasks, completed tasks, waiting queue size, and evicted tasks—achieved root-mean-square error below `5%` for all four metrics [1410.1309].

## 5. Failure characteristics and user-level heterogeneity in the 2019 trace

The 2019 Google Cluster Trace has been used to study failure characteristics at a substantially larger scale. The dataset contains traces from eight geographically distributed Borg clusters, each managing on the order of `12 000` heterogeneous machines, and during the 30 day period the clusters processed on the order of `5 million` distinct jobs composed of tens of millions of tasks, generating over one hundred million collection events in total. Analyses compare finished, killed, and failed tasks using resource usage, job priority, scheduling class, job duration, and the number of task resubmissions [2308.02358].

Several quantitative hallmarks of failure are reported. CPU usage of failed jobs is, on average, up to `13×` higher than that of successful jobs, and CPU usage for failing tasks ramps up sharply after roughly the `60th runtime percentile`. Memory usage diverges in the opposite direction: failed tasks often consume less total RAM and almost no page cache compared with successfully completed tasks. Approximately `40 %` of failed jobs run longer than the average successful job. Resubmission is strongly associated with failure: failed collections are re-submitted an order of magnitude more often than successful ones, and the worst offender was re-submitted `7 947 555` times [2308.02358].

Failure and kill rates vary across clusters, but the same study reports a typical failure range of `10 – 30 %` of all submitted jobs, with killed tasks accounting for an additional `~5 – 10 %` of events and roughly `60 – 75 %` of tasks finishing normally. High-priority, low-latency scheduling classes exhibit lower failure rates, whereas best-effort batch jobs show the highest incidence of both outright failures and preemptions. These results position GCD as a substrate for studying how resource pressure, priority, and job persistence interact in production cloud environments [2308.02358].

A distinctive contribution of the 2019 analysis is its treatment of user behavior. By linking submissions to obfuscated user identifiers, it reports that the single largest user in one cluster generated `55.53 %` of all collection events and was responsible for `75.50 %` of all `KILL` events on that cluster. In many cases, a single user accounted for over `20 %` of that cluster’s failures, and `24.9 %` of failed jobs resided within a single collection. Across the top five users, vertical scaling usage patterns ranged from fully-automated autoscaling to entirely constrained or disabled scaling, yet all five were concentrated in either production or best-effort batch classes [2308.02358].

These findings have a direct scheduling interpretation. The analysis contends that CPU ramp-up slopes, low RAM utilization, long runtime, and high resubmission rates could serve as predictive features for early failure prediction, while tenant-level skew motivates constrained scaling modes or soft throttling for users who dominate collection events or failures. This suggests that GCD is not merely a passive benchmarking resource; it also supports policy design for dynamic rescheduling and failure-aware capacity management [2308.02358].

## 6. Constraint-aware scheduling and machine-learning extensions

A more recent research direction treats GCD as a source of structured placement constraints rather than only as a workload trace. Using `clusterdata-2011` and the `AGOCS` framework, one study extracts node attributes from `machine_events.csv` and `machine_attributes.csv`, task requirements from `task_events.csv`, and placement constraints from `task_constraints.csv`. Each machine is represented with capacities and an attribute map, and each task is associated with decoded logical operators in `{Equal, NotEqual, GreaterThan, LessThan}` [2509.17695].

The core methodological problem is that a single task can have multiple constraint rows for the same attribute. The reported compaction strategy groups constraints by `attribute_name`, keeps the single maximum bound when only lower-bound predicates are present, replaces compatible lower-bound and upper-bound predicates by a `Between` constraint, collects multiple `NotEqual` exclusions into `NotEqualArray`, and lets `Equal` override any exclusions on the same attribute. Conflicting constraints are rare and lead to the task being dropped. After compaction, each task has at most one constraint per attribute in one of four classes: `Equal(attr, v)`, `NotEqualArray(attr, {v_1,\ldots,v_k})`, `GreaterEqual(attr, v_min)`, or `Between(attr, v_min, v_max)` [2509.17695].

Eligibility is then defined by evaluating each node’s attribute map and capacity against the compacted constraint set. The paper expresses this as
$$
pass(C,A) = \forall (attr \to op,v)\in task.constraints \; . \; eval(A[attr]\; op \; v),
$$
and
$$
N_{\mathrm{eligible}} = |\{node \in cluster : pass(C,node.attributes) \wedge node\ has \ge CPU\_request \wedge node\ has \ge memory\_request\}|.
$$
Tasks are grouped into `Group A` when $N_{\mathrm{eligible}} = 1`, `Group C` when `1 < N_{\mathrm{eligible}} \le 1000`, and `Group Z` otherwise [2509.17695].

The resulting statistics show how sparse highly constrained placement can be. The study reports `12 500` total cluster nodes, `48.4 million` tasks in `30 days`, and `23.29% ≈ 935 900` tasks with at least one constraint operator. After grouping by `(job_id, task_index)` and removing duplicate constraint-memory-CPU patterns, the dataset contains `27 700` unique data points. The distribution by eligible-node count is `15 400` tasks in Group A, `3 200` tasks in Group C, and `910 200` tasks in Group Z, with an average of approximately `3.2` constraints per constrained task [2509.17695].

For machine learning, compacted constraints are one-hot encoded. Although the theoretical space is up to `67 attributes × 4 operator-classes = up to 268 possible "attribute + operator" categories`, the practical vocabulary after compaction is reported as `4 404` distinct token-values. Each task becomes a binary vector in $\{0,1\}^{4404}$, augmented with two min–max normalized numeric features for CPU and memory requests, yielding a feature vector in $\mathbb{R}^{4404+2}`. The final `27 700 × 4406` matrix is split using `train_test_split` with `test_size=0.25`, `random_state=42`, and shuffling enabled, producing approximately `20 775` training points and `6 925` test points. The final ensemble voting classifier achieved `98%` accuracy and a `1.5-1.8%` misclassification rate for tasks with a single suitable node [2509.17695].

The broader significance is that GCD’s placement metadata is sufficiently rich to support predictive allocation, not only retrospective workload analysis. Earlier GCD studies emphasized utilization, backlog, and task classes; the constraint-oriented work shows that the same public trace also supports supervised learning on rare-node affinity patterns. This suggests a convergence between trace analysis, scheduler design, and data-driven placement control within a single benchmark family [2509.17695].

Source: https://www.emergentmind.com/topics/google-cluster-data-gcd