BigDataBench: Benchmarking Big Data & AI

• BigDataBench is a comprehensive benchmark suite that abstracts big data and AI workloads into reusable data motifs, ensuring realistic and representative evaluation.
• It features a modular design integrating diverse workloads, software stacks, and scalable data generation to capture both system-level and micro-architectural metrics.
• Its methodology employs workload reduction, motif-based proxy generation, and advanced statistical techniques to deliver high-fidelity, reproducible benchmarking across multiple domains.

BigDataBench is a comprehensive, open-source benchmark suite designed to evaluate big data systems, architectures, and domain-specific hardware/software co-designs. Originating from collaborative efforts with industrial partners, BigDataBench has undergone several architectural evolutions, growing from a search engine–centric suite into a unified platform for both big data and AI workloads. Its fundamental innovation is the abstraction of workloads into reusable computation primitives (“data motifs”), systematic workload diversification across data types and software stacks, and scientifically rigorous characterization at both system and micro-architectural levels. BigDataBench defines the current state of the art for workload representativeness, input veracity, and multi-platform comparability in big data and AI benchmarking (Wang et al., 2014, Gao et al., 2018, Gao et al., 2018, Gao et al., 2018, Gao et al., 2017).

1. Motivations and Benchmarking Principles

BigDataBench was developed in recognition of the limitations in prior benchmarking efforts, which typically focused on either small data sets, a narrow range of applications, or a single software stack. Suites such as PARSEC, HPCC, and SPECCPU emphasize computational kernels but fail to capture diverse data patterns and runtime stack complexities seen in Internet-scale systems. CloudSuite and DCBench are scale-out cloud/OLTP oriented but lack large-scale, realistic input data and omit crucial application domains (e.g., graph analytics, mixed OLTP/OLAP, and emerging AI workloads). TPC benchmarks center on SQL workloads and neglect semi-structured/unstructured data.

BigDataBench’s design goals are as follows:

• Comprehensiveness and representativeness: Inclusion of all four "V"s of big data—Volume, Variety, Velocity, and Veracity—across application domains such as search engines, social networks, and e-commerce.
• Software-stack coverage: Implement workloads atop major distributed frameworks and engines (Hadoop, Spark, MPI, Hive, HBase, MySQL, TensorFlow, etc.), revealing the holistic impact of middleware on performance and microarchitecture.
• Scalable input generation: Synthesize inputs at scales from GB to PB using both real-world seeds (Wikipedia, Amazon, Facebook, etc.) and sophisticated generators (BDGS) that preserve key statistical and structural properties.
• Architecture-conscious metrics: Capture both user-visible metrics (throughput, response time, data processed/sec) and full micro-architectural breakdown (instruction mix, cache/TLB/branch behavior, pipeline stalls, parallelism) required for systems and simulation research (Wang et al., 2014, Jia et al., 2014, Wang et al., 2015, Quan et al., 2013).

2. Architecture and Workload Abstractions

The BigDataBench suite is architected in a modular, extensible fashion to support continuous domain extension, workload permutation, and benchmarking reproducibility. Its core methodology evolved through the following stages:

Workload Incrementalism and Realism

The initial releases focused on web search engines—a domain with massive user concurrency, diverse query streams, and complex data pipelines—but quickly generalized to cover social networks, e-commerce, and other emerging domains (Gao et al., 2013). Each application domain is mapped to workloads reflecting its data structures and query/analysis patterns.

Operation and Pattern Abstractions (BigOP)

The BigOP abstraction, tightly integrated within BigDataBench, formalizes workloads as compositions of operations (CRUD, relational, graph, statistical, AI) and canonical workload patterns (single-operation, multi-operation, iterative), enabling systematic test prescription and automatic workload generation (Zhu et al., 2014). Each test is specified by a 5-tuple:

$$T = (\mathcal{O}_T, \mathcal{P}_T, D, \mathcal{W}, M)$$

where $\mathcal{O}_T$ is the set of operations, $\mathcal{P}_T$ is the set of patterns, $D$ is input data, $\mathcal{W}$ encodes workload parameters, and $M$ is the set of metrics to be collected.

Data Motifs ("Dwarfs") and Motif Pipelines

Subsequent methodological advances led to the identification and adoption of eight fundamental "data motifs" (a term synonymous with "dwarfs" in some publications): Matrix, Sampling, Transform, Graph, Set, Logic, Sort, and Statistic. Each complex application is constructed as a Directed Acyclic Graph (DAG) or pipeline of these motifs, parameterized by data types, scales, and interconnection patterns (Gao et al., 2018, Gao et al., 2018, Gao et al., 2017).

Motif-centric design enables:

• Motif-specific micro-benchmarking on multiple stacks (e.g., Hadoop, Spark, TensorFlow, Pthreads).
• Systematic composition of component and end-to-end benchmarks reflecting realistic workload mixes.
• Proxy-benchmarking via motif-DAGs that accurately emulate the system- and micro-architecture behavior of full workloads but with 100–1000× simulation speedup (Gao et al., 2018, Gao et al., 2017).

3. Workload Suite Composition and Data Generators

BigDataBench comprises tens of workloads, partitioned hierarchically:

• Micro-benchmarks: Isolated motif implementations (e.g., Sort, FFT, MatrixMultiply). Typical choices include Sort, Grep, WordCount, BFS, K-means, PageRank, Naive Bayes, SVM, Index, and CRUD primitives.
• Component benchmarks: Small pipelines of motifs (e.g., PageRank = Graph + Statistic + Sort; LDA = Sampling + Matrix + Statistic).
• Application/end-to-end benchmarks: Full-stack workflows such as Web search engines, e-commerce transaction pipelines, social graph analytics, and deep learning models (e.g., AlexNet, Inception-V3, GANs).

Data Generation and Scaling

Input data is produced via a combination of real-world seeds and scalable synthetic generation. The BDGS (Big Data Generator Suite) module calibrates statistical and structural models (e.g., multinomial word distributions, power-law graph degree, table column correlations, term-locality histograms) to generate petabyte-scale data while preserving veracity (Gao et al., 2013, Wang et al., 2014). For search workloads, semantic and temporal locality is enforced by modeling term frequencies (often Zipfian) and query reuse distances.

Supported data types encompass:

• Unstructured text (Wikipedia, reviews)
• Graphs (web, social)
• Structured/semi-structured tables (e-commerce transactions, resumes)
• Multimedia (images for AI models)

4. Characterization, Metrics, and Evaluation Methodology

BigDataBench supports detailed multi-level characterization via hardware performance counters, system logs, and benchmarking harnesses:

• System metrics: Requests per second (RPS), operations per second (OPS), Data Processed per Second (DPS), Data Processed per Joule (DPJ), throughput, latency, resource utilization, I/O bandwidth.
• Micro-architecture: Million Instructions per Second (MIPS), Instruction Per Cycle (IPC), Memory-Level Parallelism (MLP), operation intensity (OPINT), cache MPKI, TLB MPKI, branch misprediction ratios, pipeline slot and stall breakdown (Top-Down analysis).
• Formal definitions:
  • $$\text{DPS} = \dfrac{\text{Data Input Size (bytes)}}{\text{Run Time (s)}}$$
  • $$\text{OPINT} = \dfrac{\text{Total Operations}}{\text{Total Data Moved (bytes)}}$$
  • L1I-MPKI and L3 Hit Rate are defined as in (Wang et al., 2014).

Key findings include:

• Big data workloads are dominated by data movement (≈92% of instructions), with high branch and low floating-point intensity (Wang et al., 2015).
• Instruction footprints are large: Hadoop-based workloads typically require ≈1 MB L1I for negligible miss rate.
• L1I cache miss rates are significantly higher than traditional benchmarks (≈15–23 MPKI vs 1–5), with L3 caches providing effective working set absorption (Wang et al., 2014, Wang et al., 2015).
• Backend stalls (especially DRAM latency) dominate pipeline inefficiency in large-scale data/AI applications, while software stack complexity (deep JVM stacks in Hadoop/Spark) can impose high front-end penalties (Gao et al., 2018, Wang et al., 2015).
• Motif-proxy benchmarks, constructed as DAGs of motif components with ML-guided parameter tuning, can emulate the architectural and system characteristics of full workloads to within 90%+ accuracy, at orders-of-magnitude lower simulation cost (Gao et al., 2018, Gao et al., 2017).
• Software-stack effects often overshadow algorithmic differences—i.e., Hadoop and Spark implementations of the same analytic differ more architecturally than two different algorithms run on the same stack (Jia et al., 2014, Wang et al., 2015).

5. Multi-Tenancy, Mixed Workload, and Trace-Driven Evaluation

BigDataBench-MT addresses the challenge of mixed workload replay for realistic data center evaluation, supporting the simultaneous replay of long-running online services (e.g., search queries) and batch data analytics (e.g., Hadoop jobs) using real production traces (Han et al., 2015). Its approach is to:

• Map service traces (Sogou search queries) and analytic job traces (Google cluster) onto actual, parameterized workloads via regression and clustering.
• Preserve temporal arrival patterns and inter-tenant concurrency by scaling the number of end-user and job-submitter processes while maintaining trace-derived arrival curves:

$\lambda'(t) = s \cdot \lambda(t)$

• Enable visual selection and configuration of target machines, workloads, trace intervals, and scale factors via web-based UI.
• Facilitate rigorous evaluation scenarios for cloud/datacenter heterogeneity, scalability, and workload management research.

BigDataBench-MT sets a methodological standard for multi-tenant, trace-driven benchmarking that can be flexibly extended to further workload families and tenant policies (Han et al., 2015).

6. Methodological Extensions: Proxy Generation and Subsetting

To address simulation tractability at micro-architectural level, the BigDataBench team developed systematic reduction and proxy synthesis methodologies:

• Workload reduction: Principal Component Analysis (PCA) and K-means clustering are used on >40 architectural metrics to identify minimal sets of representative workloads. The simulation suite distills the full benchmark set into boundary workloads that span the observed behavioral space, maximizing diversity and reducing runtime (Jia et al., 2014, Wang et al., 2015).
• Motif-based proxy generation: Lightweight proxies—DAGs of motif kernels—are synthesized using machine learning to match full workload metric vectors (system + micro-architecture), enabling rapid architecture exploration while preserving ≥90% fidelity across metrics (IPC, branch/caches, bandwidth, instruction mix), and accurately predicting cross-platform trends (Gao et al., 2017, Gao et al., 2018).

7. Impact, Best Practices, and Future Directions

BigDataBench has established a generalizable, extensible infrastructure for rigorous, scalable benchmarking of big data and AI workloads. Its motif methodology anticipates future co-design directions, as domain-specific accelerators can target motifs rather than monolithic applications, and kernels can be easily extended to new computational paradigms (e.g., transformer attention, GNNs). Best practices for using the suite include:

• Scaling input sizes ≥10× the working set to reveal cache/memory transition points.
• Selecting workload mixes that include all major data types, application domains, and middleware stacks for architectural relevance.
• Exploiting proxy mode for rapid processor/interconnect exploration and motif-specific accelerator benchmarking.
• Profiling and dimensioning future hardware according to motif kernel requirements, not isolated applications (Wang et al., 2014, Gao et al., 2018, Gao et al., 2018, Gao et al., 2018, Gao et al., 2017).

As new data and AI paradigms emerge, BigDataBench’s data-motif abstraction provides a stable, interpretable, and evolvable benchmark taxonomy, ensuring that benchmarking keeps pace with the exponential evolution of workload diversity in the data-centric era.