DataBench: Scalable Benchmarking Framework

• DataBench is a family of benchmarking methodologies that rigorously evaluates big data, AI, IoT, and audit systems through motif-based workload abstraction.
• It uses a DAG composition of core computational motifs to automatically generate proxies, enabling platform-independent, repeatable, and scalable system performance measurements.
• Variants like BigDataBench, IoTDataBench, and DATABench provide domain-specific tests that enhance reproducibility and allow fair cross-system comparisons.

DataBench is a family of benchmarking methodologies and datasets that provide rigorous, multi-domain evaluation for big data systems, AI pipelines, data center scheduling, question answering over structured data, IoT time-series databases, and adversarial auditing. DataBench frameworks are designed to capture essential computational motifs or workload primitives, scale across hardware stacks and data modalities, and provide precise, repeatable measurement of system, architecture, and end-to-end task performance. Current incarnations span motific benchmarking (BigDataBench), scalable kernel-based proxies, large-scale tabular QA evaluation (SemEval DataBench), IoT-focused time-series benchmarks, fog-computing scenario testing, and adversarial audit attack suites. DataBench’s core approach emphasizes extensibility, uniform abstraction of diverse workloads, and automated, metric-driven proxy synthesis; this enables fair comparison, rapid simulation, and robust performance analysis across software and hardware environments.

1. Core Principles and Benchmarking Philosophy

DataBench frameworks adopt a motif- or operation-based abstraction of real-world workloads, enabling scalable and representative evaluation without the exponential proliferation of application-specific proxies. Each pipeline—in analytics, service, AI, data warehouse, graph, or streaming—is represented as a composition (DAG) of fundamental units of computation or "motifs":

• Typical motifs: matrix computation, sampling, logic, transform, graph, set, sort, statistical computation (Gao et al., 2018, Gao et al., 2017).
• Operations/patterns: get/put/delete, filter (σ), transform (τ), project (π), union (∪), difference (−), cross-product (×), sort, aggregation, iterative application, multi-operation chaining (Zhu et al., 2014).

This approach enables:

• Platform and stack independence: Motifs are implementable in Hadoop, Spark, MPI, SQL/NoSQL stores, cloud-native stacks, or edge/fog stacks (Gao et al., 2018, Wang et al., 2014, Pfandzelter et al., 2023).
• Scalable test synthesis: Composite benchmarks are constructed by specifying motif-DAGs and runtime weights, automatically generating proxies that mirror application-level microarchitectural metrics, data movement, and latency/throughput profiles (Gao et al., 2017).
• Data realism: Synthetic generators (e.g., BDGS) use learned semantic and locality properties from seed data to scale to TBs while maintaining real-world distributions (Wang et al., 2014).

2. DataBench Variants and Methodologies

Subsystem	Focus/Domain	Key Features
BigDataBench	General big data/AI	Motif-based, 8 core motifs, 7 workload types, multi-stack, real datasets
BigDataBench-MT	Data center co-location	Multi-tenant, trace-driven workload replay, regression-driven matching
Dwarf-based DataBench	Architecture simulation	DAG-compositional proxies, automated accuracy tuning, subminute runtime
SemEval DataBench	Tabular QA/LLM eval	80 real tables, 5 domains/types, 500k+ cells, PoT/SQL/coding baselines
IoTDataBench	IoT time-series DB	Compression-aware, scalability tests, updated metrics/cost models
Fog DataBench	Edge/fog analytics	Joint W/I spec, geo-distributed streams+ML+analytics, multi-tier SUT
DATABench	Dataset audit attacks	IF/EF taxonomy, adversarial robustness, open attack/auditor toolkit

Motif and DAG-based Benchmarks

BigDataBench and its successors model end-to-end workloads as motif-DAGs, with each node weighted by runtime fraction. This allows automated proxy generation with >90% microarchitectural fidelity and 100–700× runtime speedup over full-stack benchmarks, enabling rapid architecture simulation and system prototyping (Gao et al., 2018, Gao et al., 2017).

Realistic Mixed Workload Replay

BigDataBench-MT integrates actual service/analytics traces with real Hadoop workloads, using regression modeling and BIC-guided clustering to match trace jobs to real applications. It preserves true arrival patterns and supports multi-tenant scaling, enabling evaluation of system elasticity and mixed workload interaction under authentic load (Han et al., 2015).

3. DataBench for Tabular Question Answering and LLMs

DataBench in the QA domain (referenced as "SemEval DataBench" or "DataBench QA/Lite") provides a standard dataset and protocol for evaluating LLMs on factual and analytical queries over large, multi-domain tables:

• Dataset: 65–80 real English tables, five domains (business, health, social, sports, travel), up to 500k cells per table, test splits with 522 QA pairs, strong diversity in row/column count and answer types (boolean, category, numeric, lists) (Evangelatos et al., 1 Mar 2025, Xiong et al., 1 Sep 2025, Site et al., 1 Aug 2025, Tyagi et al., 11 Sep 2025).
• Evaluation metric: Exact-match accuracy (EM) over normalized gold/predicted answers.
• System paradigms:
  • LLM-to-code generation (Python/Pandas with error correction and CoT explanations) (Evangelatos et al., 1 Mar 2025, Site et al., 1 Aug 2025).
  • NL-to-SQL agentic pipelines with iterative verification (Tyagi et al., 11 Sep 2025).
  • Table abstraction (schema+examples in O(N) complexity), zooming (query-aware pruning) for context reduction (Xiong et al., 1 Sep 2025).
• Performance: Leading approaches reach 87%+ accuracy using LLM-driven code generation with error recovery. Baselines (textual chain-of-thought or SQL retrieval without correction) achieve ~26–70%. Schema abstraction and query-focused zooming yield up to 19 pp gain in large-table scenarios. Multi-stage prompt/reasoning frameworks and execution-based correction eliminate many numerical/semantic hallucinations (Evangelatos et al., 1 Mar 2025, Xiong et al., 1 Sep 2025, Tyagi et al., 11 Sep 2025).

4. IoT, Fog, and Domain-Specific DataBench Extensions

IoTDataBench and Fog DataBench extend DataBench to time-series ingestion and edge/fog distributed analytics:

• IoTDataBench: Quantifies ingestion throughput, system scale-out, compression efficiency, and composite cost (\$/IoTps), explicitly rewarding storage compression and linear scalability, and incorporating realistic queries (aggregation, downsampling) on variable payloads (Zhu et al., 2021).
• Fog DataBench: Jointly models workloads and infrastructure with multi-tier resource graphs, synthetic geo-distributed sensor/event/ML/analytics streams, and cross-layer metrics (latency, throughput, QoS-SLO, staleness). Enables system comparison under realistic deployment topologies, especially for heterogeneous edge/cloud SUTs (Pfandzelter et al., 2023).

5. Dataset Auditing and Model Provenance: DATABench

DATABench is an adversarial benchmarking suite for dataset auditing methods used in data provenance or copyright/privacy affirmation tasks (Shao et al., 8 Jul 2025):

• Auditor taxonomy: Internal-feature (IF; e.g., membership inference) vs. external-feature (EF; watermark-based).
• Threat models: Evasion (hide training on D); forgery (falsely implicate D).
• Attack and defense library: Decoupling, removal, detection, adversarial examples; 9 auditors, 22 attacks.
• Findings: Every evaluated auditing method is vulnerable to adaptive attacks (simple or hybrid), with ~90%+ false negatives under combined strategies. Identified an urgent need for cryptographically robust, utility-coupled auditing.
• Benchmark role: Standardizes attack protocols, metrics (confidence, p-value, WSR), and cross-comparison for future research in secure model auditing.

6. Implementation, Metrics, and Practical Guidance

DataBench platforms provide both open-source codebases (http://prof.ict.ac.cn/BigDataBench, https://github.com/shaoshuo-ss/DATABench) and detailed protocols for deployment, metric collection, and reproducibility:

• Metrics: User-level (throughput, latency), architectural (IPC, cache/TLB MPKI, OI, resource utilization), task accuracy (EM), system-level cost scaling.
• Experiment flows: Data synthesis, scalable job submission or trace replay, collection of cross-layer events.
• Platform coverage: Hadoop, Spark, MPI, SQL/NoSQL, RDBMS, TensorFlow/PyTorch, containerized cloud/fog, LLM execution sandboxes.

7. Significance, Limitations, and Future Directions

DataBench platforms have become standard evaluation substrates for research in big data systems, AI hardware/software co-design, automatic workload generation, LLM-based tabular QA, cloud-native/storage systems, IoT time-series management, and dataset audit adversarial robustness. By decomposing workloads into motif-level abstractions or operation-patterns, DataBench enables fast porting to emerging paradigms (streaming, deep neural nets, edge inference), provides composable proxies for architecture simulation, and supports rigorous cross-system comparison.

Limitations, as noted in auditor robustness (Shao et al., 8 Jul 2025) and domain coverage (Zhu et al., 2021, Pfandzelter et al., 2023), include: modality gaps (e.g., text/audio/generative models), constraint assumptions in adversary models, and incomplete integration with continuously evolving hardware–software stacks. Ongoing work in DataBench variants focuses on extending motif libraries, incorporating cryptographic robustness for auditing, supporting federated/distributed learning, and fully integrating real-time streaming and cloud-edge hybrid testing (Shao et al., 8 Jul 2025, Gao et al., 2018, Wang et al., 2014).