Plug and Play Bench for Big Data

• Plug and Play Bench is an infrastructure-aware abstraction that automates containerized benchmarking for big data clusters.
• It integrates HiBench, HDP, and Azure to streamline workload deployment, configuration, and result collection.
• PAPB tackles challenges of tool fragmentation, deployment complexity, and cost transparency in benchmarking Hadoop and Spark workloads.

Plug and Play Bench, abbreviated PAPB, is a proof-of-concept, infrastructure aware abstraction for big data benchmarking that uses containers to simplify the deployment, configuration, execution, and result collection of established benchmarking suites on clusters of machines, particularly cloud-hosted clusters. It was introduced to address a combination of fragmentation across benchmarking tools, operational complexity in real cluster deployments, and limited transparency regarding the monetary cost of benchmarking cloud-based Hadoop and Spark workloads. In its reported implementation, PAPB integrates HiBench as the benchmark suite, HDP as the cluster deployment framework, and Azure as the cloud platform, with an explicit emphasis on making benchmarking more nearly “plug-and-play” for practitioners and researchers (Ceesay et al., 2017).

1. Motivation and problem framing

PAPB was proposed in the context of rapid growth in big data processing and storage systems such as MapReduce, Spark, and NoSQL databases, alongside the practical availability of cluster-scale computation through cloud computing. The central problem identified for benchmarking in this environment is not the absence of workloads, but the difficulty of deploying and managing them consistently at scale. Existing suites are described as fragmented, complex to deploy and manage, and insufficiently transparent with respect to cost (Ceesay et al., 2017).

The paper frames three linked obstacles. The first is fragmentation: multiple benchmarking suites, including HiBench, BigDataBench, and CloudSuite, cover overlapping but not identical subsystems such as batch analytics, SQL, streaming, and graph workloads, while also relying on different configuration conventions. The second is deployment complexity: operating such suites on real clusters requires nontrivial technical work, including compiling sources, wiring Hadoop and Spark paths, setting workload-specific parameters such as Spark executor memory, generating datasets, and aligning benchmark configuration with the actual cluster distribution and software versions. The third is cost transparency: cloud users often need to compare infrastructure choices, billing meters, and VM families, yet lack an integrated way to connect benchmark executions to workload-level or suite-level cost (Ceesay et al., 2017).

The reported motivation is grounded in hands-on experience deploying HiBench and BigDataBench on an 8-node Microsoft Azure cluster, where manual setup was itself a barrier. PAPB therefore targets reduction of setup effort and reduction of configuration mismatch, rather than the invention of a new workload suite. Its role is additive: it wraps and automates existing tools rather than replacing them (Ceesay et al., 2017).

2. Infrastructure-aware abstraction and containerized design

PAPB’s defining concept is an “infrastructure-aware abstraction.” In the paper’s usage, this means a layer that interrogates the underlying cluster to discover configuration details and injects them into benchmark scripts and configuration files without requiring manual transcription by the operator. The abstraction is implemented through a Docker-based container image that packages benchmarking software and automation logic, allowing the benchmark client environment to be isolated from host-level drift while remaining network-reachable to Hadoop services and HDFS (Ceesay et al., 2017).

The architecture described in the paper implies a small set of coordinated components:

Component	Function	Inputs or outputs
Container image	Bundles Linux base, Java, HiBench, client-side Hadoop/Spark tooling, and automation scripts	Reaches cluster services and HDFS
Discovery module	Discovers cluster configuration	Environment variables, configuration files, HDP conventions
Config generator	Produces HiBench-specific configuration	Writes hibench.conf and workload configs
Orchestration and execution	Runs data generation, workloads, and metrics collection	Structured results directory
Metrics and cost module	Aggregates benchmark outputs and maps runtime to cloud cost	Runtime, throughput, pricing inputs

The discovery module is responsible for identifying installation and runtime details from the cluster. According to the description, it locates HADOOP_HOME and SPARK_HOME from environment variables or standard installation paths, parses files such as core-site.xml, yarn-site.xml, mapred-site.xml, and spark-defaults.conf, and can use HDP-specific management information such as Ambari or direct filesystem introspection to infer versions and service endpoints (Ceesay et al., 2017).

The config generator converts these discovery outputs into benchmark-ready settings. The reported generated configuration includes hadoop.home, spark.home, hibench.scale.profile, dataset generation parameters, Spark executor memory, cores, instances, HDFS input/output paths, and replication settings aligned with the target cluster (Ceesay et al., 2017). This makes PAPB “infrastructure-aware” in a concrete sense: configuration is derived from the runtime environment rather than being handwritten for each deployment.

Containerization is central to this design. The paper describes PAPB as using containers to encapsulate dependencies and remove host-level drift. Its practical image composition is described as a Linux base such as Ubuntu or CentOS, OpenJDK, Hadoop and Spark client libraries matching the cluster, HiBench sources, shell or Python tooling, and a lightweight metrics collector. Orchestration in the proof of concept is script-driven rather than dependent on a heavyweight orchestrator, which the paper presents as a portability advantage (Ceesay et al., 2017).

3. Benchmark integration with HiBench, HDP, and Azure

The proof-of-concept deployment of PAPB centers on HiBench. HiBench is treated as the workload suite that PAPB automates, while HDP provides the cluster deployment framework whose installed paths, versions, and service conventions PAPB discovers and adapts to. Azure is the cloud platform on which the implementation experience is reported (Ceesay et al., 2017).

The workload coverage attributed to HiBench in this setting includes micro benchmarks such as WordCount, Sort, and TeraSort; machine learning workloads such as KMeans, Logistic Regression, and Bayes; SQL or warehouse workloads such as Hive and Spark SQL; graph workloads such as PageRank; and streaming workloads where supported by HiBench. PAPB does not redefine these workloads. Instead, it automates their data generation, parameter selection, and execution in cluster-compatible modes such as Spark-on-YARN (Ceesay et al., 2017).

A key element of integration is configuration synthesis. PAPB populates HiBench using discovered cluster paths such as HADOOP_HOME, HADOOP_CONF_DIR, SPARK_HOME, SPARK_CONF_DIR, and HIBENCH_HOME. It selects hibench.scale.profile to match cluster size and derives Spark executor settings from aggregate CPU and memory capacity and YARN defaults. The text gives an illustrative rule in which, with $N$ worker nodes, each with $C$ cores and $M$ GiB memory, the framework may set executors per node as $E_n = \lfloor C / c_e \rfloor$, with $c_e$ representing cores per executor, and choose executor memory $m_e$ so that YARN container limits are respected (Ceesay et al., 2017).

The end-to-end automation pipeline is described as four stages:

1. install: pull the PAPB container image and ensure host connectivity to the HDP cluster;
2. configure: discover cluster settings and generate HiBench configuration;
3. run: execute selected workloads, generate datasets into HDFS, and submit Hadoop MR or Spark-on-YARN jobs;
4. collect: parse benchmark outputs such as throughput, duration, and data size, archive logs, and compute cost projections (Ceesay et al., 2017).

In Azure deployments, PAPB is described as interacting with the cloud platform indirectly through the cluster rather than through direct cloud API control. It runs on a VM that can reach Ambari, the NameNode, and the ResourceManager over the virtual network, while using HDFS for dataset storage. This means PAPB itself does not require Azure API credentials if the cluster has already been provisioned. Instead, it maps observed runtime to public price points offline (Ceesay et al., 2017).

4. Cost transparency and benchmark economics

A distinctive feature of PAPB, relative to many benchmark wrappers, is its explicit inclusion of cost as a first-class output. The paper emphasizes that users often lack visibility into the monetary cost of benchmark runs on cloud infrastructure, especially when comparing VM families, pricing models, and deployment choices. PAPB’s cost module is intended to bridge benchmark execution metrics and cloud billing abstractions (Ceesay et al., 2017).

The provided synthesis gives a cost model consistent with this objective. For a suite of $N$ runs, total cost is described as

$$C_{total} = \sum_{i=1}^{N} \Big( T_i \cdot \sum_{j=1}^{M} n_{j} \cdot P_{vm,j} \Big) + C_{storage} + C_{net} + C_{aux},$$

where $T_i$ is the wall-clock runtime of benchmark $i$, $C$0 is the number of distinct VM SKUs in the cluster, $C$1 is the count of VMs of SKU $C$2, and $C$3 is the per-hour price for SKU $C$4. The remaining terms cover storage, networking, and auxiliary metered services (Ceesay et al., 2017).

A per-run version is also given:

$C$5

with $C$6 denoting generated data volume and $C$7 denoting outbound traffic volume (Ceesay et al., 2017).

The paper also supplies normalized comparative quantities, including

$C$8

and

$C$9

These expressions are not presented as audited benchmark standards, but as a practical operationalization of PAPB’s cost-visibility objective (Ceesay et al., 2017).

In this design, PAPB provides measured runtime and data-generation information, while the operator supplies Azure pricing parameters such as per-VM-hour compute, storage, and egress charges. This yields per-workload and per-suite cost summaries. A plausible implication is that PAPB treats benchmarking as an economic activity as well as a performance activity, which distinguishes it from suites focused exclusively on throughput or completion time.

5. Deployment workflow and operational usage

PAPB is presented as a tool intended to reduce the number of manual steps required to benchmark a fresh cluster. The workflow described for an HDP cluster on Azure begins with a reachable node that has Docker installed, network access to Ambari, the NameNode, and the ResourceManager, read access to Hadoop and Spark configuration directories, and writable HDFS paths (Ceesay et al., 2017).

The operational sequence reported in the paper is:

1. Pull the image: docker pull <papb-image>
2. Launch the container with mounted Hadoop and Spark configuration directories and relevant environment variables: docker run -v /etc/hadoop/conf:/etc/hadoop/conf -v /etc/spark/conf:/etc/spark/conf -e HADOOP_HOME=/usr/hdp/current/hadoop-client -e SPARK_HOME=/usr/hdp/current/spark-client <papb-image>
3. Allow the container to discover and configure the environment.
4. Select workloads or workload profiles such as micro, ML, SQL, or graph.
5. Execute HiBench generators and benchmark scripts targeting YARN where applicable.
6. Persist results under HiBench reporting directories and optionally compute cloud cost (Ceesay et al., 2017).

The discovery logic is described algorithmically rather than as a formal pseudocode listing. It includes reading HADOOP_CONF_DIR, parsing core-site.xml for fs.defaultFS, reading YARN ResourceManager information from yarn-site.xml, setting Spark master mode to YARN, inferring Spark version from spark-submit --version, and estimating executor sizing from cluster-wide CPU and memory information, whether retrieved from Ambari or system introspection (Ceesay et al., 2017).

The paper emphasizes that PAPB is portable in the limited sense that its script-driven orchestration avoids dependence on a heavyweight scheduler or orchestrator. At the same time, its proof-of-concept implementation is explicitly aligned to HDP layout conventions, so practical portability to other distributions such as CDH, Apache Bigtop, EMR, or Dataproc would require alternative discovery adapters rather than being immediate (Ceesay et al., 2017).

6. Evaluation, trade-offs, and position within benchmarking research

PAPB is positioned as a complement to benchmark suites rather than a new suite. HiBench already provides workloads and scripts; PAPB adds auto-configuration, containerized packaging, orchestration, and cost visibility. BigDataBench and CloudSuite are treated as broader workload suites whose existence reinforces the paper’s fragmentation argument. TPCx-BB is identified as a more formal benchmark focused on standardized and audited results, whereas PAPB is aimed at deployment simplicity for practitioners and researchers (Ceesay et al., 2017).

The paper does not report a full quantitative experimental matrix for PAPB itself. It cites an 8-node Azure deployment as the context in which the manual setup burden was encountered, but it does not provide detailed tables of VM SKUs, software versions, dataset sizes, or controlled measurements of container overhead, repeatability, or cost savings. Likewise, it does not present explicit performance or cost numbers for PAPB relative to manual deployment (Ceesay et al., 2017).

This absence of quantitative validation shapes the interpretation of PAPB. Its principal contribution is architectural and operational rather than numerical: an infrastructure-aware, containerized automation layer over existing benchmark suites. The text explicitly notes several limitations: distribution specificity to HDP in the proof of concept, missing quantitative validation, security and isolation considerations when mounting configuration directories and accessing HDFS, and a single-container-driver orientation that may constrain scalability for larger deployments (Ceesay et al., 2017).

The same section suggests several best practices that emerge from the design. These include matching Spark executor memory and cores to YARN container limits, reserving headroom for operating-system and daemon processes, respecting HDFS replication defaults during data generation, pinning Java and client libraries to cluster versions, caching benchmark build artifacts, and placing generated data in HDFS layouts that favor block distribution while avoiding excessive small files (Ceesay et al., 2017). These recommendations are practical consequences of the automation logic rather than formal benchmark rules.

In research terms, PAPB occupies a specific place in the development of benchmarking infrastructure: it addresses the orchestration layer around big data benchmarks rather than benchmark semantics themselves. This suggests that its lasting significance lies less in any individual workload and more in the proposition that benchmark deployment should be treated as a systems-integration problem. Later work on modular, isolated benchmarking frameworks for other domains, such as black-box optimization, reflects a related concern with dependency isolation and reproducibility, although PAPB itself is focused on big data cluster benchmarking and cloud cost visibility rather than on a generalized RPC-based benchmark abstraction (Papenmeier et al., 27 May 2025).

7. Legacy, extensibility, and broader significance

The code for PAPB is reported as available at https://github.com/sneceesay77/papb, with repository-level details such as licensing deferred to that source (Ceesay et al., 2017). The paper describes extensibility in terms of adding new workload adapters around existing suites, adding discovery modules for other platforms such as EMR, Dataproc, or Kubernetes-based systems, and integrating additional cost meters such as storage I/O, intra-region traffic, or managed-service fees (Ceesay et al., 2017).

The broader significance of PAPB lies in the way it redefines the barrier to benchmarking. Instead of assuming that benchmarking difficulty resides primarily in workload design, PAPB identifies environment discovery, version alignment, parameter synthesis, and cost accounting as core parts of the benchmarking problem. A plausible implication is that, in operational settings, benchmark reproducibility may depend as much on deployment automation as on the workload definitions themselves.

Within that frame, PAPB can be understood as an early container-centered attempt to make big data benchmarking portable, repeatable, and cost-aware. Its proof-of-concept scope remains narrow—HiBench, HDP, Azure—but its central thesis is broader: when benchmark suites are fragmented and cluster environments are heterogeneous, an infrastructure-aware automation layer can be as important as the benchmark workloads it orchestrates (Ceesay et al., 2017).