Cloud WorkBench: IaC for Cloud Benchmarking

Updated 12 July 2026

Cloud WorkBench is a web-based framework that automates and manages IaaS benchmarks using reproducible Infrastructure-as-Code techniques.
It integrates Ruby on Rails, Vagrant, and Chef to simplify benchmark deployment, scheduling, and execution across diverse cloud environments.
The framework facilitates comparative analysis of cloud configurations through automated provisioning, robust state management, and precise performance metrics.

Searching arXiv for papers on Cloud WorkBench and directly related benchmarking ecosystems. Cloud WorkBench (CWB) is a web-based framework for automating, managing, and reusing Infrastructure-as-a-Service (IaaS) cloud benchmarks. It was proposed to address a practical problem in cloud deployment studies: users must evaluate the costs and performance of different combinations of cloud configurations to identify the best service level for a specific application, yet benchmarking cloud services is cumbersome and error-prone. CWB distinguishes itself by grounding benchmark definition and execution in Infrastructure-as-Code (IaC), using reproducible, well-defined, and testable infrastructure descriptions rather than ad hoc experimental setup (Scheuner et al., 2014).

1. Problem setting and design objectives

The motivating use case for CWB is comparative evaluation of cloud configurations under realistic deployment conditions. In the IaaS setting, performance depends not only on application code, but also on VM type, region, storage configuration, operating system image, and provisioning logic. CWB was introduced to make such evaluations less labor-intensive and less error-prone by turning benchmark execution into a managed Web service with reusable benchmark specifications (Scheuner et al., 2014).

A central design objective is the definition of reusable and representative benchmarks. In CWB, a benchmark is not treated as an isolated script, but as a compound artifact that includes infrastructure specification, provisioning logic, execution callbacks, result models, and execution control. This framing places benchmarking closer to deployment engineering than to one-off measurement. A plausible implication is that CWB was intended to reduce the gap between experimental benchmarking and operational cloud configuration management.

2. Architectural organization

CWB consists of a set of components arranged around a standard three-tier Ruby on Rails server application. The CWB server provides the web interface, business logic, and a relational database for persisting benchmark definitions and results, and it includes a scheduler, for example via Cron, to periodically execute benchmarks. The experimenter interacts primarily with the web interface to define, manage, and schedule benchmarks (Scheuner et al., 2014).

Execution depends on two further subsystems. First, a provisioning service built on Opscode Chef configures and prepares VMs for benchmark execution by means of reusable, parameterized cookbooks. Second, the server interacts with cloud providers through provider-agnostic APIs abstracted via Vagrant, which handles VM acquisition and release, storage association, and related lifecycle operations. The benchmark itself runs on the System Under Test (SUT), namely the cloud VMs on which the workload executes; these VMs are equipped with the CWB client library for status communication and result reporting (Scheuner et al., 2014).

Communication is intentionally heterogeneous. RESTful APIs connect most components to achieve loose coupling and reusability, while SSH and rsync are used for remote command execution and file transfer. This combination reflects the dual character of the system: a Web application for experiment orchestration and a deployment automation stack for benchmark realization.

3. Infrastructure-as-Code benchmark model

The defining methodological feature of CWB is its explicit use of Infrastructure-as-Code. Vagrant encapsulates VM configurations in a Ruby DSL, including provider, region, instance size, AMI, and EBS configuration, while Opscode Chef expresses provisioning steps as modular cookbooks written in Ruby. In CWB, these tools are not auxiliary conveniences; they are the substrate through which benchmarks become reproducible, parameterized, shareable, and testable (Scheuner et al., 2014).

CWB represents a benchmark as a combination of several specification layers. VM definitions describe one or multiple client VMs, including instance types, regions, operating-system images, and network or storage settings. Provisioning configurations assign roles and install required software through Chef recipes and cookbooks. Result models specify what metrics are collected and how they are reported, using nominal, ordinal, interval, or ratio scales. Each benchmark also defines callbacks, including a start callback for execution, and may be associated with scheduling and timeout parameters (Scheuner et al., 2014).

The paper’s illustrative snippets show this structure concretely. A Chef cookbook can install the FIO benchmark through the package manager, while a Vagrantfile binds a specific AWS region, AMI, SSH username, and instance type to a provisioning recipe and parameter block. This illustrates the paper’s emphasis that resource specifications and software provisioning are conflated in code for full reproducibility and reuse. In that sense, CWB treats the benchmark environment itself as a versionable artifact.

4. Execution lifecycle and state management

Benchmark execution in CWB follows a managed workflow. Execution can be triggered manually or by a schedule through the web interface. The server then provisions the specified VMs through Vagrant, after which the provisioning service applies Chef cookbooks to configure the VMs, install required software, and deploy both benchmark code and the CWB client library. The start callback is invoked on the VM via SSH, and the CWB client reports status transitions such as running back to the server (Scheuner et al., 2014).

After the benchmark runs, results are extracted, for example as CSV, and submitted to the CWB server. If postprocessing is required, corresponding scripts are executed before the server deprovisions resources after success or timeout. The workflow therefore spans acquisition, configuration, execution, reporting, postprocessing, and cleanup in a single orchestration path (Scheuner et al., 2014).

CWB also defines an explicit state model. States include Created, Preparing, Running, Finished, and Failed, with transient waiting and postprocessing states interleaved. The paper emphasizes robust handling of failures and resource leaks. This is significant because cloud benchmarking frequently fails for reasons external to the benchmarked application itself, such as provisioning errors, VM instability, or unreleased resources. By making states first-class entities, CWB turns failure handling into part of the benchmarking model rather than an afterthought.

5. Demonstration case: Amazon EC2 disk I/O benchmarking

To demonstrate the framework, the authors evaluated sequential disk write throughput on Amazon EC2. The case study used three EC2 instance types—t1.micro, m1.small, and m3.medium—and two EBS storage types, Standard/Magnetic and General-Purpose SSD (GP2), in region eu-west-1 (Ireland), on Ubuntu 14.04. FIO was used with a block size of 4 KiB and workloads of 1 GiB and 4 GiB. Bandwidth was logged every 500 ms together with hardware information, and each experiment ran 8–12 times per configuration (Scheuner et al., 2014).

The study was designed as a reusable benchmark composition within CWB. Multiple Vagrantfiles encoded the combinations of instance type and storage type, while a reusable Chef cookbook installed FIO and generated the scripts needed to execute, monitor, and report results. The paper further notes that the automation of provisioning, execution, and cleanup supports broad exploratory studies, and that in the paper, 20,000 executions were run (Scheuner et al., 2014).

The reported findings concern throughput, cost/performance tradeoffs, and variability. Upgrading from t1.micro to m1.small yields a~4x increase in write throughput for both EBS types. For t1.micro, upgrading instance yields more throughput than upgrading storage type; for m1.small, upgrading storage to GP2 SSD gives better returns than upgrading instance. Standard EBS shows higher variability than GP2 SSD across all instance types, and t1.micro instances show high intra-execution variability due to CPU scheduling and network limitations, whereas the other instance types are more stable (Scheuner et al., 2014).

Storage type	Instance type	Standard deviation as percentage of mean; intra-execution variability
Standard EBS	`t1.micro`	20% (20–50%)
Standard EBS	`m1.small`	20% (10–20%)
Standard EBS	`m3.medium`	30% (15–60%)
GP2 SSD EBS	`t1.micro`	10% (20–40%)
GP2 SSD EBS	`m1.small`	10% (5–15%)
GP2 SSD EBS	`m3.medium`	10% (5–10%)

These results illustrate the kind of measurement program CWB was built to support: parameterized, repeated, and execution-aware benchmarking in which variability is itself an object of analysis rather than incidental noise.

Within cloud benchmarking, CWB’s main contribution is methodological rather than benchmark-specific. Its key features are IaC-based, extensible, and modular benchmarking; web-based management and scheduling; support for real-world application simulation; reproducibility and portability; and automation of provisioning, execution, and cleanup (Scheuner et al., 2014). Rather than prescribing a single workload family, it supplies an execution and management substrate in which benchmarks can be encoded as reusable infrastructure-plus-software definitions.

A related line of work later presented an online ecosystem for continuous evaluation of cloud infrastructure for scientific applications. That ecosystem is organized around ExaBench, a centrally accessible results database, and an online visualization and results page, and it supports application benchmarks including HPL, VASP, and GROMACS. It emphasizes continuous benchmarking, collaborative contribution of results and code, automated upload to a central online database, and dynamic comparison across vendors, instance types, and sites (Mohammadi et al., 2018). The comparison is instructive: CWB centers on IaC-based benchmark definition and execution through Ruby on Rails, Vagrant, and Chef, whereas the later ecosystem centers on collaborative result curation and online comparison for scientific workflows. This suggests a broader trend from benchmark automation toward continuously updated, community-facing benchmarking infrastructures.

Cloud WorkBench should also be distinguished from Azure Blockchain Workbench, which is a different system concerned with blockchain deployment and key management using Azure Key Vault and HSM-backed workflows rather than IaaS cloud benchmarking methodology (Tanana, 2023). The similarity in names can cause confusion, but the systems address different technical domains, expose different threat models, and are evaluated against different criteria.