PBench: Synthetic Cloud Workload Benchmark

• PBench is a benchmarking framework that synthesizes cloud workloads by statistically replicating real production traces.
• It employs multi-objective integer linear programming to select and schedule queries, achieving significantly lower error rates compared to traditional benchmarks.
• By integrating simulated annealing for precise timestamp adjustment and LLM-driven query augmentation, PBench adapts to dynamic workload patterns and bridges catalog gaps.

PBench is a term associated with multiple technical benchmarks and evaluation frameworks across cloud analytics, embodied AI, perception, and data systems. This article focuses on PBench as introduced in the context of cloud analytics benchmarking—specifically, as a workload synthesizer for constructing synthetic workloads that statistically mirror real cloud production traces (Zhou et al., 19 Jun 2025). Connections to PBench in other AI and robotics contexts are outlined for clarity, but statistical workload synthesis remains the primary focus.

1. Problem Context: Workload Synthesis with Real Statistics

PBench directly addresses the gap between real-world cloud analytics workloads—where only execution statistics (e.g., CPU usage, operator counts) are typically observed—and the limitations of traditional fixed-pattern benchmarks (e.g., TPC-H, TPC-DS) that fail to represent production workload dynamics. The distinguishing objective is to generate a proxy workload $\widetilde W$ by combining queries and databases from public benchmarks such that, when run on a comparable system, its observed aggregate statistics closely approximate those of a real trace $W$ despite the unavailability of the original SQL and data (Zhou et al., 19 Jun 2025).

Formally, for a real workload $W = \{(t_j, q_j)\}_{j=1}^n$ (hidden queries), time windows $\Gamma^{(i)}$, and feature sets $F^{(i)} = \langle M^{(i)}, O^{(i)} \rangle$—where $M^{(i)}$ are performance metrics and $O^{(i)}$ operator distributions—the goal is to select, schedule, and, if necessary, generate synthetic queries such that the synthetic sequence's windowed statistics match $F^{(i)}$ for all windows.

2. Component Selection via Multi-Objective Integer Linear Programming

PBench frames component selection as an integer linear program (ILP) aiming to minimize the aggregate error between actual and synthesized statistics over each window (Zhou et al., 19 Jun 2025).

Given a catalog $\mathbb{C} = \{C_j = (q_j', D_j')\}_{j=1}^v$ of profiled queries and their execution features $F'_j$, the ILP for each window $W$0 seeks nonnegative integer counts $W$1:

$W$2

subject to diversity, cardinality, and time-budget constraints: $W$3 where $W$4 and $W$5.

The problem is solved using standard branch-and-cut ILP solvers, with balancing of both metric and operator distribution errors in a single unified objective.

3. Temporal Dynamics: Timestamp Assignment and Simulated Annealing

PBench further refines statistical fidelity by assigning detailed timestamps to individual synthetic query instances, better capturing burstiness and temporal localities of the real workload (Zhou et al., 19 Jun 2025). Within each window $W$6, the ILP-proposed runs are subdivided into smaller intervals (e.g., 30 seconds). A simulated annealing scheme re-shuffles query timings to minimize:

$W$7

where the tilde refer to synthesized aggregates in each subinterval. This process sharpens alignment with short-term resource usage and event spikes, which is essential for accurate emulation of dynamic system loads.

4. Catalog Augmentation with LLMs

A key challenge is that the space of available components may not cover certain observed workload distributions. To overcome convex hull gaps, PBench implements a component augmentation loop leveraging LLMs (Zhou et al., 19 Jun 2025). The process includes:

• Clustering windows with poor fits into $W$8 centroids;
• Constructing positive and negative examples from cataloged queries close to or far from these centroids;
• Prompting an LLM (using database schema, desired metrics, query examples) to synthesize new SQL queries targeting the desired profile;
• Profiling and iterative feedback (e.g., “increase scale-factor,” “add join,” “adjust selectivity”) until the synthetic query $W$9 matches requested metrics.

Typically, a handful of LLM calls and minor schema modifications suffice to bridge major statistical gaps. The augmented catalog is then re-injected into the ILP and scheduling pipeline for final workload construction.

5. Evaluation Methodology and Performance

PBench is empirically validated on production traces from large-scale cloud database vendors:

• Snowset (Snowflake): 69 million queries; five 1-hour slices with varying load shapes.
• Redset (Redshift): Multiple 1-hour and 24-hour slices.

Error is measured using joint metrics (MAE, geometric mean absolute percentage error, and geometric mean quotient error) over all windows and intervals. PBench consistently achieves up to 6$W = \{(t_j, q_j)\}_{j=1}^n$0 lower approximation error versus prior state-of-the-art techniques such as Stitcher and CAB, and yields near-zero operator MAEs. Fine-grained timestamp assignment recovers burstiness (e.g., CPU GMAPE 36% for PBench vs 154–163% for baselines at 30s interval), while catalog augmentation ensures statistical coverage even in regime extremes where benchmarks alone underfit (Zhou et al., 19 Jun 2025).

Trace / Metric	PBench GMAPE	Stitcher GMAPE	CAB GMAPE
Snowset CPU	17.4%	110%	25%
Snowset Scanned	12.4%	43.6%	45.4%

(*) Further detailed operator MAEs are consistently near zero for PBench, exceeding fivefold improvement over baselines.

6. Key Insights, Limitations, and Prospective Directions

PBench demonstrates that joint optimization of performance metrics and operator counts is essential for realistic workload emulation. Timestamp-level alignment and LLM-driven query augmentation are crucial for bridging catalog coverage gaps and matching nontrivial resource usage patterns.

Limitations noted in the paper include reliance on simplified concurrency/resource models, sensitivity to LLM prompt engineering, and assumptions of cluster homogeneity. Prospective improvements entail incorporating richer operator latency distributions, adaptive concurrency models, automated database design via LLMs, and extensions to mixed transactional-analytic (“HTAP”) workloads and multi-tenant benchmarking.

A plausible implication is that future cloud system benchmarking may routinely incorporate component augmentation and dynamic scheduling pipelines such as PBench to produce more statistically relevant and workload-realistic testbeds (Zhou et al., 19 Jun 2025).

7. PBench in Adjacent Domains

Several distinct frameworks use the acronym “PBench” or closely related terms:

• Physical AI Benchmark for World Models: In embodied and robotic AI, PBench is a video benchmark for evaluating physical plausibility and visual quality of generative world models. Scoring uses LLM or multi-metric judge systems covering spatial, temporal, and physical consistency (Zhang et al., 15 Jun 2026, Chen et al., 24 Mar 2026).
• Perception Benchmark (PBench): In perception and VLM research, PBench targets complex referring expressions, compositional reasoning (OCR, spatial, relational), and dense scene regimes, measuring zero-shot macro-F$W = \{(t_j, q_j)\}_{j=1}^n$1 segmentation scores (Bevli et al., 28 Mar 2026).
• Image Editing (PBench-Edit): PBench-Edit is a benchmark for physical and temporal consistency in image editing, using LLMs for multi-axis scoring of fidelity, identity, and coherence (Wu et al., 5 Oct 2025).
• Plug and Play Bench: PAPB refers to a containerized infrastructure for automating big data benchmarking with support for cost reporting and cluster adaptation (Ceesay et al., 2017).

Each PBench instantiation is domain-specific; the unifying theme is rigorous, multi-metric, high-fidelity evaluation of system behaviors along structure, dynamics, or statistical dimensions.

---

References

• (Zhou et al., 19 Jun 2025) PBench: Workload Synthesizer with Real Statistics for Cloud Analytics Benchmarking
• (Zhang et al., 15 Jun 2026) Qwen-RobotWorld Technical Report
• (Chen et al., 24 Mar 2026) ABot-PhysWorld: Interactive World Foundation Model for Robotic Manipulation with Physics Alignment
• (Bevli et al., 28 Mar 2026) Falcon Perception
• (Wu et al., 5 Oct 2025) ChronoEdit: Towards Temporal Reasoning for Image Editing and World Simulation
• (Ceesay et al., 2017) Plug and Play Bench: Simplifying Big Data Benchmarking Using Containers