HPE-Bench: HEP Benchmark Suite

• HPE-Bench is a modular benchmarking suite designed to assess distributed computing performance in high-energy physics experiments via pluggable metric plugins and containerized workloads.
• It employs a three-tier architecture—job submission, benchmark execution, and data aggregation—to standardize and automate performance measurement.
• The framework integrates energy efficiency analysis by correlating the HS23 throughput metric with power consumption, enabling optimization of site operations.

HPE-Bench refers to the "HEP Benchmark Suite"—a modular, extensible benchmarking framework deployed throughout the Worldwide LHC Computing Grid (WLCG) to systematically measure and analyze both the performance and operational characteristics of distributed computing resources supporting Large Hadron Collider (LHC) experiments. Developed under the guidance of the HEPiX Benchmarking Working Group, HPE-Bench integrates measurement of traditional throughput via HEPScore23 alongside fabric and sustainability metrics, providing actionable insights for site administrators, experiments, and resource planners (Szczepanek et al., 2024).

1. Architectural Overview and Core Components

HPE-Bench is architecturally structured as a lightweight Python-based orchestrator with pluggable metric "Plugins," containerized workloads, and a multi-tier data collection pipeline. The three primary tiers are:

• Job Submission Layer: Interfaces (e.g., PanDA, DIRAC, HammerCloud) initiate benchmarking jobs, submitting them to remote sites via SSH or batch schedulers.
• Benchmark Worker Layer: Executes the benchmarking job on a grid or bare-metal node. Operates in 'pre', 'during', and 'post' phases, collecting plugin metrics and running defined workloads (e.g., HS23, HS06, SPEC CPU2017, DB12).
• Data Aggregation Layer: Collects metric-rich JSON results through brokers (RabbitMQ, Kafka) into centralized backends (InfluxDB, TimescaleDB), visualized using Grafana or custom tools.

A job’s operational parameters—including workloads, plugin definitions (command, regex, interval, aggregation), and normalization references—are fully specified in a YAML configuration, facilitating reproducibility and site-local customization (Szczepanek et al., 2024).

2. HEPScore23: Measuring Canonical Performance

HEPScore23 (HS23) is the canonical performance metric within the WLCG ecosystem. It is calibrated relative to an Intel Xeon Gold 6326 @ 2.9 GHz, aggregating execution timings for $N=7$ workloads originating from five major LHC experiments.

The arithmetic mean HS23 is given by:

$$HS23 = \frac{1}{N}\,\sum_{w=1}^N \frac{T^{\mathrm{ref}}_w}{T^{\mathrm{meas}}_w}$$

where $T^{\mathrm{ref}}_w$ is the reference workload wall-clock time and $T^{\mathrm{meas}}_w$ is the measured duration on the target system. Optionally, the geometric mean can be used to diminish outlier impact:

$$HS23_{\mathrm{geo}} = \left(\prod_{w=1}^N \frac{T^{\mathrm{ref}}_w}{T^{\mathrm{meas}}_w}\right)^{1/N}$$

A calibration constant $\alpha$ can modulate HS23 for historical normalization across sites (Szczepanek et al., 2024).

3. Metric Collection and Plugin Methodology

Beyond raw execution speed, HPE-Bench systematically records machine load, memory use, swap activity, and (where hardware permits) power consumption. Each plugin encodes the metric’s shell command, result extraction regex, units, timing intervals, and aggregation strategy.

Metrics and collection tools include:

• Execution speed: UNIX time(1) or equivalent on containerized workloads.
• Machine load: uptime or /proc/loadavg polled periodically.
• Memory usage: free -b and vmstat -s pre/post-execution for total and working memory.
• Memory swap: vmstat -s or /proc/vmstat, tracking swap-in/swap-out.
• Power consumption: ipmitool sdr, vendor-specific interfaces, polled on bare-metal nodes (typically not available on batch worker nodes in production).

The metric collection loop executes each plugin's command, applies the specified regex, and appends timestamped values to a local metrics buffer, ultimately producing a structured JSON output:

{ "slotID": "atlas-42", "timestamp": 1679876543, "metrics": { "HS23": 15.32, "load_avg": 0.72, "mem_used_B": 6.1e9, "swap_in": 12345, "power_W": 210 } }

(Szczepanek et al., 2024)

4. Integration of Power Measurement and Energy Efficiency

Dedicated bare-metal testbeds are required for reliable power measurement due to restricted sensor access on typical grid worker nodes. A node is reserved, instrumented via IPMI or Redfish, and power metrics are logged alongside performance data. Post-processing yields average power $\langle P \rangle$ and enables computation of energy efficiency:

$$\eta = \frac{HS23_{\mathrm{total}}}{\sum_t P(t)\Delta t} \quad \text{[HS23 per kWh]}$$

Empirical calibration—linking HS23 throughput to power consumption via linear models $$P(\text{site}) \approx a \cdot HS23(\text{site}) + b$$—supports broader extrapolation where direct measurement is not feasible. Frequency/efficiency tradeoffs are characterized, with sites able to optimize CPU settings for sustained throughput per watt (Szczepanek et al., 2024).

5. Campaign Results, Statistical Models, and Anomaly Detection

In 2023, HPE-Bench was deployed automatically on 139 ATLAS sites (148,257 runs) and manually across 48 LHCb sites (2,100 runs), profiling a wide array of CPU models and operational settings.

A linear regression for each (CPU_model, site) pair in the low-load regime ($\text{Load} \in [0,1]$) characterizes per-core throughput:

$$HS23/\text{core} = \alpha - \beta \times \frac{\text{Load}}{\text{core}} + \varepsilon$$

Sites registering $\beta < 0$ or anomalously low $\alpha$ were flagged as misconfigured, with interventions (NUMA settings, C-states, frequency governors) yielding significant throughput recovery (e.g., a site improving from $\alpha \approx 8$ to $\alpha \approx 13$ HS23/core post-tuning). This points to the importance of correct hardware/OS configuration for performance and sustainability (Szczepanek et al., 2024).

6. Correlative Analysis and Fabric Performance Models

Correlative studies revealed:

• Strong negative correlation between HS23 and load (Pearson $r \approx -0.85$).
• Weaker negative correlation with swap-in events ($r \approx -0.42$).
• No significant dependence of HS23 on absolute memory usage until swap is triggered.

A prototypical regression model merges multiple fabric variables:

$$HS23/\text{core} \simeq \alpha - \beta_1 \left(\frac{\text{Load}}{\text{core}}\right) - \beta_2 (\text{Swap}_{\text{in}}/\text{sec}) + \beta_3 \left(\frac{\text{Free RAM}}{\text{Total RAM}}\right)$$

Empirically, architectures from more recent CPU generations exhibited smaller $\beta_1$, indicating improved tolerance to system load due to hardware advances (e.g., turbo-boost, thermal management). Heatmaps of regression parameters across CPU families support these findings (Szczepanek et al., 2024).

7. Operational Workflow, Visualization, and Best Practices

Deployment is streamlined via distribution packages (RPM/DEB, pip install hep-bench) and a CLI (hepbench submit) ingesting site-specific or global YAML configurations. The suite is self-registering, conducts pre/during/post plugin measurements, and publishes unified JSON summaries.

Visual analytics through Grafana dashboards include panels for HS23 per core, real-time load/memory/swap/power breakdowns, a "health" gauge, and outlier/anomaly indicators. Automated alerts can flag HS23 drops exceeding 10% below moving averages. For diagnostic depth, forensic time-series traces facilitate root-cause analysis.

Best-practice recommendations include:

• Running HPE-Bench with load monitoring active.
• Avoiding tests during high system contention.
• Enforcing "performance" governor settings on production grid nodes.
• Calibrating HS23-to-power models via local bare-metal campaigns.
• Routine validation for NUMA and C-state misconfigurations, with possible extension of plugin coverage.
• Employing CPU frequency lock-in to maximize energy efficiency per workload.
• Contributing site JSON metrics to the global database for benchmarking and shared diagnostics.

Comprehensive adoption enables WLCG sites to synthesize real-time operational models, perform cross-site and cross-architecture comparisons, and drive both performance improvement and sustainability efforts (Szczepanek et al., 2024).

---

References:

HEP Benchmark Suite: Enhancing Efficiency and Sustainability in Worldwide LHC Computing Infrastructures (Szczepanek et al., 2024) Using HEP experiment workflows for the benchmarking and accounting of WLCG computing resources (Valassi et al., 2020)