GlideinBenchmark: Automated Benchmarking

• GlideinBenchmark is a lightweight Python-based web application that integrates with GlideinWMS to automate benchmark data collection from diverse computing resources.
• It leverages Apptainer containers and CernVM-FS to execute customizable benchmark suites and expose results via dashboards and a REST API.
• Its architecture minimizes operational overhead by embedding benchmarking within existing pilot workflows, facilitating timely, dynamic resource provisioning.

Searching arXiv for the specified papers and related context. arxiv_search(query="(Mambelli et al., 29 Jul 2025) GlideinBenchmark collecting resource information to optimize provisioning", max_results=5, sort_by="relevance") Searching for the GlideinBenchmark paper by title and arXiv id. search_arxiv({"query":"GlideinBenchmark collecting resource information to optimize provisioning", "max_results": 5}) arxiv_search(query="GlideinBenchmark collecting resource information to optimize provisioning", max_results=5, sort_by="relevance") GlideinBenchmark is a lightweight, Python-based web application that automates the collection and publication of benchmark data across the resources managed by GlideinWMS. It was presented to close a gap in HEP, HPC, and cloud provisioning where hardware varies widely across sites and providers, performance depends sensitively on micro-architecture and system software, and prices or availability change over time. The system leverages the pilot infrastructure of GlideinWMS rather than building a separate execution substrate, executes benchmarks inside Apptainer containers using images distributed via CernVM-FS, and exposes results through dashboards and a REST API so that a scheduler such as HEPCloud’s Decision Engine can optimize resource provisioning with fresh measurements (Mambelli et al., 29 Jul 2025).

1. Problem setting and design rationale

Provisioning in HEP, HPC, and cloud environments is described as a multi-dimensional selection problem under uncertainty. Earlier HEPCloud studies showed that choosing instance types with the best price–performance mix can visibly reduce costs and accelerate science throughput, but those studies depended on thorough, manual benchmarking of resources such as Amazon EC2 instance families. That manual process was labor-intensive, slow, and quickly became obsolete as providers and sites refreshed hardware (Mambelli et al., 29 Jul 2025).

GlideinBenchmark was introduced to make benchmarking a standing service that keeps benchmark data fresh by running them routinely and on new or changed resources, reduces operator effort by reusing the existing pilot infrastructure, and makes results immediately consumable by automated provisioners via a REST API. A central point of the design is that the system does not invent a new dispatch layer: it reuses the GlideinWMS Factory to schedule special “benchmark Glideins” to the same Compute Entrypoints used for production, and it returns results through the same secure and well-understood glidein channels that GlideinWMS already uses for control and monitoring (Mambelli et al., 29 Jul 2025).

This design places benchmarking inside the existing pilot-based provisioning ecosystem rather than beside it. A plausible implication is that benchmark execution and provisioning policy can evolve together without creating an additional scheduling stack, agent framework, or data-transport path.

2. Architecture and control flow

GlideinBenchmark comprises two decoupled components: Runner and Viewer. Runner is the control and orchestration component; Viewer is the web UI and API component. Runner modifies Factory configuration, triggers reconfig or update, and submits special benchmark Glideins via the Factory’s HTCondor-based API. Viewer reads HTCondor state, parses Factory-collected glidein logs including stderr bundles, preprocesses the data, and publishes it through dashboards and a REST API for operators and external clients such as the Decision Engine (Mambelli et al., 29 Jul 2025).

Component	Primary role	Required access
Runner	Modifies Factory configuration, triggers reconfig or update, submits benchmark Glideins	Write access to the Factory’s XML configuration and ability to issue HTCondor commands on the Factory host
Viewer	Reads HTCondor state, parses glidein logs, preprocesses and publishes results	Reachability to the Factory and its HTCondor instance

The execution path is explicitly defined. An operator or automation selects target resources and a benchmark set through Runner’s GUI or CLI. Runner patches Factory configuration to enable benchmark runs on selected Factory entries, triggers reconfig, and submits benchmark Glideins. Each benchmark Glidein lands on a worker node, invokes Apptainer, and runs the chosen benchmark container. Benchmark results are written to glidein logs and transferred back over the standard error channel. Viewer ingests and parses these logs, associates results with the originating resource, and exposes them through dashboards and a REST API. HEPCloud’s Decision Engine then pulls current results and updates its resource-selection policy before instructing the Factory to provision pilots accordingly (Mambelli et al., 29 Jul 2025).

The deployment model has a simple path and an isolated path. In the simple path, Runner and Viewer run on the Factory host under the same Unix account, which simplifies file and HTCondor access. In the isolated path, they run separately but must still reach the Factory and its HTCondor instance. The architecture therefore decouples control, publication, and consumption while remaining tightly aligned with GlideinWMS operational practice (Mambelli et al., 29 Jul 2025).

3. Benchmark execution model and benchmark suite

The benchmark suite is intentionally open-ended. Benchmarks run in Apptainer containers whose images are distributed via CernVM-FS, aligning with how user payloads are typically delivered in GlideinWMS. Experiments choose “the benchmark or the set of benchmarks that most closely evaluate the performance of its workflows,” and the system does not force a single index. The runner submits arbitrary containers as benchmark payloads, and the paper does not enumerate specific CPU, memory, storage, network, or GPU tests, nor does it specify single-core versus multicore modes or run times (Mambelli et al., 29 Jul 2025).

This open-endedness is a defining property rather than an omission in the narrow sense. GlideinBenchmark is structured to support per-experiment benchmark selection aligned to workflow performance predictors, not to impose a universal scalar score. One common misconception is that resource benchmarking systems necessarily reduce all resources to one benchmark number; GlideinBenchmark explicitly does not do that (Mambelli et al., 29 Jul 2025).

The execution environment is designed to be representative of production usage. Container images are fetched from CernVM-FS, but this requirement can be relaxed because recent HTCondor tarballs carried by glideins include Apptainer, and some sites cache OSG-distributed images locally. The paper also notes that Apptainer need not be pre-installed on the worker. This suggests that the system aims for broad coverage across heterogeneous worker nodes without requiring site-by-site software installation (Mambelli et al., 29 Jul 2025).

4. Data collection, publication, and use in provisioning

What GlideinBenchmark collects is defined operationally rather than through a formal schema. Results are delivered as files via stderr and logs collected by the Factory. Glideins auto-detect node resources such as CPU cores, memory, disk, and GPUs and report monitoring and audit information; benchmark outputs therefore can be correlated with these node attributes. Results are associated with Factory entries, that is, Compute Entrypoints. Viewer parses logs and may preprocess data to speed retrieval, then makes results available to operators and clients through the REST API (Mambelli et al., 29 Jul 2025).

The paper does not formalize per-core versus per-node reporting, normalization, or aggregation, and it does not provide a canonical schema. It also does not specify compiler versions or container hashes. At the same time, container-based execution naturally supports reproducibility and versioning via image tags and CernVM-FS snapshots. This suggests that reproducibility is supported primarily through the execution substrate rather than through a dedicated benchmark-result schema (Mambelli et al., 29 Jul 2025).

The intended consumer is HEPCloud’s Decision Engine. The Decision Engine already speaks the GlideinWMS Factory protocol to request pilots, and with GlideinBenchmark’s REST API it can continuously update its view of resource performance and use that to drive provisioning toward the best-performing resources for a given budget or performance objective. The paper references earlier HEPCloud work that minimized cost-over-performance ratios using manual benchmarks in Amazon EC2, but the explicit equations shown in the materials are identified as context and not defined in the paper itself (Mambelli et al., 29 Jul 2025).

The practical significance is that benchmarking becomes part of the provisioning loop. Instead of one-off campaigns whose results are hard-coded into provisioning logic, the Decision Engine can pull current measurements from a continuously updated source.

5. Operational characteristics, limitations, and security

The paper outlines target operational characteristics rather than quantitative case-study results. Factories may have a few hundred entries; the target is to handle on the order of a thousand entries and complete a full-system test in “a few hours.” The expected cadence is no more than one benchmark run per resource per day, and intelligent sampling could avoid running on all entries each cycle. A key design goal is minimal overhead and no interference with production job execution (Mambelli et al., 29 Jul 2025).

Several limitations are stated explicitly. Benchmark representativeness remains a central issue: the system allows experiments to select their own benchmark sets, but choosing a poor proxy reduces the quality of provisioning decisions, and selecting “the best benchmarks to evaluate the resources” remains future work. Temporal and provider variability are managed by balancing freshness against interference through at most daily runs and potential intelligent sampling. Failures and preemptible or spot behavior are not explicitly discussed for GlideinBenchmark itself, and no dedicated database schema is described because results are held in Factory-collected logs and parsed by Viewer (Mambelli et al., 29 Jul 2025).

The security and compliance model is likewise bounded by what the paper states. Authentication and authorization for the Viewer’s REST API are not specified. Runner requires privileged access to the Factory host, including configuration writes and HTCondor commands. Benchmarks run unprivileged inside Apptainer containers, aligning with security practices for glideins on shared worker nodes, and data integrity relies on established glidein stderr and log channels that GlideinWMS already monitors and trusts in production workflows (Mambelli et al., 29 Jul 2025).

A second common misconception is that automating benchmarks necessarily interferes with production. The design constraint is the opposite: benchmarks are run as dedicated glideins, and their rate can be throttled by the Factory’s pressure model (Mambelli et al., 29 Jul 2025).

6. Relationship to prior GlideinWMS practice and broader context

GlideinBenchmark is best understood against the background of GlideinWMS as a pilot and late-binding system. A documented earlier large-scale example is the Snowmass Energy Frontier simulation campaign on the Open Science Grid, where HTCondor plus GlideinWMS pilots were used for distributed scheduling across many sites, mainly in the United States, and Parrot with CvmFS provided a homogeneous runtime environment on heterogeneous nodes (Avetisyan et al., 2013). In that campaign, the pilot model supported opportunistic resource use at substantial scale: 32 or more sites contributed measurable walltime, total consumed walltime reached 7,808,607 hours since February 2013, typical daily completions were approximately 14,000 jobs per day, and peak daily completions were approximately 70,000 to 80,000 jobs per day (Avetisyan et al., 2013).

Several operational patterns from that earlier GlideinWMS deployment parallel the premises of GlideinBenchmark. Snowmass used late binding to absorb queue variability and heterogeneous hardware, Parrot plus CvmFS to ensure software homogeneity without site-by-site deployment, and pilot-level caching to amortize the movement of large shared inputs. The campaign also illustrated the importance of pragmatic mitigations: a large minimum-bias input file was reduced from approximately 50 GB to approximately 1 GB and pre-staged, and where iRODS client installation was constrained, SRM was used with hard-coded SURLs to queue and throttle transfers (Avetisyan et al., 2013).

The connection is structural rather than identity of purpose. Snowmass demonstrated that GlideinWMS could sustain multi-site, late-binding production at multi-million wall-hour scale, while GlideinBenchmark applies the same pilot substrate to automated benchmarking and publication. The paper on GlideinBenchmark explicitly emphasizes that it leverages, rather than duplicates, GlideinWMS capabilities such as Factory pressure control, glidein execution, Apptainer containers, and CernVM-FS distribution (Mambelli et al., 29 Jul 2025). This suggests continuity between production execution and benchmark execution: the benchmark path is intended to measure resources in environments closely matching actual payload deployment.

7. Extensibility and research directions

Future work is framed in operational and methodological terms. The paper identifies four directions: scale to thousands of Factory entries and complete full sweeps within hours, use intelligent sampling to reduce load, curate and standardize benchmark suites that best predict experiment performance, and ensure minimal overhead and zero interference with production jobs through careful rate control (Mambelli et al., 29 Jul 2025).

It also notes that, although not stated explicitly, the architecture naturally extends to adding new providers or sites by defining Factory entries and benchmark containers, incorporating GPU or accelerator probes because glideins already auto-detect GPUs, and tightening integration with the Decision Engine and other schedulers via the REST API. These are presented as natural extensions rather than reported implemented features (Mambelli et al., 29 Jul 2025).

The overall significance of GlideinBenchmark lies in turning benchmarking into an automated, first-class part of pilot-based provisioning. Its contribution is primarily architectural and integrative: it embeds benchmark execution in the GlideinWMS Factory workflow, keeps results consumable through a REST API, and enables resource-selection policy to be refreshed as hardware, software environments, prices, and availability change. The paper is correspondingly explicit about what it does not fix: there is no mandated benchmark suite, no canonical data schema, no optimization formulae, and no quantitative case-study evaluation in the paper itself (Mambelli et al., 29 Jul 2025).