Benchmark Carpentry: A Practical Framework

• Benchmark carpentry is a socio-technical framework that treats benchmarks as dynamic, reproducible workflows rather than fixed scoreboards.
• It integrates templated experiment designs, continuous integration, and adaptable workflows to ensure fair and consistent performance evaluation.
• The approach extends Software Carpentry’s hands-on training to include benchmark competence, fostering community education and improved research practices.

Benchmark carpentry denotes a practical, educational, and infrastructural approach to designing, executing, interpreting, sharing, maintaining, and teaching benchmarks as reproducible, adaptable, and broadly accessible scientific instruments rather than static scoreboards. In the AI literature, it is presented as a community practice for benchmark specification, workflow engineering, infrastructure awareness, energy and performance measurement, result interpretation, benchmark sharing, and education; in HPC, it is framed as the craft of making benchmarks executable, adaptable, comparable, and shareable through reusable workflow templates and lightweight experiment management; and in the broader pedagogical lineage of Software Carpentry, it inherits an emphasis on hands-on competence, reproducibility, and community-based training (Laszewski et al., 12 Dec 2025, Laszewski et al., 30 Jul 2025, Wilson, 2013).

1. Origins and conceptual formation

Benchmark carpentry is explicitly modeled on Software Carpentry. Software Carpentry emerged to address a structural problem in modern science: researchers increasingly depend on software, yet “most scientists are never taught how to do this,” and the program eventually converged on short, hands-on workshops focused on a small number of high-leverage tools and habits rather than a broad software-engineering canon (Wilson, 2013). Within that lineage, benchmark carpentry extends the same pedagogical logic from coding practice to benchmarking practice. The emphasis shifts from shell use, version control, scripting, and SQL as ends in themselves to benchmark competence: defining tasks, packaging workflows, collecting metadata, interpreting outputs, and comparing systems under controlled conditions.

The term becomes explicit in two 2025 papers. In AI, benchmark carpentry is described as an emerging community practice for “designing, executing, interpreting, sharing, maintaining, and teaching AI benchmarks” in a way that is reproducible, adaptable, and broadly accessible; it is presented not as a toolkit but as a socio-technical framework (Laszewski et al., 12 Dec 2025). In HPC, benchmark carpentry is defined through a concrete activity set: “installing software,” “reserving compute resources for exclusive use,” “preparing experiments—often as many batch jobs—and executing them,” “evaluating and validating performance across CPUs, GPUs, I/O, networking, and energy,” “recording results in a uniform format for comparison,” and “submitting results back to the community,” ideally aligned with FAIR principles (Laszewski et al., 30 Jul 2025).

A broader educational precursor appears in astronomy education. “The Importance of Computation in Astronomy Education” does not use the benchmark-carpentry label, but it makes a benchmark-like normative claim that astronomy undergraduates should complete their degree with “computational literacy, familiarity with fundamental software carpentry skills, and mastery of basic numerical methods” (Zingale et al., 2016). This suggests a disciplinary analogue: benchmark carpentry is not only a benchmarking methodology but also a way of expressing graduation-level or research-readiness competencies around benchmarked computational practice.

2. Core principles and formal structure

The central premise of benchmark carpentry is that benchmarks are no longer adequately described as fixed scoreboards. In AI, the benchmark itself is treated as a “moving target” because models, datasets, workflows, and deployment contexts evolve rapidly, and because LLMs may memorize static benchmarks, creating a “drastic difference between benchmark results and real-world performance” (Laszewski et al., 12 Dec 2025). This leads to a preference for dynamic, continuous, adaptive benchmarking frameworks that preserve transparency, reproducibility, and interpretability while permitting controlled updates.

The most explicit formalization appears in the AI benchmark-democratization work, which defines a benchmark as

$B = (I, D, T \text{ or } W, M, C, R, V)$

where $I$ is infrastructure, $D$ is dataset, $T$ or $W$ is scientific task or workflow, $M$ is metrics, $C$ is constraint, $R$ is results, and $V$ is version or timestamp (Laszewski et al., 12 Dec 2025). Tasks may themselves be represented as

$T = (A, P),$

with application $I$0 and parameters $I$1, while workflows are modeled as

$I$2

with multiple tasks and their relationships. This formalization is important because it makes benchmark carpentry explicitly multi-component: benchmark design is not reducible to dataset selection or metric choice alone.

The same paper also formalizes component-specific constraints, allowing fairness or tractability restrictions to apply at the level of the benchmark, infrastructure, dataset, task, metrics, results, or application. A dynamic optimization view is then suggested through the expression

$I$3

where the benchmark varies under a fixed metric of interest and multiple solution parameter sets may exist (Laszewski et al., 12 Dec 2025). The notation is compressed in the source, but the intended implication is clear: benchmark carpentry must reason about variable configurations rather than single immutable setups.

The HPC formulation reaches similar conclusions through workflow structure rather than benchmark algebra. It distinguishes general computational workflows from experiment executions and argues that community benchmarks often revolve around multidimensional loops, arrays, grid searches, and dynamic or adaptive searches, not merely DAGs. This makes repeated, parameterized runs central to benchmark design (Laszewski et al., 30 Jul 2025). A closely related systems-engineering formulation appears in CI-inspired scientific benchmarking, where the benchmark space is described by a set of test cases

$I$4

and a set of algorithms

$I$5

with measurements defined by a mapping $I$6; the paper uses this to motivate benchmark automation because the Cartesian product expands rapidly as methods and cases are added (Toprak et al., 21 Mar 2025).

3. Workflow engineering, execution, and reproducibility

A defining feature of benchmark carpentry is its treatment of benchmarking as workflow engineering. In HPC, workflow templates are proposed as reusable recipes that can encode parameterized experiment arrays, simple pipelines, DAGs with dependencies, cyclic or adaptive workflows, and hybrid simulation–AI workflows (Laszewski et al., 30 Jul 2025). YAML is favored because communities found it accessible and expressive enough to represent hierarchy, iteration, and parameterization while remaining teachable. Cloudmesh’s Experiment Executor and HPE SmartSim are presented as two independently developed systems validating this approach, with overlap in schedulers, batch submission, Python APIs, experiment abstractions, grid search, and container support (Laszewski et al., 30 Jul 2025).

The Cloudmesh examples make this operational. A minimal YAML workflow standardizes a benchmark lifecycle of data staging, execution, and analysis, while a compact experiment specification defines a Cartesian product over epochs, GPU type, and repeat count. Given such a configuration and a batch template, Experiment Executor uses a unique directory for each experiment, takes a parameter set from the Cartesian product, instantiates the batch job template with configuration variables, and instantiates a configuration file with current experiment parameters (Laszewski et al., 30 Jul 2025). The associated templated batch script captures scheduler requests, provenance, GPU monitoring, application launch, and efficiency reporting. SmartSim expresses similar abstractions through Python objects—Experiment, Model, Ensemble, and Orchestrator—and is especially oriented toward tightly coupled simulation/AI workflows (Laszewski et al., 30 Jul 2025).

The AI benchmark-democratization paper generalizes these infrastructure concerns. It recommends open repositories, executable workflows, notebooks or scripts, package-managed dependencies, structured logging, profiling, result metadata, and especially containerization. Docker is emphasized for general environments, Apptainer for HPC settings without root access, and workflow systems such as Compute Coordinator and Experiment Executor are cited as enablers for repeated templated benchmark runs across heterogeneous infrastructures (Laszewski et al., 12 Dec 2025). Logging is stratified into infrastructure monitoring, application monitoring, training monitoring, and model-level monitoring. Profiling is treated as intrinsic to benchmark carpentry because it explains why performance differs, supports fair comparison, and exposes heterogeneous hardware usage (Laszewski et al., 12 Dec 2025).

Continuous-integration practice provides a parallel workflow model. The CI-inspired scientific benchmarking paper argues that a benchmark suite should be managed like a software product, using Git, GitLab Pipelines, protected branches, merge requests, runners, submodules, automated execution, artifacts, and staged tests (Toprak et al., 21 Mar 2025). Benchmark outputs are not just pass/fail states but numerical tables, plots, and reports. Artifacts can be downloaded, aggregated by downstream jobs, and tied to exact repository and dependency revisions. This supports benchmark evolution over time: adding a new library, metric, or case becomes a re-execution problem rather than a manual reconstruction exercise.

4. Quality criteria, adaptivity, and benchmark construction

Benchmark carpentry is also concerned with how benchmarks are built and judged. The AI benchmark-democratization work provides the clearest explicit quality rubric through the MLCommons benchmark ontology and submission workflow. New submissions are evaluated in six categories—Software Environment, Problem Specification, Dataset, Performance Metrics, Reference Solution, Documentation—and ratings are presented as summaries and radar charts (Laszewski et al., 12 Dec 2025). The same paper argues that good benchmarks should be relevant, representative, fair, repeatable, cost-effective, scalable, and transparent, and should document infrastructure, data regimes, constraints, and results in ways that support application-relevant interpretation.

BenchMake provides a concrete benchmark-construction method for scientific datasets. It aims to turn “any scientific data set into a reproducible benchmark” by deterministically isolating challenging edge cases through non-negative matrix factorisation and Euclidean matching in feature space (Barnard, 29 Jun 2025). The method converts data from tabular, graph, image, signal, and textual modalities into a non-negative numeric matrix, applies Min-Max scaling, factorizes

$I$7

and minimizes

$I$8

Actual test instances are then selected by distance to learned archetypes,

$I$9

with deterministic ordering and tie-breaking to ensure reproducibility (Barnard, 29 Jun 2025). The resulting split is validated by KS-test, $D$0-test, mutual information, KL divergence, JS divergence, Wasserstein distance, and MMD, and the paper reports stronger divergence than prescribed or random splits on 8 of 10 benchmark settings. This is benchmark carpentry in the narrow sense of split design: the benchmark is engineered to be reproducible, difficult, and informative rather than sampled arbitrarily.

A different construction paradigm appears in BenchPress, which treats benchmark generation for compiler research as feature-space search (Tsimpourlas et al., 2022). BenchPress uses a BERT-based generator with [HOLE] and [ENDHOLE] tokens so that code can be inserted with both left and right context, enabling iterative steering toward target benchmark features. It achieves an 86% compilation rate versus CLgen’s 2.33%, and can target Grewe features, InstCount, and Autophase spaces while using active learning to generate benchmarks that improve a downstream CPU-vs-GPU mapping heuristic (Tsimpourlas et al., 2022). This suggests a broader implication: benchmark carpentry can involve not only curating tasks but actively shaping benchmark suites to fill sparse or informative regions of a feature space.

The most general caution across these works is that benchmark quality is not reducible to geometric or numerical similarity alone. Dynamic datasets, living datasets, heterogeneity across platforms, and deployment-specific constraints all challenge static assumptions (Laszewski et al., 12 Dec 2025). BenchMake notes that emphasizing edge cases may improve discriminative power while reducing average-case representativeness, and that its modality-general approach depends on the quality of the chosen encoding (Barnard, 29 Jun 2025). BenchPress similarly shows that feature matching in source space does not guarantee semantic or IR-level equivalence after compilation (Tsimpourlas et al., 2022).

5. Education, dissemination, and democratization

Benchmark carpentry is presented not only as a technical workflow but as an educational program. The AI benchmark-democratization paper is explicit that democratization requires systematic education “from undergraduate to professional levels” and cannot be achieved through tools alone (Laszewski et al., 12 Dec 2025). It proposes an AI Benchmark Carpentry curriculum spanning foundational tools and practices, AI benchmarking fundamentals, reproducibility and experiment management, ethical considerations and bias mitigation, hands-on workshops and collaborative projects, and special topics such as energy efficiency, simulation, performance tuning, leaderboard management, and benchmark collections (Laszewski et al., 12 Dec 2025). Foundational content is adapted directly from Software Carpentry: Python and Jupyter, reproducible coding and documentation, Git and version control, Unix shell proficiency, data cleaning, transformation, and visualization.

The educational logic closely parallels Greg Wilson’s account of Software Carpentry. Software Carpentry’s mature model used two-day intensive workshops, live coding, task-based pre-assessment, helper support, collaborative note-taking, and open lessons versioned on GitHub, with the explicit goal of “computational competence” rather than tool exposure alone (Wilson, 2013). It also scaled through host-supported workshops, instructor training, and recruiting instructors from former participants. This history is directly relevant because benchmark carpentry papers repeatedly assume a train-the-trainer model, open materials, and community instruction rather than isolated expert workshops.

The astronomy-education paper strengthens the curricular argument in disciplinary form. It recommends that astronomy undergraduates leave with “computational literacy, familiarity with fundamental software carpentry skills, and mastery of basic numerical methods,” and argues that these skills should be embedded in astronomy courses using community-developed infrastructure, open data archives, open-source web-based tools, and Jupyter notebooks (Zingale et al., 2016). It explicitly cites Software Carpentry sessions at AAS meetings as an excellent example of training and recommends instructor training so participants can offer workshops at their own institutions. This suggests that benchmark carpentry, when translated into a discipline, is likely to be most effective when embedded in domain courses and authentic tasks rather than outsourced to general-purpose computing instruction.

Evidence for the pedagogical efficacy of the carpentry model is mixed but supportive. A retrospective interview study of Software Carpentry reports that all 24 interviewees believed the workshop shortened the learning process; 23/24 said tutors or instructors helped them get where they wanted to be faster; 19/24 used some new skills soon after the workshop; more than two thirds reported improved coding; and 20/24 said their time was definitely usefully spent (Simperler et al., 2015). The study is small, retrospective, and self-reported, but it supports the notion that workshop-based carpentry can accelerate time-to-application.

6. Limitations, tensions, and domain-specific trajectories

Benchmark carpentry is an agenda rather than a fully standardized doctrine, and the literature repeatedly identifies its limitations. The AI benchmark-democratization paper notes persistent barriers including compute cost, limited access to specialized hardware, lack of benchmark-design expertise, reproducibility problems due to software and hardware variability, interpretability challenges, and uncertainty about how to relate benchmark results to application domains (Laszewski et al., 12 Dec 2025). It also criticizes benchmark cultures centered on “leadership-class hardware” and peak performance, arguing that such practices often provide limited guidance for practitioners using campus clusters, desktop GPUs, edge devices, or commercial APIs.

The HPC workflow paper likewise emphasizes tradeoffs between simplicity and expressiveness, and between portability and platform specificity (Laszewski et al., 30 Jul 2025). Abstract schedulers and templates aid portability, but benchmark fidelity often depends on local queue policies, filesystems, security configurations, and site-specific constraints. Security and federation remain incomplete, with pragmatic SSH and split-VPN approaches substituting for full cross-site identity integration. The paper also notes licensing risk as part of benchmark carpentry because license changes can undermine reproducibility and adoption (Laszewski et al., 30 Jul 2025).

Software Carpentry’s own experience identifies several organizational analogues likely to recur. Heterogeneous skill levels produce the classic “too slow and too fast” problem; installation and environment setup are often harder than using the tools themselves; long-term impact assessment is difficult; and online-only delivery performs poorly relative to interactive workshops (Wilson, 2013). These concerns generalize directly to benchmark carpentry, where environments are often even more brittle because they involve compilers, profilers, accelerators, datasets, and scheduler policies.

Domain-specific variants illustrate both the breadth and fragmentation of the field. CI-managed scientific software benchmarking translates benchmark carpentry into repository discipline, artifacts, and pipeline execution (Toprak et al., 21 Mar 2025). Carpentry reverse engineering and co-optimization papers in wood fabrication suggest benchmark regimes centered on fabrication validity, part decomposition, assembly correctness, and multi-objective tradeoffs over material cost, fabrication time, and precision error (Noeckel et al., 2021, Zhao et al., 2021). Human-centered construction robotics and augmented manual fabrication contribute adjacent notions of carpentry benchmarking around assistive support tasks, tool-aware guidance, and geometric fidelity to execution models, but these are domain-specific extensions rather than part of the core benchmark-carpentry vocabulary (Wu et al., 2024, 2503.07473).

A cautious synthesis follows. Benchmark carpentry has a stable conceptual core—practical benchmark competence, explicit workflow design, reproducibility, structured metadata, and community education—but its concrete realization remains domain-dependent. In AI it is tied to dynamic datasets, ontology, energy measurement, and heterogeneous platforms; in HPC to experiment execution, templates, and federated infrastructure; in disciplinary education to computational literacy and train-the-trainer dissemination. The literature therefore supports benchmark carpentry most strongly as a family of aligned practices rather than a single standardized framework (Laszewski et al., 12 Dec 2025).