Comb Benchmarking Suite Proposal

• Comb Benchmarking Suite is a proposed framework for FT/FI evaluation emphasizing soft-error relevance, fault-space behavior, portability, and configurability.
• It enables resource-efficient fault injection campaigns by reducing redundant benchmark overlap and supporting representative, minimal experimental setups.
• The suite’s multi-granular structure supports both isolated algorithmic kernels and integrated systems, ensuring accurate configuration and increased comparability.

Searching arXiv for the primary paper and closely related benchmarking-suite work to ground the article in current literature. The “Comb Benchmarking Suite” denotes the proposed common benchmark collection for fault-tolerance and fault-injection studies described in “Towards a Fault-Injection Benchmarking Suite” (Wang et al., 2024). It is not a completed benchmark standard. Instead, it is a proposal to establish an FT/FI-specific suite whose composition is driven by soft-error relevance, fault-space behavior, portability, and configurability, rather than by the priorities of borrowed benchmark families such as embedded-systems or WCET-oriented suites. In that formulation, the suite is meant to make FT/FI evaluation more comparable across publications, less redundant in benchmark selection, and more resource-efficient in large FI campaigns (Wang et al., 2024).

1. Problem setting and motivation

Soft errors in memories and logic circuits are known to disturb program execution, and the FT/FI literature has responded with many fault-tolerance mechanisms and fault-injection approaches. The central problem identified for the proposed suite is that this literature lacks an agreed-upon benchmarking suite. In practice, authors therefore select benchmarks from other domains, especially suites such as MiBench or TACLeBench, even though those suites were designed for different goals and metrics (Wang et al., 2024).

The proposal isolates three shortcomings of this practice. First, there is little comparability across FT/FI papers. Different studies use different benchmark sets, and even the same benchmark can expose a very different fault space under different runtime and build configurations. Second, benchmark selections often overlap in the properties that matter for FT/FI, so large evaluation sets can appear broad while remaining behaviorally redundant. Third, off-the-shelf suites offer limited configurability for FT/FI-relevant properties such as minimum memory-access activity or maximum execution time; in practice, only the input scale can often be adjusted (Wang et al., 2024).

These deficiencies matter because FI campaigns are expensive in time and computing resources. The proposal therefore treats benchmark design not as a secondary implementation issue, but as part of the validity of FT/FI experimentation itself.

Current practice	FT/FI consequence	Proposed response
Benchmarks borrowed from other domains	Little comparability across papers	A common FT/FI benchmark collection
Large but overlapping benchmark sets	Inefficient FI campaigns	Representative, preferably minimal, experiment setups
Limited configurability	Poor control over fault space and cost	Configurable benchmarks adapted to resource constraints

2. Intended structure of the suite

The proposed suite is explicitly multi-granular. It should contain isolated algorithm implementations and program parts for targeted analysis, and integrated systems to represent realistic use cases. This requirement follows from the observation that existing FT/FI work already spans both bare-metal algorithmic kernels and larger system compositions, so a usable suite must support both modes rather than forcing a single granularity (Wang et al., 2024).

Benchmark selection is intended to support classification into groups so that experimenters can construct representative, preferably minimal, setups. Two classification axes are suggested. One is by program characteristics, including memory usage, runtime, fault-space characteristics, and related structural properties. The other is by domain, in a manner analogous to domain-grouped suites such as MiBench. The aim is to support both orthogonality and representativeness: orthogonality, in the sense of reducing redundant fault-space behavior across selected programs, and representativeness, in the sense of preserving practical relevance (Wang et al., 2024).

The suite is also designed around resource-efficient FI. Because large-scale injection experiments are costly, the infrastructure should be lightweight and benchmarks should expose configurability with respect to execution time and memory usage. This is directly tied to control of fault-space size and therefore to the number of injections required. In the same spirit, the runtime model should be self-contained and portable. The paper contrasts this objective with FI frameworks such as FAIL*, which may execute benchmarks in an instrumented virtual machine while requiring the target programs to provide their own runtime, including a barebones OS and standard library. Dependency on such external runtime support is argued to reduce portability and comparability; TACLeBench is cited as a good example of portability in contrast (Wang et al., 2024).

3. Benchmark-selection methodology and observable properties

The proposal does not present a final taxonomy or a formal benchmark-selection algorithm. Its suggested methodology is instead organized around coverage of relevant FT/FI properties and reduction of overlap. The intended suite should cover a wide range of practical applicability, different program characteristics, different granularities, and diverse fault-space behaviors, while keeping redundancy low (Wang et al., 2024).

In the preliminary evaluation, the suite’s classification logic is based on observable program characteristics. Three are used directly: number of dynamic instructions, number of memory-access locations, and Silent Data Corruption (SDC) count. The memory-access metric is defined in a footnote as follows:

“The number of unique memory locations where at least one memory read or write event is recorded during the FI campaign.” (Wang et al., 2024)

Additional useful characteristics are identified but not yet operationalized in a final taxonomy: stack usage, heap usage, branching behavior, and memory-access granularity. The proposal therefore stops short of claiming that these properties form a complete basis for benchmark selection. Its more limited claim is that FT/FI suites should be classified using observable program characteristics, and that such classification can support both better coverage and lower overlap (Wang et al., 2024).

This leaves a central methodological question unresolved: what the most relevant program properties for FT/FI actually are. The proposal treats that uncertainty as an open research problem rather than as a defect to be hidden by an apparently finished suite.

4. Empirical evidence used to motivate the proposal

To demonstrate why ad hoc benchmark choice is problematic, the paper reports an exploratory FI study using MiBench and TACLeBench. In a plot organized by dynamic instructions, unique memory-access locations, and SDC counts, the benchmarks exhibit clustering. From this, the authors infer that some benchmarks may overlap in fault-space characteristics, that outliers may be especially interesting to include, and that some benchmark classes may be missing altogether, including workloads with very high data throughput such as SIMD-heavy workloads (Wang et al., 2024).

A more concrete result concerns execution granularity and runtime environment. Some MiBench benchmarks were compiled in two settings: a full-system setting with eCos and a bare-metal setting based on picolibc. For dijkstra, susan, crc, and cutcp, the full-system variants show 37%–128% more dynamic instructions, with a similar increase in dynamic memory accesses. The FI outcomes also change substantially. For crc, the eCos variant has 97% more SDCs, 801% more timeouts, and 1491% more CPU exceptions than the picolibc variant (Wang et al., 2024).

These results are used to establish a specific FT/FI benchmarking point: benchmark identity cannot be reduced to source-level program name alone. Runtime composition and build configuration materially affect dynamic instruction count, memory-access behavior, and observed fault outcomes. Accordingly, the same nominal benchmark may instantiate meaningfully different fault spaces under different execution environments.

5. Interpretation, scope, and recurring misconceptions

The proposal’s most important limitation is also one of its defining features: it is a call to define a suite, not the presentation of a finished suite. It does not specify a final benchmark inventory, a fixed benchmark taxonomy, or a benchmark-selection algorithm. What it provides instead is a set of selection criteria and design requirements for a future FT/FI benchmark collection (Wang et al., 2024).

Several misconceptions follow naturally from the current state of the field and are addressed implicitly by the proposal. One is that comparability can be achieved merely by reusing benchmarks from well-known embedded-systems suites. The reported eCos-versus-picolibc results show that this is insufficient, because the fault space varies with runtime and build configuration. Another is that evaluation quality necessarily improves as the number of benchmarks increases. The clustering analysis suggests the opposite possibility: large benchmark sets can be behaviorally redundant and therefore inefficient in FI campaigns. A third is that benchmark configurability is a secondary convenience. In this proposal, configurability is fundamental because execution time and memory usage directly affect the size of the fault space and thus the cost of experimentation (Wang et al., 2024).

The suite’s intended balance is therefore not maximal breadth in an undifferentiated sense, but selective breadth across practical applicability, program characteristics, granularity, and fault behavior. A plausible implication is that a future FT/FI suite would be judged not only by how many programs it contains, but by how well those programs span the FT/FI-relevant design space without unnecessary overlap.

6. Position within the broader benchmarking-suite landscape

Comparable benchmark-design problems recur in other domain-specific suites, although the target metrics and workloads differ. CamBench, for cryptographic API misuse detection, makes transparency, openness, and preregistration central design principles, and structures its benchmark around real-world applications, synthetic test cases, and a crypto API coverage heuristic, with evaluation focused on precision, recall, crypto API coverage, and analysis capabilities (Schlichtig et al., 2022). SGXGauge, for Intel SGX, is presented as a comprehensive reference suite whose defining concern is SGX-specific bottlenecks such as EPC paging, enclave transitions, and TLB behavior across ten workloads (Kumar et al., 2022). RobotPerf, for robotics computing systems, uses ROS 2 as a common baseline and combines black-box testing with grey-box testing to evaluate latency, throughput, and power across heterogeneous hardware platforms (Mayoral-Vilches et al., 2023).

A different but related line of work emphasizes infrastructure and reproducibility. CATBench provides a unified interface and a fully containerized setup for compiler autotuning tasks with discrete, conditional, and permutation parameter types, known and hidden constraints, and multi-fidelity and multi-objective evaluations (Tørring et al., 2024). Bencher decouples benchmark execution from optimization logic through isolated benchmark environments and a unified, version-agnostic RPC interface, deployable via Docker or Singularity (Papenmeier et al., 27 May 2025). In quantum software, Benchpress offers an open-source framework of over 1000 tests spanning circuits with up to 930 qubits and $\mathcal{O}(10^{6})$ two-qubit gates, while microbench.py adopts the opposite scale: an approximately 200-line self-contained micro-benchmark harness intended for rapid sanity checks, nightly regression testing, and CI pipelines (Nation et al., 2024, Merilehto, 4 Jul 2025).

This comparison suggests that the proposed Comb Benchmarking Suite belongs to a broader family of domain-specific benchmark initiatives in which benchmark relevance, execution model, portability, and comparability are treated as first-order design constraints. In that context, its distinctive contribution is to articulate those constraints for the FT/FI domain, where fault-space behavior and runtime composition can be at least as important as the nominal application benchmark itself (Wang et al., 2024).