BISection Sampling (BISS) for Benchmark Minimization

• BISection Sampling (BISS) is an algorithmic framework that minimizes benchmark test suites while preserving the global ranking of software variants through variance reduction and recursive bisection.
• It employs a divide-and-conquer strategy with iterative refinement to ensure perfect ranking stability, as measured by Kendall’s τ, even under weighted test costs.
• Empirical results demonstrate that BISS reduces computational costs by 44% on average and up to 99% in some settings, significantly optimizing large-scale benchmarking.

BISection Sampling (BISS) is an algorithmic framework for minimizing benchmark test suites while preserving the global ranking of software variants. Developed primarily for benchmarking contexts in software engineering, BISS enables researchers and practitioners to reduce the computational cost of performance evaluation by strategic test selection, while ensuring that the relative ordering of software variants remains stably maintained. The method leverages test suite optimization via a combination of variance reduction, recursive bisection, and divide-and-conquer methods, with rigorous guarantees on ranking stability measured by correlation metrics such as Kendall’s τ (Matricon et al., 8 Sep 2025).

1. Formal Problem Setting and Objective

Let $V = \{v_1, \ldots, v_m\}$ denote a set of $m$ software variants and $T = \{t_1, \ldots, t_n\}$ a suite of $n$ benchmark tests. For each $(v, t)$, $\text{perf}(v, t) \in \mathbb{R}$ quantifies the performance (e.g., runtime, accuracy) of variant $v$ on test $t$. The aggregate performance on a subset $T' \subseteq T$ is $$\text{perf}(v, T') = \sum_{t \in T'} \text{perf}(v, t)$$. This induces a total order, $m$0, by sorting variants in descending aggregate performance.

The central optimization, called Ranked Test Suite Minimization (RTSM), is to identify a minimal subset $m$1 such that $m$2 for every $m$3. The problem generalizes to a weighted form (WRTSM) where each test $m$4 has a cost $m$5, and the goal is to minimize the total cost $m$6, still preserving the global ranking.

Ranking preservation is quantified using Kendall’s $m$7: $m$8 where $m$9 and $T = \{t_1, \ldots, t_n\}$0 are the counts of concordant and discordant variant pairs between $T = \{t_1, \ldots, t_n\}$1 and $T = \{t_1, \ldots, t_n\}$2, with $T = \{t_1, \ldots, t_n\}$3 and $T = \{t_1, \ldots, t_n\}$4 indicating perfect agreement.

2. Algorithmic Structure of BISS

BISS operates in three principal phases, supported by iterative refinement:

1. FindNecessary (Variance Reduction): Given a candidate set $T = \{t_1, \ldots, t_n\}$5 and "always-keep" set $T = \{t_1, \ldots, t_n\}$6, each test $T = \{t_1, \ldots, t_n\}$7 is removed, and $T = \{t_1, \ldots, t_n\}$8 is recalculated. If $T = \{t_1, \ldots, t_n\}$9, $n$0 is deemed necessary and moved to $n$1. Tests that do not affect global ranking remain in $n$2.
2. Bisection Sampling: $n$3 is recursively split into halves $n$4 and $n$5. The algorithm checks, for each half combined with $n$6, if that subset alone suffices to preserve $n$7. If so, recursion continues on the smaller half; if neither suffices, two sub-calls are made, each forcing one half into $n$8, and the smaller resulting necessary set is returned.
3. Divide-and-Conquer & Iterative Solving: To improve scalability, $n$9 is partitioned into $(v, t)$0 random chunks for parallel initial sampling. Merged pairs of reduced subsets are then processed by the bisection subroutine until only one candidate set remains. If this reduced set $(v, t)$1 is smaller than the previous best, the process restarts with $(v, t)$2. Iterative solving continues until no further reduction is found.

The workflow adheres strictly to ranking stability as measured by $(v, t)$3, and always seeks the smallest (or lowest-cost, in WRTSM) subset compatible with initial rankings.

3. Theoretical Properties

RTSM is NP-hard via reduction from Set-Cover. Each $(v, t)$4-preserving check based on normal equations incurs $(v, t)$5 for candidate set $(v, t)$6. The bisection process yields a worst-case depth of $(v, t)$7, but may double recursion in "both-fail" branches. Divide-and-conquer initialization and merging each require $(v, t)$8 subroutine calls, with cost controlled by the squared size of subsets involved.

A key monotonicity lemma holds: if $(v, t)$9, then for any superset $\text{perf}(v, t) \in \mathbb{R}$0, $\text{perf}(v, t) \in \mathbb{R}$1. This property ensures additional tests cannot introduce discordances once perfect ranking alignment is achieved.

Empirically, BISS reduces the required number of $\text{perf}(v, t) \in \mathbb{R}$2-checks by $\text{perf}(v, t) \in \mathbb{R}$3 relative to brute-force approaches (Matricon et al., 8 Sep 2025).

4. Key Parameters and Algorithm Behavior

BISS exposes several critical hyperparameters:

• Number of Chunks, $\text{perf}(v, t) \in \mathbb{R}$4: Controls initial subdivision; higher $\text{perf}(v, t) \in \mathbb{R}$5 yields more but smaller bisection calls, reducing per-process variance and wall-time per call.
• Target $\text{perf}(v, t) \in \mathbb{R}$6 ($\text{perf}(v, t) \in \mathbb{R}$7): Values of $\text{perf}(v, t) \in \mathbb{R}$8 demand strict preservation; values such as $\text{perf}(v, t) \in \mathbb{R}$9 relax constraints and admit further reduction at a minor cost in ranking fidelity.
• Iteration–Restart Threshold: Defines the number of restart rounds before halting, affecting convergence to potentially smaller solutions.
• Random Seed: Governs the stochastic partitioning; repeated runs can reduce variance in the reduced subset size.

Parameter choices directly influence both final cost savings and computation time in large-scale settings.

5. Illustrative Example

Consider three variants $v$0, four tests $v$1, and the matrix:

Test	A	B	C
$v$2	1	2	3
$v$3	2	1	4
$v$4	3	4	1
$v$5	5	5	5

Aggregate performances are $v$6, $v$7, $v$8, yielding $v$9. The FindNecessary step identifies $t$0 as unnecessary because it is invariant across all variants. Bisection and recursion over $t$1, $t$2, $t$3 confirm all three are required: any further removal changes the ranking. Thus, BISS returns $t$4, i.e., a $t$5 reduction.

6. Experimental Findings

Extensive evaluation on 50 benchmarks from domains such as LLM code generation (HumanEval, RepairBench), SAT competitions (ASlib, SAT ’18/’20), and configurable system benchmarks (x264, SQLite) reveals:

• Mean reduction: BISS cuts original benchmark cost to $t$6 on average; in reducible tasks, to $t$7.
• Cost drops up to $t$8: Over half of reducible benchmarks see near-total reduction without error ($t$9).
• Relaxed $T' \subseteq T$0 criteria: Lowering $T' \subseteq T$1 to $T' \subseteq T$2 yields an additional $T' \subseteq T$3 cost saving on average.
• Benchmark comparison: BISS outperforms random removal, greedy variance, PCA, and MILP (Wilcoxon $T' \subseteq T$4, effect size $T' \subseteq T$5).
• Scalability: Divide-and-conquer with iteration lowers timeout rates by $T' \subseteq T$6 compared to plain bisection; iterative solving improves $T' \subseteq T$7 of reducible cases.

7. Context, Limitations, and Applications

BISS provides an efficient, principled method for pruning benchmark suites in a variety of contexts where reliable software variant ranking is computationally costly. It is especially suitable for settings such as LLM leaderboards, SAT solver competitions, and performance modeling of configurable systems, offering negligible ranking error for significant cost reduction. The method’s design as a divide-and-conquer, bisection-based framework enables scaling to large test suites and accommodates weighted test costs.

A practical implication is that BISS can optimize ongoing benchmarking infrastructures, notably improving resource utilization in settings involving costly or large-scale testing, without sacrificing the stability or interpretability of ranking outputs (Matricon et al., 8 Sep 2025). The monotonicity property provides theoretical assurance on the algorithm’s behavior under test set augmentation.

Future directions may investigate extensions to additional ranking preservation metrics, further acceleration for extremely large test suites, or integration with adaptive benchmarking protocols. No significant controversies were identified regarding the methodological soundness or experimental advantage of BISS relative to standard baselines.