SliceBench: Static Slicing Benchmark

• SliceBench is a benchmark for static program slicing, annotated with 2,200 Java and Python instances to test statement-level backward slicing.
• It integrates contest-style programs from CodeNet with real-world buggy examples from Defects4J and BugsInPy, highlighting scalability challenges.
• The benchmark employs rigorous annotation protocols and standard metrics like precision, recall, and exact-match accuracy for evaluating slicers.

Searching arXiv for the SliceMate/SliceBench paper and closely related slicing-benchmark work. SliceBench is a static program slicing benchmark introduced alongside "SLICEMATE: Accurate and Scalable Static Program Slicing via LLM-Powered Agents" (Chang et al., 25 Jul 2025). It comprises 2,200 manually annotated Java and Python programs and is designed for evaluating backward static slicing at the statement level. The benchmark combines small programming-contest programs from IBM CodeNet with larger real-world buggy programs from Defects4J and BugsInPy, and it provides adjudicated ground-truth slices defined with respect to a specified slicing criterion such as a final print(...) or a failing assert(...) (Chang et al., 25 Jul 2025). In the reported formulation, SliceBench functions as both an evaluation suite for static slicers and a corpus for learning-based slicing experiments, while also exposing the practical difficulties of annotation, scale, and language coverage.

1. Definition and scope

SliceBench is a benchmark for static program slicing, where the task is to extract the executable portions of a program that affect the values at a specific location. Its instances are annotated for backward slicing only, and every instance is associated with a slicing criterion that is either a final print(...) in the CodeNet-derived subset or a failing assert(...) in the GitHub-derived subset (Chang et al., 25 Jul 2025).

The benchmark contains 1,100 Java programs and 1,100 Python programs, for a total of 2,200 instances. These instances are drawn from two qualitatively different sources. The CodeNet portion supplies short programs, typically a single function, with lengths from 5 to 150 LOC and average size approximately 30 LOC. The GitHub portion supplies larger, multi-file or multi-function buggy examples from Defects4J and BugsInPy, with average code size approximately 2,106 statements and maximum size up to 8,577 LOC (Chang et al., 25 Jul 2025).

This composition gives SliceBench a mixed character. It includes both compact algorithmic programs and substantially larger real-world code. A plausible implication is that the benchmark is intended to probe both canonical dependence-tracing behavior and the scalability of slicers under more realistic software structures.

2. Construction, sampling, and dataset composition

SliceBench is constructed from IBM CodeNet, Defects4J, and BugsInPy. For CodeNet, a round-robin sampler is applied over all problems, selecting the first solution of each, then the second of each, and so on, until 1,000 Python and 1,000 Java programs are obtained. These programs are described as almost always single-function and 5–150 LOC in size, with average length approximately 30 LOC (Chang et al., 25 Jul 2025).

For the real-world component, the benchmark includes 100 Java programs from Defects4J and 100 Python programs from BugsInPy. These examples are characterized as larger, multi-file or multi-function buggy programs, and their failing test assertions serve as slicing criteria. Their average code size is approximately 2,106 statements, with a maximum up to 8,577 LOC (Chang et al., 25 Jul 2025).

The dataset breakdown can be summarized as follows:

Subset	Programs	Characteristics
CodeNet	2 × 1,000	Java/Python, 5–150 LOC, avg ≈ 30 LOC, typically 1 function
GitHub (real)	2 × 100	Java/Python, 150–8,577 LOC, avg ≈ 2,106 LOC, multi-function
Total	2,200	Java and Python backward-slicing instances

The benchmark also reports approximate size buckets: small programs of at most 50 LOC constitute approximately 91%, medium programs from 51 to 500 LOC approximately 5%, and large programs above 500 LOC approximately 4% (Chang et al., 25 Jul 2025). This distribution indicates that most instances are short, even though the benchmark includes a long tail of substantially larger programs.

3. Ground truth and annotation protocol

The ground-truth slice for each instance is defined by expert annotation. Two annotators, described as having PhD+Master’s level training and at least four years of coding experience, independently mark every statement that possibly indirectly contributes via data, control, or inter-procedural dependencies to the specified slicing criterion (Chang et al., 25 Jul 2025).

The annotation workflow is specified in four steps. First, the annotator identifies the slicing criterion variable or program point, for example a print in CodeNet or an assert in a failing test. Second, the annotator traces all statements that directly or indirectly affect that criterion along any static execution path. The specified dependency classes are data dependencies, control dependencies, and inter-procedural calls, including caller-to-callee and return-value flows. Third, import, package, or class declarations are included if needed to make the slice syntactically valid. Fourth, statements that are provably irrelevant are excluded, such as print statements unrelated to the criterion (Chang et al., 25 Jul 2025).

Inter-annotator agreement is measured with Cohen’s $\kappa$. The reported values are $\kappa = 0.927$ for the CodeNet subset, described as almost perfect, and $\kappa = 0.893$ for the GitHub subset, described as high. Disagreements are adjudicated by one of two supervising professors, and the final adjudicated slice becomes the benchmark ground truth (Chang et al., 25 Jul 2025).

These procedures establish SliceBench as a manually verified benchmark rather than a synthetic or automatically derived one. The emphasis on data, control, and inter-procedural dependencies also clarifies that the intended notion of relevance is broader than simple def-use tracing.

4. Descriptive statistics and structural properties

SliceBench reports descriptive statistics for program size, slice size, and function counts. Across all programs, the minimum size is 5 LOC, the maximum is 8,577 LOC, the median is 30 LOC, and the 90th percentile is 2,500 LOC (Chang et al., 25 Jul 2025). This confirms that the benchmark is highly skewed toward small programs while still containing a nontrivial set of large instances.

Ground-truth slice sizes vary substantially by subset. CodeNet slices range from 1 to 45 statements with average size approximately 12 statements. GitHub slices range from 10 to 450 statements with average size approximately 112 statements (Chang et al., 25 Jul 2025). Reported overall descriptive statistics also include a slice-size minimum of 1, median of 12, 90th percentile of 200, and maximum of 450, as well as function-count statistics with minimum 1, median 1, 90th percentile 8, and maximum 25 (Chang et al., 25 Jul 2025).

Language-level weighted averages are also reported. Java has 1,100 programs with average LOC 300 and average slice size 60, while Python has 1,100 programs with average LOC 290 and average slice size 58; these are explicitly identified as weighted averages over short CodeNet and large GitHub examples (Chang et al., 25 Jul 2025).

A compact summary is given below:

Statistic	Value
LOC (all)	Min 5, Median 30, 90th percentile 2,500, Max 8,577
Slice size	Min 1, Median 12, 90th percentile 200, Max 450
Number of functions	Min 1, Median 1, 90th percentile 8, Max 25

These statistics suggest a benchmark that is structurally heterogeneous in a deliberate way: most instances are compact, but the upper tail includes large, multi-function programs whose slices are correspondingly larger.

5. Evaluation methodology

Evaluation on SliceBench is performed at the statement level by comparing an automatically generated slice $S$ against the ground truth $G$. The benchmark defines the standard confusion components as follows: $TP = |S \cap G|$, $FP = |S \setminus G|$, $FN = |G \setminus S|$, and $$TN = \text{total statements in program} - TP - FP - FN$$ (Chang et al., 25 Jul 2025).

The reported metrics are precision, recall, $F_1$ score, accuracy, and Exact-Match Accuracy (Acc-EM). Precision is defined as

$\kappa = 0.927$0

recall as

$\kappa = 0.927$1

and accuracy as

$\kappa = 0.927$2

The benchmark also defines Exact-Match Accuracy as the fraction of instances where $\kappa = 0.927$3 exactly equals $\kappa = 0.927$4 as sets of statements (Chang et al., 25 Jul 2025).

The printed form of the $\kappa = 0.927$5 definition in the supplied benchmark summary is:

$\kappa = 0.927$6

and this is the formula as given in the source summary (Chang et al., 25 Jul 2025).

This evaluation setup emphasizes exact statement inclusion and exclusion rather than semantic equivalence under execution. A plausible implication is that SliceBench is especially suitable for comparing tools that emit explicit source-level slices and that its metrics reward both completeness and conciseness.

6. Example instance and benchmark semantics

The benchmark summary includes a small CodeNet-style Java example, MyMax.java, where the slicing criterion is the printed value at line 12:

$\kappa = 0.927$7

The annotated ground-truth slice consists of the import, the global list initialization, the max method, and the main method with the call to max(nums):

$\kappa = 0.927$8

The benchmark commentary notes that irrelevant code is dropped and only the import, global data, the max method, and its invocation remain (Chang et al., 25 Jul 2025). This example makes the benchmark’s semantics concrete: a valid backward slice includes all statements that influence the criterion through data flow, control flow, or inter-procedural propagation, while still allowing syntactic support declarations when necessary.

7. Usage, strengths, and limitations

SliceBench is presented for three uses: as a zero-shot evaluation suite for static slicers, including traditional PDG-based systems and LLM-powered systems; as a fine-tuning corpus via cross-validation splits for learning-based slicing models; and for downstream tasks such as debugging and bug-localization, where high-precision slices translate into better fault isolation (Chang et al., 25 Jul 2025).

The benchmark itself is published unlabeled by split, and users may define training, validation, and test partitions as they wish. In the experiments reported with the associated paper, splitting is used only for NS-Slicer Pro, which requires fine-tuning, via 10-fold cross-validation with each fold divided 8:1:1 for train, validation, and test. No split is enforced for evaluating SliceMate under zero-shot GPT-4o inference (Chang et al., 25 Jul 2025).

Several limitations are stated explicitly. Although the GitHub subset introduces realistic multi-file and multi-procedure slices, it is limited to 200 programs. Rare language constructs such as dynamic class-loading and multi-threading are under-represented. Only Java and Python are included, with extension to C/C++ or JavaScript left as future work. Only backward slicing is annotated; forward slices for impact analysis are not covered (Chang et al., 25 Jul 2025).

The benchmark summary also notes that at 2,200 instances, SliceBench is an order of magnitude larger than earlier manually verified static slicing benchmarks, yet still small for end-to-end ML fine-tuning (Chang et al., 25 Jul 2025). This suggests a dual interpretation. On one hand, SliceBench is comparatively large within manually curated slicing evaluation. On the other hand, its scale remains limited relative to data-hungry training paradigms, especially if one aims to cover broader language ecosystems or rarer semantic phenomena.

Taken together, these properties position SliceBench as a rigorously annotated benchmark with verified statement-level ground truth, substantial variation in code size, and a mixed distribution of contest-style and real-world programs. Within the associated study, it serves as the empirical basis for evaluating SliceMate and for comparing static slicing methods under a common, manually adjudicated standard (Chang et al., 25 Jul 2025).