DE-Benchmark: Evaluating Java Debloating
- DE-Benchmark is a micro-benchmark that measures Java debloating by balancing soundness (preservation of required constructs) and precision (removal of bloated code) using 59 test cases over 13 language features.
- It utilizes a manually curated JSON ground truth to validate classes, methods, and fields at class, method, and field levels, enabling precise diagnostics of debloating failures.
- The benchmark exposes specific failure modes in tools like Deptrim, JShrink, and ProGuard, with notable challenges in handling dynamic class loading, annotations, and reflective operations.
DE-Benchmark, also described as Deblometer, is a micro-benchmark for evaluating Java debloating tools by measuring the central trade-off between soundness and precision. In this setting, soundness means never removing code that is still needed at runtime, while precision means removing all code that is truly unnecessary. The benchmark was introduced because prior evaluations were often too application-oriented, too coarse-grained, or lacked a clear oracle for which Java constructs should remain. DE-Benchmark addresses that gap with 59 test cases, 13 Java language features, and a manual ground truth for classes, methods, and fields, enabling direct measurement of whether a tool preserves required constructs and removes bloated ones (Klauke et al., 23 Oct 2025).
1. Origin and evaluation objective
DE-Benchmark was created to evaluate debloating correctness rather than only code-size reduction or aggregate runtime behavior. The motivating concern is that Java programs make extensive use of dynamic and implicit mechanisms such as reflection, annotations, lambdas, dynamic class loading, serialization, and related features. A debloating tool may therefore fail in two opposite ways: it may remove too much and break the program, or it may keep too much and fail to reduce attack surface (Klauke et al., 23 Oct 2025).
The benchmark formalizes this tension using two axes. Soundness asks whether required constructs are preserved. Precision asks whether unnecessary constructs are removed. This framing is important because debloating is not treated as a pure minimization problem. A smaller output JAR is not necessarily better if it has lost required runtime behavior, and a safer output is not necessarily better if it retains large amounts of bloated code.
The benchmark was also designed to make failures diagnosable. Instead of relying on large end-to-end applications where the cause of failure is hard to localize, it isolates specific language features so that unsoundness or imprecision can be attributed to particular Java mechanisms. This makes the benchmark feature-oriented rather than purely aggregate in its diagnostic structure.
2. Benchmark structure and feature coverage
DE-Benchmark contains 59 test cases, grouped into 13 Java language features, and evaluated at three debloating levels: class, method, and field (Klauke et al., 23 Oct 2025).
Each feature has its own bloated JAR. Each test case includes both required and bloated classes, methods, and fields. A main class invokes all test cases so that the JARs are executable and not trivially pruned away by reachability-based tools. A validation harness then compares the tool’s output JAR against the expected ground truth.
| Java language feature | Test cases |
|---|---|
| Abstract classes | 6 |
| Annotations | 7 |
| Deserialization | 2 |
| Dynamic class loading | 2 |
| Exception handling | 4 |
| Externalization | 2 |
| Generics | 7 |
| Interfaces | 4 |
| Lambdas | 4 |
| Overloading | 6 |
| Overriding | 4 |
| Reflection | 6 |
| Serialization | 5 |
This organization is intentionally fine-grained. Each case is designed around one language feature, so a failure can be tied to that feature rather than being obscured by unrelated application complexity. A plausible implication is that the benchmark is meant less as a proxy for average production performance than as a controlled instrument for exposing known failure modes in Java debloating.
3. Ground truth and formal metrics
A defining component of DE-Benchmark is its manually curated oracle. The ground truth is stored in JSON, organized by debloating level—[CLASS](https://www.emergentmind.com/topics/colorado-learning-attitudes-about-science-survey-class), METHOD, and FIELD—and within each level into required and bloated entries. Class entries include package name and class name. Method entries include declaring type/class, method name, return type, and parameter list. Field entries include declaring class and field name. The paper states that this ground truth was manually created and independently verified by three authors (Klauke et al., 23 Oct 2025).
The benchmark treats the debloated JAR as output and compares its remaining program constructs with the ground truth. For each debloating level, the harness counts:
- TP (true positives): required constructs that remain
- FP (false positives): bloated constructs that remain
- FN (false negatives): required constructs that were removed
The paper defines the metrics as
Under this definition, soundness answers “Of the required constructs, how many were preserved?” and precision answers “Of the retained constructs, how many were actually required?” The benchmark reports these metrics separately for classes, methods, and fields, and then summarizes them per Java feature. This separation matters because some tools may appear adequate at class level while failing at method or field level, especially for reflection-heavy or dynamically loaded constructs.
4. Evaluated tools and comparative behavior
The benchmark evaluates three Java debloating tools: Deptrim (v0.1.2), JShrink, and ProGuard (v7.7). The paper initially considered J-Reduce, but excluded it because it produced empty JARs for the test cases. In terms of strategy, ProGuard relies on static analysis, while Deptrim and JShrink combine static and dynamic techniques, although JShrink’s default configuration is largely static (Klauke et al., 23 Oct 2025).
The paper’s high-level result is that all three tools are unsound on at least some test cases, and all three also show precision weaknesses. Their failure profiles differ. Deptrim is the most conservative tool of the three. It generally has the best soundness, often preserves required constructs successfully, and performs especially well on annotations and generics compared with the others. Its main weakness is lower precision: it often leaves bloated code behind, favoring safety over aggressiveness.
JShrink is described as the most fragile tool in the study. It has limited support for annotations and produced corrupted JARs in several cases, especially for annotations, lambdas, and interfaces. The corruption was traced to incomplete annotation removal: annotation references were removed from the constant pool, but dangling references to deleted constant-pool entries remained. Some JARs could not even be parsed by SootUp, and corrupted class files had to be manually removed to continue evaluation.
ProGuard is more aggressive than Deptrim. It usually removes more bloated code and often achieves better precision than Deptrim, but it is more likely to remove required constructs and therefore has lower soundness in several cases. The paper reports unsoundness for ProGuard especially on annotations, deserialization, overloading, generics, and method-level behavior in dynamic settings.
5. Feature-specific failure modes and behavioral consequences
The clearest benchmark-wide failure mode is dynamic class loading. The paper states that all evaluated tools fail on dynamic class loading. Required classes loaded at runtime are incorrectly removed because static analysis cannot fully infer them. The effects include class-level unsoundness, incorrect field removal, method-level failures, and runtime crashes or altered behavior. In the reported results, dynamic class loading is particularly severe at the field level, where soundness and precision both drop to 0% for all tools (Klauke et al., 23 Oct 2025).
Annotations are another major problem area, especially for JShrink. The benchmark exposes failures involving corrupted JARs, incomplete class-file rewriting, inability to parse or execute the resulting artifacts, and cascading effects on lambda and interface test cases that include annotation metadata.
Reflection is also difficult for static analysis. The paper reports that Deptrim reaches 100% soundness in reflection-heavy method-level cases thanks to dynamic analysis support, whereas JShrink and ProGuard perform much worse, particularly at method level, where they fail to preserve required reflective methods.
Additional weaknesses are reported for overloading, generics, serialization/deserialization, interfaces, and lambdas. These cases show that the benchmark is not limited to one pathological Java feature; it spans several mechanisms that routinely complicate reachability and retention analysis.
The paper also goes beyond metric summaries and executes unsound outputs. It reports 1 unsound JAR from Deptrim, 5 unsound JARs from JShrink, and 7 unsound JARs from ProGuard. Among these, Deptrim and JShrink unsound cases caused exceptions and crashes. For ProGuard, 3 unsound JARs crashed, while the remaining 4 ProGuard JARs changed runtime behavior. One particularly important observation is that when an overriding method is removed, Java’s dynamic dispatch may fall back to a superclass implementation, causing a semantic change rather than a crash. That failure mode is especially significant because the program may continue to run while no longer preserving original behavior.
6. Limitations, scope, and significance
The paper identifies several limitations. First, the test cases are artificially constructed, so they may not fully reflect real-world applications, even though they are useful for exposing feature-level weaknesses. Second, coverage is intentionally selective: the benchmark does not cover every Java language feature, but focuses on features known to cause unsoundness or imprecision. Third, the manual oracle carries a human error risk, although this was mitigated by independent verification from three authors. Fourth, results remain sensitive to tool configuration, even though the authors contacted tool developers to use appropriate settings (Klauke et al., 23 Oct 2025).
Within those limits, DE-Benchmark’s significance lies in making debloating correctness directly measurable. It can pinpoint which Java features a tool cannot handle, distinguish unsoundness from imprecision, expose corrupted outputs and runtime failures, and guide systematic tool improvement. Rather than presenting debloating as a single scalar reduction objective, it treats it as a correctness-sensitive transformation problem in which the preservation of required classes, methods, and fields is as important as removal of bloated ones.
The benchmark’s central lesson is that dynamic Java features—especially dynamic class loading and annotations—remain major obstacles to reliable debloating. In that sense, DE-Benchmark functions as a feature-oriented oracle for Java debloating research: it measures not merely whether code disappears, but whether the right code remains.
