---
title: 'Decompile-Bench-Eval: Decompilation Benchmarks'
url: https://www.emergentmind.com/topics/decompile-bench-eval
type: topic
---

# Decompile-Bench-Eval: Decompilation Benchmarks

Decompile-Bench-Eval is a set of formal benchmark suites and evaluation protocols designed to rigorously measure the effectiveness of decompilers—systems that recover high-level source code from low-level binaries, bytecode, or stripped executable programs. Widely adopted for academia and industry, Decompile-Bench-Eval frameworks span multiple programming languages (C/C++, Java, Rust, WebAssembly, EVM, quantum circuits), support a diverse range of binary formats and compilation scenarios, and provide standardized, reproducible methodologies for functional, structural, and human-centric assessment of decompilation systems. These benchmarks anchor the comparison of commercial decompilers, end-to-end neural models, LLM-based tools, and hybrid symbolic-neural pipelines in reverse engineering, malware analysis, binary translation, and code generation contexts.

## 1. Benchmark Suite Design and Composition

Decompile-Bench-Eval encompasses both contest-style micro-benchmarks and large-scale, real-world corpora. The suite covers:

- **HumanEval and MBPP:** 164–1000 hand-written algorithmic problems (originally Python, ported to C/C++), each with unit tests. Binaries are compiled at multiple GCC/Clang optimization levels (O0–O3), producing paired (assembly, original-source) data [2406.17233][2503.07215][2505.12668].
- **ExeBench and GitHub2025:** Thousands to hundreds of thousands of functions extracted from OSS-Fuzz projects or post-2025 GitHub repos. Functions are isolated with coverage sanitization and compiled with various options, emphasizing leakage-resistance in dataset splits [2505.11340][2505.12668].
- **Java Bytecode:** 14 open-source Java projects (N=2041 classes), compiled using multiple JVM compilers, with coverage and structural diversity for bytecode decompilation tools [1908.06895].
- **Rust/EVM/WebAssembly:** Rust suite exercises generics, traits, concurrency, and error-handling at both debug and release build modes [2507.18792]. EVM benchmarks use thousands of Ethereum smart contracts compiled across mainnet and Yul/viaIR pipelines [2409.11157]. WASM benchmarks synthesize functions (DecFuzzer, PolyBenchC, CHStone) and decompile to C for comparative assessment [2411.02278].
- **Quantum Circuits:** DeQompile targets canonical quantum functions (GHZ, QFT, QPE) in parametric gate representation for reverse-engineering interpretable Qiskit programs [2504.08310].

Dataset construction typically ensures disjointness between training and evaluation splits and includes features such as DWARF debug info for source-binary alignment.

## 2. Evaluation Methodologies and Metrics

Decompile-Bench-Eval standardizes the assessment of decompilers using several rigorous metrics:

- **Re-Compilability:** Fraction of functions whose decompiled output successfully compiles (e.g., with gcc –std=c17 –O2 –c or Clang) [2406.17233][1908.06895].
- **Re-Executability:** Measures functional correctness by executing recompiled output against the original unit tests; only code passing all test cases is credited [2406.17233][2505.12668]. Formula:
  $$
  \text{Re-Executability} = \frac{ \#\{ \text{functions compiles \& pass all tests} \} }{ \#\{\text{total functions}\} } \times 100\%
  $$
- **Test Case Pass Rate (TCP):** Fraction of individual test cases passed across all functions:
  $$
  \mathrm{TCP} = \frac{1}{\sum_{i=1}^N T_i} \sum_{i=1}^N \sum_{j=1}^{T_i} \mathbf{1}[ out_{\mathrm{gen}}^{(i,j)} = out_{\mathrm{ref}}^{(i,j)} ] \times 100\%
  $$
- **Structural Readability and Similarity:**
  - **Edit Similarity:** Normalized Levenshtein edit distance between generated and reference code [2305.12520][2505.12668].
  - **Relative Readability Index (R2I):** Weighted AST feature score capturing nested depth, identifier length, and idiomatic structures [2505.12668][2509.22114].
  - **Tree-Edit Distance Score (TED):** Normalized tree-edit operations to reconcile code ASTs [2411.02278].
- **Semantic Equivalence:** Modulo inputs (Java), or via program dependence graph isomorphism (neural decompilation) [1908.06895][1905.08325].
- **Other Dimensions:** Variable/type recovery accuracy, identifier naming fidelity, code coverage equivalence (CER), recompilation success rate (RSR), and LLM-as-Judge human-centric code assessment [2503.07215][2505.11340][2507.18792].

Evaluations frequently incorporate ablations, e.g., end-to-end vs. step-by-step alignment, CFG/data-mapping prompt augmentation, or reward schemes for structure and identifier phases [2406.17233][2503.07215][2509.22114].

## 3. Notable Benchmark Outcomes and Comparative Analysis

Benchmarking reveals pronounced advances in LLM-driven decompilation, especially with structured context, fine-grained alignment, and quality-aware fine-tuning:

| Model/Method               | Re-Exec (HumanEval, O0–O3) | Notable Metric/Comment              |
|---------------------------|----------------------------|-------------------------------------|
| GPT-4o (prompt)           | 14.6%                      | Baseline general-purpose LLM [2406.17233] |
| DeepSeek-chat             | 6.6%                       | LLM baseline                        |
| llm4decompile-6.7B        | 47.68%                     | Prior SOTA                          |
| +FAE                      | 52.28%                     | Statement-level alignment [2406.17233] |
| +sc2dec                   | 51.52%                     | Self-constructed context [2406.17233] |
| +FAE+sc2dec               | 55.03%                     | SOTA with both alignment/context [2406.17233] |
| ReF Decompile             | 61.43% (SOTA)              | Relabeling + Function Call strategies [2502.12221] |
| SK2Decompile              | 69.00% (SOTA)              | Two-phase RL with structure+identifier [2509.22114] |
| SALT4Decompile            | TCP=70.4% (SOTA)           | Source-level logic tree abstraction, robust to obfuscation [2509.14646] |
| SLaDe (x86/O3)            | 66%                        | Transformer + type inference (ExeBench) [2305.12520] |
| CodeInverter Suite        | 41.5% (CIM-1.3B)           | CFG/data-mapping augmented prompts (HumanEval-64) [2503.07215] |
| Ghidra (rule-based)       | 20.12%                     | Industry decompiler                 |

LLM-based tools that incorporate explicit control-flow, source-binary alignment, or dual-phase RL pipelines consistently outperform vanilla LLMs and commercial decompilers in functionality and readability. However, commercial engines (Hex-Rays, Ghidra) still lead in coverage equivalence and recompliation robustness for production binaries [2505.11340].

## 4. Technical Innovations in Benchmark-Driven Decompilation

Benchmarked research has introduced several algorithmic and workflow innovations:

- **Self-Constructed Context Decompilation (sc²dec):** In-context exemplars generated from first-pass model outputs and recompiled/disassembled code to bridge compiler/version gaps [2406.17233].
- **Fine-grained Alignment Enhancement (FAE):** DWARF-driven pairing of assembly and source code statements, enabling stepwise alignment objectives in neural fine-tuning [2406.17233].
- **Relabeling and Function Call Preprocessing:** Explicit label mapping for control-flow and variable inference, recovering jump targets and data constants [2502.12221].
- **Two-phase RL Pipelines:** Decoupling structure recovery (generic placeholders, compiler-driven rewards) and identifier naming (semantic embedding similarity rewards) [2509.22114].
- **Source-level Abstract Logic Tree (SALT):** Hierarchical abstraction of stable control-flow features, including loop back-edges, providing LLM-guidance for semantic recovery and resilience to obfuscation [2509.14646].
- **CFG/Data Mapping Prompt Engineering:** Augmenting input with control-flow graphs and variable-table mappings for improved function recovery and readability [2503.07215].
- **D-SCORE and CodeAlign Metrics:** Integrated symbolic, semantic, and readability scoring enforcing accuracy before readability feedback, and fine-grained SSA instruction-level alignment [2506.10125][2501.04811].

## 5. Language and Domain Coverage

Decompile-Bench-Eval spans traditional binary formats and domains:

- **C/C++ and Assembly (x86, ARM, MIPS, x86-64/32):** Extensive coverage including ExeBench, HumanEval, MBPP, and GitHub2025 functions with source-binary mapping across optimization levels [2305.12520][2505.12668].
- **Java Bytecode:** Syntactic correctness, distortion, and semantic equivalence measured across a large multi-project corpus with multiple JVM compilers [1908.06895].
- **Rust:** Feature-oriented benchmarks (generics, traits, concurrency, error semantics) at debug and release builds, showing marked fidelity drops under optimization [2507.18792].
- **WebAssembly:** DecFuzzer synthetic programs, PolyBenchC/CHStone, with comparison to wasm2c, w2c2, wasm-decompile, Ghidra, RetDec; metrics include Halstead effort and AST similarity [2411.02278].
- **Ethereum Smart Contracts:** EVM and Yul pipelines, emphasizing block completeness, imprecision, and external-call/event coverage; advanced static-analysis context (Shrnkr) significantly outperforms prior symbolic/execution engines [2409.11157].
- **Quantum Circuits:** Genetic-programming decompilation of OpenQASM to high-level Qiskit with functional, structural, and interpretability scoring [2504.08310].

## 6. Limitations, Open Challenges, and Recommendations

Benchmark-driven methodology exposes remaining challenges:

- **Scale and Context:** Most benchmarks focus on per-function recovery; multi-file and large-scale whole-program decompilation is prohibitively costly for current models [2505.12668].
- **Optimization and Semantic Loss:** Aggressive compiler optimizations (inlining, flattening, dead-code elimination) systematically obscure high-level semantics, particularly in Rust and C binaries [2507.18792].
- **Type and Name Recovery:** Nontrivial loss of identifier fidelity and type structure persists, especially in large, complex, or deeply optimized code; explicit type inference and data-mapping augmentation are ongoing research [2305.12520][2503.07215].
- **Human Judgment and Metric Correlation:** While metrics such as R2I and LLM-as-Judge show high agreement with expert evaluations, approximation and domain-specific proxies remain—further user studies and hybrid metrics are recommended [2505.11340][2509.22114][2509.14646].
- **Obfuscation and Robustness:** Control-flow obfuscation (bogus CF, flattening, instruction substitution) dramatically degrades standard methods; explicit logic tree modeling and control-flow abstraction mitigate effects [2509.14646].
- **Legal and Licensing Constraints:** Only permissively licensed code included; closed/proprietary binaries excluded, potentially limiting relevance for some applications [2505.12668].

Prominent recommendations for future Decompile-Bench-Eval suites include richer feature categories (cross-language, concurrency, obfuscation), broader metric sets (semantic symbolic equivalence, panic-path recovery, trait preservation), and standardized artifact releases for reproducibility and broad adoption [1908.06895][2411.02278][2507.18792][2505.12668].

## 7. Impact and Future Directions

Decompile-Bench-Eval has become the bedrock of comparative decompiler research, enabling reproducible, multi-dimensional assessment of symbolic, analytic, neural, and LLM-based systems. Modern findings indicate that context-driven, structurally aligned, and RL-optimized pipelines push the frontier of decompilation, yielding dramatic advances in both functional fidelity and code readability. The modularity and extensibility of these frameworks continue to support emerging domains (quantum, Rust, WASM, EVM), algorithmic improvements (two-phase RL, hybrid neural-symbolic), and standardization of evaluation protocols.

Ongoing research will likely address multi-function/project recovery, richer type/name/semantic abstraction, deeper metric instrumentation, enhanced obfuscation resilience, and improved user-centric assessments to further advance the reliability, accuracy, and utility of decompilation science.

Source: https://www.emergentmind.com/topics/decompile-bench-eval