---
title: 'Decompile-Eval: Decompiler Benchmark'
url: https://www.emergentmind.com/topics/decompile-eval
type: topic
---

# Decompile-Eval: Decompiler Benchmark

Decompile-Eval is a unifying evaluation methodology and benchmark suite formulated to rigorously assess the functional correctness, readability, and practical usability of automated decompilation systems across diverse programming languages, instruction sets, compiler optimizations, and decompilation paradigms. It is foundational in modern research on LLM-powered, neurosymbolic, and classical static-analysis-based decompilers, providing standardized metrics, datasets, and protocols that enable reproducible and quantitative comparison between tools and algorithms in academic and applied reverse engineering.

## 1. Definition and Scope

Decompile-Eval specifies both the evaluation task—reconstructing high-level source code from compiled binaries or assembly such that the result is syntactically correct and semantically equivalent—and the protocol by which success is measured. Its influence extends to native binaries (C/C++, Rust), virtual machine code (WASM), DNN executables, and quantum circuits, and enables comparison of neural, rule-based, hybrid, and human-oriented decompiler workflows [2403.05286][2505.11340][2502.12221][2507.18792][2509.22114][2509.14646].

## 2. Core Metrics and Evaluation Protocols

Decompile-Eval introduces and standardizes a family of metrics that directly quantify decompilation quality.

- **Re-Executability Rate ($R_{exec}$):** 
  $$
  R_{exec} = \frac{\#\;\mathrm{outputs\ that\ compile\ \&\ pass\ all\ unit\ tests}}{\mathrm{total\;samples}}
  $$
  This is the primary functional metric: correct decompilation requires not just recompilation but behavioral equivalence.

- **Re-Compilability Rate ($R_{comp}$):**
  $$
  R_{comp} = \frac{\#\;\mathrm{outputs\ that\ compile}}{\mathrm{total\;samples}}
  $$
  Used to separate syntactic from semantic errors.

- **Coverage Equivalence Rate (CER):**
  Runtime side-effect comparison: proportion of decompiled functions whose execution under test corpus matches the branch coverage profile of the original [2505.11340].

- **Readability Metrics:**
  - **Relative Readability Index (R2I):** Weighted function of AST-derived features normalized to $[0,1]$, indicating idiomatic similarity to human-written code [2509.22114][2505.12668].
  - **Edit Similarity:** $1 - \mathrm{Lev}(d, s)/\max(|d|, |s|)$, with $\mathrm{Lev}$ the Levenshtein distance between decompiled and reference code.
  - **Human or LLM-Judge Elo Ratings:** Pairwise code comparison on understandability/structure, aggregated into rankings [2505.11340].

- **Advanced Measures (context-specific):**
  - **AST Edit Distance and Cyclomatic Complexity Similarity** for functional/structural preservation (notably in WASM domain) [2406.11346].
  - **Quantitative program recovery (token/program accuracy)** in early neural models [1906.12029].

## 3. Datasets, Task Construction, Benchmarks

Decompile-Eval systematically incorporates a range of test data varying in complexity and realism:

| Dataset        | Domain(s)         | Function Count | Characteristics                     |
|:--------------:|:----------------:|:-------------:|:------------------------------------|
| HumanEval      | C/C++, Rust       | 164           | Algorithmic, unit-tested, handpicked|
| MBPP           | C/C++             | 923           | Algorithmic, multi-case test harness|
| ExeBench       | x86, ARM, etc.    | 4000+         | System-level, diverse optimization  |
| GitHub2025     | C, C++            | 60,000        | Real-world OSS, post-2025, unseen   |
| Real-world OSS | C, C++            | 23,400        | Fuzz-extracted, multi-O-level       |

Each source is compiled at multiple optimization levels (O0–O3), often with debug info for alignment, and evaluated in both single-function and multi-file settings. For WASM, HumanEval-X and MBXP adapt the methodology to the C++ and web domains [2406.04568].

## 4. Methodological Best Practices

Best practices for Decompile-Eval, as established in leading studies, include:

- **End-to-End Harnessing:** Automated routines compile, disassemble, decompile, recompile the result, and run all reference test cases with sandboxing [2403.05286][2505.11340].
- **Strict Partitioning:** Rigidly separate training, retrieval, and evaluation sets to mitigate data leakage.
- **Optimized Compilation Matching:** Always apply the same toolchain, flags, and optimization level as the ground-truth binary in both training and evaluation.
- **Robust Error Taxonomies:** Catalog failure types (syntax, assertion, type, runtime, coverage misses) for actionable diagnosis.
- **Human Study Integration:** Employ controlled user studies or LLM-based judges for subjective readability when appropriate.
- **Configurable Pipelines:** Expose fine-grained toggles (e.g., control-flow handling, cast elision) for analyst-in-the-loop experiments [2205.06719].

## 5. Impact on System Design and Research Progress

Decompile-Eval has catalyzed the development and objective assessment of fundamentally new decompiler designs:

- **LLM-Driven and Hybrid Systems:** LLM4Decompile, SK²Decompile, ReF Decompile, SALT4Decompile, and context-augmented approaches (ICL4Decomp, sc²dec/FAE) have shown >16–70% relative gains in $R_{exec}$ vs. previous SOTA [2403.05286][2509.22114][2502.12221][2509.14646][2406.17233][2511.01763].
- **Neurosymbolic and Two-Stage Pipelines:** Integration of static analysis with CoT prompting or explicit logic-tree extraction (StackSight, SALT4Decompile) improves both correctness and code interpretability, especially for WASM and obfuscated binaries [2406.04568][2509.14646].
- **Robustness Evaluation:** Obfuscation, cross-architecture, and quantized/optimized code scenarios inform generalizability and are incorporated in Decompile-Eval [2509.14646][2509.06402][2406.11346].
- **Human and Automated Readability Measures:** Multi-dimensional metrics ensure research aligns tool development with both functional and analyst-centric needs [2505.11340][2505.12668][2205.06719].

## 6. Limitations, Extensions, and Controversies

Key constraints and open issues in current Decompile-Eval practice include:

- **Context Sensitivity:** Scaling to global compilation, link-time optimizations, or highly inlined code remains limited; current toolchains focus on function-level equivalence [2511.01763][2505.12668].
- **Human Studies vs. Automated Metrics:** Some studies suggest subjective readability diverges from semantic correctness, motivating hybrid evaluation schemes [2505.11340][2205.06719].
- **Architecture and ABI Coverage:** While x86 and C predominate, ongoing work expands Decompile-Eval to ARM, AArch64, RISC-V, Rust, WASM, DNN executables, and quantum circuits [2507.18792][2406.11346][2509.06402][2504.08310].
- **Optimizations and Obfuscation Handling:** Aggressive compiler optimizations and adversarial obfuscation introduce semantic drift and evaluation ambiguity; advanced symbolic/static analysis and hybrid dynamic/LLM strategies are under active exploration [2509.14646][2507.18792].
- **Potential for Hallucinations:** Pure LLM-based approaches can invent logic not present in the binary; alignment strategies (debug info, logic trees, explicit context) are considered best practice [2502.12221][2509.22114][2406.04568].
- **Reproducibility and Data Leakage:** Careful curation of evaluation sets (post-date training, debug info, no overlap) is essential to valid comparison [2505.12668].

## 7. Outlook and Future Directions

Decompile-Eval is continually evolving to address the needs of emerging domains (e.g., transformer-based DNNs, quantum circuits), new programming languages, and more realistic attack/defense scenarios. Anticipated directions include:

- **Closed-Loop Self-Debugging and Repair Loops:** Integrating LLM error correction via re-compilation feedback [2509.14646][1906.12029].
- **Semantic-Aware Metric Development:** Moving beyond token or AST similarity to richer models of behavioral and dataflow equivalence [2507.18792][2509.06402].
- **Multi-Function and Project-Scale Decompilation:** Handling global program properties, linking, and cross-file optimization artifacts in the evaluation loop.
- **Human-in-the-Loop and Analyst-Tunable Workflows:** Configurability and human-centric evaluation to bridge practical reverse engineering with automated pipelines [2205.06719][2505.11340].

By anchoring evaluation in rigorous, quantitative, and reproducible protocols, Decompile-Eval has become a de facto standard and reference point for the assessment and advancement of modern decompilation research and tooling.

Source: https://www.emergentmind.com/topics/decompile-eval