---
title: 'Decompile-Bench: Million-Scale Binary Functions'
url: https://www.emergentmind.com/topics/decompile-bench-dataset
type: topic
---

# Decompile-Bench: Million-Scale Binary Functions

Decompile-Bench is the first open-source, million-scale corpus of precisely paired binary and source functions for C and C++, systematically curated from real-world permissively licensed software with rigorous provenance, deduplication, and function boundary recovery. It is designed for empirical study and advancement of large language model (LLM) based binary decompilation, offering scale, fidelity, and evaluation benchmarks that address the limitations of synthetic or partial datasets in prior art [2505.12668].

## 1. Corpus Scope, Provenance, and Licensing

Decompile-Bench comprises 2,000,000 binary–source function pairs, distilled from an initial collection of approximately 100 million binary functions (≈450 GB of compiled artifacts) [2505.12668]. The underlying source code is drawn from C and C++ repositories in the “Stack V2” collection, selected strictly for permissive licensing (MIT, BSD, Apache 2.0 as detected via ScanCode/Blue Oak Council), nontriviality (at least one star and a valid CMakeLists.txt), and public availability.

All binaries are compiled directly from these public repositories under the original licenses, with non-permissive, commercial, or system/external code strictly excluded at both source and header dependency levels. This guarantees both legal clarity and ethical use for academic research. The dataset can be freely used for research under the terms of the original permissive licenses.

## 2. Data Collection and Compile-Trace-Filter (CTF) Pipeline

The CTF pipeline ensures robust function-level matching and noise suppression across three orchestrated stages:

**2.1 Automatic Compilation (“Compile”)**  
Clang is forked and patched to forcibly embed DWARF debug information (with `-g`) and employ one of four optimization levels (`-O0` … `-O3`) on every invocation. All binaries are built using CMake-driven build systems, with missing dependencies resolved via single-shot LLM queries and recipes cached per-project. This robust environment, applied to 3,961 GitHub repositories, yields ≈85,000 binaries and ∼100 million raw binary functions.

**2.2 Binary–Source Function Matching (“Trace”)**  
DWARF debugging data provides line-level mappings, but inlined and optimized code typically fragments or reorders these source line links. Decompile-Bench’s “Source-Trace” algorithm collects, for each binary function $f_b$, the full set of DWARF-mapped source locations (`func_segment`). Using Tree-sitter, it retrieves all enclosing source functions $f_s(\ell)$ for every $\ell$ in `func_segment` and selects the candidate with maximal overlap in line numbers—restoring canonical function boundaries and grouping inlined elements.

**2.3 Noise Removal (“Filter”)**  
Precise filtering is applied in three tiers:
- **Project-scope**: discard any source not defined in the target repo (eliminates system/dependency headers, trivial getters/setters).
- **In-binary deduplication**: for multiple binary functions mapping to the same source (e.g., template instantiations), retain only the function with largest DWARF overlap.
- **Cross-binary deduplication**: apply MinHash-LSH over disassembled binary and corresponding source to remove near-duplicates globally.

This yields a final dataset of 2 million high-quality, project-rooted function pairs—retaining only approximately 2% of the initial function pool [2505.12668].

## 3. Dataset Splits and Evaluation Suite (Decompile-Bench-Eval)

Decompile-Bench provides canonical splits for training, validation, and test, based on explicit repository provenance and creation chronology.

- **Experimental protocol**: In published results, 10% (200,000 pairs) are allocated for training, with all repositories published post-2025 (designated “GitHub2025”, 121 repos, ~60,000 functions) strictly held out from training and used exclusively for final test to preclude data leakage.
- **Decompile-Bench-Eval**: The companion benchmark suite is constructed for rigorous, non-leaky, and interpretable evaluation. It comprises three disjoint sets, each compiled at all optimization levels:
  1. **HumanEval** (C/C++): 164 C/C++ problems manually adapted from the Python HumanEval benchmark plus harnesses.
  2. **MBPP** (C/C++): 200 C/C++ problems likewise hand-translated from the Python MBPP suite.
  3. **GitHub2025**: 60,000 functions extracted via the CTF pipeline (with identical filtering), from repositories introduced after 2025.

## 4. Metrics and Performance Results

Multiple metrics are employed to measure decompilation quality: correctness, readability, and textual similarity. Key definitions and absolute results are as follows.

**4.1 Re-Executability Rate (Functional Correctness)**  
Given a decompiled function $d$, the function is "re-executable" if:
$$
\forall x \in T, \quad s(x) = d(x)
$$
where $s$ is the original source and $T$ is the test input set supplied by HumanEval or MBPP. The re-executability rate is:
$$
\text{Rate} = \frac{\text{number of functions passing all tests}}{\text{total functions}}
$$
Main results (averaged over optimization levels):

| Model                    | HumanEval | MBPP    |
|--------------------------|-----------|---------|
| LLM4Decompile-End        | 16.22%    | 20.54%  |
| +Fine-tune on DCBench    | 20.89%    | 24.93%  |
| Relative improvement     | +28.8%    | +21.4%  |

**4.2 Relative Readability Index (R2I)**  
R2I ℝ ∈ [0,1] quantifies readability via AST-derived features and learned weights; higher scores reflect superior structure, indentation, and identifiers:
- LLM4Decompile-End: 60.47
- LLM4Decompile-DCBench: 73.18 (+21% relative) (GitHub2025 average, O0–O3)

**4.3 Edit Similarity**  
Defined as $1 - \frac{\text{LevenshteinDistance}(d,s)}{\max(\text{len}(d), \text{len}(s))}$, measuring normalized edit proximity. On GitHub2025 (average): LLM4Decompile-End yields 21.57%; LLM4Decompile-DCBench, 29.51% (+36.8% relative).

**4.4 Additional Metrics**  
Embedding similarity (CodeSage embeddings + cosine) and CodeBLEU (hybrid n-gram BLEU, AST subtree match, data-flow match) consistently show absolute improvements of ≈15–20% when LLM decompilers are fine-tuned on Decompile-Bench.

A plausible implication is that representational quality and functional recoverability of LLM decompilers benefit more from real-world, large-scale pairing than from synthetic or line-level benchmarks.

## 5. Data Format, Availability, and Applications

Each example in Decompile-Bench encodes:
- `asm`: disassembled binary function (with DWARF-resolved symbols removed),
- `src`: original C/C++ function (full signature and body),
- `project`: repository name, optimization, and relevant build metadata.

Public access is provided via HuggingFace (https://huggingface.co/datasets/LLM4Binary/decompile-bench) and source/metadata/Eval suite via GitHub (https://github.com/albertan017/LLM4Decompile).

Recommended research uses include:
- Sequence-to-sequence fine-tuning of any LLM, encoder–decoder, or transformer-based model for binary→source or source retrieval,
- Contrastive learning for embedding alignment and retrieval tasks,
- Method development for function boundary recovery, inlining analysis, or noise-robust source matching.

## 6. Ethics and Licensing Considerations

All content is governed by its originating permissive license. No non-permissive or commercial code is included; commercial binaries—often obfuscated—are excluded by design and not suitable for standard decompilation research. The dataset is strictly intended and recommended for academic research in fields including binary decompilation, reverse engineering, and program understanding [2505.12668].

## 7. Relation to Prior Decompilation Benchmarks and Datasets

Decompile-Bench is distinguished from earlier datasets by both its scale and methodology. Prior datasets, such as those used for Java bytecode decompiler evaluation [1908.06895], typically cover orders of magnitude fewer code units (e.g., ≈2,000–25,000 classes or functions) and are often constrained to syntactic correctness or partial semantic equivalence. Recent benchmarks like DecompileBench [2505.11340] for C/C++ focus on runtime-aware validation (Coverage Equivalence Rate) and LLM-based code understanding assessment over ~23,400 functions. In contrast, Decompile-Bench provides comprehensive binary–source alignment using real-world C/C++ code, incorporates robust inlining and optimization handling, and spans two million function pairs. This suggests that Decompile-Bench is the current largest resource enabling both large-scale model training and practical, leakage-resistant evaluation for LLM-based decompilation research.

---
**References:**  
[2505.12668]: Decompile-Bench: Million-Scale Binary-Source Function Pairs for Real-World Binary Decompilation  
[2505.11340]: DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios  
[1908.06895]: The Strengths and Behavioral Quirks of Java Bytecode Decompilers

Source: https://www.emergentmind.com/topics/decompile-bench-dataset