AutoDecompiler: RL-Based Binary Decompilation

Updated 5 July 2026

AutoDecompiler is a decompilation LLM that treats binary-to-source translation as iterative refinement using reinforcement learning.
It accepts x86_64 assembly or pseudo-code input and employs compilation, execution, and I/O testing feedback to ensure behavioral consistency.
By internalizing multi-turn corrections, it improves re-executability and structural accuracy over successive decompilation iterations.

AutoDecompiler is a decompilation-specialized LLM family trained with reinforcement learning for feedback-driven multi-turn binary decompilation, designed to treat decompilation as iterative refinement rather than single-turn code generation (Liu et al., 15 Jun 2026). It accepts x86_64 assembly or decompiler-produced pseudo-code and aims to produce readable, compilable, executable C-like functions while using compilation, execution, and input/output testing feedback to improve behavioral re-executability (Liu et al., 15 Jun 2026). In the broader research landscape, it belongs to a line of work that treats decompilation not as exact inversion of compilation, but as constrained reconstruction under information loss, verification, and ambiguity (Katz et al., 2019, Wong et al., 2023).

1. Position in the decompilation literature

Automatic decompilation has evolved from rule-based structuring and type recovery toward hybrid systems that combine symbolic analyses, learned translation, and dynamic validation. Katz, Olshaker, Goldberg, and Yahav’s "Towards Neural Decompilation" framed decompilation as verified neural inverse compilation, using compilation for supervision and recompilation plus program-dependence-graph isomorphism for validation (Katz et al., 2019). Later work split along several lines: retargetable plain-text translation across source languages (Hosseini et al., 2022), post-decompilation semantic repair (Wong et al., 2023), human-centric readability optimization (Enders et al., 2022), and dynamically validated recompilation against original binaries (Cui et al., 16 Mar 2026).

A recurrent misconception in this literature is that a decompiler should be judged primarily by textual similarity to original source. Multiple papers reject that criterion. Decompiled code may be readable yet semantically wrong, or textually different yet behaviorally equivalent, which is why recompilability, test-based execution, structural similarity, and human comprehension recur as distinct targets rather than a single metric (Katz et al., 2019, Wong et al., 2023, Thurnherr et al., 2024).

System	Input	Principal mechanism
TraFix	LLVM IR or x86 assembly	NMT plus recompilation and PDG-based structural equivalence
DecGPT	IDA-Pro output	Compiler-in-the-loop and ASAN-guided repair
PCodeTrans	Hex-Rays output	In-situ substitution and BP-Diff
AutoDecompiler	x86_64 assembly or IDA pseudo-code	RL-trained multi-turn refinement with program feedback

Within this lineage, AutoDecompiler is distinguished by making iterative repair a learned behavior rather than an external orchestration layer. Earlier systems already showed the value of feedback; AutoDecompiler internalizes that loop through multi-turn RL, progress-aware trajectory rewarding, and turn-aware advantage reweighting (Liu et al., 15 Jun 2026).

2. Representations, preprocessing, and alignment

A central theme in automatic decompilation is that input representation determines how much recoverable semantics reaches the model. "Beyond the C" deliberately treated assembly and source as plain text with BPE tokenization to maximize retargetability across Go, Fortran, OCaml, and C, avoiding parsers and ASTs where possible (Hosseini et al., 2022). That strategy reduced language-specific engineering, but it also left the model responsible for inferring control structure and semantics from minimally structured input.

ReF Decompile argues that excessive simplification of assembly is itself destructive. Its Relabelling strategy replaces jump targets with labels such as L1 and memory references with labels such as D1, preserving control-flow clarity and memory-reference identity while removing irrelevant absolute addresses (Feng et al., 17 Feb 2025). Its Function Call strategy then lets the model request values behind labeled memory accesses, using inferred types such as string, float, or double to retrieve constants from the binary, especially from .rodata (Feng et al., 17 Feb 2025). This directly targets two sources of failure in end-to-end decompilation: loss of branch topology and loss of literal values.

Fine-grained Alignment Enhancement takes a different route. By compiling C with DWARF, disassembling with objdump -S, and reorganizing source-interleaved disassembly into aligned assembly/source supervision, it converts implicit correspondence into a step-by-step training signal (Feng et al., 2024). The associated sc $^2$ dec method then exploits a decompilation-specific property at inference time: if a first-pass decompilation is compilable, it can be recompiled and disassembled to form a self-constructed in-context example for a second pass (Feng et al., 2024). This suggests that the compiler can serve not only as an evaluator but also as an inference-time context generator.

Language-specific binaries stress these representation choices further. The Rust study shows that decompilers tuned to C/C++ assumptions tend to emit low-level C-like pseudocode that loses the semantics of Rust abstractions, especially after optimization (Zhou, 24 Jul 2025). In that benchmark-driven evaluation with Ghidra 10.3.3 on Mach-O ARM64 binaries built by rustc 1.71.1, generic types, trait methods, Result/Option error handling, and complex pattern matching were especially damaging, and average decompilation scores were generally between 1.2 and 1.4 out of 8 across programs (Zhou, 24 Jul 2025). This suggests that an AutoDecompiler cannot rely on structural lifting alone when the source language encodes semantics through high-level layouts, niche encodings, and monomorphization.

AutoDecompiler formalizes decompilation as a sequential decision problem. At turn $t$ , the state is

$s_t = (x, y_{<t}, f_{<t}),$

where $x$ is assembly or pseudo-code input, $y_{<t}$ are prior decompilation outputs, and $f_{<t}$ are prior diagnostic feedback messages; the action is a full decompiled function $y_t \sim \pi_\theta(\cdot \mid s_t)$ (Liu et al., 15 Jun 2026). The environment validates each turn through C-like validity checking, recompilation, execution, and I/O testing, then emits both feedback and reward (Liu et al., 15 Jun 2026).

Its reward is explicitly multi-dimensional. The execution-oriented component is piecewise:

$R_{\mathrm{exe}}(y_t)= \begin{cases} -1, & \text{recompilation fails},\ -0.6, & \text{runtime error},\ -0.3, & \text{partial I/O mismatch},\ 1, & \text{all I/O tests pass}. \end{cases}$

This is combined with C-like validity, AST-subtree syntactic reward, and normalized data-flow semantic reward using weights $w_C=0.5$ , $w_{\mathrm{exe}}=1$ , $t$ 0, and $t$ 1 (Liu et al., 15 Jun 2026). Progress-aware trajectory rewarding adds an improvement term across turns, and turn-aware advantage reweighting redistributes credit to the turns that actually improved the trajectory (Liu et al., 15 Jun 2026).

This architecture generalizes earlier feedback-driven repair systems. DecGPT already used a static augmenting stage to make IDA-Pro output compilable and a dynamic repairing stage to fix runtime defects using compiler diagnostics, ASAN reports, and output mismatches (Wong et al., 2023). PCodeTrans went further by compiling repaired functions into dynamic libraries, hot-swapping them into the unmodified original binary through an in situ substitutable engine, and collecting Breakpoint-Matched Differential Tracing to localize semantic divergence under authentic process context (Cui et al., 16 Mar 2026). D-LiFT similarly insisted that readability gains must be conditioned on preserved correctness, using D-SCORE to penalize syntax and semantic errors before awarding readability improvements, and then training its backend with GRPO (Zou et al., 11 Jun 2025).

A second misconception is that readability and correctness can be optimized independently. Several systems show the opposite. DecGPT’s plain LLM baseline often compiled by introducing casts or deleting problematic code, while PCodeTrans and D-LiFT both treat such behavior as semantic regression rather than improvement (Wong et al., 2023, Zou et al., 11 Jun 2025). AutoDecompiler inherits that lesson by weighting execution correctness above auxiliary syntactic and semantic proxies (Liu et al., 15 Jun 2026).

4. Evaluation regimes and empirical results

Modern decompilation evaluation has shifted from surface similarity toward executable validation. Decompile-Eval measures re-compilability and re-executability on HumanEval-derived C functions compiled at O0 through O3, and it is the basis for several recent LLM studies (Feng et al., 2024). On that benchmark, llm4decompile-6.7b-v1.5 improved from 47.68 to 52.28 average re-executability with Fine-grained Alignment Enhancement, and to 55.03 when FAE was combined with sc $t$ 2dec (Feng et al., 2024). ReF Decompile reported a new state-of-the-art average re-executability of 61.43 with average readability 3.69, outperforming LLM4Decompile-Ref at the same 6.7B scale and improving over FAE Decompile by 10.36 in re-executability (Feng et al., 17 Feb 2025).

AutoDecompiler’s strongest gains appear when iterative refinement is learned rather than externally imposed. On HumanEval, AutoDecompiler-E2E 6.7B reached 63.11 average Re-exe, while AutoDecompiler-Pscode 6.7B reached 70.58 and the 30B pseudo-code model reached 75.30; construction-stage analysis showed a progression from 45.73 for the vanilla DeepSeek-Coder pseudo-code model to 65.09 after SFT, 68.75 after 1-turn RL, and 70.58 after 3-turn RL (Liu et al., 15 Jun 2026). The same paper reports that removing compilation/execution reward reduced HumanEval Re-exe for AutoDecompiler-E2E 1.3B from 53.51 to 50.07, while removing progress-aware trajectory rewarding or turn-aware advantage reweighting reduced it to 50.30 and 50.46, respectively (Liu et al., 15 Jun 2026).

Where whole-binary fidelity is needed, PCodeTrans sets a different reference point. When rectifying raw Hex-Rays outputs on unstripped Coreutils and Binutils, it achieved 100% function-level compilability and 99.55% and 99.89% test-validated behavioral consistency, while maintaining over 96% behavioral consistency even on fully stripped binaries (Cui et al., 16 Mar 2026). DecGPT, in a less invasive compiler-and-sanitizer loop around IDA-Pro output, reached a 75% recompilation-plus-test-passing success rate at conversation-chain length $t$ 3, whereas none of the original IDA-Pro outputs were directly recompilable (Wong et al., 2023).

Dataset construction has become equally central. Decompile-Bench contributes two million high-quality binary-source function pairs condensed from 100 million collected pairs, derived from 3,961 permissively licensed GitHub C/C++ repositories, about 85K binaries, and roughly 450GB of executables (Tan et al., 19 May 2025). Fine-tuning LLM4Decompile-End-1.3B on only 10% of Decompile-Bench improved HumanEval average re-executability from 16.22 to 20.89 and MBPP from 20.54 to 24.93, while improving GitHub2025 R2I from 60.47 to 73.18 (Tan et al., 19 May 2025). The paper also shows that raw unfiltered real-world pairings can hurt correctness, whereas aggressively filtered function-level alignment improves it (Tan et al., 19 May 2025).

5. Semantic recovery and language-specific challenges

Control-flow lifting is only part of decompilation. A large fraction of source-level meaning resides in identifiers, types, layouts, and language-specific abstractions. DIRE addresses identifier naming as a post-processing stage over decompiled code, combining lexical context and AST structure to recover developer-like variable names; on a corpus of 164,632 unique x86-64 binaries, it exactly recovered developer names 74.3% of the time, with lexical-only and structural-only ablations at 72.9% and 64.6% accuracy (Lacomis et al., 2019). This does not solve semantics completely, but it materially improves analyst comprehension.

Type recovery remains comparably important. A supervised classifier for return-type inference from x86-32 binaries compiled with Microsoft cl achieved macro-F1 0.791 on high-level return types, while the best baseline decompiler achieved 0.30 macro-F1 (Escalada et al., 2021). The paper also documented specific binary patterns for bool, char, short, pointer, struct, long long, float, double, and void, suggesting that learned predictors can be converted back into deterministic decompiler heuristics (Escalada et al., 2021).

Rust exposes the limits of generic C-like lifting most starkly. In the Rust benchmark, function naming was the worst-preserved dimension, variable naming performed best at around 1.2/2.0, and release builds often improved local control-flow clarity while reducing type retention (Zhou, 24 Jul 2025). Representative failures showed Result<(), ValidationError> collapsing into integer status codes, Option::unwrap() degrading into null-check-like logic, and release inlining erasing helper boundaries and trait context (Zhou, 24 Jul 2025). A plausible implication is that Rust-aware decompilation needs MIR- or DWARF-informed lifting, panic-pattern recognition, and reconstruction of Option<T>/Result<T,E> layouts rather than just better CFG recovery.

For GPU kernels, "A method for decompilation of AMD GCN kernels to OpenCL" reconstructed system values, address spaces, control flow, and arithmetic templates well enough to translate a large subset of AMD GCN instructions into OpenCL, supporting get_global_id, get_local_id, get_group_id, and related built-ins through ABI-aware recovery (Mihajlenko et al., 2021). Human-centric work such as dewolf complements these semantic passes by showing that analysts often prefer configurable restructuring, fewer casts, and readable switch and for constructs even when those diverge from assembly-imposed shape (Enders et al., 2022).

6. Extensions beyond native C decompilation and outstanding problems

Automatic decompilation now extends beyond native CPU binaries. An early hardware-oriented example decompiled software binaries into annotated CDFGs for hardware/software partitioning on microprocessor/FPGA platforms, reporting average application speedup 5.4 and average energy savings 69% relative to a 200 MHz MIPS processor (0710.4700). DeQompile treated quantum-circuit decompilation as genetic-programming-based synthesis of high-level Qiskit from OpenQASM families, achieving perfect fitness on simple regular circuit families and more limited performance on GHZ, QFT, and QPE (Xie et al., 11 Apr 2025). "Neural Decompiling of Tracr Transformers" treated transformer weights as compiled artifacts and recovered RASP programs, obtaining over 30% exact recovery in teacher-forced evaluation and 73% functional equivalence in autoregressive mode (Thurnherr et al., 2024). These cases suggest that AutoDecompiler, in the broad sense, has become a general methodology for recovering symbolic structure from compiled or lowered representations.

The major unresolved issues are consistent across domains. Function boundaries disappear under inlining; optimization and obfuscation distort both control flow and data layout; multiple source programs can remain equally valid explanations of one compiled artifact; test-based validation depends on coverage; and dynamic repair systems still rely on strong frontends, accurate dependency extraction, and workable execution environments (Katz et al., 2019, Cui et al., 16 Mar 2026, Liu et al., 15 Jun 2026). For real-world binaries, stripped symbols, indirect calls, calling-convention mismatches, and global data reconstruction remain especially stubborn (Cui et al., 16 Mar 2026). For LLM-based systems, long-context degradation, hallucinated symbol or type changes, and the gap between compilability and semantic correctness are still persistent failure modes (Wong et al., 2023, Zou et al., 11 Jun 2025).

Taken together, the literature suggests that AutoDecompiler is best understood not as a single architecture, but as an increasingly layered stack: source- or binary-aware preprocessing, structured representations of control and data, model-based translation or repair, compiler and runtime oracles, and evaluation centered on behavioral consistency rather than text similarity. The strongest current systems succeed when they combine these layers rather than treating decompilation as isolated sequence generation (Liu et al., 15 Jun 2026).