---
title: 'AutoDecompiler: RL-Based Binary Decompilation'
url: https://www.emergentmind.com/topics/autodecompiler
type: topic
---

# AutoDecompiler: RL-Based Binary Decompilation

AutoDecompiler is a decompilation-specialized LLM family trained with reinforcement learning for feedback-driven multi-turn binary decompilation, designed to treat decompilation as iterative refinement rather than single-turn code generation [2606.16162]. It accepts x86\_64 assembly or decompiler-produced pseudo-code and aims to produce readable, compilable, executable C-like functions while using compilation, execution, and input/output testing feedback to improve behavioral re-executability [2606.16162]. In the broader research landscape, it belongs to a line of work that treats decompilation not as exact inversion of compilation, but as constrained reconstruction under information loss, verification, and ambiguity [1905.08325] [2310.06530].

## 1. Position in the decompilation literature

Automatic decompilation has evolved from rule-based structuring and type recovery toward hybrid systems that combine symbolic analyses, learned translation, and dynamic validation. Katz, Olshaker, Goldberg, and Yahav’s "Towards Neural Decompilation" framed decompilation as verified neural inverse compilation, using compilation for supervision and recompilation plus program-dependence-graph isomorphism for validation [1905.08325]. Later work split along several lines: retargetable plain-text translation across source languages [2212.08950], post-decompilation semantic repair [2310.06530], human-centric readability optimization [2205.06719], and dynamically validated recompilation against original binaries [2603.14855].

A recurrent misconception in this literature is that a decompiler should be judged primarily by textual similarity to original source. Multiple papers reject that criterion. Decompiled code may be readable yet semantically wrong, or textually different yet behaviorally equivalent, which is why recompilability, test-based execution, structural similarity, and human comprehension recur as distinct targets rather than a single metric [1905.08325] [2310.06530] [2410.00061].

| System | Input | Principal mechanism |
|---|---|---|
| TraFix | LLVM IR or x86 assembly | NMT plus recompilation and PDG-based structural equivalence |
| DecGPT | IDA-Pro output | Compiler-in-the-loop and ASAN-guided repair |
| PCodeTrans | Hex-Rays output | In-situ substitution and BP-Diff |
| AutoDecompiler | x86\_64 assembly or IDA pseudo-code | RL-trained multi-turn refinement with program feedback |

Within this lineage, AutoDecompiler is distinguished by making iterative repair a learned behavior rather than an external orchestration layer. Earlier systems already showed the value of feedback; AutoDecompiler internalizes that loop through multi-turn RL, progress-aware trajectory rewarding, and turn-aware advantage reweighting [2606.16162].

## 2. Representations, preprocessing, and alignment

A central theme in automatic decompilation is that input representation determines how much recoverable semantics reaches the model. "Beyond the C" deliberately treated assembly and source as plain text with BPE tokenization to maximize retargetability across Go, Fortran, OCaml, and C, avoiding parsers and ASTs where possible [2212.08950]. That strategy reduced language-specific engineering, but it also left the model responsible for inferring control structure and semantics from minimally structured input.

ReF Decompile argues that excessive simplification of assembly is itself destructive. Its Relabelling strategy replaces jump targets with labels such as `L1` and memory references with labels such as `D1`, preserving control-flow clarity and memory-reference identity while removing irrelevant absolute addresses [2502.12221]. Its Function Call strategy then lets the model request values behind labeled memory accesses, using inferred types such as string, float, or double to retrieve constants from the binary, especially from `.rodata` [2502.12221]. This directly targets two sources of failure in end-to-end decompilation: loss of branch topology and loss of literal values.

Fine-grained Alignment Enhancement takes a different route. By compiling C with DWARF, disassembling with `objdump -S`, and reorganizing source-interleaved disassembly into aligned assembly/source supervision, it converts implicit correspondence into a step-by-step training signal [2406.17233]. The associated sc\(^2\)dec method then exploits a decompilation-specific property at inference time: if a first-pass decompilation is compilable, it can be recompiled and disassembled to form a self-constructed in-context example for a second pass [2406.17233]. This suggests that the compiler can serve not only as an evaluator but also as an inference-time context generator.

Language-specific binaries stress these representation choices further. The Rust study shows that decompilers tuned to C/C++ assumptions tend to emit low-level C-like pseudocode that loses the semantics of Rust abstractions, especially after optimization [2507.18792]. In that benchmark-driven evaluation with Ghidra 10.3.3 on Mach-O ARM64 binaries built by `rustc 1.71.1`, generic types, trait methods, `Result`/`Option` error handling, and complex pattern matching were especially damaging, and average decompilation scores were generally between 1.2 and 1.4 out of 8 across programs [2507.18792]. This suggests that an AutoDecompiler cannot rely on structural lifting alone when the source language encodes semantics through high-level layouts, niche encodings, and monomorphization.

## 3. Feedback-driven refinement architecture

AutoDecompiler formalizes decompilation as a sequential decision problem. At turn $t$, the state is
$$
s_t = (x, y_{<t}, f_{<t}),
$$
where $x$ is assembly or pseudo-code input, $y_{<t}$ are prior decompilation outputs, and $f_{<t}$ are prior diagnostic feedback messages; the action is a full decompiled function $y_t \sim \pi_\theta(\cdot \mid s_t)$ [2606.16162]. The environment validates each turn through C-like validity checking, recompilation, execution, and I/O testing, then emits both feedback and reward [2606.16162].

Its reward is explicitly multi-dimensional. The execution-oriented component is piecewise:
$$
R_{\mathrm{exe}}(y_t)=
\begin{cases}
-1, & \text{recompilation fails},\\
-0.6, & \text{runtime error},\\
-0.3, & \text{partial I/O mismatch},\\
1, & \text{all I/O tests pass}.
\end{cases}
$$
This is combined with C-like validity, AST-subtree syntactic reward, and normalized data-flow semantic reward using weights $w_C=0.5$, $w_{\mathrm{exe}}=1$, $w_{\mathrm{syn}}=0.3$, and $w_{\mathrm{sem}}=0.2$ [2606.16162]. Progress-aware trajectory rewarding adds an improvement term across turns, and turn-aware advantage reweighting redistributes credit to the turns that actually improved the trajectory [2606.16162].

This architecture generalizes earlier feedback-driven repair systems. DecGPT already used a static augmenting stage to make IDA-Pro output compilable and a dynamic repairing stage to fix runtime defects using compiler diagnostics, ASAN reports, and output mismatches [2310.06530]. PCodeTrans went further by compiling repaired functions into dynamic libraries, hot-swapping them into the unmodified original binary through an in situ substitutable engine, and collecting Breakpoint-Matched Differential Tracing to localize semantic divergence under authentic process context [2603.14855]. D-LiFT similarly insisted that readability gains must be conditioned on preserved correctness, using D-SCORE to penalize syntax and semantic errors before awarding readability improvements, and then training its backend with GRPO [2506.10125].

A second misconception is that readability and correctness can be optimized independently. Several systems show the opposite. DecGPT’s plain LLM baseline often compiled by introducing casts or deleting problematic code, while PCodeTrans and D-LiFT both treat such behavior as semantic regression rather than improvement [2310.06530] [2506.10125]. AutoDecompiler inherits that lesson by weighting execution correctness above auxiliary syntactic and semantic proxies [2606.16162].

## 4. Evaluation regimes and empirical results

Modern decompilation evaluation has shifted from surface similarity toward executable validation. Decompile-Eval measures re-compilability and re-executability on HumanEval-derived C functions compiled at `O0` through `O3`, and it is the basis for several recent LLM studies [2406.17233]. On that benchmark, llm4decompile-6.7b-v1.5 improved from 47.68 to 52.28 average re-executability with Fine-grained Alignment Enhancement, and to 55.03 when FAE was combined with sc\(^2\)dec [2406.17233]. ReF Decompile reported a new state-of-the-art average re-executability of 61.43 with average readability 3.69, outperforming LLM4Decompile-Ref at the same 6.7B scale and improving over FAE Decompile by 10.36 in re-executability [2502.12221].

AutoDecompiler’s strongest gains appear when iterative refinement is learned rather than externally imposed. On HumanEval, AutoDecompiler-E2E 6.7B reached 63.11 average Re-exe, while AutoDecompiler-Pscode 6.7B reached 70.58 and the 30B pseudo-code model reached 75.30; construction-stage analysis showed a progression from 45.73 for the vanilla DeepSeek-Coder pseudo-code model to 65.09 after SFT, 68.75 after 1-turn RL, and 70.58 after 3-turn RL [2606.16162]. The same paper reports that removing compilation/execution reward reduced HumanEval Re-exe for AutoDecompiler-E2E 1.3B from 53.51 to 50.07, while removing progress-aware trajectory rewarding or turn-aware advantage reweighting reduced it to 50.30 and 50.46, respectively [2606.16162].

Where whole-binary fidelity is needed, PCodeTrans sets a different reference point. When rectifying raw Hex-Rays outputs on unstripped Coreutils and Binutils, it achieved 100% function-level compilability and 99.55% and 99.89% test-validated behavioral consistency, while maintaining over 96% behavioral consistency even on fully stripped binaries [2603.14855]. DecGPT, in a less invasive compiler-and-sanitizer loop around IDA-Pro output, reached a 75% recompilation-plus-test-passing success rate at conversation-chain length $C=15$, whereas none of the original IDA-Pro outputs were directly recompilable [2310.06530].

Dataset construction has become equally central. Decompile-Bench contributes two million high-quality binary-source function pairs condensed from 100 million collected pairs, derived from 3,961 permissively licensed GitHub C/C++ repositories, about 85K binaries, and roughly 450GB of executables [2505.12668]. Fine-tuning LLM4Decompile-End-1.3B on only 10% of Decompile-Bench improved HumanEval average re-executability from 16.22 to 20.89 and MBPP from 20.54 to 24.93, while improving GitHub2025 R2I from 60.47 to 73.18 [2505.12668]. The paper also shows that raw unfiltered real-world pairings can hurt correctness, whereas aggressively filtered function-level alignment improves it [2505.12668].

## 5. Semantic recovery and language-specific challenges

Control-flow lifting is only part of decompilation. A large fraction of source-level meaning resides in identifiers, types, layouts, and language-specific abstractions. DIRE addresses identifier naming as a post-processing stage over decompiled code, combining lexical context and AST structure to recover developer-like variable names; on a corpus of 164,632 unique x86-64 binaries, it exactly recovered developer names 74.3% of the time, with lexical-only and structural-only ablations at 72.9% and 64.6% accuracy [1909.09029]. This does not solve semantics completely, but it materially improves analyst comprehension.

Type recovery remains comparably important. A supervised classifier for return-type inference from x86-32 binaries compiled with Microsoft `cl` achieved macro-F1 0.791 on high-level return types, while the best baseline decompiler achieved 0.30 macro-F1 [2101.08116]. The paper also documented specific binary patterns for `bool`, `char`, `short`, `pointer`, `struct`, `long long`, `float`, `double`, and `void`, suggesting that learned predictors can be converted back into deterministic decompiler heuristics [2101.08116].

Rust exposes the limits of generic C-like lifting most starkly. In the Rust benchmark, function naming was the worst-preserved dimension, variable naming performed best at around 1.2/2.0, and release builds often improved local control-flow clarity while reducing type retention [2507.18792]. Representative failures showed `Result<(), ValidationError>` collapsing into integer status codes, `Option::unwrap()` degrading into null-check-like logic, and release inlining erasing helper boundaries and trait context [2507.18792]. A plausible implication is that Rust-aware decompilation needs MIR- or DWARF-informed lifting, panic-pattern recognition, and reconstruction of `Option<T>`/`Result<T,E>` layouts rather than just better CFG recovery.

For GPU kernels, "A method for decompilation of AMD GCN kernels to OpenCL" reconstructed system values, address spaces, control flow, and arithmetic templates well enough to translate a large subset of AMD GCN instructions into OpenCL, supporting `get_global_id`, `get_local_id`, `get_group_id`, and related built-ins through ABI-aware recovery [2107.07809]. Human-centric work such as dewolf complements these semantic passes by showing that analysts often prefer configurable restructuring, fewer casts, and readable `switch` and `for` constructs even when those diverge from assembly-imposed shape [2205.06719].

## 6. Extensions beyond native C decompilation and outstanding problems

Automatic decompilation now extends beyond native CPU binaries. An early hardware-oriented example decompiled software binaries into annotated CDFGs for hardware/software partitioning on microprocessor/FPGA platforms, reporting average application speedup 5.4 and average energy savings 69% relative to a 200 MHz MIPS processor [0710.4700]. DeQompile treated quantum-circuit decompilation as genetic-programming-based synthesis of high-level Qiskit from OpenQASM families, achieving perfect fitness on simple regular circuit families and more limited performance on GHZ, QFT, and QPE [2504.08310]. "Neural Decompiling of Tracr Transformers" treated transformer weights as compiled artifacts and recovered RASP programs, obtaining over 30% exact recovery in teacher-forced evaluation and 73% functional equivalence in autoregressive mode [2410.00061]. These cases suggest that AutoDecompiler, in the broad sense, has become a general methodology for recovering symbolic structure from compiled or lowered representations.

The major unresolved issues are consistent across domains. Function boundaries disappear under inlining; optimization and obfuscation distort both control flow and data layout; multiple source programs can remain equally valid explanations of one compiled artifact; test-based validation depends on coverage; and dynamic repair systems still rely on strong frontends, accurate dependency extraction, and workable execution environments [1905.08325] [2603.14855] [2606.16162]. For real-world binaries, stripped symbols, indirect calls, calling-convention mismatches, and global data reconstruction remain especially stubborn [2603.14855]. For LLM-based systems, long-context degradation, hallucinated symbol or type changes, and the gap between compilability and semantic correctness are still persistent failure modes [2310.06530] [2506.10125].

Taken together, the literature suggests that AutoDecompiler is best understood not as a single architecture, but as an increasingly layered stack: source- or binary-aware preprocessing, structured representations of control and data, model-based translation or repair, compiler and runtime oracles, and evaluation centered on behavioral consistency rather than text similarity. The strongest current systems succeed when they combine these layers rather than treating decompilation as isolated sequence generation [2606.16162].

Source: https://www.emergentmind.com/topics/autodecompiler