LLM4Decompile-End: Transformer for Code Decompilation

• The paper introduces an end-to-end Transformer that directly maps disassembled binary code to compilable C source, eliminating the need for traditional refinement-based decompilers.
• It employs autoregressive decoding with combined next-token and sequence-to-sequence losses to generate syntactically correct and semantically faithful source code.
• Performance metrics show that while the 6.7B model optimally balances re-compilability and re-executability, scaling beyond this size faces diminishing returns.

LLM4Decompile-End Backbone refers to the end-to-end, Transformer-based LLM architecture specifically designed and fine-tuned for binary code decompilation, as formalized in "LLM4Decompile: Decompiling Binary Code with LLMs" (Tan et al., 2024). This approach represents a shift from refinement-based decompilers, which post-process outputs of traditional decompilers such as Ghidra, toward LLMs that directly learn the mapping from disassembled assembly code to compilable, readable high-level source code (e.g., C). Such backbones are fundamental in advancing decompilation fidelity, readability, and re-executability in reverse engineering, vulnerability assessment, and malware analysis.

1. Core Architecture and Initialization

LLM4Decompile-End uses an autoregressive, decoder-only Transformer architecture initialized from DeepSeek-Coder checkpoints. Three primary model sizes are provided:

Variant	Layers (L)	Hidden Dim (H)	Heads (A)	Parameter Count
LLM4Decompile-1B	≈24	≈2048	≈16	1.3B
LLM4Decompile-6B	≈28	≈4096	≈32	6.7B
LLM4Decompile-33B	≈40	≈6144	≈48	33B

The backbone uses standard BPE tokenization over a joint vocabulary spanning C source and x86 assembly (≈50K subwords). Positional encoding is realized through learned embeddings. No task-specific modules such as adapters, syntax-directed attention, or binary-aware architectural modifications are added—the model’s inductive bias derives solely from massive-scale fine-tuning on paired assembly/C samples (Tan et al., 2024).

2. Training Objectives and Prompt Engineering

The training regime alternates or combines two key losses:

• Next-Token Prediction (NTP): Applied over the sequence [assembly tokens] + [C source tokens].

$$\mathcal{L}_{\mathrm{NTP}} = -\sum_{i=1}^N \log P_\theta(x_i \mid x_{<i})$$

• Sequence-to-Sequence Prediction (S2S): Applied only to target source code tokens, where assembly is consumption-only.

$$\mathcal{L}_{\mathrm{S2S}} = -\sum_{i=s}^{t} \log P_\theta(y_i \mid x_1, \dots, x_{s-1}, y_{<i})$$

Training data comprises ≈4 billion tokens of aligned assembly/source pairs. Prompts include explicit markers for compiler optimization levels, e.g., “# This is the assembly code with [O2] optimization: … # What is the source code?”, to inform the model of idiosyncrasies induced by compiler settings (Tan et al., 2024).

3. Layerwise Architecture: Transformer Specification

Within each Transformer block, standard deep learning primitives are used:

• Multi-head Self-Attention:

$$\mathrm{MHAttn}(X) = \operatorname{concat}(H_1, \dots, H_A) W^O$$

where for each head $j$,

$$H_j = \operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

• Feed-Forward Network:

$$\mathrm{FFN}(x) = \operatorname{max}(0, xW_1 + b_1) W_2 + b_2$$

• Embedding and Final Logits: Learned embeddings $\mathrm{Emb}$, positional encodings $P$, and a projection to output token logits conclude the stack. All details, including norm layers, dropout, and activations, are inherited from the corresponding DeepSeek-Coder checkpoint.

4. Pipeline Workflow and Inference Paradigm

The high-level pipeline for LLM4Decompile-End is:

1. Input: Raw assembly (disassembled from binary via objdump or a comparable tool).
2. Tokenization: Convert to BPE subword tokens using the joint vocabulary.
3. Prompt Construction: Prefix with human-readable task marker and, optionally, optimization-level annotation.
4. Autoregressive Decoding: Feed all tokens into the transformer. Autoregressive prediction continues until an <eos> or sufficiently probable code block is generated.
5. Post-processing: Validate output via compilation and, where available, run test suites for re-executability.

No external control flow, syntax rule, or binary-specific heuristics are injected into the backbone itself. The architecture is agnostic to programming language beyond the tokenization scope—readability and semantic alignment rely on the scale and diversity of the multi-billion token training corpus (Tan et al., 2024).

5. Performance Characteristics and Scaling Law

Extensive evaluation on HumanEval and ExeBench shows:

Model	Re-compilability	Re-executability
1.3B	≈86%	≈10%
6.7B	≈87%	≈21%
33B	≈82%	≈21%

Scaling from 1B to 6.7B doubles the ability to recover code that is syntactically and semantically correct (as measured by re-executability), but further growth to 33B parameters shows diminishing returns, likely due to optimization challenges in extremely large models for sequence generation (Tan et al., 2024). In practical use, the 6.7B model provides the best tradeoff.

6. Comparative Analysis and Limitations

Compared to refinement-based (LLM4Decompile-Ref) or graph-augmented LLMs, LLM4Decompile-End does not leverage explicit control/data flow structure or auxiliary tool outputs. Its principal advantage is complete “tool removal”—the model can, in principle, be deployed on any architecture or instruction set reflected in its training data. However, the lack of inductive bias for binary structure remains a bottleneck: flattening internal binary graph representations into linear prompts often results in lost control flow details, variable misalignment, and occasional hallucinations, particularly on highly optimized or obfuscated binaries.

Moreover, end-to-end backbones are sensitive to the diversity of compiler settings, instruction variants, and “domain shift” between training and deployment binaries—a phenomenon also noted in downstream classification tasks such as LLM4Decompile-Driven malware detection (Chawla et al., 14 Jan 2026).

7. Impact and Future Directions

LLM4Decompile-End has precipitated a new class of LLM applications in binary analysis, obsolescing some reliance on legacy decompilers while introducing new research challenges. Hybrid approaches now explore mix-and-match methodologies, combining Transformer backbones with LoRA-adapted modules (Feng et al., 17 Feb 2025), graph-based hierarchical prompting (Achamyeleh et al., 21 Jan 2026), or reinforcement-learned pipelines (Tan et al., 26 Sep 2025) to further increase re-executability and semantic faithfulness. Limitations remain in handling large binaries (context-window failures), domain adaptation (malware and obfuscation robustness), and functional correctness for highly optimized or adversarially crafted binaries.

A plausible implication is that the LLM4Decompile-End backbone will increasingly serve as the foundation for modular, plug-in architectures, where structure-aware, semantic-guided, or hybrid decoders are invoked atop general-purpose end-to-end Transformer models for specialized task or domain generalization (Tan et al., 2024, Feng et al., 17 Feb 2025, Tkachenko et al., 26 May 2025, Chawla et al., 14 Jan 2026).