NeuComBack: Neural Compilation Benchmark

• NeuComBack is an IR-to-assembly benchmark and evaluation framework that formalizes neural compilation, ensuring both semantic correctness and competitive performance.
• It integrates a two-level curated benchmark with a structured workflow combining self-debugging and iterative prompt optimization to refine assembly code generation.
• Empirical results on x86_64 and AArch64 show improved correctness, enhanced performance metrics, and reduced self-debug rounds compared to baseline approaches.

NeuComBack is an IR-to-assembly benchmark and evaluation framework introduced in "QiMeng-NeuComBack: Self-Evolving Translation from IR to Assembly Code" (Fang et al., 3 Nov 2025). It is designed for neural compilation, in which a model translates a source representation such as LLVM IR into target assembly while being evaluated not only for executable correctness but also for runtime performance. The work combines a curated two-level benchmark, a formal neural compilation workflow, explicit correctness and performance metrics, and a self-evolving prompt optimization method in which an LLM updates its prompt strategy by extracting recurring patterns from its own self-debugging traces.

1. Neural compilation problem setting

NeuComBack is situated in the broader claim that compilers are "notoriously complex systems" that require "prohibitively expensive human expertise to develop and maintain," whereas recent advances in LLMs offer a possible neural compilation paradigm that could simplify compiler development for new architectures and facilitate the discovery of optimization techniques (Fang et al., 3 Nov 2025). The paper identifies two obstacles to practical adoption: the lack of dedicated benchmarks and robust evaluation methodologies, and the difficulty of systematically improving the reliability and performance of LLM-generated assembly.

The neural compilation task is formalized as

$$A_{\text{target}} = f_{\text{NC}}(P_{\text{source}}, \mathcal{M}; \theta),$$

where $$P_{\text{source}} \in \{\text{LLVM IR}, \text{high-level language}\}$$ and $A_{\text{target}}$ is assembly for architecture $\mathcal{M}$. Two requirements are imposed. The first is functional correctness,

$$\llbracket A_{\text{target}} \rrbracket_{\mathcal{M}} \equiv \llbracket P_{\text{source}} \rrbracket,$$

defined as the same observable behavior for all valid inputs. The second is performance optimization,

$c(A_{\text{target}}) \le c(A_{\text{clang-O3}}).$

This formulation separates semantic equivalence from optimization quality. A plausible implication is that NeuComBack is intended to evaluate neural compilation as a compiler-like activity rather than as a text generation exercise: generating compilable assembly is insufficient unless the output is both semantically valid and competitive with an optimized conventional toolchain.

2. Benchmark composition and data curation

NeuComBack consists of two benchmark levels with distinct emphases (Fang et al., 3 Nov 2025).

Level	Programs	Role
NeuComBack-L1	200 C programs selected from ExeBench	Cleaning, filtering, and retention of the longest LLVM IR sequences
NeuComBack-L2	151 programs from the TSVC suite	Stress-testing loop-centric and vectorization-heavy kernels

The source representation is LLVM IR in .ll files. Target assembly is provided in two architecture-specific forms: x86_64 in AT&T syntax using GNU as format, and AArch64 (ARMv8) assembly.

The preprocessing pipeline is unusually explicit. All C sources were rewritten for full C/C++ standard compliance and recompiled with clang to verify LLVM IR generation. ExeBench entries with nonstandard extensions or overly short IR traces were excluded, and the 200 longest-IR cases were retained. For TSVC, fixed-value array initializations were replaced with pseudorandom values under a fixed seed to avoid degenerate test results.

These design choices define the benchmark’s scope. NeuComBack-L1 emphasizes cleaned IR examples selected for long LLVM IR sequences, whereas NeuComBack-L2 emphasizes kernels that are loop-centric and vectorization-heavy. This suggests that the benchmark is intended both to test raw IR-to-assembly generation and to probe whether an LLM can produce performance-competitive code in optimization-sensitive settings.

3. Evaluation harness, workflow, and metrics

NeuComBack specifies a concrete evaluation harness rather than leaving evaluation to ad hoc execution checks (Fang et al., 3 Nov 2025). Functional correctness is checked against clang-O0 outputs for L1 and against TSVC’s enhanced test runners for L2. Performance comparisons are conducted only on L2. For performance measurement, each program is run 11 times; the first 3 and last 3 runs are discarded as warm-up and cool-down, and the median of the middle 5 runs is reported.

The workflow has three stages. First, the model produces an initial assembly candidate $A_{\text{target}}^{(0)}$ from $P_{\text{source}}$. Second, a self-debugging stage compiles and tests $A^{(0)}$; if it is incorrect, the system allows up to $D$ self-fix rounds. Third, an iterative optimization stage runs for $$P_{\text{source}} \in \{\text{LLVM IR}, \text{high-level language}\}$$0, generating $$P_{\text{source}} \in \{\text{LLVM IR}, \text{high-level language}\}$$1 from $$P_{\text{source}} \in \{\text{LLVM IR}, \text{high-level language}\}$$2, self-debugging as needed, and measuring performance.

The primary metrics are rate-based:

$$P_{\text{source}} \in \{\text{LLVM IR}, \text{high-level language}\}$$3

$$P_{\text{source}} \in \{\text{LLVM IR}, \text{high-level language}\}$$4

where $$P_{\text{source}} \in \{\text{LLVM IR}, \text{high-level language}\}$$5 counts programs that are both correct and faster than clang-O3. Single-program speedup is

$$P_{\text{source}} \in \{\text{LLVM IR}, \text{high-level language}\}$$6

so $$P_{\text{source}} \in \{\text{LLVM IR}, \text{high-level language}\}$$7 means the LLM-generated program runs faster than the clang-O3 baseline.

A central point of interpretation follows directly from these definitions: high-performance results are conditioned on correctness. The benchmark therefore does not treat raw runtime reduction as meaningful if semantic validity has not already been established.

4. Self-evolving prompt optimization

NeuComBack’s main methodological contribution is a self-evolving prompt optimization procedure in which the model refines its own prompt by analyzing complete self-debug traces (Fang et al., 3 Nov 2025). The key idea is not merely to let the LLM debug outputs, but to let it update the prompting policy that governs future generations.

The method has an offline prompt learning phase and an online inference phase. In offline learning, the process begins with an empty or generic prompt $$P_{\text{source}} \in \{\text{LLVM IR}, \text{high-level language}\}$$8. For each mini-batch of tasks, code is generated using $$P_{\text{source}} \in \{\text{LLVM IR}, \text{high-level language}\}$$9, and cases that were corrected after self-debugging are collected with their full traces $A_{\text{target}}$0. A meta-prompt then asks the model to analyze $A_{\text{target}}$1 and extract recurring error patterns and successful fixes, denoted $A_{\text{target}}$2. Proposed edits are then confirmed and stabilized to produce $A_{\text{target}}$3.

The update is summarized compactly as

$A_{\text{target}}$4

where $A_{\text{target}}$5 controls how aggressively new insights reshape the prompt.

At online inference time, the learned prompt $A_{\text{target}}$6 is used to guide both initial generation and iterative optimization. For each test IR, the model generates $A_{\text{target}}$7 with $A_{\text{target}}$8, self-debugs until the result is correct or the budget is exhausted, performs up to $A_{\text{target}}$9 iterative optimization rounds conditioned on $\mathcal{M}$0 plus $\mathcal{M}$1, and then outputs the final $\mathcal{M}$2.

Methodologically, this reframes prompt engineering as a learned artifact derived from debugging history rather than as a fixed hand-crafted instruction. A plausible implication is that the approach attempts to internalize recurrent compiler-like failure modes—such as repairable code generation errors—into the prompt itself.

5. Empirical results on x86_64 and AArch64

The paper reports comparisons between a baseline prompt and the learned prompt on DeepSeek-R1 for NeuComBack-L2, using a 25-case test split for each architecture (Fang et al., 3 Nov 2025).

Setting	Baseline prompt	Learned prompt
x86_64 initial ACC	44% (11/25)	64% (16/25)
x86_64 initial ACC+Perf	24% (6/25)	40% (10/25)
x86_64 after 2 optimization rounds, ACC+Perf	28% (7/25)	56% (14/25)
AArch64 ACC	36% (9/25)	58% (14/25)
AArch64 ACC+Perf	8% (2/25)	28% (7/25)

For x86_64, the reported relative improvements are ACC +45% and ACC+Perf +100%. Among the 16 correct x86_64 cases produced by the learned prompt, 14, or 87.5%, beat clang-O3; the average Speedup is approximately $\mathcal{M}$3, with a peak of $\mathcal{M}$4. For AArch64, correctness improves from 36% to 58%, while the number of high-performance cases rises from 8% to 28%.

The method also reduces self-debug effort. On x86_64 L2, the average number of self-debug rounds required for convergence drops from 1.09 to 0.25.

These results support two distinct claims. First, learned prompt evolution improves functional correctness relative to the baseline prompt. Second, the same learned prompt improves the rate of correct programs that are also faster than clang-O3, especially on x86_64 after two optimization rounds.

6. Scope, interpretation, and research significance

NeuComBack is not merely a dataset; it is a benchmark coupled to a formal workflow and a prompt-learning methodology (Fang et al., 3 Nov 2025). Its significance lies in making neural compilation measurable under criteria that resemble those used in compiler research: semantic validity, optimization efficacy, and iterative refinement under an execution-based harness.

Several scope conditions are important. Performance comparisons are only conducted on NeuComBack-L2, not on L1. The strongest performance claims therefore attach specifically to the TSVC-derived subset, which was chosen to stress-test loop-centric and vectorization-heavy kernels. This suggests that the benchmark’s performance component is intentionally focused on optimization-sensitive workloads rather than on general-purpose program distributions.

A common misunderstanding would be to read the reported speedups as unconditional. The benchmark definitions explicitly prevent that interpretation: $\mathcal{M}$5 counts only programs that are both correct and faster than clang-O3. Another potential misunderstanding would be to treat prompt optimization as equivalent to ordinary self-debugging. In NeuComBack, these are separate mechanisms: self-debugging repairs a current candidate, whereas self-evolving prompt optimization updates the prompt using accumulated traces so that future generations and optimization rounds are guided differently.

Within the paper’s framing, NeuComBack addresses the two bottlenecks identified at the outset: the absence of a dedicated benchmark and the challenge of improving LLM-generated assembly. The reported results suggest that learning from self-debugging history can improve both correctness and performance across x86_64 and AArch64, while also reducing the number of self-debug rounds required for convergence.