LegoFuzz: Modular Compiler Testing Framework

• LegoFuzz is a compiler testing framework that decouples LLM usage into an offline phase for extracting rich, valid C functions and an online phase for synthesizing large, diverse test programs.
• It rigorously validates functions via parsing, type checking, sanitizers, and CompCert, building a reusable database from real-world code snippets.
• The online synthesis assembles tests with deep call chains and shared globals to trigger advanced compiler optimizations, uncovering bugs and miscompilations in GCC and LLVM.

Searching arXiv for the LegoFuzz paper and closely related compiler-testing work to ground the article in current literature. LegoFuzz is a compiler testing framework for C compilers that decouples LLM usage into an offline phase and an online phase. In the offline phase, it uses a single LLM to build a reusable database of small but feature-rich, valid C functions; in the online phase, it composes those functions into large, diverse test programs without any further LLM invocations. The framework was implemented for GCC and LLVM and reported 66 bugs, including 30 miscompilations, while also emphasizing computational efficiency relative to LLM-in-the-loop fuzzing (Ni et al., 26 Aug 2025).

1. Problem setting and design rationale

LegoFuzz is motivated by two stated limitations of existing LLM-based compiler fuzzers. First, prompting a LLM to emit complete C programs produces many tiny or semantically invalid snippets that are too simple to stress modern compilers. Second, invoking the LLM in the inner loop of fuzzing increases both latency and monetary cost. The paper contrasts this with prior LLM fuzzers such as Fuzz4All, which generate on average approximately 20 lines of C, of which only 37% compile, and which nearly always exercise only front-end behavior such as crashes rather than miscompilations (Ni et al., 26 Aug 2025).

The central design decision is therefore architectural rather than merely prompt-based. LegoFuzz interleaves LLM calls into an offline database-construction stage and a fast online synthesis stage. This organization is intended to preserve the diversity of LLM-derived code fragments while avoiding repeated model invocation during high-throughput test generation. A plausible implication is that the framework treats LLMs primarily as a source of reusable semantic building blocks rather than as a direct generator of end-to-end test cases.

2. Offline database construction

The offline phase uses a single LLM, GPT-4o-mini, to transform real-world functions into small, standalone, numeric-I/O C functions (Ni et al., 26 Aug 2025). Each candidate function $F_i$ must satisfy two conditions.

First, it must be Expressive: it must contain non-trivial control flow, including loops and nested conditionals, and non-trivial data flow, including pointer arithmetic and struct fields. Second, it must be Valid: it must pass parsing, type checking, and a suite of sanitizers, plus CompCert validation for no undefined behavior. The retained entries are stored as

$E_i=\{F_i,\mathrm{prof}_i\},$

where $\mathrm{prof}_i$ is a one-time runtime profile under safe inputs, recording expression values per line.

Snippet selection is biased by a diversity score

$$D(F_i)=\sum_{k=1}^K \alpha_k\,\mathrm{feat}_k(F_i),$$

where $\mathrm{feat}_k$ counts occurrences of feature $k$, such as branches, loops, and pointer operations, and $\alpha_k$ weights its testing value. Sampling uses

$$P(F_i)=\frac{\exp(\lambda D(F_i))}{\sum_j \exp(\lambda D(F_j))}.$$

The implementation sets $\lambda=0.1$ to avoid over-concentrating on a few “monster” functions. After filtering, over 553 K functions from 146 projects survive, corresponding to approximately 53% of AnghaBench (Ni et al., 26 Aug 2025).

This phase establishes the reusable substrate on which the rest of the framework depends. The emphasis on real-world functions, validation, and feature weighting indicates that the database is intended to preserve compiler-relevant structure rather than merely syntactic well-formedness.

3. Online iterative synthesis

The online phase performs iterative program synthesis without further LLM calls (Ni et al., 26 Aug 2025). Its purpose is to assemble test programs by inserting dependencies among functions so that compilers are forced to inline, vectorize, constant-propagate, and otherwise expose back-end optimization behavior.

The high-level synthesis procedure begins by generating a pool of global variables $\mathcal{G}$ consisting of random numeric arrays and scalars. It then selects a seed function $E_i=\{F_i,\mathrm{prof}_i\},$0 and builds a driver main that calls $E_i=\{F_i,\mathrm{prof}_i\},$1 with a safe input from $E_i=\{F_i,\mathrm{prof}_i\},$2. For each iteration $E_i=\{F_i,\mathrm{prof}_i\},$3, it chooses a target function from the used list, extracts “stable” subexpressions $E_i=\{F_i,\mathrm{prof}_i\},$4 whose runtime value $E_i=\{F_i,\mathrm{prof}_i\},$5 is known from the offline profile, and then performs one of two transformations. With probability $E_i=\{F_i,\mathrm{prof}_i\},$6, it inserts a call to a fresh function $E_i=\{F_i,\mathrm{prof}_i\},$7, synthesizing parameters so that the call evaluates to $E_i=\{F_i,\mathrm{prof}_i\},$8, and replaces

$E_i=\{F_i,\mathrm{prof}_i\},$9

to preserve semantic equivalence. Otherwise, it shares a global variable $\mathrm{prof}_i$0 by replacing

$\mathrm{prof}_i$1

or by inserting after $\mathrm{prof}_i$2 a write of the form

$\mathrm{prof}_i$3

If a new function is used, it is added to the used list, and the final output is a complete C file with synthesized main.

The paper expresses the resulting invariants as

$\mathrm{prof}_i$4

for combined validity, and

$\mathrm{prof}_i$5

for feature coverage, with coverage growing roughly linearly with $\mathrm{prof}_i$6. In practice, the system sets $\mathrm{prof}_i$7, producing programs of 10 K–15 K LOC in about $\mathrm{prof}_i$8 (Ni et al., 26 Aug 2025).

A common misconception in LLM-based compiler testing is that complexity must come from a single generated artifact. LegoFuzz instead constructs complexity compositionally by weaving together deep call chains, shared globals, and preserved-value substitutions across many validated functions.

4. Implementation for GCC and LLVM

The reported implementation targets GCC and LLVM (Ni et al., 26 Aug 2025). During offline compilation, snippets are validated with gcc -std=c18 -O0 -fsyntax-only and clang -std=c18 -O0 -fsyntax-only, together with sanitizers -fsanitize=address,undefined,integer,thread, and are cross-checked under CompCert. Earlier in the description, validity is also characterized as passing parsing, type checking, and sanitizers including ASan, UBSan, MSan, and TypeSan, plus CompCert validation for no undefined behavior.

During online fuzzing, each synthesized program is compiled using gcc -O0,-O1,-O2,-O3 and clang -O0,-O1,-Os,-O2,-O3. Crash diagnostics are captured through --crash-on-warning together with exit code observation. Miscompilation detection uses an output-differencing oracle: main() is instrumented to print a single checksum combining global variables and function return values, and outputs are compared across compilers and optimization levels.

This implementation strategy is significant because it separates three concerns that are often conflated in fuzzing systems: structural validity, semantic safety, and behavioral divergence under optimization. LegoFuzz relies on the first two as preconditions for the third.

5. Evaluation and reported results

The evaluation uses an AMD EPYC 7742 system running Ubuntu 20.04 and a single LLM API key; AnghaBench database construction cost $394 (Ni et al., 26 Aug 2025). The primary reported metrics are bug count, coverage, speed, and GPU-related LLM cost.

Metric	Reported result
Bug count	66 total: 23 GCC, 43 LLVM
Miscompilations	30
Coverage gain	GCC line coverage +12.5% (+110 945 lines); LLVM +2.9% (+80 295 lines)
Generation efficiency	10 K tests in 193 s
Comparison to Fuzz4All / WhiteFox	12 121 s / 28 284 s for 10 K tests
Bug status	56 fixed, 8 duplicates, 2 still unconfirmed

The coverage gains are reported from 1 K seeds expanded into 10 K synthesized programs. The speed comparison is summarized as a 62×–146× speedup over Fuzz4All and WhiteFox. The paper also states that nearly half of the 66 bugs are silent miscompilations, which are described as serious and hard-to-find bugs that existing LLM-based tools could not find (Ni et al., 26 Aug 2025).

The evaluation places particular emphasis on the distinction between crash finding and miscompilation discovery. This suggests that the framework’s value is not only in producing more tests quickly, but in producing tests with the control-flow, data-dependency, and optimization-triggering structure needed to expose backend defects.

6. Miscompilation profile, boundaries, and broader implications

The case studies attribute the discovered bugs to specific optimization categories (Ni et al., 26 Aug 2025). The reported breakdown includes 15 LLVM bugs and 1 GCC bug in loop transformations, 8 in vectorization passes, and 8 in peephole optimizations. One GCC crash is described as stemming from a nested-loop reduction misclassification, while one LLVM miscompilation is attributed to an invalid InstCombine after inlining a pointer-arithmetic function chain.

The paper explains why earlier LLM fuzzers miss such failures. They rarely generate deep function call chains of approximately 90 calls, global-variable sharing across functions, or test cases that combine complex nested loops with pointer arithmetic in the same program. LegoFuzz’s iterative synthesis is described as naturally weaving these features together.

The framework also states several limitations. It is currently restricted to numeric-I/O functions, which simplifies semantic equivalence; extending to structs, floats, and custom types is identified as a route to wider coverage. It operates at function-level granularity, although inter-procedural optimizations such as global constant propagation could benefit from whole-program synthesis or selective inlining. The prompt used for real-world-aligned extraction achieves approximately 53% valid snippet extraction, and better chain-of-thought or retrieval-augmented prompting is proposed as a possible improvement. The paper further suggests portability to Rust, Swift, or GPU kernels with new grammar checkers and type sanitizers.

Taken together, these points position LegoFuzz as a specific paradigm for AI-enabled compiler testing: LLM power is concentrated in a one-time database construction step, while large-scale test generation is handled by a deterministic, high-throughput synthesizer. A plausible implication is that its main contribution is methodological modularization—separating expensive semantic harvesting from cheap combinatorial exploration—rather than relying on ever-larger models or more frequent prompting alone.