---
title: 'ReFuzzer: Refining LLM-Generated Compiler Tests'
url: https://www.emergentmind.com/topics/refuzzer
type: topic
---

# ReFuzzer: Refining LLM-Generated Compiler Tests

ReFuzzer is a framework for refining LLM-generated test programs in compiler fuzzing by systematically detecting and correcting compilation and runtime violations through a feedback loop with a local LLM. It is designed for C/C++ compiler-testing pipelines in which generated programs are often syntactically invalid, semantically invalid, or dynamically unsafe, and therefore fail to exercise optimizer and backend logic effectively. In the reported LLVM/Clang experiments, ReFuzzer improved test-program validity from 47.0–49.4% to 96.6–97.3% on GPU configurations, with an average processing time of 2.9–3.5 s per test program on a dual-GPU machine, while also increasing coverage in optimization and IR-generation components [2508.03603].

## 1. Problem setting and objectives

ReFuzzer arises from a specific limitation of recent LLM-based compiler fuzzers. Systems such as WhiteFox and Fuzz4All use LLMs to generate C/C++ test inputs for compilers, but the resulting programs frequently contain static invalidity, such as syntax or type errors, and dynamic invalidity, such as division by zero or out-of-bounds memory accesses. The reported static validity rates for LLM-generated programs are as low as 23%–49%. This matters because statically invalid programs never compile and therefore never exercise middle-end or back-end compiler optimizations, while dynamically invalid programs can produce false positives in sanitizer-based crash detection or mask genuine compiler bugs [2508.03603].

The framework is organized around four explicit objectives. It aims to detect and categorize compilation and sanitizer errors in LLM-generated programs, iteratively refine those programs via a local LLM loop until they become both statically and dynamically valid, improve overall validity from approximately 47% to above 96% while keeping per-test processing time to a few seconds on GPU hardware, and enable richer coverage of compiler components, particularly optimizer and code-generation passes. A common misconception is to treat ReFuzzer as a generator of new tests. Its function is narrower and more specific: it sits downstream of an existing LLM-based fuzzer and refines the generated outputs so that they can actually traverse deeper compiler subsystems.

## 2. System architecture

ReFuzzer is intended to slot into an existing LLM-based fuzzing pipeline and is composed of four core modules: an LLM Generator, a Compilation/Runtime Checker, an Error Analyzer, and a Refiner. The generator can be instantiated by a black-box fuzzer, a grey-box fuzzer, or a white-box fuzzer. The paper evaluates three such front ends: BlackBox via Ollama+LLaMA 3.2, Fuzz4All, and WhiteFox [2508.03603].

| Module | Role | Specifics |
|---|---|---|
| LLM Generator | Produces raw C/C++ programs | BlackBox, Fuzz4All, WhiteFox |
| Compilation/Runtime Checker | Evaluates static and dynamic validity | Clang `-O0`, ASan, UBSan, MSan, TSan |
| Error Analyzer | Categorizes failures | syntax, type, memory safety, arithmetic safety, inline-asm |
| Refiner | Repairs invalid programs | local LLM loop, up to `n = 2` attempts |

The Compilation/Runtime Checker invokes Clang `-O0` to collect compiler errors and warnings for static validity, and then uses AddressSanitizer, UndefinedBehaviorSanitizer, MemorySanitizer, and ThreadSanitizer to assess dynamic validity. The Error Analyzer parses compiler logs and sanitizer outputs and classifies failures into static versus dynamic categories, with finer classes including syntax, type, memory safety, arithmetic safety, and inline-asm. The Refiner receives the original program together with the error log and optimization level, issues a fix request to a local LLM, and rechecks the candidate output. If the result remains invalid after the permitted number of attempts, the program is demoted to a `crash_only_folder`.

This modularization is significant because it isolates generation from repair. A plausible implication is that ReFuzzer can be adopted without changing the upstream fuzzing strategy, provided that the generated artifacts are C/C++ programs and the downstream toolchain can supply compiler and sanitizer diagnostics.

## 3. Feedback-driven refinement and error correction

The core algorithm is an iterative feedback loop over a set of generated programs. For each program, ReFuzzer first compiles with Clang `-O0`. If compilation fails, the error type is treated as a compilation error. If compilation succeeds, the framework executes sanitizer-based dynamic checks. If both static and dynamic checks pass, the program is admitted into the valid set. Otherwise, and provided that the current attempt count is below the configured maximum, ReFuzzer constructs a fix prompt from the program, the error type, and the compiler or sanitizer log, and asks the local LLM to produce a corrected version. After at most two refinement attempts per program, unresolved cases are moved to `crash_only_folder` [2508.03603].

The correction logic is class-sensitive rather than uniform. For syntax and type errors, the local LLM is asked to correct the AST or add missing includes. For inline assembly errors, it standardizes asm constraints or comments out unsupported sections. For buffer overflows, it increases buffer sizes or adds boundary checks. For null-pointer risks, it injects `if (ptr != NULL)` guards. For division by zero, it adds the precondition `if (denom != 0) denom = 1;`. For sanitizer warnings, it may remove or rewrite unsafe library calls, such as replacing misuse of `vsprintf_s` with `vsnprintf`.

The sample prompt template is correspondingly diagnostic-driven: the model is told the optimization level, the reported compiler or sanitizer failure, and is asked to return a corrected version that resolves the errors without altering the high-level intent. This framing is important because the framework is not merely filtering bad programs; it is attempting repair while preserving the structural features that may still be useful for fuzzing. The paper also formalizes `ValidityRate` and `CoverageDelta`, underscoring that the primary evaluation target is not crash count alone but the proportion of generated programs that survive static and dynamic validation.

## 4. Implementation and experimental protocol

The implementation uses C++ for the orchestrator and Python for glue scripts, including the Dockerfile and logging infrastructure. The local LLM is LLaMA 3.2 via Ollama v0.5.7. Three hardware configurations are reported: a CPU-only system based on Intel Xeon D-1548 with 8 cores at 2.0 GHz and 64 GB RAM; a single-GPU system based on Intel Xeon Silver 4114 plus an NVIDIA Tesla P100 with 12 GB; and a dual-GPU system based on 2× AMD EPYC 7542 plus 2× NVIDIA Tesla V100S with 32 GB each. The timeout per compile or refuzz attempt is 60 s, the memory limit is 16 GB, and the number of refinement attempts per program is fixed at 2 [2508.03603].

The evaluation is structured around three fuzzing configurations: black-box using BlackBox, grey-box using Fuzz4All, and white-box using WhiteFox. The stated research questions ask how much ReFuzzer improves validity rate, how CPU versus GPU affects validity and throughput, and what effect the system has on deep compiler code coverage. The experimental setup uses 24-hour seed generation per fuzzer with a 60 s per-test timeout, persistent storage of the resulting suites, and one refuzzing pass per suite with ReFuzzer on each hardware configuration. Function coverage is measured with GCOV/LCOV on Clang 21.0.0.

The reported average per-test processing times are 14.2–15.1 s on CPU, 4.6–5.2 s on GPU×1, and 2.9–3.5 s on GPU×2. These timings make the hardware dependence explicit: the framework is feasible on CPU-only systems, but much higher throughput is obtained on GPU-backed local-LLM configurations.

## 5. Empirical performance

The central quantitative result is that ReFuzzer raises validity from approximately 47–49% to approximately 96.6–97.3% on GPU, while GPU×2 achieves approximately 3 s per test and is 4–5× faster than CPU. CPU-only operation remains slower, at 14–15 s per test, and reaches lower final validity, between 55% and 80% depending on the upstream fuzzer [2508.03603].

| Setting | Validity before → after | Time/test |
|---|---|---|
| BlackBox, CPU | 24.0% → 80.7% | 14.2 s |
| BlackBox, GPU×1 | 47.0% → 96.8% | 4.9 s |
| BlackBox, GPU×2 | 47.0% → 96.8% | 3.1 s |
| Fuzz4All, CPU | 47.9% → 55.9% | 15.1 s |
| Fuzz4All, GPU×1 | 48.5% → 96.6% | 4.6 s |
| Fuzz4All, GPU×2 | 48.5% → 96.6% | 2.9 s |
| WhiteFox, CPU | 12.3% → 68.3% | 14.5 s |
| WhiteFox, GPU×1 | 49.4% → 97.3% | 5.2 s |
| WhiteFox, GPU×2 | 49.4% → 97.3% | 3.5 s |

Coverage gains extend beyond mere validity restoration. On GPU×2, function coverage improvements are reported across multiple compiler subsystems. In the frontend parser, the gains are +2.0%, +1.0%, and +0.8% for BlackBox, Fuzz4All, and WhiteFox respectively. In AST/semantics, the gains are +1.7%, +0.7%, and +0.3%. IR generation shows a notably asymmetric effect: +10.2% for BlackBox and +0.4% for both Fuzz4All and WhiteFox. The most pronounced improvements occur in optimization passes. Loop optimization improves by +12.7%, +2.3%, and +10.8%; vectorization by +9.2%, +2.3%, and +7.1%; inlining by +21.2%, +5.7%, and +16.8%; and DCE by +17.0%, +3.6%, and +13.1%. Backend code generation gains are smaller, at +0.5%, +0.3%, and +1.1%.

These coverage changes are statistically supported. Paired t-tests on per-function coverage before and after ReFuzzer yield \(p < 0.01\) across all major optimization passes. The pattern suggests that the principal benefit of refuzzing valid programs is not confined to frontend parsing, but propagates into passes that require well-formed IR and semantically executable inputs before their behavior can be meaningfully exercised.

## 6. Interpretation, limitations, and related systems

ReFuzzer’s contribution is specific to validity enhancement in LLM-driven compiler fuzzing. It does not primarily optimize seed scheduling, reuse historical test corpora, or model exploration policy. This distinguishes it from similarly named work such as ReFuzz, which targets processor fuzzing and uses a contextual-bandit framework to reuse highly effective tests from prior processors for a processor-under-test within a given ISA [2512.04436]. The resemblance in names can obscure the fact that the two systems address different failure modes: ReFuzzer repairs invalid generated programs for compiler testing, whereas ReFuzz reorders and mutates prior hardware tests to accelerate RTL coverage and vulnerability discovery.

The reported limitations of ReFuzzer are correspondingly concrete. Its dependence on C/C++ sanitizers limits immediate applicability to other languages without analogous dynamic-analysis tools. The two-attempt cap may miss complex multi-error repairs, although the paper reports diminishing returns beyond two iterations. The local LLM, specifically LLaMA 3.2, can occasionally inject semantic changes irrelevant to fuzzing goals. These limitations matter because they define the envelope within which the measured gains should be interpreted: the framework improves the executability of generated programs, but it does not guarantee semantic preservation in a strict program-equivalence sense.

Several extensions are identified. The work proposes incorporating test-case minimization and corpus reduction to maximize coverage per synthetic program and reduce computational cost; exploring multi-model ensembles, including retrieval-augmented GPT-4 combined with a local LLM, for higher fix accuracy; extending the approach to Java, Go, and Rust through language-specific sanitizer frameworks; and evaluating distributed GPU clusters and ChatGPT-style remote APIs to compare fix quality and throughput against local deployments. This suggests a broader research direction in which LLM-based fuzzing pipelines are decomposed into generation, diagnosis, repair, and coverage optimization stages, with ReFuzzer occupying the diagnosis-and-repair layer.

Source: https://www.emergentmind.com/topics/refuzzer