R1-Fuzz: RL-Enhanced Textual Fuzzing

• R1-Fuzz is a reinforcement learning framework that integrates a cost-efficient language model into coverage-guided textual fuzzing for compilers, interpreters, and databases.
• It employs innovative coverage-slicing question construction and a distance-based reward mechanism to guide branch-targeted input generation, achieving up to 75% higher coverage.
• The framework has demonstrated significant practical impact by uncovering 29 previously unknown vulnerabilities, with 24 confirmed or fixed by upstream developers.

R1-Fuzz is a reinforcement-learning-based framework for textual fuzzing that specializes a small, cost-efficient LLM (LM) and integrates it into a coverage-guided fuzzing workflow for targets such as compilers, interpreters, and database engines that accept syntactically and semantically constrained textual inputs (Lin et al., 21 Sep 2025). It introduces two central mechanisms—coverage-slicing-based question construction and a distance-based reward calculation—and uses RL-based post-training to produce a model, R1-Fuzz-7B, that is embedded into AFL++ to generate branch-targeted inputs during live fuzzing. On ten real-world textual targets, the reported system achieves up to 75\% higher coverage than state-of-the-art fuzzers and discovers 29 previously unknown vulnerabilities, with 24 confirmed or fixed by upstream developers (Lin et al., 21 Sep 2025).

1. Problem setting and design rationale

R1-Fuzz is motivated by a recurrent limitation of conventional fuzzing on complex textual targets: compilers, interpreters, and database engines often require inputs that satisfy intricate syntactic and semantic constraints, so random mutation and purely grammar-driven generation may fail to penetrate deep program logic (Lin et al., 21 Sep 2025). The framework is presented as the first to leverage reinforcement learning to specialize cost-efficient LMs for complex textual fuzzing input generation, rather than relying on a large general-purpose model at inference time.

The design addresses two challenges identified in the paper. The first is insufficient exploration of deep program logic among real-world codebases. The second is the high cost of leveraging larger models. R1-Fuzz answers these by post-training a lightweight pretrained model on branch-targeted tasks derived from real executions, then coupling the model to AFL++ so that LM-generated inputs and traditional mutations coexist in one fuzzing loop. This makes the LM a task-specific input synthesizer rather than a standalone test generator (Lin et al., 21 Sep 2025).

A central premise is that branch-local reasoning can be made learnable if the model is shown only the executable code slice from program entry to a target branch, together with an input that currently fails to flip that branch. This constrains the prompting problem and supplies a direct fuzzing objective: generate a new input that inverts the branch outcome. The reinforcement-learning stage then aligns the model with that objective by rewarding inputs according to how close they come to reaching and flipping the target branch.

2. Three-stage architecture

R1-Fuzz is described as a three-stage system: dataset construction, RL post-training, and an LLM-powered fuzzing loop (Lin et al., 21 Sep 2025).

In the first stage, a target program $P$ and an initial seed corpus $X=\{x_1,\dots,x_n\}$ are executed. For each seed, the framework records covered branches $\mathrm{Cov}_i$ and uncovered branches $\mathrm{Uncov}_i$. For every branch $b \in \mathrm{Uncov}_i$ that is covered by some other seed, R1-Fuzz slices out the code along the call stack from entry to $b$ and packages that slice plus $x_i$ into a training “question.” This produces a supervision source grounded in actual program behavior rather than synthetic instruction-following data.

In the second stage, a pretrained lightweight LM $\pi_\theta$—the reported instantiation uses Qwen2.5-7B-Instruct—is fine-tuned with Group Relative Policy Optimization (GRPO). Given a code-slice question, the model generates a candidate input $y$ intended to flip the branch condition. The reward is distance-based rather than purely binary, so the model receives graded feedback even when it fails to flip the branch outright.

In the third stage, the fine-tuned model is embedded into AFL++’s mutation loop. On each new seed, the system extracts newly uncovered branches, builds questions for them, queries the LM for targeted inputs, and feeds those inputs back into AFL++ alongside conventional AFL++ mutations. The resulting workflow is hybrid: coverage-guided mutation remains in place, but branch-targeted textual generation is delegated to the RL-specialized model (Lin et al., 21 Sep 2025).

3. Coverage slicing and branch-targeted question construction

The question-construction mechanism is based on branch coverage and dynamic slicing. Let $C$ denote the set of all branch conditions in $X=\{x_1,\dots,x_n\}$0. As a seed $X=\{x_1,\dots,x_n\}$1 executes, the framework partitions $X=\{x_1,\dots,x_n\}$2 into $X=\{x_1,\dots,x_n\}$3 and $X=\{x_1,\dots,x_n\}$4. It then collects a global branch set $X=\{x_1,\dots,x_n\}$5 consisting of branches that are uncovered by one seed but reachable by some other seed (Lin et al., 21 Sep 2025).

For each $X=\{x_1,\dots,x_n\}$6, R1-Fuzz defines a slice

$X=\{x_1,\dots,x_n\}$7

and forms the set

$X=\{x_1,\dots,x_n\}$8

Each slice is converted into a question–prompt pair $X=\{x_1,\dots,x_n\}$9. The system prompt states: “Here is the executable code slice from program entry to branch b, and an original input x that takes the false branch. Generate a new input x′ that inverts b’s outcome (False→True).” The user content contains the text of $\mathrm{Cov}_i$0 and the original seed $\mathrm{Cov}_i$1. During training, one question is generated for each target branch. During fuzzing, equivalent questions are reconstructed on the fly for newly observed uncovered branches (Lin et al., 21 Sep 2025).

This formulation narrows the model’s reasoning burden. Instead of synthesizing valid whole-program inputs from scratch under a broad natural-language instruction, the model is asked to solve a localized program-analysis problem expressed in the target program’s own code. A plausible implication is that the method benefits from the LM’s latent code-reasoning ability while avoiding the context blow-up that would result from prompting with entire codebases.

4. Reward model and reinforcement-learning objective

R1-Fuzz uses a distance-based reward to provide dense feedback during RL post-training. Let $\mathrm{Cov}_i$2 and $\mathrm{Cov}_i$3 denote the runtime sequences of function calls triggered by inputs $\mathrm{Cov}_i$4 and $\mathrm{Cov}_i$5, and let $\mathrm{Cov}_i$6 be their longest common prefix. The normalized function-level coverage distance is defined as

$\mathrm{Cov}_i$7

The single-step reward is then

$\mathrm{Cov}_i$8

The paper states that, by normalizing $\mathrm{Cov}_i$9 into the GRPO framework, the model receives a fine-grained, smoothly varying reward that encourages deeper exploration while rewarding successful branch flips most heavily (Lin et al., 21 Sep 2025).

The LM is treated as a policy $\mathrm{Uncov}_i$0 that samples an answer $\mathrm{Uncov}_i$1 for a question $\mathrm{Uncov}_i$2. For a rollout $\mathrm{Uncov}_i$3 with rewards $\mathrm{Uncov}_i$4, the RL objective is

$\mathrm{Uncov}_i$5

Using GRPO, the gradient estimate is

$\mathrm{Uncov}_i$6

where $\mathrm{Uncov}_i$7. A KL-divergence regularizer keeps the learned policy close to the reference pretrained model:

$\mathrm{Uncov}_i$8

with $\mathrm{Uncov}_i$9 (Lin et al., 21 Sep 2025).

The paper reports 1,000 RL steps. This optimization regime turns branch flipping into a policy-learning problem rather than a prompt-engineering problem, which is a defining methodological feature of R1-Fuzz.

5. Implementation, integration, and empirical performance

The reported implementation uses Qwen2.5-7B-Instruct as the base model, Qwen’s BPE tokenizer with approximately 50 K tokens, and a dataset of 16,338 questions drawn from ten real-world textual targets: PHP, CPython, Lua, mruby, NJS, QuickJS, Solidity, SQLite, Sql-parser, and DuckDB. The split is 90/10 train/test. RL hyperparameters are batch size 128, 8 rollouts per question with temperature 1.0, KL coefficient $b \in \mathrm{Uncov}_i$0, and 1,000 training steps. Full post-training is reported to complete within a few hours on a single GPU (Lin et al., 21 Sep 2025).

Inside AFL++, the fuzzing workflow proceeds by maintaining a seed corpus and an accumulated coverage set. In each round, a seed is selected, AFL++ mutations are applied, the mutated input is executed on the target, and coverage is updated if new branches are hit. From that execution, the framework lists branches that remain unexplored, builds questions for those branches, and enqueues them into a priority queue. The LM consumes the queue concurrently, generates targeted textual inputs, executes them, and adds them to the corpus if they yield new coverage. Question scheduling lowers the priority of branches that have already been asked frequently, thereby biasing the system toward new branches (Lin et al., 21 Sep 2025).

On the static test split of 1,640 questions, the paper reports the following Pass@1 and Pass@5 results (Lin et al., 21 Sep 2025):

Model	Pass@1	Pass@5
Qwen2.5-7B	8%	16%
Qwen2.5-32B	20%	33%
DeepSeek-V3 (13 B)	23%	39%
GPT-04mini (14 B)	35%	49%
R1-Fuzz-7B	50%	55%

For 24-hour fuzzing, averaged over five runs, AFL++ + R1-Fuzz-7B reaches approximately $b \in \mathrm{Uncov}_i$1 relative to baseline AFL++ coverage $b \in \mathrm{Uncov}_i$2, with up to +75\% coverage on some targets. The paper states that the system outperforms AFL++ combined with other LMs on nearly every target and exceeds libFuzzer, Polyglot, Nautilus, and Gramatron by approximately 75\% average coverage (Lin et al., 21 Sep 2025).

The vulnerability results are likewise central to the framework’s significance. R1-Fuzz reports 29 previously unknown bugs in PHP, Lua, mruby, NJS, QuickJS, Solidity, and DuckDB, with 24 confirmed or fixed by upstream developers. By comparison, AFL++ and libFuzzer are reported to find 2 bugs, while the referenced grammar fuzzers find at most 5 (Lin et al., 21 Sep 2025).

6. Relationship to adjacent fuzzing methods

R1-Fuzz occupies a distinct position within the fuzzing literature because it combines coverage-guided fuzzing, branch-targeted code slicing, and RL-specialized language modeling. This differs from grammar-aware software fuzzing such as AFL++-based testing of OAI5G configuration files, where inputs are mutated through a JSON grammar and evaluation emphasizes edge coverage, unique paths, and crash triggers (Wu et al., 2023). It also differs from hardware fuzzing through RTL translation, where a Verilator-generated C++ model is fuzzed and assertion violations define “crashes,” yielding two orders of magnitude faster convergence on full coverage than conventional dynamic verification in the reported setup (Trippel et al., 2021).

The framework is also distinct from black-box RL fuzzing of REST APIs. FuzzTheREST formulates fuzzing as a Markov Decision Process over HTTP status-code classes, uses a Multi-Table Q-Learning algorithm, and records 55\% code coverage with six unique vulnerabilities on the Petstore API (Dias et al., 2024). R1-Fuzz, by contrast, uses source-level code slices and branch-flip rewards tied to execution traces, so the LM is optimized against program-internal control-flow objectives rather than external response classes.

A closer methodological relative is branch-directed fuzzing for smart contracts. IR-Fuzz uses function invocation ordering, branch-distance-based seed evolution, and energy allocation for rare and vulnerable branches, achieving 28\% higher branch coverage than prior smart-contract fuzzers and improving average vulnerability-detection accuracy by 7\% (Liu et al., 2023). R1-Fuzz shares the emphasis on branch proximity and targeted exploration, but replaces symbolic or arithmetic branch-distance heuristics with LM-mediated reasoning over executable code slices.

Another adjacent line is the use of fuzzing outputs to enrich LM representations for program understanding. “Fuzz Tuning” augments CodeBERT and UniXcoder with AFL++-harvested input–output pairs for code clone detection and code classification, improving MAP@R and reducing classification error across POJ104 and CodeNet subsets (Zhao et al., 2023). R1-Fuzz inverts that direction: instead of using fuzzing to help an LM understand programs, it uses RL to help an LM generate fuzzing inputs that explore deeper semantics (Lin et al., 21 Sep 2025).

7. Limitations, extensions, and terminology

The paper identifies three primary limitations. First, context windows constrain how much code can be shown to the model, so deeper slices may be truncated. Second, simple function-level slicing may omit important dependencies such as globals, macros, and types. Third, the reward considers one branch at a time, so multi-branch interactions are not jointly optimized (Lin et al., 21 Sep 2025). The proposed extensions are correspondingly direct: smarter slicing through dataflow or taint analysis, hierarchical RL that first plans branch sequences and then generates inputs, binary-level slicing for targets without source code, and application to API fuzzing, network-protocol fuzzing, or structured file formats.

The term “R1-Fuzz” is not entirely unique across the broader material. The label also appears in a distinct cyber-physical setting as “Physical Differential Fuzz Testing (R1-Fuzz),” where an untampered sensor system is challenged with pseudo-random physical parameter sequences and its noisy output time series is compared against a stored baseline using statistics such as modified reduced $b \in \mathrm{Uncov}_i$3 per spectrum pair or KL divergence (Vavrek et al., 2024). In that formulation, the purpose is tamper detection in measurement systems rather than vulnerability discovery in textual parsers, interpreters, or databases. This suggests that the designation has been used for more than one fuzzing paradigm, even though the 2025 framework names it explicitly as a reinforcement-learning system for textual fuzzing.

Within the textual-fuzzing sense, however, R1-Fuzz denotes a specific synthesis of coverage-guided fuzzing and RL-specialized LM reasoning: branch-focused prompt construction, distance-shaped rewards, lightweight post-training, and tight integration with AFL++ in a live campaign. Its reported results position it as a concrete instance of a broader shift in fuzzing research from unguided mutation toward learned, execution-aware input generation (Lin et al., 21 Sep 2025).