Reinforcement Learning for Fuzzing

• Reinforcement Learning for Fuzzing is a method that frames fuzzing as a sequential decision problem, enabling adaptive mutation selection and seed prioritization.
• It employs diverse RL formulations—from value-based methods to bandit strategies—to maximize coverage and improve bug discovery rates.
• Methodologies utilize shaped semantic rewards and structured-input synthesis to address challenges in distributed systems, API testing, and complex input domains.

Reinforcement Learning for Fuzzing (RLF) denotes a family of methods that cast fuzzing, test generation, or guided program exploration as a sequential decision problem in which an agent learns from execution feedback to improve coverage expansion, target reachability, bug discovery, or related objectives. In the recent literature, RLF spans mutation-operator selection, seed scheduling, directed/path-sensitive exploration, format-constrained generation, language-model-guided input synthesis, and harness- or task-level scheduling. A systematic review covering studies from 2015 to 2026 identifies fuzzing and guided exploration as the dominant RL application in vulnerability analysis, with 15 of 21 primary studies falling into this category (Caro-Vásquez et al., 24 Jun 2026).

1. Historical emergence and scope

Early RLF work formalized fuzzing as an MDP or closely related online decision process in which the agent observes a representation of the current fuzzing context, chooses a mutation or scheduling action, executes the program under test, and updates its policy from coverage- or behavior-derived rewards. "Deep Reinforcement Fuzzing" framed input mutation for structured binary formats as a reinforcement learning problem and used Deep Q-learning to select probabilistic mutation operators conditioned on local byte windows, reporting improvements of $+7.75\%$ for coverage-only reward, $+7.0\%$ for time-only reward, and $+11.3\%$ for a combined reward on PDF-processing benchmarks (Böttinger et al., 2018). "FuzzerGym" then integrated OpenAI Gym with libFuzzer and LLVM Sanitizers, treating mutator selection as a throughput-preserving asynchronous POMDP and showing higher maximum coverage on all five evaluated benchmarks and higher average coverage on four out of five (Drozd et al., 2018).

Subsequent work diversified the locus of control. Some systems retained the classical fuzzing loop and learned only one decision layer, such as operator selection or seed scheduling. Others moved toward generation-based or domain-constrained fuzzing for structured inputs such as HTML, JavaScript, REST APIs, and compiler inputs. A distinct line of work applied bandit methods rather than full value-function approximation, especially when the fuzzing subproblem could be reduced to arm selection over seeds, targets, or operators. The review literature therefore treats RLF not as a single algorithmic template but as a task family spanning greybox, black-box, directed, grammar-aware, and LM-based fuzzing settings (Caro-Vásquez et al., 24 Jun 2026).

This breadth also changed the meaning of “state.” In early software-input settings, state often meant a local byte window or the current input itself. In later systems, it became a coverage bitmap, a masked structured seed, an abstract distributed-system configuration, a transformer embedding of an HTTP response, or a code-slice question that asks a LLM to invert a specific branch condition. A plausible implication is that the field has shifted from “RL for low-level mutator control” toward “RL for semantically informed exploration,” especially in domains where validity and long-range constraints dominate.

2. Core RL formulations

Across the literature, the standard formal substrate is an MDP or POMDP $\langle S, A, P, R, \gamma \rangle$, but the specific semantics vary substantially. In mutational greybox fuzzing, the environment is the instrumented target under a coverage-guided loop; actions are mutators or seed choices; and rewards are usually derived from coverage deltas, unique edges, or “interesting” inputs. FuzzerGym, for example, defines the observation as a 32768-dimensional bit vector encoding an input capped at 4096 bytes, uses 13 libFuzzer mutators as discrete actions, and sets reward to $R_t = cov_t - cov_{t-1}$ (Drozd et al., 2018). Deep Reinforcement Fuzzing similarly uses runtime properties such as basic-block coverage and execution time, with a combined reward

$$r_3(\tilde{x}, a) = E_1(\tilde{x}, \emptyset) + 10^6 \cdot T(\tilde{x}),$$

where $E_1$ is the absolute number of unique basic blocks hit in that execution and $T(\tilde{x})$ is execution time (Böttinger et al., 2018).

In seed scheduling, the problem is often simplified to a bandit. "T-Scheduler" models each coverage feature as an arm, assigns binary rewards based on whether mutating the corresponding favored input yields an interesting input, and updates Beta–Bernoulli posteriors via Thompson sampling. Its parameter update is

$$(\alpha_k^{(t)}, \beta_k^{(t)}) = (\alpha_k^{(t-1)}, \beta_k^{(t-1)}) + (r_k^{(t)}, 1-r_k^{(t)}),$$

with $\alpha_k^{(0)}=\beta_k^{(0)}=1$, and it introduces a self-balancing rareness correction to avoid over-favoring already popular coverage features (Luo et al., 2023). In distributed-systems testing, the reward may be entirely synthetic: "Reward Augmentation in Reinforcement Learning for Testing Distributed Systems" augments sparse terminal failures with a decaying novelty bonus $+7.0\%$0 and waypoint-conditioned updates, replacing the usual additive Bellman target by a max-combination to prioritize novelty more aggressively (Borgarelli et al., 2024).

LM-centered textual fuzzing introduces another formulation. "R1-Fuzz" defines each episode around a question $+7.0\%$1 built from a code slice, a target branch condition, and an original input $+7.0\%$2, and the policy $+7.0\%$3 generates a complete candidate textual input $+7.0\%$4. Its objective is

$+7.0\%$5

with a distance-shaped reward based on function-trace proximity and branch inversion success (Lin et al., 21 Sep 2025). "CovRL" similarly treats masked-span completions as actions, but couples them to a PPO-style objective and a TF-IDF weighted coverage reward over AFL-style edge coverage (Eom et al., 2024).

3. Main methodological families

A useful taxonomy, consistent with the review literature, separates RLF systems by the decision variable they optimize (Caro-Vásquez et al., 24 Jun 2026).

Family	Representative systems	Learned decision
Mutation/operator selection	FuzzerGym, Deep Reinforcement Fuzzing	Which mutator to apply
Seed scheduling / prioritization	T-Scheduler, hierarchical seed scheduling in the review	Which seed or coverage feature to fuzz next
Directed/path-sensitive exploration	DeepGo, R1-Fuzz	Which path, branch, or target to pursue
Structured or domain-constrained generation	HTML DDQN+TCN, CovRL, REST fuzzers	How to generate valid structured inputs
LM-based semantic synthesis	R1-Fuzz, CovRL, APIRL	How to use learned representations or LMs to produce semantically targeted inputs

Value-based deep RL remains common when actions are discrete and moderately sized. FuzzerGym uses DDQN with prioritized experience replay and an LSTM to cope with asynchronous partial observability (Drozd et al., 2018). The HTML rendering-engine work uses DDQN to choose which HTML tag to insert next while a TCN generates the actual content, obtaining a best DDQN-guided coverage of 57,993 unique basic blocks, a $+7.0\%$6 increase over the best grammar-based baseline set (Sablotny et al., 2023). APIRL uses a DQN with prioritized replay and a target network over a 772-dimensional state combining a 768-dimensional RoBERTa response embedding with four structured REST features (Foley et al., 2024).

Bandit methods dominate when the problem is essentially allocation rather than long-horizon planning. T-Scheduler explicitly argues for a parameter-free multi-armed bandit formulation to avoid the tuning burden and throughput reduction of heavier RL approaches, and reports improvements over 11 schedulers over 35 CPU-years of fuzzing (Luo et al., 2023). The same reduction appears in some distributed-systems and scheduling settings, where explicit context is deliberately abstracted away.

Policy-gradient and actor–critic methods appear when the action space is continuous, hierarchical, or sequence-generative. CovRL uses a PPO-style update to online-finetune an LLM-based mutator from coverage feedback (Eom et al., 2024). DeepGo uses Soft Actor-Critic over predicted path transitions to select mutation locations, then combines the resulting policy with a 27-dimensional action group optimized by particle swarm methods (Lin et al., 29 Jul 2025). R1-Fuzz uses GRPO to post-train a 7B LM for branch-targeted textual input synthesis (Lin et al., 21 Sep 2025). In autonomous-driving safety fuzzing, MARL-OT adopts MADDPG with centralized training and decentralized execution to coordinate multiple surrounding vehicles as dynamic adversaries, although this setting broadens “fuzzing” toward online scenario generation rather than traditional software-input mutation (Liang et al., 24 Jan 2025).

4. Coverage shaping, novelty, and semantic guidance

Reward design is the central technical difficulty in RLF because natural rewards are sparse. Coverage delta is the simplest and most common signal. FuzzerGym uses line-coverage increment, while many greybox formulations in the review use edge or basic-block novelty and optional crash bonuses (Drozd et al., 2018, Caro-Vásquez et al., 24 Jun 2026). However, later work increasingly replaces raw coverage counts with shaped or semantically weighted rewards.

CovRL addresses the “many-covered” bias by weighting edges with a TF-IDF-style rarity score. For a valid test case, it computes

$+7.0\%$7

then normalizes this into $+7.0\%$8, while invalid tests receive fixed penalties of $+7.0\%$9 for syntax error and $+11.3\%$0 for semantic error (Eom et al., 2024). This mechanism rewards rare coverage rather than just large coverage.

R1-Fuzz turns branch inversion into a dense sequence-level objective by defining

$+11.3\%$1

and then assigning reward $+11.3\%$2 if the target function is not reached, $+11.3\%$3 if the branch is reached with the original outcome, $+11.3\%$4 if the branch outcome is inverted, and $+11.3\%$5 if $+11.3\%$6 (Lin et al., 21 Sep 2025). This is notable because it operationalizes a directed-fuzzing notion of proximity within LM post-training rather than only inside the fuzzing scheduler.

DeepGo generalizes this idea for directed greybox fuzzing through a predictive path-transition model. Its transition reward is the change in a seed-value function balancing target distance, estimated difficulty of unexplored sibling branches, execution speed, and favoredness, and its Virtual Ensemble Environment predicts both next paths and rewards with mean predictive accuracies of $+11.3\%$7 for transition probabilities and $+11.3\%$8 for rewards on UniBench (Lin et al., 29 Jul 2025). The shift here is from “reactive” fitness metrics to model-based foresight.

Semantic rewards can also be hand-designed. In distributed protocols, waypoint predicates such as leader election, term divergence, or committed-entry differences produce staged shaping rewards that let policies reliably reach deep semantic states even without execution caching (Borgarelli et al., 2024). In REST API fuzzing, FuzzTheREST uses HTTP status classes as the only reward signal—$+11.3\%$9 for 1XX, $\langle S, A, P, R, \gamma \rangle$0 for 2XX/3XX, $\langle S, A, P, R, \gamma \rangle$1 for 4XX, and $\langle S, A, P, R, \gamma \rangle$2 for 5XX—whereas APIRL uses a simpler but denser scheme of $\langle S, A, P, R, \gamma \rangle$3 for 5XX, $\langle S, A, P, R, \gamma \rangle$4 for 2XX, and $\langle S, A, P, R, \gamma \rangle$5 otherwise, combined with rich response embeddings in the state rather than the reward (Dias et al., 2024, Foley et al., 2024).

5. Textual, structured, and API-centric fuzzing

RLF is especially active where inputs must satisfy strong syntactic and semantic constraints. JavaScript-engine fuzzing is a prominent case. CovRL mutates masked spans using Insert, Overwrite, and Splice operations, instruments engines with AFL-style edge coverage, and online-finetunes a CodeT5+-based mutator and rewarder. In one-week, three-core campaigns it found 48 bugs across engines, including 39 previously unknown vulnerabilities and 11 CVEs (Eom et al., 2024).

R1-Fuzz pushes this direction further by specializing a small LM through RL rather than merely using an untuned model in the fuzzing loop. It constructs branch-targeted questions from LLVM source-based coverage and execution traces, trains Qwen2.5-7B-Instruct with GRPO, and produces R1-Fuzz-7B. On a static test set of 1,640 questions, R1-Fuzz-7B reaches Pass@1 $\langle S, A, P, R, \gamma \rangle$6 and Pass@5 $\langle S, A, P, R, \gamma \rangle$7, compared with Pass@1 $\langle S, A, P, R, \gamma \rangle$8 and Pass@5 $\langle S, A, P, R, \gamma \rangle$9 for the untuned 7B base model (Lin et al., 21 Sep 2025). In 24-hour fuzzing over 10 real-world textual systems, AFL++ + R1-Fuzz-7B achieves up to $R_t = cov_t - cov_{t-1}$0 higher code-region coverage than state-of-the-art fuzzers and discovers 29 previously unknown vulnerabilities, 24 of which were confirmed or fixed (Lin et al., 21 Sep 2025).

REST API fuzzing illustrates a different structured-input regime. FuzzTheREST uses tabular Q-learning with one Q-table per data type and reports six unique vulnerabilities with $R_t = cov_t - cov_{t-1}$1 code coverage on Petstore (Dias et al., 2024). APIRL uses a transformer pre-trained on HTTP responses from 103 APIs, a 23-action schema-aware mutation space, and a DQN policy trained once on a single API and then applied zero-shot to 26 unseen APIs. It reports that across studies it achieves $R_t = cov_t - cov_{t-1}$2 higher coverage per request on average, uses $R_t = cov_t - cov_{t-1}$3 fewer requests than Rand-APIRL while covering at least $R_t = cov_t - cov_{t-1}$4 more code, and finds significantly more bugs than several baselines (Foley et al., 2024).

Structured generation also appears in browser fuzzing. The HTML-rendering-engine work couples a TCN generator with DDQN over tag-selection actions and reports up to $R_t = cov_t - cov_{t-1}$5 improved code coverage compared to the baseline grammar-based fuzzer for Firefox’s HTML rendering engine (Sablotny et al., 2023). This suggests that RL is particularly valuable when the main difficulty is not raw mutation throughput but deciding where to perturb structure without collapsing validity.

6. Empirical performance, limitations, and open directions

The empirical record is heterogeneous because targets, metrics, and budgets vary substantially. The systematic review explicitly notes limited comparability across benchmarks, incomplete hyperparameter reporting, and scarce use of standardized source-graph states such as CFGs or ASTs (Caro-Vásquez et al., 24 Jun 2026). Even so, several recurring patterns are visible.

First, RL often improves peak or late-stage exploration when the baseline heuristic saturates. FuzzerGym reports that RL massively surpassed libFuzzer on sqlite, with best runs reaching 2209 covered lines versus 905 for libFuzzer (Drozd et al., 2018). T-Scheduler reports that its Sample variant achieved the highest aggregate total and unique bug counts on Magma, while Rare and COE were strongest on coverage metrics depending on the benchmark slice (Luo et al., 2023). DeepGo reached 73 of 80 targets within 24 hours on UniBench, versus 22 for AFLGo, 11 for BEACON, 19 for WindRanger, and 9 for ParmeSan, with mean TTR speedups ranging from $R_t = cov_t - cov_{t-1}$6 to $R_t = cov_t - cov_{t-1}$7 over these baselines (Lin et al., 29 Jul 2025). R1-Fuzz, CovRL, and APIRL all show that incorporating learned semantic priors can materially raise both coverage and bug yield in structured-input domains (Lin et al., 21 Sep 2025, Eom et al., 2024, Foley et al., 2024).

Second, reward shaping and representation quality matter at least as much as the RL algorithm. CovRL’s TF-IDF coverage weighting outperforms simpler coverage rewards, and APIRL’s transformer state improves both coverage and bug yield relative to ablations without embeddings (Eom et al., 2024, Foley et al., 2024). Distributed-systems testing shows that waypoint predicates can dominate purely novelty-driven exploration for deep semantic goals (Borgarelli et al., 2024). R1-Fuzz shows that compact coverage slices plus dense branch-distance rewards enable a 7B model to rival or exceed much larger LMs in fuzzing (Lin et al., 21 Sep 2025).

Third, throughput and overhead remain a persistent constraint. FuzzerGym’s asynchronous circular buffer was designed specifically to prevent neural inference from stalling libFuzzer’s microsecond loop (Drozd et al., 2018). T-Scheduler emphasizes modest scheduling overhead and zero tuning as practical advantages over heavier RL schemes (Luo et al., 2023). CovRL incurs approximately 10 minutes of finetuning per 2.5 hours of fuzzing, counted in wall-clock evaluation (Eom et al., 2024). DeepGo requires concurrent GPU training for its predictive environment and actor–critic components (Lin et al., 29 Jul 2025). A plausible implication is that RLF remains most compelling when the input domain is so structured, target-directed, or semantically constrained that intelligent decisions outweigh raw execution rate.

Common limitations recur across the literature: sparse and delayed rewards, non-stationarity from corpus evolution, partial observability, reward hacking, and target-specific tuning sensitivity. The review highlights a broader research gap: RL fuzzers rarely use source-level CFGs or ASTs as agent states, despite their potential value for directed exploration and vulnerability localization (Caro-Vásquez et al., 24 Jun 2026). Other explicitly proposed directions include curriculum learning, offline RL, hierarchical policies, distillation into smaller models, multi-objective reward design, richer target scheduling, and better integration of static analyses or graph-derived structure (Lin et al., 21 Sep 2025, Eom et al., 2024, Caro-Vásquez et al., 24 Jun 2026).

In this sense, RLF is best understood not as a settled technique but as an evolving synthesis of coverage-guided testing, adaptive control, and learned semantic modeling. Its most successful instantiations are those that align the RL abstraction with the real bottleneck in fuzzing: operator choice in high-throughput loops, seed prioritization under corpus explosion, semantic validity in structured inputs, or long-horizon planning in directed exploration.