FdLoop: Feedback-Guided Grammar Fuzzing

• FdLoop is a directed grammar-based test generation method that uses an iterative feedback loop to steer input mutations toward explicit testing goals.
• It adaptively learns probabilistic grammars from seed inputs and refines production probabilities based on execution feedback to enhance testing outcomes.
• Empirical evaluations reveal that FdLoop outperforms conventional fuzzers in unique exception discovery and overall testing effectiveness.

FdLoop is a directed grammar-based test generation method proposed in “Directed Grammar-Based Test Generation” (Kirschner et al., 2 Aug 2025). The name stands for “Feedback Loop for Directed Grammar-based Test Generation.” It is designed for the generation of syntactically valid inputs that are not merely grammar-conforming, but are steered toward explicit testing goals such as unique code coverage, input-to-code complexity, program failures, and long execution time. Its central mechanism is an iterative feedback loop that learns a probabilistic grammar from seed inputs, generates and mutates candidate inputs, executes the subject program, selects the best-performing input under a goal-specific fitness function, and then re-learns and mutates grammar probabilities from that selected input.

1. Concept and problem setting

FdLoop addresses a limitation of conventional grammar-based fuzzing: a grammar can enforce syntactic validity, but it does not by itself identify which structural choices are relevant to a downstream testing objective. The method is therefore “directed” in the sense that generation is steered toward one or more explicit testing goals rather than toward validity or diversity alone (Kirschner et al., 2 Aug 2025).

The motivating claim is that standard grammar-based generators usually rely on uniform production choices or on fixed probabilistic choices learned from seeds. Such choices may preserve realism or validity, but they do not adapt to execution feedback about which structures actually induce the desired behavior. FdLoop was proposed to bridge that gap by combining probabilistic grammar learning, evolutionary test generation, input mutation, grammar mutation, and a fitness function over one or more testing goals.

The paper summarizes the method as one that “iteratively selects, evolves and learns the input distribution of goal-specific test inputs via test feedback and a probabilistic grammar.” In that sense, FdLoop belongs to the line of work that treats structured-input generation as a closed-loop optimization problem rather than as one-shot sampling from a fixed grammar. A plausible implication is that FdLoop should be understood less as a static fuzzer than as an adaptive grammar-guided search procedure.

2. Representations, grammar model, and objective

FdLoop assumes a subject program that accepts structured inputs, an input grammar describing valid inputs, a set of seed inputs, and a fitness function corresponding to the testing goal or goals (Kirschner et al., 2 Aug 2025). Inputs exist simultaneously in several representations: as concrete strings generated from the grammar, as parse trees, and as UTF-8 text encoded as bit arrays for mutation.

The grammar is represented as a probabilistic grammar whose structure remains fixed while each production alternative receives a probability. The paper gives the genetic representation of these probabilities as

$$[[p_{0,0},p_{0,1},\dots],[p_{1,0},p_{1,1},\dots],\dots]$$

where $p_{i,j}$ is the probability of the $j$-th grammar alternative under the $i$-th grammar expansion rule. Initial probabilities are learned from seed inputs by parsing them and counting expansion choices. The paper illustrates this with a Euclid example in which productions such as integer, number, and digit receive nonuniform probabilities derived from the seed corpus.

The optimization target is a scalar fitness value formed from normalized execution feedback. FdLoop uses four feedback channels: number of mappings, number of covered functions, program runtime, and exception behavior. The overall objective is

$$\text{Fitness } F = \mu_1x_1 + \mu_2x_2 + \mu_3x_3 + \mu_4x_4$$

with $a$ the number of mappings, $a_{max}$ a pre-estimated maximum number of mappings, $b$ the number of covered functions, $b_{tot}$ the total number of functions, $c$ the program runtime, $p_{i,j}$0 the number of exceptions, $p_{i,j}$1 the number of unique exceptions, and $p_{i,j}$2 user-chosen weights. The paper gives

$p_{i,j}$3

For the input-to-code complexity goal, the method defines a mapping between input features and executed methods as

$p_{i,j}$4

where $p_{i,j}$5 is the set of input features extracted from the parse tree and $p_{i,j}$6 is the set of executed functions. The paper emphasizes that these features are derived from rule types and from all subsets of the parse tree up to a certain depth.

This formulation makes FdLoop explicitly scalarized rather than Pareto-based. The paper also supports several weighting modes: single-goal mode assigns target weight $p_{i,j}$7 and all others $p_{i,j}$8; multiple-goals mode sets all weights to $p_{i,j}$9; ignore-goal mode sets the ignored goal to $j$0 and the others to $j$1. This suggests that FdLoop is intended to support both focused objective pursuit and coarse multi-objective balancing without changing the underlying search loop.

3. Iterative feedback loop and search procedure

FdLoop begins by learning an initial probabilistic grammar from seed inputs,

$j$2

and then iterates over generations (Kirschner et al., 2 Aug 2025). In each generation it generates inputs from the current grammar,

$j$3

mutates those inputs,

$j$4

collects parse-tree information and execution feedback for each candidate, and selects the best-performing input according to the fitness function: $j$5

The selected input is then added to the accumulated set,

$j$6

and used to update the grammar: $j$7 This produces the next-generation sampling distribution. The paper’s explicit pseudocode makes clear that the grammar update is based on the selected best input rather than on the entire candidate population.

Operationally, the loop can be read as alternating between exploitation and exploration. Exploitation enters through grammar relearning from the current best-performing input. Exploration enters through both input mutation and grammar mutation. The method repeats for a certain number of generations, until a goal is reached, or until an arbitrary budget is exhausted. In the reported experiments, typical budgets were 50 generations for effectiveness and sensitivity experiments and 48 generations for baseline comparison, usually with 5 inputs per generation.

A common misconception about grammar-based testing is that once a probabilistic grammar has been learned from seeds, further improvement is mainly a matter of generating more samples. FdLoop explicitly rejects that premise: its search dynamics depend on continual execution-guided redistribution of production probabilities, not on static probabilistic sampling.

4. Mutation, feedback, and supported testing goals

FdLoop uses both an input mutator and a grammar mutator (Kirschner et al., 2 Aug 2025). On the input side, the paper describes two mutation operators. The first is a bit-flip on the UTF-8 bit-array representation. The paper illustrates this with ASCII 52 (110100) mutating to 48 (110000), changing digit 4 to 0. The second is parse-tree swap: the input is parsed into a derivation tree, two distinct subtrees whose roots correspond to the same grammar rule index are selected, and the subtrees are swapped. Because the roots match, the resulting input remains syntactically valid after unparsing.

Grammar mutation does not alter the grammar rules or tree structure; it alters probabilities only. If a rule with index $j$8 has $j$9 alternatives, mutation resets that rule to a uniform distribution: $i$0 This is a targeted flattening operation rather than a structural grammar rewrite.

The method is instantiated for four testing goals. For code coverage, the feedback is the number of covered functions. For input-to-code complexity, the feedback is the number of mappings $i$1. For failures, the paper uses both total exceptions and unique exceptions, with the normalized term

$i$2

which weights unique exceptions more strongly than repeated ones. For long execution time, runtime is normalized as

$i$3

The paper’s component analysis reports that grammar mutation contributes the most, “up to 20%,” especially for coverage, mappings, and unique exceptions, while input mutation contributes less overall, typically up to about 1.7% in the reported aggregated results (Kirschner et al., 2 Aug 2025). The paper also states that mappings and exceptions contribute the most, 86%, to FdLoop’s effectiveness in simultaneously achieving multiple testing goals. Both mutation components contribute little or negatively for runtime, which the paper interprets as evidence that runtime may require more code-aware feedback than the current design provides.

This design clarifies that FdLoop is not merely grammar-guided random testing with a fitness wrapper. Its distinctive feature is that feedback is used twice: once for candidate ranking and once for distribution learning.

5. Evaluation, baselines, and empirical findings

The evaluation uses 3 input formats—JSON, CSS, and JavaScript—and 20 open-source Java subject programs: 13 JSON systems, 5 CSS systems, and 2 JavaScript systems (Kirschner et al., 2 Aug 2025). The seed corpus was built by crawling 300,000 JSON files, 5,500 CSS files, and 200,000 JavaScript files from GitHub, then randomly selecting 1000 files; default experiments used 500 randomly selected seeds. The implementation comprises about 5.3K lines Python and about 3.9K lines Java, is built on Tribble, uses ANTLR for parsing, and was evaluated in Docker on Debian Bullseye.

FdLoop is compared against five baseline families: a random baseline, a random grammar-based fuzzer, a probabilistic grammar-based fuzzer, an inverse probabilistic grammar-based fuzzer, EvoGFuzz, and DynaMOSA via EvoSuite. The headline result is that FdLoop outperforms the baselines in 86% (43/50) of all tested settings (Kirschner et al., 2 Aug 2025).

Against seed inputs, FdLoop outperforms or matches seeds in 73 out of 100 cases in single-goal mode and in 76% of cases in multiple-goal mode. Aggregate improvements over seeds are reported as +5.38% coverage, +47.08% exceptions, and +62.50% unique exceptions in single-goal mode, and +6.33% coverage, +43.02% exceptions, and +58.33% unique exceptions in multiple-goal mode. The paper notes that mappings and runtime can remain weaker than rich human-written seeds, especially for CSS and JavaScript, because seed inputs may encode semantic structures not captured by context-free grammars.

In the baseline comparison over 10 common subjects, FdLoop (Single) achieves Coverage 35.89, Mappings 11,147,735, Exceptions 937, Unique Exceptions 14, and Runtime 2847.10 sec.; FdLoop (Multiple) achieves Coverage 36.25, Mappings 10,901,060, Exceptions 865, Unique Exceptions 13, and Runtime 2210.51 sec. The best baselines are Probabilistic baseline for coverage at 32.30, EvoGFuzz for mappings at 13,173,177, EvoGFuzz for exceptions at 575, EvoGFuzz for unique exceptions at 7, and Random baseline for runtime at 2871.63. Relative to the best baseline, FdLoop (Single) reports +11.11% coverage, -15.38% mappings, +62.97% exceptions, +100% unique exceptions, and -0.85% runtime; FdLoop (Multiple) reports +12.23% coverage, -17.25% mappings, +50.43% exceptions, +85.71% unique exceptions, and -23.02% runtime.

The paper’s strongest bug-finding claim is that FdLoop is twice (2X) as effective as the best baseline, EvoGFuzz, in inducing unique exceptions. Concretely, FdLoop triggered 18 unique exceptions, EvoGFuzz triggered 7, and DynaMOSA triggered 5. The paper further states that 6 exceptions were triggered only by FdLoop, while 1 exception was triggered only by DynaMOSA. It also reports a statistically significant difference against EvoGFuzz mainly for runtime, with Mann–Whitney $i$4.

These findings establish a characteristic performance profile. FdLoop is strongest on coverage and exception discovery, weaker in aggregate on mappings and runtime, and comparatively expensive: the paper notes that it is about 2X the time of EvoGFuzz in one reported comparison table. A plausible implication is that FdLoop’s feedback loop is most beneficial when the target behavior is sparse and structurally mediated, as in failure-inducing input generation.

6. Scope, limitations, and place within grammar-based fuzzing

FdLoop’s main strength is that it turns grammar-based generation into a goal-directed adaptive process while preserving syntactic validity (Kirschner et al., 2 Aug 2025). It supports arbitrary goal-directed testing through fitness weighting, works with structured inputs through grammars, combines validity-preserving generation with feedback-driven adaptation, and supports single-goal, multiple-goal, and ignore-goal strategies. The paper positions it as especially suitable for parsers and processors of structured textual inputs such as JSON, CSS, and JavaScript.

The paper is also explicit about its limitations. FdLoop depends on grammar quality and is sensitive to grammar ambiguity, recursion, and the gap between grammar coverage and real accepted inputs. The authors manually adapted ANTLR grammars to mitigate ambiguity. The method mainly exploits context-free structure rather than semantic constraints such as ordering or variable dependencies, which helps explain why human seed inputs can outperform it on some JavaScript and CSS mappings or runtime behaviors. Runtime remains hard to optimize robustly, and the weighted-sum scalarization assumes that the user can express objective preferences through fixed weights; the paper notes that if Pareto-optimal tradeoffs among conflicting objectives are desired, a many-objective method might be preferable.

It is therefore inaccurate to treat FdLoop as a universal replacement for grammar-based fuzzers or for many-objective search. The paper instead supports a narrower and more technical interpretation: FdLoop is a feedback-driven, evolutionary, probabilistic grammar-based method for generating goal-specific structured inputs, with particularly strong performance on exception discovery and coverage. Its defining contribution is not grammar inference alone, mutation alone, or fitness-guided selection alone, but the closed loop connecting execution feedback to grammar redistribution.

In that sense, FdLoop represents a shift in emphasis within grammar-based test generation. Rather than asking only how to generate valid inputs from a grammar, it asks how to learn, from observed executions, which grammar choices matter for a specified testing objective and then to bias future generations toward those choices.