BASFuzz: Robust Fuzz Testing for LLMs

• BASFuzz is an automated fuzz testing methodology for LLM-based NLP software, combining semantic-aware mutations with dynamic quality control.
• It employs a hybrid beam-annealing algorithm that merges beam search with simulated annealing to efficiently explore high-dimensional input spaces.
• It integrates text consistency metrics such as BLEU scores to guide mutations, achieving 90.335% testing effectiveness with reduced time overhead.

BASFuzz is an automated fuzz testing methodology designed specifically for robustness evaluation of LLM-based NLP software. The approach focuses on coupling the fuzzing process with the behavioral patterns characteristic of LLM-based systems—particularly in open-ended natural language generation (NLG) scenarios. BASFuzz achieves efficient coverage and adversarial sample generation by integrating advanced search algorithms, semantic-aware mutation strategies, and dynamic quality control mechanisms, validated through experiments on prominent NLG and natural language understanding (NLU) tasks (Xiao et al., 22 Sep 2025).

1. Fuzz Testing Framework and Input Modeling

BASFuzz treats each test instance as a composition of a prompt and one or more examples, reflecting the interaction paradigm in contemporary LLM-based applications. The initial step is to "filter" the input, identifying essence words through stop-word removal and a word importance ranking (WIR) procedure. This WIR is defined mathematically as:

$$WIR(w_i) = \text{softmax}\Big(\mathcal{L}(I_{ori}^i, T_y) - \mathcal{L}(I_{ori}, T_y)\Big) \cdot \Big(\mathcal{L}(I_{ori}^i, T_y) - \mathcal{L}(I_{ori}, T_y)\Big)$$

where $w_i$ is the word under consideration, and $\mathcal{L}$ denotes the negative BLEU score loss post-word masking.

After sensitive words are identified, BASFuzz employs a two-stage construction of its perturbation space: lexical retrieval (from multilingual resources like WordNet for synonyms, hypernyms, and hyponyms) followed by LLM-based high-dimensional vector encoding with cosine similarity filtering, ensuring semantic proximity of all candidate substitutions.

2. Hybrid Beam-Annealing Search Algorithm

The core of BASFuzz's search strategy is a hybrid algorithm that combines beam search with simulated annealing. Beam search maintains multiple mutation trajectories in parallel, preventing premature convergence on local optima typical of single-path greedy strategies. Simulated annealing introduces stochasticity, allowing suboptimal perturbations to be explored probabilistically, governed by an acceptance probability:

$$p_{SA}(I') = \begin{cases} 1, & \text{if } \Delta \mathcal{L} > 0 \ \exp\left(\frac{\Delta \mathcal{L}}{tem_{SA}}\right), & \text{if } \Delta \mathcal{L} \le 0 \end{cases}$$

where $\Delta \mathcal{L}$ is the objective function change and $tem_{SA}$ is the temperature, updated with logarithmic decay:

$$tem_{SA}(t) = \frac{tem_{SA}(0)}{1 + \gamma \cdot \ln(1+t)}$$

This hybrid approach efficiently explores the discrete, high-dimensional input space encountered in LLM-based NLG tasks.

3. Text Consistency Metrics and Mutation Guidance

BASFuzz incorporates a text consistency metric—most notably the BLEU score—to quantify the semantic and structural deviation of LLM-generated outputs relative to reference texts. The BLEU score, defined as:

$$BLEU = BP \cdot \exp \left(\sum_{n=1}^{N} w_n \log P_n\right)$$

where brevity penalty (BP) is:

$$BP = \begin{cases} 1, & \text{if } c > r \ \exp\left(1 - \frac{r}{c}\right), & \text{if } c \le r \end{cases}$$

with $c$ and $r$ representing candidate and reference lengths respectively, acts as the objective function guiding mutations:

$$\operatorname*{arg\,max}_{\delta \in C} \; \mathbb{E}_{(T_x,T_y) \in \mathcal{D}} \mathcal{L}[ \mathcal{F}([P; T_x] + \delta), T_y]$$

where $\mathcal{L}$ is the negative BLEU score and $\delta$ is the perturbation within constraints $C$.

4. Adaptive Exploration: Information Entropy and Elitism

To avoid stagnation and support dynamic focus, BASFuzz adaptively adjusts its beam width based on the entropy $H_{beam}$ of candidate losses. The beam width update formula is:

$$b_{t+1} = \max(b_{min}, \min(b_{max}, b_t \times (1 + (H_{beam} / b_{max})) / (b_t + \sigma) ))$$

where $b_{min}$, $b_{max}$ are bounds and $\sigma$ is a smoothing factor.

An elitism strategy is implemented to probabilistically retain the best candidate test input, with retention probability:

$$p^* = \frac{\exp(\mathcal{L}(I^*, T_y))}{\sum_i \exp(\mathcal{L}(I_i, T_y))}$$

and overall elitism probability:

$p_e = p_0^* + (1 - p_0^*) \cdot p^*$

This maintains diversity while ensuring that high-impact test cases are preserved across iterations.

5. Experimental Evaluation and Comparative Effectiveness

BASFuzz has been evaluated on representative datasets for both NLG (machine translation—CS2EN, DE2EN, RU2EN) and NLU (Financial Phrasebank, AG’s News, MR). It achieved a testing effectiveness of 90.335% and reduced the average time overhead by 2,163.852 seconds relative to leading baselines such as ABS, ABFS, GreedyFuzz, and MORPHEUS (Xiao et al., 22 Sep 2025). The method successfully produces adversarial test cases that induce significant output degradation (lower BLEU) with minimal semantic distortion—only approximately 3–4% of words are perturbed per input. Quality metrics including perplexity (fluency) and grammar error rates underscore that BASFuzz’s mutants are linguistically plausible and stealthy, aligning with practical robustness testing requirements for LLM-based NLP deployments.

6. Significance and Distinctive Characteristics

BASFuzz’s approach is distinguished by tight integration of semantic-based mutation selection, a dynamic hybrid search loop (beam-annealing), and explicit consistency metrics that couple mutation guidance to LLM behavioral patterns. Unlike traditional fuzzing strategies which treat input–output mappings in a static or category-driven fashion, BASFuzz evaluates mutations in the context of realistic prompt-plus-example input modalities and leverages LLM embeddings for meaning-preserving substitutions. The entropy-based adaptive adjustment and elitism strategies further enhance exploration efficiency, making the method highly suitable for pre-deployment robustness audits in complex LLMing systems.

A plausible implication is that BASFuzz's methodology could be extended to robustness analysis for other generative AI systems where the output space is high-dimensional and mutation search must respect semantic plausibility.

7. Limitations and Prospective Research Directions

While BASFuzz outperforms existing methods in efficacy and efficiency, the current design does mandate access to word-level semantic resources and LLM embeddings for meaningful mutation ranking. This suggests possible limitations when applied to resource-scarce languages or domain-specific jargon. Future research might address variants of the text consistency objective suitable for multimodal or cross-lingual systems, deeper coupling with black-box LLM APIs, or automated parameter tuning of beam width and annealing schedules.

BASFuzz represents a state-of-the-art example of robustness-oriented fuzz testing in intelligent NLP software, rigorously aligning input mutation strategies with model behavior and output evaluation for maximal impact in both research and industrial deployment contexts (Xiao et al., 22 Sep 2025).