LLM4FP: Differential Testing for Floating-Point Software
- LLM4FP is a framework that leverages LLMs to generate diverse floating-point C programs, targeting differential compiler inconsistencies.
- It combines grammar-based generation with feedback-driven mutation to create valid and varied test cases covering key numerical constructs.
- Empirical evaluations show LLM4FP detects over twice as many inconsistencies as Varity, revealing subtle real-valued discrepancies across multiple compilers.
LLM4FP is a framework for differential testing of floating-point software that uses LLMs to generate floating-point programs specifically designed to trigger inconsistencies across compilers (Wang et al., 29 Aug 2025). It combines Grammar-Based Generation and Feedback-Based Mutation to produce diverse and valid programs, then compiles and executes them under multiple compilers and optimization levels to detect bitwise output mismatches. Within the reported evaluation, LLM4FP detects over twice as many inconsistencies compared to the state-of-the-art tool, Varity, and most of the inconsistencies involve real-valued differences, rather than extreme values like NaN or infinities (Wang et al., 29 Aug 2025).
1. Origin, objective, and problem setting
LLM4FP was introduced in the paper "LLM-Based Program Generation for Triggering Numerical Inconsistencies Across Compilers" (Wang et al., 29 Aug 2025). Its stated objective is to address floating-point inconsistencies across compilers, which can undermine the reliability of numerical software. The framework is presented as the first that uses LLMs to generate floating-point programs specifically designed to trigger such inconsistencies (Wang et al., 29 Aug 2025).
The immediate baseline context is differential testing for floating-point behavior. Varity is described as the state-of-the-art tool for discovering such inconsistencies using random program generation and differential testing, but it suffers from efficiency and coverage limitations (Wang et al., 29 Aug 2025). LLM4FP is positioned against this background as an LLM-guided alternative that aims to improve the yield, coverage, and usefulness of generated test programs.
The target phenomenon is not restricted to conspicuous failures. The reported results emphasize that most detected inconsistencies are subtle real-valued mismatches rather than exceptional outputs such as NaN, +Inf, or -Inf (Wang et al., 29 Aug 2025). This is significant because such discrepancies are easier to miss in ordinary testing pipelines while still being relevant to the reliability of scientific and engineering software.
2. Framework architecture and testing workflow
The workflow of LLM4FP consists of five stages (Wang et al., 29 Aug 2025). First, the system selects a generation strategy, either Grammar-Based Generation or Feedback-Based Mutation. Second, it prompts an LLM to generate a new C floating-point program. Third, it prepares and compiles the program for host and device compilers under various optimization levels. Fourth, it performs differential testing by executing the resulting binaries and comparing their outputs. Fifth, if inconsistencies are found, the generated program is added to the set of successful seeds for future mutation (Wang et al., 29 Aug 2025).
The generated programs follow a constrained structure. Each program consists of two functions, main and compute. The compute function takes scalar floating-point arguments, performs arithmetic and possibly calls Math library routines, and returns a scalar result printed by main (Wang et al., 29 Aug 2025). The allowed constructs are governed by an explicit context-free grammar supporting arithmetic operations {+, -, *, /}, conditionals, nested loops, temporary variables, arrays, standard math functions, and precision specification as float or double (Wang et al., 29 Aug 2025).
This grammar constraint is central to the framework design. It is used to steer the LLM toward well-formed floating-point programs, avoid invalid or trivial outputs, and enforce coverage of interesting floating-point constructs (Wang et al., 29 Aug 2025). A plausible implication is that LLM4FP treats the LLM less as an unrestricted code synthesizer than as a guided generator within a formally delimited test space.
3. Generation strategies
LLM4FP combines two complementary generation strategies: Grammar-Based Generation and Feedback-Based Mutation (Wang et al., 29 Aug 2025).
Grammar-Based Generation prompts the LLM to create random, valid floating-point C programs while specifying the precision, the high-level and grammar-based internal structure, coding robustness requirements, and the requirement to output plain code without explanations (Wang et al., 29 Aug 2025). The aim is to ensure both diversity and validity while keeping the generated programs aligned with floating-point constructs likely to expose compiler differences.
Feedback-Based Mutation starts from a previously successful inconsistency-triggering program and instructs the LLM to generate a different valid variant (Wang et al., 29 Aug 2025). The mutation prompt includes the same structural requirements as Grammar-Based Generation, but adds mutation-oriented guidance such as reordering or nesting expressions, changing constants, adding new loops or conditionals, and using different math functions. The previously successful program is embedded in the prompt, and the model is explicitly required to generate a different program (Wang et al., 29 Aug 2025).
The two strategies are randomly selected per new generated program, typically with 50% probability (Wang et al., 29 Aug 2025). This coupling of unconstrained exploration and success-driven mutation is one of the defining properties of LLM4FP. The paper’s framing suggests that Grammar-Based Generation expands coverage, while Feedback-Based Mutation exploits discovered fault-inducing patterns (Wang et al., 29 Aug 2025).
4. Differential testing methodology
For each generated program, LLM4FP compiles host code with gcc and clang, and device code with nvcc (Wang et al., 29 Aug 2025). It evaluates the resulting binaries under six optimization levels: O0_nofma, O0, O1, O2, O3, and O3_fastmath (Wang et al., 29 Aug 2025). O0_nofma is described as most compliant to IEEE 754 and disables Fused Multiply-Add (Wang et al., 29 Aug 2025).
The framework defines inconsistency by bitwise output mismatch. Outputs are compared via hex-encoded floating-point values across every pair of compilers at each optimization level (Wang et al., 29 Aug 2025). The reported formulation is
where is the number of compilers, the number of optimization levels, and the number of programs (Wang et al., 29 Aug 2025). The inconsistency rate is then given by
The paper also classifies mismatches by output type, including {Real, Real}, {Real, Zero}, {Real, ±Inf}, and {Real, NaN} (Wang et al., 29 Aug 2025). This classification permits analysis beyond raw counts by distinguishing subtle numeric divergence from exceptional behavior. The paper further tracks the occurrence and distribution of each class across optimization levels (Wang et al., 29 Aug 2025).
5. Empirical results and comparative performance
The evaluation uses a budget of 1,000 program generations per approach, compiled and run across three compilers and six optimization levels, totaling 18,000 program executions per tool (Wang et al., 29 Aug 2025). The environment includes gcc 9.4, clang 12.0, nvcc 12.3, x86_64, Ubuntu 20.04, and an NVIDIA RTX A6000 GPU (Wang et al., 29 Aug 2025).
The main quantitative comparison is summarized below.
| Approach | Inconsistency Rate | Total Inconsistencies | Time Cost |
|---|---|---|---|
| Varity | 11.93% | 2,147 | 00:30:42 |
| Direct-Prompt | 13.43% | 2,417 | 03:23:40 |
| Grammar-Guided | 15.80% | 2,844 | 03:16:44 |
| LLM4FP | 26.56% | 4,781 | 03:22:00 |
These results show that LLM4FP detects over twice as many inconsistencies as Varity, with an inconsistency rate of 26.56% versus 11.93% (Wang et al., 29 Aug 2025). It also outperforms two LLM baselines: a Direct-Prompt baseline and a Grammar-Guided baseline without feedback-based mutation (Wang et al., 29 Aug 2025).
The nature of the detected inconsistencies is also important. LLM4FP triggers more than 92% of inconsistencies in the {Real, Real} class (Wang et al., 29 Aug 2025). Extreme-value mismatches involving NaN or infinities are reported as rare and mainly associated with O3_fastmath (Wang et al., 29 Aug 2025). The paper additionally states that LLM4FP inconsistencies often involve a large number of differing floating-point digits, with average differences greater than 10 digits, suggesting more severe discrepancies (Wang et al., 29 Aug 2025).
In diversity analysis, Varity has a CodeBLEU score of 0.3581, while LLM4FP has 0.3610 (Wang et al., 29 Aug 2025). Direct-Prompt and Grammar-Guided are reported as less diverse, with higher CodeBLEU values of 0.4213 and 0.5099 respectively (Wang et al., 29 Aug 2025). LLM4FP slightly increases code clone occurrence, but only 6.2% of its outputs are close clones, compared with 0% for Varity and 2% for Grammar-Guided (Wang et al., 29 Aug 2025).
6. Cross-compiler behavior, significance, and limitations
A major empirical finding is that most inconsistencies occur between host and device compilers, specifically gcc versus nvcc and clang versus nvcc (Wang et al., 29 Aug 2025). LLM4FP also exposes more host-host inconsistencies than Varity, but host-device mismatches dominate (Wang et al., 29 Aug 2025). The framework further uncovers inconsistencies across a wider range of optimization levels, rather than concentrating only at aggressive settings such as O3_fastmath (Wang et al., 29 Aug 2025). The paper states that Varity mainly finds inconsistencies only at aggressive optimization, whereas LLM4FP discovers discrepancies even at low and moderate optimizations such as O1, O2, and O3 (Wang et al., 29 Aug 2025).
The broader significance claimed for LLM4FP is that LLM-guided program generation improves the detection of numerical inconsistencies (Wang et al., 29 Aug 2025). The paper argues that the framework enables deeper and broader detection of subtle, silent numerical inconsistencies, especially in heterogeneous computing environments involving mixed CPU/GPU workflows (Wang et al., 29 Aug 2025). A plausible implication is that LLM4FP is not merely a code-generation benchmark, but a testing methodology for probing compiler- and optimization-induced numerical instability.
The paper also states explicit limitations. Not all detected inconsistencies are bugs, and the framework does not yet distinguish harmful from benign differences (Wang et al., 29 Aug 2025). Time cost remains higher than Varity, mainly due to LLM API call latency (Wang et al., 29 Aug 2025). Future work is described as targeting semantic-guided diversity beyond grammar-based structure, faster LLM inference for scalability, and integration with static analysis or undefined behavior checkers to reduce false positives (Wang et al., 29 Aug 2025).
In this sense, LLM4FP occupies a specific position in the landscape of testing tools: it preserves grammar-based validity, augments it with LLM-driven synthesis and mutation, and uses differential execution across compilers and optimization levels to reveal floating-point mismatches that are often real-valued, non-crashing, and operationally relevant (Wang et al., 29 Aug 2025).