StaAgent: LLM-Powered Static Analyzer Testing

• StaAgent is an LLM-powered testing framework that uses rule-driven synthesis and metamorphic testing to assess static analyzer rule implementations.
• It employs a four-agent pipeline—seed generation, code validation, mutation creation, and analyzer evaluation—to systematically generate and test bug-inducing programs.
• Experimental results reveal that StaAgent uncovered 64 problematic rules across five analyzers, demonstrating its effectiveness over baseline systems like Statfier.

StaAgent is an LLM-powered, agentic testing framework for static analyzers that uses rule-driven synthesis, validation, semantic-preserving mutation, and metamorphic testing to identify problematic rule implementations. Its central premise is that semantically equivalent programs should elicit consistent behavior from a static analyzer; when analyzer behavior diverges across equivalent variants, the inconsistency is treated as evidence of a faulty rule implementation. In the reported evaluation, StaAgent was instantiated with five state-of-the-art LLMs and applied to five widely used static analyzers, revealing 64 problematic rules, of which 53 were not detected by the state-of-the-art baseline Statfier; all identified bugs were reported to developers, with two already fixed and three confirmed at the time of writing (Nnorom et al., 20 Jul 2025).

1. Concept and motivation

StaAgent was proposed in response to a specific weakness in the quality assurance of static analysis tools: although static analyzers are widely used to detect bugs, security issues, and performance problems, their rule implementations are often under-tested, inconsistent, prone to false negatives, and difficult to validate manually at scale (Nnorom et al., 20 Jul 2025). Prior metamorphic testing systems, especially Statfier, depend on human-created seed programs or curated test suites. That dependence makes them effective but labor-intensive and harder to generalize across analyzers and rules.

The framework addresses this gap by treating rule descriptions themselves as inputs to an agentic synthesis-and-testing pipeline. Rather than beginning from a manually assembled corpus of buggy programs, StaAgent extracts rule descriptions from analyzer documentation, generates bug-inducing programs intended to realize those descriptions, validates whether the synthesized programs actually manifest the intended bug behavior, mutates them into semantically equivalent variants, and then checks whether the analyzer behaves consistently across the original and mutated programs. This suggests a shift from manually curated metamorphic testing toward rule-driven, LLM-mediated generation.

The system is therefore not merely a code generator. Its intended function is to test the reliability of rule implementations in static analyzers by turning natural-language rule descriptions into executable evidence about analyzer behavior. A plausible implication is that StaAgent operationalizes rule documentation as a test specification, thereby reducing the manual effort required to scale analyzer testing across many rules.

2. Multi-agent architecture and execution pipeline

StaAgent comprises four specialized agents arranged in a fixed workflow (Nnorom et al., 20 Jul 2025).

Agent	Function
Seed Generation Agent	Translates bug detection rules into concrete, bug-inducing seed programs
Code Validation Agent	Ensures the correctness of these seeds
Mutation Generation Agent	Produces semantically equivalent mutants
Analyzer Evaluation Agent	Performs metamorphic testing by comparing analyzer behavior on seeds and mutants

The workflow begins by extracting rule descriptions from the static analyzer’s documentation. The Seed Generation Agent receives a rule title and description and asks an LLM to generate a small, self-contained Java seed program that exhibits the bug described by the rule. The paper specifies three seed requirements: the program should be compilable, self-contained, and free of third-party libraries. The bug is to be isolated in a simple function such as showBug(). Compilation is part of the synthesis loop: if compilation fails, the error message is fed back to the LLM, and refinement repeats up to five times. Uncompilable candidates are discarded. Metadata for each generated seed includes the corresponding rule and the buggy lines in the program.

Compilation alone is not treated as sufficient evidence that the seed is valid. The Code Validation Agent generates tests for the seed program and checks whether the seed genuinely manifests the intended rule violation. Its loop is analogous: it generates a test case, compiles the test and seed together, feeds test, seed, and error trace back to the LLM on failure, retries up to five times, and then executes the test. Validation may inspect failure messages, exception traces, method-call sequences, and runtime behavior. If the bug cannot be reliably triggered, the seed is discarded.

Only validated seeds are passed to mutation. The Mutation Generation Agent determines which mutation operators are applicable, generates $N$ distinct mutants per applicable operator, compiles them, repairs failed mutants iteratively via LLM feedback, and executes the same tests on both mutant and original seed. A mutant is retained only if behavior is identical in output, exceptions, and execution trace. The Analyzer Evaluation Agent then runs the static analyzer on the seed and its validated mutants and flags inconsistent detection behavior as a potential bug in the rule implementation.

3. Metamorphic testing principle and mutation system

StaAgent is built around a simple metamorphic relation. Let $S$ denote a seed program intended to trigger a rule, $M(S)$ a semantically equivalent mutant, and $A(\cdot)$ the static analyzer. The desired consistency condition is

$A(S) = A(M(S))$

When the seed and mutant are semantically equivalent, analyzer behavior should ideally remain consistent (Nnorom et al., 20 Jul 2025). A mismatch is interpreted as a potential rule defect.

The paper distinguishes two forms of problematic behavior. Type 1: Inconsistent Detection occurs when the analyzer detects the seed bug but not all semantically equivalent mutants. Type 2: False Negatives Across Variants occurs when the analyzer fails to detect both the seed and at least one equivalent mutant. The first indicates brittleness under semantically preserving variation; the second indicates that the rule is not robust enough even on the intended seed instance and its equivalent variants.

The mutation stage reuses nine semantic-preserving operators:

• Op1: dead store
• Op2: duplication / obfuscating numerical expressions
• Op3: unreachable if
• Op4: unreachable if-else
• Op5: unreachable switch
• Op6: unreachable for
• Op7: unreachable while
• Op8: renaming
• Op9: equivalent do-while

These operators vary syntax and structure while aiming to preserve meaning. The paper gives representative examples: Op1 inserts an unused variable declaration; Op2 rewrites a numeric expression such as $x = 1.0$ into $x = 1.0 + 0.1 - 0.1$; Op3 through Op7 insert dead branches or loops with false conditions; Op8 renames a local variable; and Op9 replaces a loop with a semantically equivalent do-while form. Some mutation operators from Statfier were intentionally not adopted because they may not change compiled bytecode enough to be useful for bytecode-level analyzers such as SpotBugs, Infer, ErrorProne, and SonarQube.

A key technical point is that StaAgent does not assume semantic equivalence from the mutation template alone. Mutants are filtered by dynamic equivalence tests, and only those with identical output, exceptions, and execution trace are kept. This gives the metamorphic relation an executable validation layer rather than leaving equivalence as a purely syntactic assumption.

4. Experimental setting and quantitative results

The reported evaluation used five state-of-the-art LLMs—CodeLlama-34B-Instruct-hf, DeepSeek-Coder-33B-Instruct, Codestral-22B-v0.1, Qwen2.5-Coder-32B, and GPT-4o—and five static analyzers: SpotBugs version 4.9.3, SonarQube version 25.5, ErrorProne version 2.40.0, Infer version 1.2.0, and PMD version 7.14.0 (Nnorom et al., 20 Jul 2025). The selected rule counts were 199 for SpotBugs, 173 for SonarQube, 177 for ErrorProne, 26 for Infer, and 125 for PMD. The study focused on rules related to correctness, security, performance, and malware detection, excluding style, maintainability, framework-specific rules, and test-code rules. Experimental parameters included generation temperature 0.75, validation temperature 0.1, maximum refinement iterations 5, and 3 mutants per operator per seed. The hardware environment was a GPU cluster with 4 NVIDIA L40S GPUs and an Intel Xeon Gold 6442Y CPU.

The central empirical result is that StaAgent uncovered 64 problematic rules across the five analyzers: 28 in SpotBugs, 18 in SonarQube, 6 in ErrorProne, 4 in Infer, and 8 in PMD. Across all tools, these comprised 43 Type 1 bugs and 21 Type 2 bugs. The comparison with Statfier is one of the paper’s main claims: 53 of the 64 bugs, or 83%, were not detected by the state-of-the-art baseline. The tool-wise overlap data suggest partial complementarity: for SpotBugs, StaAgent found 28 total bugs while Statfier found 12, with 8 common; for SonarQube, StaAgent found 18 and Statfier found 0; for Infer, StaAgent found 4 and Statfier found 2, with 0 common; and for PMD, StaAgent found 8 while Statfier found 11, with 3 common.

The paper also reports internal performance characteristics of the four-agent pipeline. For the Seed Generation Agent, most LLMs achieved 95.22%–98.29% compilation success, while CodeLlama achieved 40.92%. The Code Validation Agent’s best validation accuracy came from GPT-4o and Qwen, both around 79%; the plotted approximate values were 0.3966 for CodeLlama, 0.6131 for Codestral, 0.5742 for DeepSeek, 0.7912 for GPT-4o, and 0.7978 for Qwen. For the Mutation Generation Agent, 68.86% of generated mutants were semantically valid overall. By operator, the valid-mutant rates were approximately 74% for Op1, 66% for Op2, 71% for Op3, 66% for Op4, 62% for Op5, 67% for Op6, 67% for Op7, 53% for Op8, and 72% for Op9.

The most effective operators for finding Type 1 bugs were those that introduced control-flow complexity. The paper reports 5 bugs from Op1, 6 from Op2, 17 from Op3, 16 from Op4, 13 from Op5, 17 from Op6, 20 from Op7, 6 from Op8, and 7 from Op9. This supports the paper’s observation that structural complexity helps expose analyzer flaws.

An additional experiment replaced StaAgent’s synthesized seeds with Statfier’s seed programs, thereby skipping the Seed Generation and Code Validation agents and retaining only mutation and analyzer evaluation. Under this setting, StaAgent found 10 bugs for SpotBugs versus 12 by Statfier, and 2 bugs for Infer versus 2 by Statfier, with 0 common bugs in both cases. This suggests that StaAgent’s mutation-and-evaluation stages can complement Statfier even when both begin from the same seeds.

5. Validation outcomes, limitations, and technical significance

All 64 identified bugs were reported to developers. At the time reported in the paper, 2 had been fixed, 3 had been confirmed, and the rest were awaiting response (Nnorom et al., 20 Jul 2025). These outcomes are limited in number, but they provide external evidence that at least part of the reported inconsistency set corresponds to actionable defects rather than only experimental artifacts.

The paper also states several limitations. Not all bugs are easily test-triggerable; some valid bugs require specific runtime conditions or external dependencies. The experimental scope is limited to Java and five analyzers, so results may differ for other languages and tools. Performance depends on the chosen LLMs, and the paper explicitly notes model-dependent behavior. False positives remain an issue, especially for models such as DeepSeek, which sometimes accidentally fixed the seed while mutating it or changed the test logic rather than following the prompt. The authors also suggest improved prompting strategies, including CoT, as future work to reduce false positives.

These constraints matter for interpreting the framework’s contribution. StaAgent is not presented as a proof that LLM-driven analyzer testing is universally reliable. Rather, it is presented as a scalable and adaptable framework that combines rule-driven generation, LLM-based validation, semantic-preserving mutation, and metamorphic testing into a single automated pipeline. A plausible implication is that the framework’s main novelty lies less in any individual stage than in the composition of those stages into an end-to-end system for turning analyzer documentation into bug evidence.

6. Terminological ambiguity and related uses of “StaAgent”

The name “StaAgent” is not unique in the literature, and this has produced some ambiguity. In "Exploratory Test Agents for Stateful Software Systems" (Karlsson, 2020), StaAgent is not a finished tool for static analyzer testing but a proposed exploratory test agent approach for stateful software systems. That paper frames StaAgent as a research vision involving autonomous or semi-autonomous agents that explore software systems to find faults and gain knowledge, including possible agent types such as fuzzing agent, smoke-test agent, chaos agent, client agent, and security agent. It is concerned with long-running stateful interactions, interaction models such as GUI, FSM, and Swagger/API artifacts, and property-based exploratory assertions, rather than metamorphic testing of static analyzer rules.

A second homonymous usage appears in "An LLM Agent-Based Complex Semantic Table Annotation Approach" (Geng et al., 18 Aug 2025), where StaAgent denotes a ReAct-based LLM agent for Semantic Table Annotation, specifically CTA and CEA. In that work, the system integrates five external tools for preprocessing, column topic detection, DBpedia-based knowledge graph enhancement, CTA candidate ranking, and context-supported CEA selection. Its problem setting is semantic annotation of complex tables rather than software testing.

Accordingly, “StaAgent” can refer to materially different systems across subfields. In the software-engineering sense established by the 2025 paper, the term denotes the agentic framework for testing static analyzers through rule-driven synthesis and metamorphic comparison (Nnorom et al., 20 Jul 2025). In other contexts, it denotes either a conceptual exploratory testing approach for stateful systems or a semantic table annotation agent. This distinction is important because surface similarity in naming does not imply methodological continuity.