Less Is More: Measuring How LLM Involvement affects Chatbot Accuracy in Static Analysis

Abstract: LLMs are increasingly used to make static analysis tools accessible through natural language, yet existing systems differ in how much they delegate to the LLM without treating the degree of delegation as an independent variable. We compare three architectures along a spectrum of LLM involvement for translating natural language to Joern's query language \cpgql{}: direct query generation (\approach{1}), generation of a schema-constrained JSON intermediate representation (\approach{2}), and tool-augmented agentic generation (\approach{3}). These are evaluated on a benchmark of 20 code analysis tasks across three complexity tiers, using four open-weight models in a 2(\times)2 design (two model families (\times) two scales), each with three repetitions. The structured intermediate representation (\approach{2}) achieves the highest result match rates, outperforming direct generation by 15--25 percentage points on large models and surpassing the agentic approach despite the latter consuming 8(\times) more tokens. The benefit of structured intermediates is most pronounced for large models; for small models, schema compliance becomes the bottleneck. These findings suggest that in formally structured domains, constraining the LLM's output to a well-typed intermediate representation and delegating query construction to deterministic code yields better results than either unconstrained generation or iterative tool use.

• The paper demonstrates that structured intermediate representation (A#1{2}) improves result match rates by 15–25 percentage points and ensures 100% execution success for large models.
• It compares three architectures and reveals that increasing LLM autonomy, as seen in direct and agentic methods, leads to error compounding despite higher token consumption.
• The study highlights that decomposing semantic parsing from deterministic query generation significantly enhances reliability in safety-critical static analysis applications.

Architectural Analysis of LLM Delegation in Static Analysis Chatbots

Motivation and Problem Statement

The integration of LLMs into static analysis pipelines addresses a fundamental challenge: bridging natural language requirements from developers to formal, domain-specific query languages used by tools such as Joern’s CPGQL. Despite advances in LLMs for code synthesis and retrieval, their efficacy in generating syntactically and semantically correct queries for formally structured DSLs remains undermined by stochastic decoding, hallucinations, and limited exposure to niche languages in pretraining. There is a lack of empirical rigor regarding how much of the pipeline should be delegated to the LLM—a crucial design variable affecting translation fidelity and reliability. The paper approaches this gap by benchmarking three architectures with varying levels of LLM autonomy and establishing concrete correlations between involvement and outcome accuracy.

Architectural Spectrum and Experimental Design

Three architectures are instantiated to systematically control and measure LLM involvement:

• A#1{1} (Direct Generation): The LLM generates CPGQL queries directly, leveraging retrieval-augmented prompts with syntactic references and examples. The output space is unconstrained; the LLM must perform fluency in a low-resource, Scala-based DSL outside typical pretraining corpora.
• A#1{2} (Structured Intermediate Representation): The LLM produces a schema-constrained JSON object, specifying all parameters of the analysis task. Query construction is handled by deterministic code, reducing the LLM’s task to selecting among well-typed fields—effectively shifting the problem towards classification within a compact output space.
• A#1{3} (Tool-Augmented Agentic Generation): Analysis operations are exposed as discrete tools via function-calling APIs. The LLM orchestrates multi-step tool invocations and argument selection using ReAct-style loops, compounding uncertainty across steps.

The architectures are evaluated across 20 natural-language-to-query translation tasks stratified by structural, data flow, and composite complexity tiers. Four open-weight models—Qwen and Llama, each at two scales (7–8B and 70–72B)—are tested in a controlled 2×2 design, producing 720 trials. Each trial is assessed on execution success, result match, token consumption, and architectural robustness, ensuring all approaches operate under uniform retry policies for recoverable errors.

Empirical Results and Numerical Patterns

Performance Comparison

• Structured Intermediate (A#1{2}) demonstrates superior result match rates across all models. Large models (Llama 70B, Qwen 72B) achieve a 15–25 pp improvement over direct generation, with 100% execution success due to consistent schema compliance. For smaller models, gains are modest (3–5 pp), but schema compliance becomes the limiting factor (execution success drops to 53–65%).
• Direct Generation (A#1{1}) yields high execution success (>98%) across all models but lower result matches; it is resilient to superficial syntax errors but susceptible to semantic mismatches.
• Agentic Generation (A#1{3}) produces the lowest result match rates (15–25%), even with 6–8× greater token consumption and multi-step reasoning, confirming that increased computational budget does not offset error compounding. The set of tasks solved by A#1{3} is a strict subset of those solved by A#1{2}, providing no complementary coverage or practical justification for its higher cost.

Tier-Specific Findings

• Structural queries are most tractable across all architectures and models due to their alignment with static schema features.
• Data flow and composite queries introduce multi-hop reasoning that small models and agentic approaches fail to operationalize effectively, particularly when coordination across tool calls is required.

String Matching vs. Semantic Evaluation

Result match metrics—defined by execution outputs—reveal that exact string matches grossly understate correctness; for instance, Qwen 72B sees a 16.6 pp gap between exact and result matches in direct generation. The deterministic mapper in A#1{2} enforces canonical syntax, minimizing such discrepancies.

Architectural Implications and Theoretical Reflections

Constraining Output Spaces

The empirical superiority of the structured intermediate architecture reflects fundamental limitations in LLMs' representations for cross-category relations (syntactic tokens vs. identifiers) when generating DSL queries [anand2024critical]. By enforcing typed schema constraints, the task reduces to field selection and entity classification, which is more congruent with LLM capabilities and reduces hallucination pathways. This finding aligns with independent observations in CodeQL translation systems [mocq2025, iris2025], where output-space reduction through schema restriction or prompt engineering boosts reliability beyond end-to-end code synthesis.

Agentic Generation and Error Propagation

Multi-step agentic frameworks suffer from compounded uncertainty; even high per-step accuracy does not circumvent exponential decay in end-to-end reliability. Empirical evidence from MCP-Universe and QLCoder work [qlcoder2025, mcpuniverse2025] further supports the argument, especially as tool-use length increases. Additionally, A#1{3}'s lack of unique coverage negates any modular benefit for system hybridization.

Model Scale as a Bottleneck

Model scale is a critical mediator: large models accurately fill structured schemas, thus fully realizing the architectural advantage. Small models, though able to generate plausible freeform queries, fail the schema compliance threshold and underperform in intermediate representation. Constrained decoding and guided generation [structeval2025] offer prospective remedies for schema alignment in low-resource LLMs.

Practical and Theoretical Impact

In practical deployment, imposing strict schema constraints via structured intermediates yields deterministic behavior and high reliability, suitable for safety-critical environments such as vulnerability detection and compliance audits. On the theoretical front, the decomposition—LLM for semantic parsing/classification, deterministic code for DSL translation—generalizes to any domain where a formally typed query language and schema can be enumerated. The benchmark constructed for CPGQL translation is novel and extends the methodological landscape previously dominated by text-to-SQL benchmarks [spider2018, bird2023].

Future Directions

• Schema-Guided Decoding: Implementing token-level enforcement for intermediate representation in small models could bridge the compliance bottleneck.
• Benchmark Expansion: Broadening the task dataset to cover additional DSLs and complexity tiers, enabling comprehensive evaluation of translation architectures.
• Tool Protocol Alternatives: Systematic analysis of MCP, HuggingFace, and OpenAI function-calling implementations to further isolate infrastructural impacts from architectural decisions.

Conclusion

Empirical evidence from a controlled evaluation of three LLM-based architectures for NL-to-static analysis translation demonstrates that limiting LLM autonomy via structured intermediate representations yields maximal result correctness, particularly for large models capable of reliable schema compliance. Direct freeform generation and agentic tool use are less robust and less efficient, with agentic approaches offering no unique coverage despite increased resource consumption. The architectural principle—delegate semantic classification to the LLM, formal query generation to deterministic code—should guide system design in any structured domain with formal schemas and can substantially increase adoption and usability of static analysis tools. The benchmark and infrastructure provided underpin ongoing research and enable standardized evaluation for future developments in AI-assisted program analysis.

---

Relevant citations:

• "Less Is More: Measuring How LLM Involvement affects Chatbot Accuracy in Static Analysis" (2604.21746)
• "IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities" [iris2025]
• "QLCoder: A Query Synthesizer For Static Analysis of Security Vulnerabilities" [qlcoder2025]
• "Automated Static Vulnerability Detection via a Holistic Neuro-symbolic Approach" [mocq2025]
• "A Critical Study of What Code-LLMs (Do Not) Learn" [anand2024critical]
• "StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs" [structeval2025]
• "S pider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task" [spider2018]
• "Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs" [bird2023]
• "Modeling and Discovering Vulnerabilities with Code Property Graphs" [joern2014]