Rethinking Complexity Metrics for LLM-Integrated Applications: Beyond Source Code

Abstract: LLM-integrated applications blend natural language prompts with program code, and much of their runtime behavior originates in the prompt layer rather than in the code itself. Existing complexity metrics, however, operate solely at the code level and therefore overlook this behavioral logic entirely. We present HECATE, the first tool designed to assess complexity in both the prompt and code layers of such applications. Central to HECATE is Prompt-as-Specification, a Hoare-logic-inspired formalism that interprets every prompt as a specification of intended behavior. Grounded in 25 complexity dimensions identified across published taxonomies, the tool generates 52 candidate metrics. We assess each metric against 118 components collected from 18 open-source repositories, relying on maintenance activity derived from version history as an empirical proxy for complexity, and discard any metric that loses significance once code size is accounted for. Only ten metrics withstand this test. Seven belong to our newly introduced set; rather than measuring sheer volume, each tallies structurally distinct elements, such as LLM call sites, memory attributes, and prompt templates, an attribute we call structural breadth. Of the three surviving conventional metrics, RFC exhibits a similar breadth-oriented character, while Halstead N and V survive only as a residual effect of size; our top-performing metrics exceed all three. Crucially, the prompt-layer metrics retain significance even when the strongest code-level metric is added as a covariate, establishing prompt complexity as a dimension in its own right. A final validation on 20 components spanning six held-out repositories shows that the two best-performing metrics continue to predict maintenance effort, supporting their generalizability beyond the training set.

• The paper introduces Hecate, a static analysis system that quantifies 'structural breadth' by integrating complexity metrics for both code and natural language prompts.
• It employs a taxonomy-driven approach to derive 52 candidate metrics, with 10 showing significant correlation with maintenance complexity after size control.
• Empirical validation on 118 components from 18 repositories demonstrates that NL-derived metrics and structural breadth are key predictors of software maintainability.

Rethinking Software Complexity Metrics for LLM-Integrated Applications: An Analysis of Hecate

Overview and Motivation

The proliferation of LLMs as integral components within software systems has induced a structural paradigmatic shift for software engineering. LLM-integrated applications entwine prompt-based natural language specifications with executable code, interleaving behavioral contracts across modalities. Traditional software complexity metrics—focused exclusively on code through syntactic constructs such as cyclomatic complexity, Halstead metrics, or the Chidamber–Kemerer suite—fail to account for the substantial behavioral logic now encoded in natural-language prompts. This paper introduces Hecate, a static analysis system designed to quantitatively measure complexity spanning both code and prompts, articulating metrics for "structural breadth" rather than conventional artifact volume.

Figure 1: Two components of similar code size that traditional metrics score alike but Hecate metrics differentiate via their structural breadth.

Theoretical Foundations: Prompt-as-Specification

Central to Hecate is the Prompt-as-Specification formalism, rooted in Hoare logic. Unlike free-form text, prompts in LLM-integrated systems function as programmatic specifications, defining rules, global invariants, and required state predicates. Each behavioral rule operates as the natural-language analog of a Hoare triple, encompassing explicit conditions, required actions, and result constraints. By extracting conditional logic, constraints, and context dependencies from prompts, Hecate enables specification-level reasoning about complexity within the NL layer.

Figure 2: A real component prompt decomposed into behavioral rules, invariants, and state predicates under Prompt-as-Specification.

Metric Derivation and Structural Breadth

Hecate employs a systematic, taxonomy-driven approach for metric construction. Drawing from 25 empirically validated complexity dimensions across code (e.g., number of LLM call sites, memory management strategies), prompt (conditional branching, compositional constraints), and code–prompt interface layers (injection points, output parsing), the approach yields 52 candidate metrics, including both adaptations of traditional SE metrics and original measures tied to LLM-driven architectural features.

The explicit requirement is that effective metrics retain a statistically significant association with maintenance complexity even after controlling for code size—thus directly handling the notorious size confound that renders classical metrics (e.g., McCabe’s CC, Halstead) ineffective for defect prediction in this domain.

Empirical Validation

The study evaluates these metrics over 118 components from 18 high-profile, open-source LLM agent frameworks and applications, with maintenance complexity proxied by version-control-based ground truth (re-fix recurrence, temporal activity, contributor diversity). Only 10 metrics retain significant predictive effect after size control: 7 are newly proposed (structural breadth) metrics, and 3 are classical baselines (RFC, Halstead N, Halstead V). Notably, Cyclomatic Complexity loses all significance post size-control ($\rho = +0.06$). The strongest metrics—n_mem_refs (distinct memory references, $\rho = +0.40$), n_llm_calls (LLM call sites, $\rho = +0.38$), and inject_surf (prompt injection channels, $\rho = +0.27$)—persist independently of code size or volume, differentiating structurally distinct components even at fixed LOC.

Figure 3: Dataset distributions showing the diversity in component sizes and complexity dimensions used as ground truth for metric validation.

Structural breadth, i.e., the count of distinct architectural elements—memory attributes, LLM invocation points, prompt templates—not volume or token count, emerges as the prime correlate with maintenance effort. Moreover, the persistence of NL-derived metrics (P_dec_ratio: fraction of conditional prompt instructions; P_ev: complex conditionals; n_prompts: number of prompt templates) after joint control for code complexity confirms prompt complexity as an independent dimension for software risk and maintainability.

Comparative Analysis and Generalization

A direct comparison reveals that almost all traditional code metrics collapse under size control. RFC (response for class) is the only classical metric to retain moderate effect, owing to its alignment with structural breadth. All Hecate-proposed metrics surpass RFC. Furthermore, the metrics generalize to previously unseen repositories, with primary breadth measures (n_mem_refs, n_attrs, P_dec_ratio) maintaining signal, while interface-oriented metrics degrade in frameworks encapsulating LLM orchestration behind advanced abstractions.

Figure 4: Size-control effect comparison—signal from traditional code metrics dissipates under LOC control, while Hecate’s metrics persist.

Practical and Theoretical Implications

This study demonstrates that for LLM-integrated software, measurement of logical specification and coordination complexity must cross the code–prompt boundary. The findings suggest that future SE metrics should operationalize architectural diversity and specification modal coupling, rather than artifact volume, as first-order complexity determinants.

Practically, Hecate’s static, execution-independent computation renders it immediately suitable for integration into CI/CD pipelines as a complexity regression tool, facilitating early detection of architectural entropy and maintainability risk in LLM-augmented systems. As LLM abstractions proliferate, there is likely to be an increasing need for dynamic and multi-agent complexity measurements that extend beyond static artifact specification.

Limitations and Validity

Construct validity is anchored by cross-replication with metric validation literature, but potential threats include imperfect recall in component extraction and reliance on version-control maintenance signals as proxies. External validity is constrained by a focus on Python and open-source repositories; however, the methodological framework is language-agnostic.

Conclusion

The work underlying Hecate establishes that modern LLM-integrated applications fundamentally invalidate size-driven complexity metrics. Instead, structural breadth—capturing the distinct architectural and specification constructs governing application behavior—is the major determinant of downstream maintainability. Hecate introduces a static analysis technique that precisely quantifies this new form of complexity, surpassing classical metrics and generalizing effectively across diverse repositories and application contexts. This blueprint lays the groundwork for comprehensive, specification-aware complexity analysis of next-generation AI-driven software.