How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval

Abstract: Large-language-model code generation has shifted from single-shot prompting to multi-agent orchestrations - analyst, coder, tester, and debugger pipelines - and is evaluated almost exclusively on functional correctness. Whether these architectures also affect the structural complexity of the code they produce, and which orchestration layers carry the cost, remains largely unexamined: prior work has documented prompt-level effects on code complexity, but the architecture-level question is open. We compare six widely-used multi-agent configurations (Basic, AC, ACT, Debugger, AC+Debugger, ACT+Debugger) under two models from the GPT-4o family across all 164 HumanEval tasks - 1,968 paired observations - using the five RADON complexity metrics (SLOC, cyclomatic complexity, and Halstead Volume, Difficulty, and Effort). We apply a paired non-parametric statistical pipeline (Friedman omnibus, Wilcoxon signed-rank post-hoc with Holm correction, Kendall's $W$ and matched-pairs rank-biserial effect sizes) in both all-completions and passing-only conditions. The six architectures collapse into two indistinguishable complexity clusters separated by a 50-130% gap, the same partition in both models and under both conditions; among the architectural layers, the analyst-coder split inflates complexity, the runtime debugger does not - and on the analyst-coder background actively deflates it - and the tester re-inflates it. The heavy cluster's additional complexity buys no pass@1 advantage: the leanest architectures match or beat the heaviest on accuracy. Architectural elaboration in LLM code generation should therefore be justified by measured benefit on the dimensions that matter, not assumed.

• The paper shows that generation architecture is the dominant contributor to code complexity, with heavy pipelines inflating complexity by 50–130% compared to lean configurations.
• The study employs a paired design across 1,968 code samples using five static code complexity metrics and rigorous non-parametric tests to isolate architectural effects.
• The findings indicate that added collaboration roles, such as Analyst and Tester, increase code verbosity without improving functional accuracy, urging simpler system designs.

Architectural Determinants of Code Complexity in Multi-Agent LLM Systems

Introduction

The study "How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval" (2606.00308) systematically dissects how architectural choices in multi-agent configurations affect the structural complexity of LLM-generated code, using HumanEval as the benchmark and five established static code complexity metrics. Whereas previous work has largely centered on pass@1 as an evaluative measure, this paper broadens the scope to consider maintainability-relevant attributes, resolving a notable gap in empirical LLM code generation research. The authors rigorously compare six multi-agent orchestration pipelines—ranging from minimalist single-shot prompting to sophisticated analyst/coder/tester/debugger arrangements—under two OpenAI GPT-4o family models, leveraging a strictly paired experimental design and robust non-parametric statistical inference.

Methodological Overview

The experiments pivot on six multi-agent architectures: Basic, AC (Analyst+Coder), ACT (Analyst+Coder+Tester), Debugger, AC+Debugger, and ACT+Debugger. These span permutations of three orthogonal architectural layers: role decomposition (R), static critique-driven testing with bounded iteration (T), and execution-grounded runtime debugging (D). Each configuration is operationalized as a specific multi-agent workflow with clearly defined agent roles and data/feedback flows (see Figure 1).

(Figure 1)

Figure 1: The ACT+Debugger pipeline illustrates the union of architectural layers ($R$, $T$, $D$); each architecture is a strict subset, concretely specifying agent roles and iteration/feedback regimes.

Two GPT-4o family models (gpt-4o and gpt-4o-mini) were systematically benchmarked, each generating code for all 164 HumanEval tasks under every architecture, totaling 1,968 paired code samples. Complexity was measured using five radon-derived metrics: Source Lines of Code (SLOC), Cyclomatic Complexity (CC), and the Halstead Volume, Difficulty, and Effort measures. For reliability, only parse-valid, entry-point-matching samples were processed—listwise completeness was achieved across tasks and runs. Main analyses were conducted both unconditionally (all completions) and within the intersection of tasks all architectures solved (passing-only subset), allowing robust separation of behavioral (failure-mode) artifacts from general architectural effects.

Statistical inference leveraged the Friedman test for block-wise (task-wise) repeated measures, with Wilcoxon signed-rank post-hoc tests and stringent Holm correction, reporting both statistical significance and matched-pairs rank-biserial correlations for effect size. This paired analysis enables precise attribution of observed complexity shifts to architectural factors, rather than task or model confounders.

Empirical Results

Emergence of Complexity Clusters

All metrics and both models exhibited pronounced and highly significant effects of generation architecture on code complexity, with SLOC, CC, and Halstead metrics showing consistent, replicable shifts. However, the effect is not smoothly distributed; architectures collapsed into two sharply delineated clusters:

• Lean cluster: Basic, Debugger, AC+Debugger
• Heavy cluster: AC, ACT, ACT+Debugger

As demonstrated in Figure 2, every measured metric separated cleanly across this taxonomy, with heavy architectures producing code that was 50–130% more complex (in SLOC, CC, Halstead Volume), a difference robust to both model variant and pass conditioning.

Figure 2: Distributions of all five complexity metrics by architecture; clear separation into lean/heavy clusters for both GPT-4o variants.

The median SLOC, for instance, jumps from 5–6 in the lean cluster to 8–10 in the heavy cluster. This gap is not an isolated artifact; correspondingly large effects are observed in all core complexity metrics (Table 1 in the paper). The mean-rank separation is visually captured in Figure 3, highlighting the exclusive intra-cluster homogeneity and inter-cluster separation.

(Figure 3)

Figure 3: Matched-pairs rank-biserial correlations with statistical significance; only differences crossing the cluster boundary are significant, verifying the block structure.

Causal Attribution: Layer-Wise Effects

Layer-isolated analysis reveals that, contrary to additive intuition, layers combine nonlinearly:

• Role decomposition (R): Introducing Analyst inflates complexity (e.g., Basic vs AC $r_{rb} = -0.90$).
• Debugger (D): On an Analyst-Coder background, adding Debugger actively reduces complexity (AC vs AC+Debugger $r_{rb} = +0.87$); this effect is context-sensitive and does not apply outside the R context.
• Tester (T): Testing re-inflates complexity when added to an AC+Debugger chain.

Importantly, the Debugger layer exhibited a complexity-deflating effect only when not combined with Tester. The partitions and effects are robust across statistical approaches, completeness conditions, and both evaluated model scales.

Complexity vs. Correctness Tradeoff

A principal finding is that the observed complexity inflation does not yield increased functional accuracy. Across all models and runs, the highest pass@1 scores are held by configurations in the lean cluster (Debugger, AC+Debugger), while heavy architectures are as likely to perform worse as better.

Figure 4: Mean SLOC vs. pass@1 reveals no correlation between higher complexity and increased accuracy; lean architectures attain top accuracy at minimal complexity.

This decisively counters any assumption that elaborated agentic orchestration, with more roles or critique layers, improves correctness or produces "safer," better-engineered code; in reality, it mainly induces verbosity and greater control-flow and data-complexity without tangible functional benefit.

Broader Implications

Practical Guidance for System Designers

• Correctness-Only Benchmarks Are Insufficient: Pass@1 obscures substantial downstream maintenance and comprehension costs. Structural complexity metrics must be included in future head-to-head system evaluations.
• Elaborate Orchestrations Often Detrimental: Pipelines that add Analyst and Tester roles without runtime debugging are strictly dominated on both complexity and accuracy/cost axes for this benchmark.
• Execution-Grounded Iteration is Preferable: Architectures deploying runtime debugging (execution feedback) outperform those with static critique loops in both correctness and maintainability metrics.

Theoretical Ramifications

• Prompt Patterns vs. Architectures: While previous prompt-pattern manipulations showed only local line-count effects (e.g., Della Porta et al. [dinucci2024unlocking]), architectural interventions shape the deeper structural dimensions of code—halstead measures, control flow, operator/operand richness.
• Complexity as a Style Attractor: Architectures act as attractors inducing canonical (concise, functional) or elaborated (verbose, defensive) styles, with high within-cluster lexical similarity, especially in the minimal pipeline.

Limitations and Directions for Future Research

• Level of Granularity: Findings pertain to function-level generation. Complexity drift across class-, module-, and system-level architectures remains unmeasured.
• Model Panel Scope: Only GPT-4o variants were included; direct generalization to OSS models or alternate providers awaits replication.
• Prompt–Orchestration Interactions: Synergistic or antagonistic effects between architectural choices and specific prompt engineering patterns are as yet unstudied.

Conclusion

This study provides definitive empirical evidence that in multi-agent LLM code generation, architectural design is the dominant lever on code complexity, exerting a far greater influence than previously reported prompt-level interventions. The findings unequivocally refute the notion that increased architectural complexity buys functional correctness, showing instead that additional dialogic roles primarily generate maintainability debt. The practical directive is clear: system orchestrators should minimize agentic pipeline elaboration unless complexity increases are strictly justified by measured gains in task performance.

By rigorously characterizing the causal structure of architecture–complexity relationships, this work establishes a robust empirical baseline for future investigations into both the controllability and quality of LLM-generated software artifacts.

---

• Ashrafi, N. (2025). How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval (2606.00308)
• Della Porta, A., et al. Unlocking Code Simplicity: The Role of Prompt Patterns in Managing LLM Code Complexity [dinucci2024unlocking]
• Ashrafi, N. S., et al. Enhancing LLM Code Generation: A Systematic Evaluation of Multi-Agent Collaboration and Runtime Debugging for Improved Accuracy, Reliability, and Latency (Ashrafi et al., 4 May 2025)