LLM-as-a-Developer in Software Engineering

• LLM-as-a-Developer is a paradigm where LLMs act as active collaborators in coding, testing, debugging, and documentation across the software lifecycle.
• It employs structured prompt engineering, versioned artifact management, and cost-controlled batch generation to enhance code quality and productivity.
• Empirical studies show these AI tools boost development speed and efficiency while requiring essential human oversight for requirements validation and security.

LLM-as-a-Developer denotes a class of software-engineering workflows in which a LLM is treated not as a one-off code generator but as an active collaborator across coding, testing, debugging, documentation, design, prompt revision, and evaluation. In this framing, the model may function as an “AI pair programmer,” a “junior-developer collaborator,” a reviewer, a test runner, an evaluator, or, in controlled experiments, a replacement for the human developer role itself (Mohamed et al., 3 Jul 2025, Huang et al., 4 Jun 2026). Empirical studies nevertheless converge on an important constraint: most practitioners treat LLMs as assistive tools rather than standalone solutions, and core requirements-engineering, validation, and security practices remain necessary (Santos et al., 10 Oct 2025, Ullrich et al., 10 Jul 2025).

1. Conceptualization and scope

LLM-assistants are generative AI tools powered by LLMs that act as on-demand “AI pair programmers.” Reported use spans coding, testing, debugging, documentation, and design, rather than merely completing local code snippets (Mohamed et al., 3 Jul 2025). A practitioner survey of 131 software professionals shows that this role is already distributed across the software lifecycle: 87.8% reported using ChatGPT, 57.3% Copilot, 40.5% Bard/Gemini, and 29.0% Claude for coding-specific work; 83% reported generating code fragments and then iteratively modifying them; 36% reported copy-pasting verbatim, principally for trivial or well-scoped tasks; about 40% used LLMs to scaffold unit tests; and 55% used them as creative partners for brainstorming API names, comments, or usage examples (Santos et al., 10 Oct 2025).

The research literature describes several stronger formulations of the same idea. In LLARS, the model is integrated into an end-to-end workflow in which it participates in prompt engineering, output generation, and hybrid evaluation, with “AI commits” appearing beside human-authored changes (Steigerwald et al., 11 May 2026). In ADK Arena, “LLM-as-a-Developer” is a controlled-variable methodology that replaces human developers with an LLM coding agent that learns a framework’s API from documentation and source code, writes an agent, and iteratively repairs it through a validate-and-feedback loop until tests pass (Huang et al., 4 Jun 2026).

These formulations do not imply that traditional software engineering disappears. A systematic review instead characterizes the practical role shift as moving from “writer” to “reviewer” and “prompt engineer,” while keeping human oversight to guard quality and prevent complacency (Mohamed et al., 3 Jul 2025). This suggests that LLM-as-a-Developer is primarily a redistribution of development labor rather than a removal of development responsibilities.

2. Requirements transformation, prompt construction, and versioned artifacts

A central finding in research on LLM-assisted software engineering is that high-level requirements are usually too abstract for direct use as prompts. Ullrich et al. formalize this as a pipeline in which requirements artifacts $\mathcal{R}$ are first decomposed into programming tasks $\mathcal{T}$ by a function $\delta$, then enriched with design decisions and architectural constraints by $\epsilon$, then combined with code context $C$ by a prompt-construction function $\phi$, yielding a prompt $\pi$ and, finally, generated code:

$$T = \delta(r), \quad E = \epsilon(T, D), \quad \pi = \phi(E, C), \quad \text{code} = \mathrm{LLM}(\pi).$$

Developers in that study reported that raw user stories lack function names, error handling, data formats, testability details, and architectural context, and therefore cannot simply be handed to the model (Ullrich et al., 10 Jul 2025).

LLARS operationalizes prompt work as a versioned engineering activity rather than an informal chat transcript. In its Collaborative Prompt Engineering module, each prompt is represented as editable blocks, consisting of one optional system-block and one or more user-blocks, with commit history $G_{\text{block}}=(V,E)$ based on a CRDT implemented with Yjs. Branching is supported through branch pointers, rollback is available for any past commit, diff comparison works across branches, and every keystroke propagates to collaborators with sub-100 ms latency via WebSockets and Yjs. A single “Test” click substitutes sample values for template variables and streams the model reply back into the editor pane in real time (Steigerwald et al., 11 May 2026).

This versioned treatment extends the “prompt as artifact” idea into ordinary developer infrastructure. LLARS exports prompt block histories as JSON patches suitable for Git, allows the backend to be mounted as a Git remote via an SSH+Git wrapper, and supports CI/CD hooks so that batch generation and hybrid evaluation can be triggered automatically after prompt changes are merged (Steigerwald et al., 11 May 2026). The recommendation from requirements-centric work is aligned: prompts and outputs should be treated as first-class artifacts—versioned, documented, and linked to requirements and commits (Ullrich et al., 10 Jul 2025).

3. End-to-end orchestration: generation, cost control, and hybrid evaluation

LLARS presents one of the most explicit end-to-end realizations of LLM-as-a-Developer. Its architecture consists of three tightly integrated modules—Collaborative Prompt Engineering, Batch Generation, and Hybrid Evaluation—arranged as a pipeline: co-authored prompt versions are materialized into a Cartesian-product generation matrix, generated outputs with full provenance are lifted into evaluation, and judgments are aggregated to rank model–prompt pairs (Steigerwald et al., 11 May 2026).

The batch-generation stage is defined over prompt versions $P=\{p_1,\dots,p_{|P|}\}$, models $\mathcal{T}$0, and input data items $\mathcal{T}$1, so that the total number of generated outputs is

$\mathcal{T}$2

Each output $\mathcal{T}$3 incurs a cost

$\mathcal{T}$4

and the total estimated cost is

$\mathcal{T}$5

Users may supply a budget cap $\mathcal{T}$6, and generation pauses automatically when accumulated cost reaches the cap (Steigerwald et al., 11 May 2026).

Hybrid Evaluation defines an evaluation scenario $\mathcal{T}$7 by a set of items $\mathcal{T}$8, a set of evaluators $\mathcal{T}$9 combining human and LLM judges, and an assessment type chosen from Likert multi-dimensional, bucket ranking, pairwise, categorical labelling, mail rating, and authenticity detection. Provenance is hidden during assessment, inter-rater reliability is computed automatically, and for categorical or Likert scales the system computes Krippendorff’s $\delta$0. Provenance analysis then aggregates a top-bucket hit rate

$\delta$1

which is used to rank prompt–model pairs (Steigerwald et al., 11 May 2026).

Concrete deployment results illustrate the scale of this workflow. In the subject-line study, $\delta$2 prompt variants, $\delta$3 LLMs, and $\delta$4 email threads yielded approximately 6,413 outputs, with total estimated cost of \$\delta$5\alpha_{\text{humans}}=0.72$\delta$6\alpha_{\text{all}}=0.75$\delta$7h_{\text{best}}=0.87$. Interviews with six domain experts and three developers further reported an “intuitive interface,” prompt-iteration cycles dropping from days to hours, end-to-end prompt-to-evaluation turnaround reduced by approximately 50%, prompt-review meetings reduced from 3 to 1 per iteration, and routine execution of 200+ combinations in under five minutes (Steigerwald et al., 11 May 2026).

4. Human oversight, provenance, and real adoption behavior

Empirical studies of LLM-generated code validation show that provenance matters. In a lab study with 28 participants validating and repairing Copilot-generated Java code, developers without explicit provenance cues often failed to identify the LLM origin of the code. Participants informed that the code was generated by Copilot achieved a higher mean bug-fix success rate ($\delta$8, $\delta$9) than non-informed participants ($\epsilon$0, $\epsilon$1), with $\epsilon$2. The informed group also reported higher NASA-TLX effort ($\epsilon$3 versus $\epsilon$4, $\epsilon$5), showed more fixation time, more search effort, and more Copilot invocations (Tang et al., 2024).

Observed validation and repair strategies were not arbitrary. Developers performed full-subroutine read-throughs for context comprehension, step-by-step logic tracing, and “running for output” with ad-hoc print statements more often than debugger use. For repair, they used in-place edits for small syntax bugs but often adopted a delete-and-rewrite strategy when the generated code was tangled. The study also reports frequent code-to-comment gaze switches and a shifted focus in which developers tended to trust syntax while double-checking logic (Tang et al., 2024). Design implications proposed in that work include provenance indicators, prompt–code linking, explainable logic, and multimodal context support.

Repository mining of refactoring commits exposes a related pattern in real-world adoption. Analysis of 169 GitHub commits, 440 file-level datapoints, and 368 distinct ChatGPT prompts found that developers mostly accept and use refactoring suggestions without modifications. When changes occur, they are often major and follow five patterns: full adoption without change, deletion-driven pruning, minor integration patches, structural adaptation, and “recipe” adoption with content drift (Schön et al., 6 May 2026). The study links these patterns to refactoring complexity, prompt design strategies, semantic validity of LLM responses, and conversation length.

Across broader practice, the dominant interaction model remains human-guided. Survey data report that 83% of practitioners never deploy LLM output without human review, and 53% would refuse to ship any AI-generated code into production without manual vetting (Santos et al., 10 Oct 2025). The systematic review reaches a similar conclusion in prescriptive form: treat LLM outputs as drafts, validate against unit tests and code reviews, and preserve pair programming, design reviews, and scrum ceremonies to offset reduced peer interactions (Mohamed et al., 3 Jul 2025).

5. Empirical outcomes: productivity, security, and timing-aware interaction

The current evidence base on productivity is mixed but substantial. A systematic literature review of 37 peer-reviewed studies published between January 2014 and December 2024 reports that 92% examine at least two SPACE dimensions, while only 14% extend beyond three dimensions; Satisfaction, Performance, and Efficiency are most frequently investigated, whereas Communication and Activity remain underexplored (Mohamed et al., 3 Jul 2025). Reported benefits include up to 30% faster information retrieval compared to web search, 20%–35% better completion rates than Stack Overflow in controlled tasks, reductions in task completion time of 15%–25%, a case-study effort reduction from 75 to 22 person-days, approximately 40% fewer keystrokes for boilerplate generation, a 51% reduction in code-translation error rate, and broad survey findings of a 24% average throughput gain with a 26% quality gain when GenAI is used in coding (Mohamed et al., 3 Jul 2025).

The same review emphasizes persistent risks. Cognitive offloading and over-reliance appear in both novices and professionals; field studies show reduced team collaboration when developers consult chatbots rather than peers; throughput gains can correlate negatively with quality at approximately $\epsilon$6; some users report disrupted flow and increased frustration from competing suggestions; and requirement mismatches remain common when outputs are verbose, out of context, or fail non-functional requirements (Mohamed et al., 3 Jul 2025). This evidence does not support a one-dimensional “productivity” interpretation.

Security outcomes also depend strongly on workflow design. In a quasi-experimental study of 12 professional Java developers using a fixed model configuration to implement an identity-centric Spring Boot backend, switching from free-form prompting to a structured prompt template grounded in the six-layer MA-CoT model was associated with a significant paired reduction in the severity-weighted validated-weakness score under an exact Wilcoxon signed-rank test ($\epsilon$7). Aggregate validated weaknesses decreased from 162 to 111 (31.5%), the severity-weighted burden decreased from 432 to 267 (38.2%), and critical findings decreased from 24 to 5 (79.2%). The largest reductions were in authorization and object access (53.3%) and in authentication, credential policy, and recovery weaknesses (44.7%) (Kharma et al., 20 Apr 2026). The authors explicitly note that these results do not support replacing secure defaults, static analysis, expert review, or operational hardening.

Another line of work treats the timing of LLM intervention as part of the development problem. In a production-grade Visual Studio Code plugin, a lightweight pre-filter model using only pre-invocation developer telemetry—typing speed, file navigation, editing activity, warnings/errors, and session context—was inserted before any LLM call. Deployed over four months, the system nearly doubled acceptance rates from 18.4% to 34.2% while suppressing 35% of low-value LLM calls. Offline evaluation reported $\epsilon$8, $\epsilon$9, and at threshold $C$0 achieved 96.5% recall on accepted events while filtering approximately 41.6% of rejections (Awad et al., 24 Nov 2025). This suggests that LLM-as-a-Developer is not only a question of what the model generates, but also when it should generate.

6. Domain-specific extensions and benchmarked instantiations

In model-driven development, LLM-as-a-Developer is used to translate formalized models into deployable code. The Agile Model Driven Development approach organizes inputs into structural, behavioral, and constraint layers; translates UML diagrams into PlantUML text; adds OCL invariants and FIPA ontology schemas; and prompts GPT-4 to generate Java and Python code targeting JADE and PADE. Evaluation compares OCL-only and OCL-plus-ontology constraints using cyclomatic complexity $C$1, with all reported classes remaining in the “low risk” range and all reported values at or below $C$2. Behavioral evaluation further found that generated systems matched the intended interaction flow, although GPT-4 introduced an extra “DiscoverUVs” message as a logical refinement (Sadik et al., 2024).

In software architecture design, the ADD-based approach provides the model with an explicit ADD process description, an architect persona, and prioritized architectural drivers. The workflow proceeds through review of drivers, domain-model creation, iteration-plan creation, and ADD iterations, producing a continuously updated Architecture.md and per-iteration IterationN.md artifacts. In case studies on a Hotel Pricing System and an Event Ticketing System, the full ADD-based process generated architectures closely aligned with established solutions, while “without ADD” conditions produced noticeably less coherent and less detailed designs (Cervantes et al., 27 Jun 2025). The same work identifies context management, dynamic retrieval, continuous evaluation, and bi-directional synchronization between code and architecture as future directions.

Developer-support systems also extend the paradigm beyond code generation itself. “Never Come Up Empty” constructs a retrieval corpus of 3,410,872 Java- and Python-related Stack Overflow accepted-answer documents and evaluates 7 RAG pipelines and 63 pipeline variants. The HyDE→full-answer pipeline performs best, and adaptive similarity-threshold lowering raises coverage from approximately 48% under a static threshold to 100% on unseen queries, while improving helpfulness, correctness, and detail relative to zero-shot baselines across four open-source LLMs (Lei et al., 22 Jul 2025). In a different direction, DevAssistLlama instruction-tunes LLaMA 2 13B on a 45,000-example software-engineering instruction dataset spanning NER, RE, link prediction, forum answer ranking, and QA, and is exposed through IDE plugins, a local REST endpoint, and a CLI for tasks such as summarization, documentation, and test creation (Banerjee et al., 2023).

A stricter benchmark formulation appears in ADK Arena. Evaluating 51 Python ADK frameworks across 204 framework–benchmark pairs, the study reports that code generation succeeds for 57% of runs, per-agent generation cost varies 5.6$C$3 from \$C$43.4, median per-framework task resolution is 32%, and the best single-benchmark framework resolves up to 80% of tasks. Information-source ablations show genuine framework usage in a narrow 28%–40% band across docs-only, source-only, and no-reference conditions, leading the authors to conclude that documentation, source code, and parametric knowledge are largely substitutable rather than any one being a hard bottleneck (Huang et al., 4 Jun 2026).

7. Limitations, controversies, and unresolved research questions

A recurring misconception is that high-level requirements can simply be “fed” to an LLM to obtain production-ready software. Empirical interviews with practitioners contradict this: requirements, as typically documented, are too abstract for direct input and must be decomposed into programming tasks, enriched with design decisions and architectural constraints, and supplied with the right slice of code context (Ullrich et al., 10 Jul 2025). This is consistent with survey evidence that developers generally use models as assistive tools rather than autonomous substitutes (Santos et al., 10 Oct 2025).

Another unresolved issue concerns the evaluation horizon. Most published studies remain exploratory: 64% in the systematic review are short-term lab experiments, and the review explicitly calls for longitudinal and team-based studies that instrument real projects over months and capture code-review comments, developer behaviors, and defect rates before and after LLM adoption (Mohamed et al., 3 Jul 2025). Communication and Activity, the two least studied SPACE dimensions, are especially relevant for determining whether LLM-as-a-Developer weakens or reshapes collective engineering practice.

Context management remains a technical bottleneck. Architecture-design studies report that large master documents can cause the model to forget earlier drivers, steps, or file references, and that growing documents can lead to inconsistent edits or skipped sections (Cervantes et al., 27 Jun 2025). Benchmarking studies likewise show that no single framework dominates all workloads, and that lower generation cost does not by itself predict success (Huang et al., 4 Jun 2026). A plausible implication is that future progress will depend less on monolithic model capability than on artifact management, validation loops, retrieval, and interface design.

Finally, the strongest positive results still come with explicit guardrails. Security training improves outcomes without changing the model, but the authors state that the findings do not support replacing secure defaults, static analysis, expert review, or operational hardening (Kharma et al., 20 Apr 2026). The literature therefore supports a bounded interpretation: LLM-as-a-Developer is most defensible when prompts, generated outputs, provenance, validation signals, and review practices are engineered as first-class components of the software process, rather than treated as informal side channels.