A Quasi-Experimental Developer Study of Security Training in LLM-Assisted Web Application Development

Abstract: This paper presents a controlled quasi-experimental developer study examining whether a layer-based security training package is associated with improved security quality in LLM-assisted implementation of an identity-centric Java Spring Boot backend. The study uses a mixed design with a within-subject pre-training versus post-training comparison and an exploratory between-subject expertise factor. Twelve developers completed matched runs under a common interface, fixed model configuration, counterbalanced task sets, and a shared starter project. Security outcomes were assessed via independent manual validation of submitted repositories by the first and second authors. The primary participant-level endpoint was a severity-weighted validated-weakness score. The post-training condition showed a significant paired reduction under an exact Wilcoxon signed-rank test ($p = 0.0059$). In aggregate, validated weaknesses decreased from 162 to 111 (31.5\%), the severity-weighted burden decreased from 432 to 267 (38.2\%), and critical findings decreased from 24 to 5 (79.2\%). The largest reductions were in authorization and object access (53.3\%) and in authentication, credential policy, and recovery weaknesses (44.7\%). Session and browser trust-boundary issues showed minimal change, while sensitive-data and cryptographic weaknesses showed only marginal improvement. These results suggest that, under the tested conditions, post-training runs reduce validated security burden in LLM-assisted backend development without modifying the model. They do not support replacing secure defaults, static analysis, expert review, or operational hardening.

• The paper demonstrates that targeted developer training in security-oriented prompting significantly reduces both the quantity and severity of vulnerabilities in LLM-assisted code.
• It employs a quasi-experimental design with pre-training/post-training comparisons on a Java Spring Boot backend, revealing up to 79.2% reduction in critical vulnerabilities.
• The study highlights the MA-CoT model for articulating multi-layer security constraints, notably improving authorization and credential management controls.

Security-Oriented Developer Training in LLM-Assisted Web Application Development

Problem Context and Motivation

LLMs have been widely adopted in code generation and developer assistance, but empirical evidence demonstrates persistent vulnerabilities and inconsistent security quality in generated code, particularly when security constraints are inadequately specified. Prior studies reveal high rates of insecurity in LLM-generated programs across languages and frameworks, even with advanced models and security-aware explanations. Although prompt engineering and automated interventions can reduce vulnerabilities, these improvements depend fundamentally on prompt structure and are often produced by researchers, not practitioners. This paper addresses a vital gap by examining whether targeted developer training in security-oriented prompting can improve the security of LLM-assisted code without modifying the underlying model.

Experimental Design and Methodology

The research employs a controlled, quasi-experimental mixed-design involving twelve developers stratified by experience (six junior, six senior), each completing matched development tasks in a Java Spring Boot user-management backend. The study utilizes a within-subject pre-training/post-training comparison: participants first use unstructured prompts (baseline), then receive a standardized training intervention and apply a structured prompting protocol reflecting the Mitigation-Aware Chain-of-Thought (MA-CoT) architectural-layer model. This model guides developers in explicitly expressing security constraints across layers, including language-core semantics, built-in APIs, ecosystem frameworks, platform/OS interfaces, toolchain behavior, and application security logic.

Security outcomes are rigorously assessed via independent manual reviews using a predefined guide, recording validated weaknesses with associated CWE mappings and severity ratings. The principal endpoint is a severity-weighted weakness score ($$S_i^{(c)} = 4C_i^{(c)} + 3H_i^{(c)} + 2M_i^{(c)} + L_i^{(c)}$$), analyzed via exact Wilcoxon signed-rank tests. Task sets are counterbalanced to mitigate order effects, and all code is generated from identical starter projects under a fixed LLM configuration (gpt-5.2-2025-12-11).

Empirical Results and Numerical Analysis

Post-training developer runs exhibited significant reduction in validated security weaknesses and severity burden. Aggregate findings declined from 162 (pre) to 111 (post), a 31.5% reduction; the severity-weighted burden dropped from 432 to 267, a 38.2% reduction. Most notably, critical vulnerabilities decreased from 24 to 5 (79.2% reduction), with paired participant-level analysis showing a median improvement of 11.5 weighted points ($p = 0.0059$, Holm-adjusted $p = 0.0117$, $r_{rb}=0.93$), indicating a large and consistent effect.

Descriptive stratification by expertise level showed both junior and senior developers improved post-training, though junior participants had a larger median reduction in severity-weighted burden. Examining weakness family distributions demonstrated the strongest gains in authorization/object access (53.3% reduction) and authentication/credential policy/recovery (44.7% reduction). However, session and browser trust-boundary weaknesses remained unchanged, and operational hardening (availability, abuse resistance) showed minimal improvement.

MA-CoT layer mapping highlighted that reductions were concentrated at the ecosystem library/framework layer (52.1% reduction) and application security logic/policy layer (23.4% reduction), reinforcing that explicit security reasoning and structured prompting most effectively mitigate vulnerabilities at these architectural levels.

Theoretical Implications and Practical Significance

The findings formally substantiate that developer-side training enables meaningful security improvements in LLM-assisted code generation without model modification. Gains are strongest in domains where security constraints are readily articulable—authorization, authentication, policy, and boundary control. Conversely, domains reliant on secure framework defaults, rate limiting, operational configuration, and cryptographic enforcement require further intervention. Importantly, these results do not support the displacement of secure defaults, static analysis, expert review, or deployment hardening; developer training acts as a complementary safeguard.

From a machine learning alignment perspective, the study demonstrates that user-level interaction design and structured prompting can effectively shape LLM outputs, independent of core model architecture. This shifts the locus of control from model-centric to developer-centric, suggesting that improved reliability in LLM code generation may be achieved through user training, interaction scaffolds, and mental models, augmenting the evolving landscape of automated security interventions.

Limitations and Threats to Validity

The quasi-experimental design, fixed pre/post sequence, and limited sample size restrict causal inference and subgroup power, particularly for expertise-by-intervention effects. Manual review, though structured and independent, may still reflect reviewer bias and contextual variability in severity scoring. Environmental control is achieved via standardization of project, interface, and LLM configuration, but external validity to other languages, frameworks, and models remains to be established. The bundled nature of the training package precludes isolation of individual component effects (instructional content, MA-CoT model, prompting protocol).

Implications for Future AI Developments

The demonstrated efficacy of developer-centered, structured security training underscores the need for scalable, context-sensitive intervention mechanisms that empower practitioners to elicit robust LLM outputs. Expansion to broader domains (e.g., operational hardening, log management, external system integration) and frameworks, incorporation of real-world artifact logs, and larger replications are indicated. Integration of security-centric prompting protocols within IDEs and developer tooling may further enhance practical security outcomes. The MA-CoT model may serve as a foundation for layered prompting checklists and interaction pattern mining.

Conclusion

Structured layer-based security training significantly reduces both the quantity and severity of validated security weaknesses in LLM-assisted Java backend development, with strongest reductions in authorization and credential management failures. Developer-side intervention supplements but does not substitute established security review and operational safeguards. Improvements are concentrated at the application and framework configuration layers, emphasizing the role of explicit boundary reasoning and constraint articulation. These results present actionable directions for developer-aligned security interventions and highlight a complementary path to improving LLM reliability through interaction-level alignment, with broad implications for future AI-assisted software engineering workflows.

• "A Quasi-Experimental Developer Study of Security Training in LLM-Assisted Web Application Development" (2604.17763)