Trust but Verify? Uncovering the Security Debt of Autonomous Coding Agents

Abstract: The increasing adoption of autonomous coding agents accelerates software development but also introduces scoped security risks within high-impact file paths that can outpace traditional human review capacity. While prior research has primarily evaluated these systems in terms of functional correctness and productivity, this paper presents a large-scale empirical study using the AIDev dataset to systematically characterize security code smells in agent-generated pull requests (PRs). Through a combination of a validated LLM-as-a-judge framework and manual qualitative analysis, we identify and classify security misconfigurations across 16,112 file changes spanning 4,022 pull requests. Our results reveal that 38.9% of agent-generated PRs contain at least one security smell, with supply chain integrity issues accounting for 82.3% of all detected security smells. Furthermore, hard-coded credentials constitute 99.6% of all critical-severity security smells. Crucially, we find that human collaborators are responsible for introducing 67.6% of genuine leaked secrets within these agent-assisted workflows, while existing automated and human review processes fail to detect 81.1% of these credentials prior to integration. These findings highlight substantial security risks in agent-assisted software development workflows and suggest a potential reduction in developer vigilance. They also underscore the urgent need for context-aware security guardrails implemented directly at the point of human-AI collaboration.

• The paper provides a systematic quantification of security smells in agent-generated PRs, revealing a 38.9% flagging rate and an 82.3% share of supply chain misconfigurations.
• Using a validated LLM-as-a-judge pipeline along with manual reviews, the study detects security flaws across high-risk file categories in software engineering workflows.
• The analysis shows that human collaborators are responsible for most critical hard-coded credentials, highlighting significant gaps in current review processes.

Empirical Characterization of Security Debt in Autonomous Coding Agents

Introduction

"Trust but Verify? Uncovering the Security Debt of Autonomous Coding Agents" (2607.12428) provides a rigorous empirical investigation into the security posture of agent-generated code in contemporary Software Engineering 3.0 workflows. Autonomous coding agents are increasingly responsible for generating, testing, and submitting PRs—a paradigm shift that has accelerated code production but has also created conditions favoring the introduction of structural security flaws. Prior literature largely emphasized functional correctness, productivity, and user experience, leaving significant gaps regarding the distribution and severity of security code smells within the dynamic paths affected by agentic workflows.

This paper employs the AIDev dataset, comprising 16,112 file changes across 4,022 PRs, to systematically audit selected high-risk file categories for security smells. Using a validated LLM-as-a-judge pipeline, cross-model validation, and manual qualitative review, the authors quantify the prevalence, category, severity and review dynamics of smells introduced by autonomous agents and human collaborators.

Figure 1: Proposed methodology architecture for detection, labeling, and validation of security smells in agent-generated PRs.

Methodology

The study focuses on file path patterns associated with elevated security risk, such as CI definitions, Dockerfiles, shell scripts, infrastructure-as-code, and configuration files. Detection is grounded in a six-category taxonomy derived from OWASP, CIS Benchmarks, and GitHub hardening documentation: secrets_identity, cleartext_transport, over_privilege_execution, permissive_network, misconfig_hardening, and supply_chain_integrity.

Security smell detection is executed via zero-shot prompting of quantized LLMs—Qwen3.6-35B-A3B-FP8 and Gemma-4-26B-A4B-IT-FP8—each receiving file-specific diff, context, and metadata. Responses label files as clean or flagged, assign categories and severity, and provide rationale. Model outputs are merged in a union protocol and deduplicated for analysis.

Validation employs a manually coded gold standard, yielding precision, recall, F1, and inter-rater agreement, with subsequent manual verification of hard-coded credentials in flagged secrets_identity instances. Stratified analysis investigates agent type, repo language, file category, and PR size effects.

Security Smell Prevalence and Category Distribution

The LLM-as-a-judge system demonstrated a PR-level flagging rate of 38.9\%—that is, nearly 2 out of every 5 agent-generated PRs contained at least one security smell within selected high-risk paths. Supply_chain_integrity misconfigurations represent 82.3\% of all detected smells, primarily mutable action/image tags and globally unpinned dependency installations. Over_privilege_execution and secrets_identity follow at 9.6\% and 3.4\%, respectively. Crucially, 99.6\% of critical-severity smells correspond to hard-coded credentials.

Figure 2: Severity composition of smells by category, highlighting the dominance of supply_chain_integrity and concentration of critical smells in hard-coded secrets.

Disaggregating prevalence by agent and language reveals key insights. GitHub Copilot PRs exhibited the highest smell rate at 45.5\%, while OpenAI Codex PRs registered the lowest at 34.9\%. JavaScript repositories were particularly vulnerable, flagging 55.3\% of agentic PRs versus Python's 31.8\%. Smell occurrence correlates strongly with development activity: PRs that modify more lines are increasingly likely to manifest smells, scaling from 16.2\% (XS) to 53.6\% (XL) with the volume of code changed.

Figure 3: Summary of RQ2 investigation outcomes for credential leaks, attribution, and review interventions.

Figure 4: PR-level smell-flagging rate by agent type and primary language relative to corpus average.

Figure 5: Smell-flagging rate by PR size confirms upward trend with increasing lines changed.

Attribution and Review Dynamics of Critical Smells

The critical-severity smells primarily involve genuine hard-coded credentials, which pose immediate exploitability risk upon integration. Manual review confirmed the authenticity of 27.2\% of labeled secrets_identity smells. Attribution analysis reveals a significant departure from intuition: human collaborators, not agents, are responsible for introducing 67.6\% of genuine leaked secrets in agent-assisted workflows. Automated security bots and human reviewers only flagged 18.9\% of leaked credentials prior to integration; thus, 81.1\% of genuine leaks were missed during review.

Figure 6: LLM-as-a-judge prompt used for security smell classification.

Figure 7: Representative response sent to LLM for file change evaluation.

Developers leveraging agentic workflows appear to bypass standard security hygiene, potentially due to reduced vigilance or cognitive offloading behaviors emerging from increased trust in the agentic code pipeline.

Implications for Agentic Software Engineering

The study demonstrates that the velocity and scale of agent-generated PRs overwhelm traditional review pipelines, increasing repository security debt through marginal detection coverage. The concentration of high-severity smells in privileged paths and CI artifacts, combined with the predominance of human-induced critical leaks, suggests a compromised review environment and insufficient guardrails at the human-AI interface.

Practically, this calls for the adoption of context-aware, in-line security interventions embedded within agentic workflows, targeting both agent and collaborator behaviors. Architecturally, push-down automated credential scanning and supply chain hygiene enforcement must operate at PR submission and review, not solely in post-commit auditing.

Theoretically, the findings inform future research on developer cognition during human-agent collaboration, behavioral economics of trust in agentic outputs, and adaptive guardrail design for both open-source and enterprise contexts. Further studies are warranted to differentiate generalizability across agent models, environments, and code domains.

Figure 8: Sample LLM server setup command for local model hosting during empirical study.

Figure 9: Sample manually verified secret detected via GitGuardian.

Figure 10: Sample credential leak detected and flagged by Gemini Code Assist.

Conclusion

This empirical analysis substantiates the significant security debt present in agent-assisted software development, characterized by high rates and critical concentrations of structural security smells. The dominant prevalence of supply chain misconfigurations and the near-exclusive association of critical smells with hard-coded credentials underline systemic weaknesses in modern AI-augmented workflows. The finding that human collaborators are the major source of credential leaks, coupled with ineffective review layers, exposes infrastructural gaps in both agent and human guardrails.

Addressing these operational risks necessitates targeted, context-sensitive security interventions at the agent-human interface. Future advancements in agentic software engineering must prioritize workflow-integrated security enforcement, real-time credential detection, and behavioral analytics to counteract cognitive offloading in mixed-initiative development pipelines. The study's methods and taxonomy serve as a foundation for ongoing enterprise-scale audits and agentic workflow hardening.