Quality and Security Signals in AI-Generated Python Refactoring Pull Requests

Abstract: As AI agents increasingly contribute to code development and maintenance, there is still limited empirical evidence on the quality and risk characteristics of their changes in real-world projects, particularly for refactoring-oriented contributions. It remains unclear how agent-authored refactoring edits affect maintainability, code quality, and security once merged into GitHub repositories. To address this gap, we conduct an empirical study of Python refactoring pull requests (PRs) from the AIDev dataset. We analyze agentic refactoring PRs using PyQu, an ML-based quality assessment tool for Python, to quantify changes across five quality attributes, and we complement PyQu with domain-independent static analysis (Pylint and Bandit) to measure code quality and security issues before and after each change. Our results show that, on average, agentic commits improve a quality attribute in 22.5% of the studied changes, with usability improving most frequently (36.5%). At the same time, 24.17% of modified files introduce new Pylint issues predominantly convention level violations such as long lines-while 4.7% introduce new Bandit findings. From the observed diffs, we derive a taxonomy of 24 recurring change operations and map them to the lint and security findings they most commonly affect. Despite these mixed outcomes, developer acceptance is high: 73.5% of the analyzed PRs are merged, including cases that introduce new lint or security findings, often alongside the removal of existing issues. Overall, these findings highlight both the promise and current limitations of agentic refactoring, and motivate stronger tool-in-the-loop quality and security gating for AI-driven development workflows.

• The paper presents systematic empirical analysis of AI-generated refactoring PRs using PyQu, Pylint, and Bandit to assess quality improvements and regressions.
• It finds a modest 22.5% rate of quality enhancements, with usability most improved (36.5%), while highlighting significant static-analysis churn.
• The study shows high developer acceptance despite risks, advocating advanced gating tools to better balance automated improvements with security.

Empirical Characterization of Quality and Security in AI-Generated Python Refactoring PRs

Study Context and Methodology

This work systematically interrogates the quality and risk profiles of AI agent-generated refactoring pull requests (PRs) in real-world Python repositories, addressing an under-explored dimension of AI-in-the-loop software maintenance workflows. The study leverages the AIDev dataset, comprising nearly 1,000 AI-generated pull requests targeting repositories with over 100 GitHub stars, filtered to include 438 Python refactoring PRs from prominent agentic tools: OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code.

Quality assessment was performed using PyQu, an ML-based framework that infers attribute-level quality deltas from low-level Python program metrics, mapping to understandability, reliability, maintainability, modularity, and usability. Complementary static analyses—Pylint and Bandit—quantify code-quality and security issues, respectively, pre- and post-change. The study operationalizes these signals at both the commit and file levels, enabling fine-grained analysis of issue churn and the taxonomy of code change operations. Manual inspection by expert raters provides depth beyond automated metric labels.

Key Empirical Findings

Quality Gains and Attribute-Specific Patterns

Agentic PRs induce a measurable, though limited, improvement in software quality: 22.5% of refactoring commits enhanced at least one PyQu-tracked quality attribute. Usability is most commonly improved (36.5% of commits), followed by reliability (27.6%) and understandability (24.0%). Maintainability and modularity are rarely enhanced (14.9% and 9.5% of commits, respectively), reflecting the relative difficulty of realizing structural improvements through automated refactoring at scale.

Statistical interrogation confirms construct validity of the PyQu labels—enhancement and non-enhancement cohorts exhibit significant, attribute-aligned shifts in metric deltas (e.g., Cliff’s Delta for understandability = 0.500, $q = 3.71 \times 10^{-23}$). Exploratory analysis indicates a small number of cross-attribute metric signals but reinforces that major gains are concentrated in usability and readability, rather than deeper architectural properties. Manual review exposes that many "not enhanced" changes are mechanical (e.g., renaming, trivial cleanup) and do not measurably shift higher-level maintainability or modularity.

Static-Analysis Churn: Code-Quality and Security

Analysis of 2,528 Python files modified across the PRs reveals significant issue churn as captured by static analyzers. Pylint issues are introduced in 24.17% of files and removed in 19.94%, with the majority concentrated in stylistic, convention, and documentation compliance categories (e.g., line length, missing docstrings, import order). Only 4.7% of files introduce new Bandit-detected security issues—these tend toward risky patterns (use of assert, weak pseudo-randomness, improper subprocess invocation) rather than high-severity vulnerabilities.

Manual classification of high-impact code changes yields a taxonomy spanning direct security remediations (e.g., removing shell=True, unsafe serialization, /tmp path hardcoding), idiomatic and maintainability refactorings (extract method/class, replace repeated code with dynamic dispatch), API alignment, style and documentation adherence, and import hygiene. It is crucially observed that not all reductions in warnings are true remediations; warnings frequently disappear due to code relocation or suppression, not substantive fixes.

Developer Acceptance and Workflow Implications

Despite mixed quality and security signals, AI-generated refactoring is widely accepted: 73.5% of studied PRs were merged, even though 53.9% of these introduced new Pylint issues and 11.3% introduced new Bandit issues. Acceptance is heterogeneous across agents, with OpenAI Codex submissions enjoying the highest merge rates.

Static-analysis regressions are not reliable blockers—merged PRs routinely exhibit both the addition and removal of issues, indicating that improving some aspects of code quality or security may offset regressions in others in reviewer assessment. Notably, PRs introducing Bandit findings are merged without visible discussion or CI gating in a substantial fraction of cases, exposing a gap in automated security enforcement.

Implications and Theoretical Considerations

These findings reveal both the promise and limitation of agentic refactoring in production OSS workflows:

• Engineering Workflows: High merge rates, despite static-analysis regressions, expose a need for more robust tool-in-the-loop enforcement. Automated gating—integrating both quality and security analysis in pre-merge workflows—could align agentic code contributions with community standards and mitigate risk propagation.
• Metric-Driven Evaluation: The multidimensionality of code quality is poorly captured by test-passing alone—quality attribute assessment and static-analysis signals (particularly maintainability and modularity) need to be foregrounded in LLM evaluation and development pipelines.
• Security Posture: While high-severity vulnerabilities are rare, persistent introduction of risky practices by agents highlights the necessity of integrating feedback-driven static analysis and active hardening of agentic code generation.
• Human-AI Review Synergy: The observed tolerance for regressions suggests reviewers prioritize net change over strict regression avoidance; further study of reviewer rationales and risk perception is warranted.
• Future Directions: A controlled comparison with human-authored refactoring in the same repositories could illuminate systematic differences, bias, and acceptance trends, enabling calibrations in both agent design and review strategy.

Conclusion

AI-generated Python refactoring PRs result in modest but consistent improvements in select software quality attributes. However, they introduce considerable churn in code-quality and, less frequently, security signals, with widespread developer acceptance in spite of regressions. The evidence supports the incorporation of advanced tool-based gating mechanisms for both quality and security and highlights the necessity of moving beyond functional correctness toward holistic quality assurance in agentic and LLM-driven software development workflows. The methodology and replication package established in this study provide a foundation for the ongoing empirical evaluation of automated code change agents in large-scale, real-world environments.

Reference: "Quality and Security Signals in AI-Generated Python Refactoring Pull Requests" (2605.21453).