Generalizability to Interactive Coding-Assistant Workflows

Determine whether the observed persistence of exploitability in statically clean LLM-generated Python programs extends to code produced by assistants in interactive development workflows involving differing context, iteration, tool use, and human oversight.

Background

The study evaluates security-sensitive benchmark code and LLM-generated vulnerable Python samples rather than code produced by an interactive coding assistant. Its results show that some LLM-generated programs can pass the composite Bandit–Semgrep static-analysis gate while remaining exploitable under runtime verification.

The authors explicitly state that it is unresolved whether this finding generalizes to assistant-in-the-loop development workflows. Such workflows may differ substantially because developers and assistants can iteratively revise code, use external tools, incorporate broader contextual information, and receive human oversight.