SCAFFOLD-CEGIS: Preventing Latent Security Degradation in LLM-Driven Iterative Code Refinement

Abstract: The application of LLMs to code generation has evolved from one-shot generation to iterative refinement, yet the evolution of security throughout iteration remains insufficiently understood. Through comparative experiments on three mainstream LLMs, this paper reveals the iterative refinement paradox: specification drift during multi-objective optimization causes security to degrade gradually over successive iterations. Taking GPT-4o as an example, 43.7 % of iteration chains contain more vulnerabilities than the baseline after ten rounds, and cross-model experiments show that this phenomenon is prevalent. Further analysis shows that simply introducing static application security testing (SAST) gating cannot effectively suppress degradation; instead, it increases the latent security degradation rate from 12.5% under the unprotected baseline to 20.8 %. The root cause is that static-analysis rules cannot cover structural degradations such as the removal of defensive logic or the weakening of exception handling. To address this problem, we propose the SCAFFOLD-CEGIS framework. Drawing on the counterexample-guided inductive synthesis (CEGIS) paradigm, the framework adopts a multi-agent collaborative architecture that transforms security constraints from implicit prompts into explicit verifiable constraints. It automatically identifies and solidifies security-critical elements as hard constraints through semantic anchoring, enforces safety monotonicity through four-layer gated verification, and continuously assimilates experience from failures. Comparative experiments against six existing defense methods show that the full framework reduces the latent security degradation rate to 2.1% and achieves a safety monotonicity rate of 100%.

• The paper introduces SCAFFOLD-CEGIS, a CEGIS-inspired framework that combines semantic security anchors, static analysis, invariant checks, diff budgets, and structured failure feedback to constrain iterative LLM code refinement.
• The paper finds that unprotected refinement can increase vulnerabilities in 43.7% of GPT-4o chains, while post-hoc SAST gating raises latent degradation from 12.5% to 20.8%, revealing a pseudo-safety effect.
• The paper shows that SCAFFOLD-CEGIS achieves 0% degradation, 100% safety monotonicity, 2.1% latent-degradation rate, and 77.14% task completion, though validation remains limited to 24 single-file tasks in Python and Java.

The iterative refinement paradox

This paper examines a property of LLM-assisted development that prior work on single-shot code security has largely left unmeasured: how the security posture of a program evolves across successive refinement iterations. The authors model iterative refinement as a state-transition system over programs, define safety monotonicity as non-increasing vulnerability counts $V(P_{i+1}) \leq V(P_i)$ along an iteration chain, and measure degradation rate (DR) and safety monotonicity rate (SMR) over chains. Their central empirical claim is that, absent strong external constraints, multi-turn refinement degrades security rather than improving it. In an observational study of 96 GPT-4o iteration chains using Semgrep with the p/default rule set, 43.7% of chains contain more vulnerabilities than baseline after ten rounds when all severities are counted ($DR_{all}$). Even under explicit security-hardening prompts, the degradation rate remains 28.6%, and feature-enhancement prompts produce the worst outcome at 52.8%. Notably, correlation between code churn and degradation is consistently low (0.072–0.156), indicating that degradation volume alone does not explain the effect. The authors attribute the phenomenon to specification drift: when security exists only as soft prompt-level constraints among competing objectives (functionality, performance, maintainability), the optimization trajectory gradually departs from the intended security specification.

SAST gating and the pseudo-safety effect

A natural mitigation—rejecting any change that introduces new static-analysis findings—turns out to be counterproductive for latent defects. The paper defines latent security degradation as removal or weakening of security-critical elements (deleted validation functions, weakened exception handling, bypassed permission checks) that trigger no SAST rule because the remaining code matches no vulnerability signature. Three representative cases illustrate this: deletion of validate_user_input while parameterized queries remain (invisible to both Semgrep and CodeQL); replacement of concrete exception handling with an empty except block; and introduction of an administrator shortcut path bypassing permission checks.

The most striking result in the paper is that Post-hoc SAST gating raises the latent-security degradation rate (SSDR, measured by independent LLM semantic review with Claude Opus 4.5) from 12.5% under the unprotected baseline to 20.8%. The authors call this a pseudo-safety effect: the model treats "passing static scanning" as license for larger refactors that introduce more out-of-coverage degradation. This directly contradicts the assumption that adding a SAST gate monotonically improves security outcomes, and it motivates the paper's central design decision to anchor defensive structure explicitly rather than rely on pattern-based detection.

Framework design

SCAFFOLD-CEGIS adapts the CEGIS loop—candidate generation, verification, feedback, regeneration—to settings where full formal specification is impractical. Formal verifiers are replaced by a gate combining static analysis with semantic-anchor checking; counterexamples become structured natural-language failure reports. Four agents collaborate:

• SecurityArchitectAgent mines the security specification $$\Phi = \langle \mathcal{I}, \mathcal{A}, \mathcal{R} \rangle$$ from four levels: function-level anchors (naming conventions plus data-flow analysis tracing guard functions between input boundaries and sensitive sinks such as SQL execution or deserialization calls), pattern-level anchors (e.g., parameterized-query patterns), and invariant-level anchors per task category. Anchors carry lock level, priority, confidence, and are managed by TTL and migration protocols so protected logic can be legitimately refactored with equivalence evidence.
• ImplementerAgent generates candidates under hard anchor constraints.
• GatekeeperAgent enforces four layers: test-suite correctness, safety monotonicity ($\Delta_{CH} \leq \delta_{max}$ and risk-density change $\Delta_\rho \leq \epsilon$), diff budget, and anchor integrity. Experiments use $\delta_{max} = 0$, which yields the monotonicity guarantee by construction.
• AssimilatorAgent converts rejections into structured remediation rules stored in a knowledge base and delivered as developer messages, and dynamically adjusts anchor priorities based on violation history.

The key architectural move is converting security constraints from implicit prompts into explicit, machine-checkable constraints—a direct response to the specification-drift diagnosis.

Experimental results

The evaluation uses 24 single-file task samples across six security scenarios (SQL injection defense, input validation, session/token validation, resource limits, cryptography, path traversal) in Python and Java, spanning LOC 25–392 and cyclomatic complexity 4–145, yielding 288 chains and 2,880 iteration steps across GPT-5-Nano, Claude Sonnet 4.5, and DeepSeek-V3.

RQ1 (external validity): Degradation reproduces across models and languages. Overall $DR_{CH}$ ranges from 10.5% (GPT-5-Nano) to 4.2% (DeepSeek-V3), but reliability degradation detected by LLM review runs far higher (42.5%–52.5%), exposing a systematic gap between static-analysis metrics and semantically reviewed quality.

RQ2 (defense comparison): Against six baselines on GPT-5-Nano, SCAFFOLD-CEGIS achieves DR = 0%, SMR = 100%, and SSDR = 2.1% with $\Delta S = -0.35$, versus SSDR of 12.5% (baseline), 20.8% (Post-hoc SAST), and 22.9% (Test-driven Guard). It also preserves reasonable evolution capacity (CEV 58,216 lines versus 71,998 for baseline).

RQ3 (ablation): The components decompose cleanly. Gate-Only and No-Assimilation both reach DR = 0% and SMR = 100%, confirming gated verification as the source of monotonicity. Anchor-Only reduces DR to 2.50% but cannot enforce rejection, showing anchors as soft guidance are insufficient. Anchors extend detection coverage: No-Assimilation's rejection rate (52.08%) exceeds Gate-Only's (41.67%). Failure assimilation restores evolution capacity under strict gating, raising fix success rate from ~21–23% to 52.71% and task completion rate from 66.19% to 77.14%.

Setting	DR	SMR	TCR	FSR
Baseline	11.5%	96.2%	100.0%	—
Anchor-Only	2.50%	97.62%	100.0%	—
Gate-Only	0.00%	100.0%	72.38%	20.83%
No-Assimilation	0.00%	100.0%	66.19%	23.19%
SCAFFOLD-CEGIS	0.00%	100.0%	77.14%	52.71%

Limitations and open questions

The authors are explicit about several constraints on their claims. The benchmark comprises only 24 samples restricted to two languages and single-file modules; absolute degradation rates may shift with larger and more diverse corpora, and extension to multi-file industrial codebases requires further validation. SSDR depends on LLM semantic review and inherits reviewer noise; the authors mitigate this through differencing against the same code's baseline and structured-output requirements, but position SSDR as a trend indicator rather than an exact count—their claims rest on relative rankings across methods. Gate effectiveness remains bounded by Semgrep rule coverage, and computational cost grows with code scale due to per-iteration AST parsing, data-flow analysis, and anchor verification. Anchor mining still leans on naming conventions, motivating learned classifiers for security-critical element identification. Fully automated gate decisions leave no mechanism for developer override of anchor violations, and explainability of gate failures and anchor semantics remains unresolved.

Conclusion

This paper establishes that LLM-driven iterative code refinement carries an inherent risk of security degradation driven by specification drift, demonstrates that SAST gating alone can worsen latent degradation through a pseudo-safety effect, and shows that explicit semantic anchoring combined with four-layer gated verification and failure assimilation suppresses it—reducing SSDR from 20.8% to 2.1% while achieving 100% safety monotonicity and retaining 77.14% task completion. The results argue that security properties in iterative synthesis must be enforced as verifiable hard constraints rather than communicated as prompts. Whether these guarantees hold at industrial codebase scale, across broader language toolchains, and under human-in-the-loop override policies remains open.