Disentangle the mechanisms behind higher regression rates in syntactically clean timelines

Disentangle whether the higher security-regression rate of syntactically clean iterative Terraform repair timelines is caused by their greater number of valid-to-valid transitions, by more aggressive restructuring in response to security-only feedback, or by both mechanisms, through an ablation study varying feedback types.

Background

The study reports that scenarios with consistently valid syntax regress more often than scenarios that experience syntax errors: 20.8% versus 12.1%. The authors propose two non-mutually-exclusive explanations. Clean timelines provide more opportunities for Checkov-level regressions because they contain more evaluable valid-to-valid transitions, while security-only feedback may encourage the LLM to restructure already-valid Terraform code more aggressively and thereby break previously passing checks.

The paper does not determine which explanation, if either, accounts for the observed difference. An ablation study that independently varies syntax and security feedback would be needed to distinguish the opportunity effect from the feedback-purity effect.