Adversarial Bug Reports in APR Systems

• Adversarial Bug Reports are intentionally crafted natural-language submissions designed to manipulate APR systems into producing insecure or malicious patches.
• They exploit the APR pipeline by embedding contextually plausible yet malicious information to reintroduce vulnerabilities, exfiltrate secrets, or waste compute resources, achieving a high success rate in empirical studies.
• Mitigation strategies include layered defenses such as structured LLM filtering, sandboxed execution environments, and human-in-loop reviews to secure the automated repair workflow.

Searching arXiv for papers on adversarial bug reports and closely related defect-report reliability work. arXiv_search_query: "adversarial bug reports automated program repair" Adversarial bug reports are user-visible issue submissions intentionally engineered to appear as legitimate bug reports while steering a LLM–based Automated Program Repair (APR) system toward insecure, unintended, or malicious code changes. In the reported formulation, the attack exploits the trust boundary created when APR systems ingest natural-language issue reports and automatically synthesize patches; the report is adversarial precisely when the APR system maps it to a patch that fulfills the attacker’s objective, such as reintroducing a CVE, injecting test-time secrets exfiltration code, or wasting compute (Przymus et al., 4 Sep 2025).

1. Definition and formal scope

The core definition is semantic rather than stylistic. An adversarial bug report is not merely low quality, ambiguous, or mistaken; it is intentionally constructed to induce an APR system to generate a patch aligned with an attack goal. Let $R$ denote the space of natural-language reports, $P$ the space of patches produced by an APR system $A$, and $r^* \in R$ a crafted report. The report is adversarial when

$$A(r^*) = p_{\text{attack}} \quad\text{and}\quad p_{\text{attack}} \models \text{malicious},$$

where “$\models \text{malicious}$” indicates violation of integrity or security invariants (Przymus et al., 4 Sep 2025).

This formalization places adversarial bug reports at the interface between prompt injection, social engineering, and code-generation security. The report must remain plausible as an issue submission—using correct style, context snippets, and plausible reproduction steps—while causally shaping the patch search process toward a malicious outcome. The attack therefore targets the APR system’s natural-language conditioning channel rather than its model weights or training corpus.

A common misconception is to equate adversarial bug reports with ordinary false reports. The distinction is that ordinary false reports need not carry an attacker objective, whereas adversarial bug reports are goal-directed inputs crafted to induce a particular harmful repair behavior. In this sense, the bug report functions as an attack payload encoded in issue-tracker discourse.

2. Threat model and attack surface

The threat model treats LLM-based APR systems deployed in continuous integration and software maintenance workflows as security-relevant targets. The protected assets are explicitly identified as source code integrity, the CI/CD environment, and operational resources. Correspondingly, the adversary’s goals are to introduce vulnerabilities, exfiltrate secrets, or exhaust resources (Przymus et al., 4 Sep 2025).

The adversary is assumed to control only the bug report text. No model poisoning is assumed. The attacker may or may not have source code access, but can submit an unlimited number of high-quality bug reports, often via LLMs. This is significant because the attack is feasible under comparatively weak capabilities: the manipulation channel is the same public issue-reporting interface already intended for routine maintenance traffic.

The attack surface is described as a pipeline: public issue tracker entry, APR ingestion, patch generation, and CI/CD test execution. The downstream impact is correspondingly broader than code synthesis alone. A successful adversarial report can lead to malicious code in production, secret leakage, or denial of service through wasted compute and reviewer effort. In STRIDE terms, the concrete threat categories are tampering, information disclosure, denial of service, and elevation of privilege (Przymus et al., 4 Sep 2025).

This threat model exposes an asymmetry characteristic of many LLM security settings: the attacker injects a single carefully composed natural-language artifact, while the defender must secure report triage, patch generation, static analysis, dynamic execution, and human review simultaneously. A plausible implication is that APR security cannot be reduced to model robustness in isolation; it is a workflow-security problem.

3. Construction of adversarial bug reports

The reported attack generation spectrum ranges from manual curation to fully automated pipelines. A generic pipeline consists of four stages: context retrieval, prompt construction, report synthesis via LLM, and optional validation and filtering (Przymus et al., 4 Sep 2025).

For context-sensitive attacks, particularly those targeting known security fixes, the generation process uses a context-building mechanism based on locality-sensitive hashing (LSH) and fuzzy string matching. Given an original diff $D$, the sanitized context is

$$C = \text{sanitize}(D), \qquad \text{sanitize}(\delta) := \delta \setminus (\text{tests} \cup \text{docs}),$$

so that tests and documentation are removed before embedding the diff into the synthesized issue report. The corresponding pseudocode centers on retrieving relevant commits, sanitizing the seed diff, and producing a context snippet from the resulting fragment (Przymus et al., 4 Sep 2025).

The implemented attack families span distinct attacker objectives and degrees of automation.

Attack type	Objective	Automation level
Naive APR	Trigger irrelevant code	Semi-automatic
CI/CD Exploits	Exfiltrate secrets or RCE in tests	Semi-automatic
Vulnerability Injection	Reintroduce insecure logic	Manual + context
Revert CVE Fix	Undo known security patch	Automatic + context
Deceptive Noise	Waste APR and review cycles	Automatic + context

The “Revert CVE Fix” template is particularly direct: the prompt asks the model to “Write a bug report that reverts this commit...” with the sanitized diff inserted as context (Przymus et al., 4 Sep 2025). That construction is notable because it converts repository history into adversarial scaffolding. Rather than attacking APR through arbitrary instructions alone, it exploits the model’s tendency to treat patch-like contextual evidence as legitimate maintenance guidance.

The prototype framework released for scalable evaluation automates seed selection, context retrieval, prompt injection, issue submission to a private GitHub mirror, APR invocation and pull-request collection, and post-APR analysis using Copilot and CodeQL. This automation closes the attack-feedback loop and makes continuous red-teaming operationally feasible (Przymus et al., 4 Sep 2025).

4. Empirical evaluation and measured effectiveness

The reported empirical study evaluates 51 adversarial bug reports against an APR backend consisting of SWE-agent v1.1.0 with SW-ReX v1.2.2, using Claude Sonnet for patch synthesis. The repositories are five Python projects from SWE-bench: django, flask, requests, pip, and scikit-learn (Przymus et al., 4 Sep 2025).

Adversarial success is measured at the report level. For each submitted report $r_i$, $\mathrm{PatchCreated}_i = 1$ when SWE-agent generated a pull request, and $P$0 when the generated patch executed the attacker’s intent. The attack success rate is then

$P$1

Empirically, all 51 reports led to patches, and 46 of 51, or 90.2%, fulfilled malicious objectives (Przymus et al., 4 Sep 2025).

The defense results are comparably specific. Among pre-APR filters, the best reported result is o4-mini with unstructured output, detecting 24 of 51 reports, or 47.1%. GPT-4.1-mini with unstructured output detects 12 of 51, or 23.5%. PromptGuard variants detect 3 of 51, or 5.9%; LlamaGuard variants detect 0 of 51; Granite-Guardian detects 5 of 51, or 9.8%. For post-APR defenses, CodeQL detects 0 of 51, while GitHub Copilot review, aggregating all comment levels, detects 30 of 51, or 58.8%. Ensemble strategies improve blocking rates but do not close the gap: the full pre-APR ensemble blocks 32 of 51, the best pre-APR plus Copilot blocks 35 of 51, and the full pre-APR ensemble plus Copilot reaches 37 of 51, or 72.5% (Przymus et al., 4 Sep 2025).

These results directly challenge the assumption that existing moderation or code-review tools are sufficient when APR is driven by untrusted issue text. Another common misconception is that static analysis alone provides an adequate backstop; the measured CodeQL result of 0 of 51 is an explicit counterexample in this setting.

5. Defensive posture and operational mitigations

The reported recommendations favor layered controls rather than any single detector. At the input boundary, the work recommends structured LLM-based filtering, specifically using small LLMs such as o4-mini with structured outputs to detect malicious intent at low cost, reported as less than \$0.003 per issue (Przymus et al., 4 Sep 2025). This recommendation is operationally important because it addresses scale: issue trackers can receive many reports, and triage cost is itself part of the attack surface.

At execution time, APR and CI/CD processes are recommended to run in sandboxed environments with strict quotas and privilege separation. This follows directly from the threat model’s inclusion of CI/CD exploits, test-time secret exfiltration, and resource-exhaustion objectives. The recommendation is not limited to preventing bad patches from merging; it also aims to prevent damage during validation and testing.

For downstream review, the work recommends improving the visibility of Copilot’s low-confidence and file-level security comments, while warning against heavy reliance on large ensembles because they increase false positives. Human-in-the-loop triage remains explicitly recommended, especially for APR-generated pull requests that touch security-sensitive files (Przymus et al., 4 Sep 2025). In other words, human review is treated not as a fallback after all automation succeeds, but as a necessary control for high-risk repair contexts.

The future research agenda is correspondingly structural. The identified directions are Policy-Aware APR, which would integrate repository-specific security constraints such as “never revert CVEs”; Certified Defenses, which would apply formal methods to guarantee patch safety with respect to known invariants; Adversarial Training of APR Models using adversarial bug reports; and Automated Red-Teaming as a Service embedded into CI pipelines (Przymus et al., 4 Sep 2025). Together, these directions imply that robustness must eventually be expressed not only in terms of model behavior, but also in terms of enforceable repair policies.

6. Distinctions, adjacent work, and unresolved questions

Adversarial bug reports should be distinguished from a nearby but different problem: plausible-but-wrong defect reports produced during LLM-assisted defect discovery. In the Refute-or-Promote framework, the central problem is a precision crisis in which plausible-but-wrong reports overwhelm maintainers and degrade credibility for real findings. The proposed remedy is an adversarial, stage-gated multi-agent pipeline using Stratified Context Hunting, adversarial kill mandates, context asymmetry, a mandatory empirical validation gate, and a Cross-Model Critic. Over a 31-day campaign across seven targets, the pipeline killed roughly 79% of 171 candidates before disclosure; on a consolidated-protocol subset, the prospective kill rate was 83% (Agarwal, 21 Apr 2026).

The distinction is conceptually important. Adversarial bug reports are malicious inputs injected into an APR workflow, whereas Refute-or-Promote addresses false-positive elimination in a defect-discovery workflow. The former is an attack on the repair process; the latter is a reliability architecture for finding real defects. This suggests that “adversarialism” has at least two roles in the literature: as an attacker strategy against APR and as a defender strategy for filtering suspect findings before disclosure.

The most instructive failure in Refute-or-Promote further sharpens this point. Ten dedicated reviewers unanimously endorsed a non-existent Bleichenbacher padding oracle in OpenSSL’s CMS module, and the candidate was killed only by a single empirical test, motivating a mandatory empirical gate (Agarwal, 21 Apr 2026). A plausible implication for adversarial bug reports is that consensus among LLMs or reviewers about the plausibility of an issue report is not a reliable proxy for safety. If APR systems are to accept natural-language reports as triggers for automated code modification, they require policy checks and empirical verification steps that are independent of the linguistic persuasiveness of the report.

A second adjacent lesson comes from adversarial robustness evaluation. In the analysis of Sabre, a one-line bug in the evaluation code induced severe gradient masking, and removing that line reduced the reported robust accuracy to 0% on CIFAR-10 and to 0% under PGD-100 on MNIST; further patches introduced additional failure modes that again collapsed robustness under adaptive attack (Carlini, 2024). This does not concern bug reports directly, but it underscores a broader methodological point: security claims about AI systems are vulnerable to implementation artifacts, incomplete threat modeling, and non-adaptive evaluation. For adversarial bug reports, that observation supports the need for attack-aware testing rather than reliance on nominal filter performance alone.

The principal unresolved issue is structural asymmetry. Generating adversarial inputs is inexpensive, while detecting or mitigating them remains costly and error-prone (Przymus et al., 4 Sep 2025). Until APR systems can enforce repository-specific invariants and maintain trustworthy execution isolation, adversarial bug reports remain a fundamental security concern for automated repair pipelines rather than a narrow prompt-engineering edge case.