SecureForge: Black-box Hardening for Code LLMs

• SecureForge is a black-box hardening pipeline for code-generating LLMs that uses a three-stage process—falsification, MCMC amplification, and prompt optimization—to systematically reduce security vulnerabilities.
• It employs static analysis tools like Semgrep to identify exploitable CWEs and iteratively refine system prompts based on observed failure distributions without modifying model weights.
• The approach achieves up to a 48% reduction in vulnerabilities while maintaining unit test performance, demonstrating practical improvements in secure code generation.

SecureForge is a black-box hardening pipeline for code-generating LLMs that audits security risks of frontier models and produces auditing-informed secure system prompts that reduce output security vulnerabilities while maintaining unit test performance (Liu et al., 8 May 2026). It is explicitly an inference-time defense—no finetuning, no weights access, only API access plus a static analyzer—and it targets benign, realistic user requests rather than adversarial jailbreaks. Its central premise is that modern coding models can generate functionally correct code that passes tests, looks normal to a developer, and still contains exploitable CWEs; SecureForge therefore treats model-specific vulnerability elicitation and prompt hardening as a coupled auditing-and-mitigation problem rather than as generic “secure prompting” alone (Liu et al., 8 May 2026).

1. Problem setting and formal objective

SecureForge studies benign user prompting as the primary threat model. The concern is not overtly malicious requests, but ordinary software-development tasks that elicit insecure implementations. Across a corpus of 250 benign coding prompts, frontier models “still produce verifiable vulnerabilities on average 23\% of the time,” even when prompted to be secure and even when told the CWE families on which they will be judged; moreover, “12.7\% of the time, outputs are both vulnerable and yet still pass unit tests.” The paper therefore treats security failures as distributions over benign prompts rather than isolated examples (Liu et al., 8 May 2026).

The formalization indexes weakness families by failure modes $F_j$. For a fixed system prompt $I$, SecureForge defines the vulnerability rate of failure $F_j$ as the marginal probability that prompts exercising that failure mode yield any statically-verifiable vulnerability. The hardening objective is to find a new system prompt $I'$ such that

$$p_{\theta}\big(\tau_{\text{fail}\mid I',F_j}\big) \ll p_{\theta}\big(\tau_{\text{fail}\mid I,F_j}\big) \quad \text{for all weaknesses } F_j.$$

This formulation makes two commitments. First, the reward signal is static-analysis failure, not human preference or adversarial jailbreak resistance. Second, the defense is model-specific: the prompt is optimized against the observed benign failure distribution of a particular model rather than against an abstract notion of secure behavior (Liu et al., 8 May 2026).

The paper also argues that generic secure instructions underperform because they do not target how a given model actually fails. Its prompting baselines are a plain coding prompt, a “Secure” prompt asking for production-quality secure code, and a “CWE” prompt that additionally says, “We will be evaluating the code on CWE static analysis, so be sure to write code that is secure!” Even with these stronger prompts, models still exhibit “double-digit rates of vulnerabilities even for test passing cases,” which the paper interprets as evidence that current safety tuning improves resistance to overtly malicious requests more than to unintentional benign requests that induce insecure code (Liu et al., 8 May 2026).

2. Single-shot falsification and seed-prompt discovery

SecureForge is organized into three stages: single-shot falsification, MCMC amplification, and prompt optimization. The first stage discovers benign prompts that reliably elicit vulnerabilities, thereby constructing the seed set from which the rest of the pipeline proceeds.

Stage	Mechanism	Output
Single-shot falsification	MITRE CWE examples, benign prompt filtering, Semgrep	seed prompts
MCMC amplification	Metropolis-Hastings-style MCMC with LLM rephrasing kernel	synthetic prompt corpus
Prompt optimization	GEPA with Semgrep reward and Pareto selection	hardened system prompt

Seed construction begins from MITRE CWE examples and existing human-written prompts from Pearce et al. For each specified CWE, SecureForge generates a realistic coding task prompt, then revises it to remove explicit mentions of attacks or vulnerabilities. This “benign prompt filtering” adds production-quality language such as “ensure input is properly sanitized,” filters out mentions of CWEs, and applies a hotword filter based on the CWE title to prevent leakage. The target model is then sampled at temperature $1.0$ on each prompt, and outputs are analyzed with Semgrep; prompts whose outputs trigger static-analysis failures are retained as seeds. The appendix exposes the prompt-template modules used here: DescribeScenario, ExtractScenarios, StripVulnerability, and SuggestLibraries (Liu et al., 8 May 2026).

This stage is important because SecureForge does not treat human prompt writing as the main source of failure discovery. Instead, it programmatically synthesizes benign requests from weakness descriptions and uses static analysis to identify which requests actually induce insecure code on the target model. A plausible implication is that SecureForge is closer to failure-distribution discovery than to conventional red-teaming: the aim is not to find a single high-profile exploit prompt, but to characterize a broad, benign elicitation regime for each weakness family (Liu et al., 8 May 2026).

3. Failure-distribution discovery with MCMC amplification

The seed prompts are then expanded into a much larger synthetic corpus by a Markovian procedure inspired by Metropolis-Hastings. The desired target is the conditional failure-prompt distribution

$p(\cdot \mid F_j) = \frac{p(\cdot, F_j)}{p(F_j)},$

but because $p(F_j)$ is intractable, SecureForge samples from the corresponding unnormalized density

$$\bar{p}(x \mid F_j) = p_{\text{fail}(x)}\, p_{\text{user}(x)}.$$

The paper assumes $p_{\text{user}(x)}$ is uniform over fluent benign prompts, so the acceptance criterion is driven by estimated failure probability. The proposal kernel $I$0 is an LLM rephrasing prompt; symmetry is assumed, $I$1. Failure probability is estimated by repeated rollouts at temperature $I$2, modeled with a Beta posterior $I$3, where $I$4 is failure count and $I$5 success count; sampling continues until posterior variance is below $I$6 (Liu et al., 8 May 2026).

Operationally, the Markovian aspect matters because each new prompt is generated from the previous accepted prompt rather than repeatedly paraphrasing the original seed. The appendix names the rephrasing kernel GenerateConditionedPrompt, which is instructed to create a meaningfully different coding task that exercises the same tools, libraries, skills, and techniques as the previous one. Implementation details are explicit: 3 burn-in steps per prompt, 32 sampling steps per prompt, repeated rollout until Beta posterior variance is below $I$7, and a resulting optimization corpus size of 80,000 prompts (Liu et al., 8 May 2026).

The paper’s key ablation in this stage compares MCMC against repeated rephrasing of the original discovered prompt. MCMC “discovers new vulnerable prompts up to 2 times more successfully than simply rephrasing the original discovered prompt.” This suggests that local exploration of the failure neighborhood preserves both high failure propensity and prompt diversity better than seed-centric paraphrasing, and it is one of the few places where the paper isolates a concrete design choice empirically (Liu et al., 8 May 2026).

4. Hardening through iterative system-prompt optimization

After amplification, all discovered prompts are collected into a corpus

$I$8

which is then used to optimize a new system prompt. The optimization objective is presented as minimizing model failure probability over the hard-negative corpus under the target model $I$9. SecureForge instantiates this with GEPA, described as a zeroth-order prompt-evolution algorithm. Starting from a generic system prompt $F_j$0, GEPA rolls out the model on $F_j$1, scores each rollout using Semgrep output, reflects on the errors, mutates promising candidate prompts, and uses Pareto selection to preserve candidates that trade off objectives well (Liu et al., 8 May 2026).

The reward signal is highly concrete. Reflective feedback is generated from Semgrep findings: if vulnerabilities are found, the feedback is score: -1 together with each line number, rule id, and Semgrep message; if none are found, the feedback is score: +1 with “The generated trace does not contain any vulnerabilities. Good job!” GEPA settings are also specified: light preset, $F_j$2, Pareto selection, reflection minibatch 3, merge enabled, reflection model gpt-5.4 at temperature $F_j$3, target rollouts at temperature $F_j$4, max token budget $F_j$5, and reward $F_j$6 from Semgrep plus CWE-labeled critique (Liu et al., 8 May 2026).

The stated outcome is a Pareto improvement in the two-dimensional space of security and test performance. The paper does not formalize a Pareto equation, but repeatedly describes the geometric effect as movement “up and to the right” in the $F_j$7 plane: security rises without meaningful leftward movement in capability. This is central to the system’s identity. SecureForge is not framed as a pure security filter that sacrifices utility; rather, it is a prompt-optimization procedure whose objective is to reduce statically detectable vulnerabilities while preserving unit test performance (Liu et al., 8 May 2026).

5. Evaluation methodology and empirical findings

The evaluation uses the CWE Top 25 for Python/Semgrep settings. The experimental design is explicit: 20 scenarios per CWE under test, 10 scenarios used for MCMC/tuning, 10 held out for evaluation, total 500 prompts, 5,000 seed rollouts, 250 prompts used for MCMC amplification, and 250 held-out prompts used for evaluation. The held-out evaluation therefore covers 25 CWEs, 10 held-out scenarios each, and 10 rollouts each, for a total of 2,500 held-out evaluation samples. Vulnerability probability is estimated in a Bayesian way: for each prompt, the model is repeatedly rolled out until the Beta posterior variance over failure probability is less than $F_j$8, and reported uncertainty uses 95\% Beta-posterior credible intervals (Liu et al., 8 May 2026).

Model coverage spans both open and closed systems. The open models are Qwen 2.5 Coder 7B, Qwen 2.5 Coder 14B, Qwen3 30B-A3B MoE, CodeLlama 7B, and Kimi K2. The closed/frontier models are GPT 5.4-nano, GPT 5.4-mini, GPT 5.4, GPT 5.3-codex, and Claude Sonnet 4.6. All are sampled with temperature $F_j$9, max token budget $I'$0, and left truncation. Static vulnerability detection uses the Semgrep Community Python ruleset in the main paper evaluations, with rule-to-CWE mapping used to align findings to the weakness taxonomy; the released toolkit also supports CodeQL, but the main reported results use Semgrep (Liu et al., 8 May 2026).

Utility is tracked through LLM-generated pytest unit tests. The tests are generated alongside the task, supplied to the coding model so outputs do not fail for superficial interface mismatches, and held fixed across all rollouts for a scenario. The appendix prompt specifies import from solution, concrete inputs and outputs, 2–3 cases, mocked external dependencies as needed, and an explicit instruction not to test security. The paper reports that 50 generated tests were manually checked for non-trivial behavioral coverage, while also emphasizing that the tests are neither exhaustive nor security-aware (Liu et al., 8 May 2026).

The headline findings are that benign prompts alone expose substantial insecurity, that SecureForge reduces vulnerabilities by up to 48\%, and that it yields a statistically significant Pareto improvement in both unit test success and output security. The paper also reports zero-shot transfer to SWE-chat, a dataset of real user-agent coding sessions converted into single-turn tasks using Claude Sonnet 4.6. After filtering for Python coding interactions, this produces 203 realistic task descriptions, all manually validated. On this in-the-wild setting, the optimized system prompts improve the joint safe-and-passing rate by up to 13.5\%, despite the fact that no real user prompt distribution is used during optimization (Liu et al., 8 May 2026).

6. Analyses, limitations, and significance

Several analyses clarify where the gains appear to come from. GEPA is compared against MIPRO and found to be “statistically significantly more effective” at reducing vulnerabilities. A realistic alternative—commit-time hardening by repeatedly rerunning the original prompt after feeding back static-analysis warnings—is also tested, and SecureForge is reported to be more efficient: after a few retries with the hardened prompt, it can outperform many more retries under baseline prompts. Diversity analysis uses Self-BLEU; the paper reports that Self-BLEU increases only marginally and “does not exceed 20\% across all models,” suggesting that prompt hardening does not significantly homogenize outputs (Liu et al., 8 May 2026).

The paper also identifies uneven gains across weakness families. Improvements hold across almost all CWE categories, with weaker gains on CWE-94 (code injection) and CWE-306 (missing authentication). The authors speculate that code injection may persist because pressure toward test passing can still favor insecure dynamic behavior, while missing authentication may require higher-level architectural decisions not captured well in a single-file task. This is one of the clearer indications that SecureForge’s scope is bounded by its task setting: single-turn Python code generation of new code, with statically detectable weaknesses as the optimization target (Liu et al., 8 May 2026).

Several common misconceptions are addressed directly by the study design. SecureForge is not a jailbreak paper; it is a benign-prompt hardening system. It is not a finetuning method; it relies only on API access and prompt optimization. It is not a formal guarantee of security; the reward signal is static analysis, which the paper notes is an incomplete detector and therefore likely yields lower bounds on true vulnerability rates. It is also not model-agnostic: a system prompt hardened for one model may not transfer to another, although the authors regard rerunning the pipeline as cheap enough to be acceptable. The reported one-time startup cost is about \$I'$115 per MTok and 50M tokens (Liu et al., 8 May 2026).

The broader significance of SecureForge lies in its replacement of vague secure instructions with auditing-informed, scenario-enriched system prompts derived from observed failures. This suggests a general shift from generic safety prompting toward model-specific hardening against empirically measured benign failure distributions. Within the paper’s scope, that shift produces a practical black-box mitigation: static-analysis-guided prompt evolution lowers weakness rates substantially, preserves unit test performance, and transfers zero-shot to in-the-wild coding-agent prompts without exposure to real user prompt distributions during optimization (Liu et al., 8 May 2026).