CodeSpear: Exploiting GCD for Jailbreaks

• CodeSpear is a jailbreak attack that exploits grammar-constrained decoding to force LLMs to generate malicious code by excluding natural-language refusals.
• It restricts the model’s output to a programming grammar, significantly boosting attack success rates over conventional, unconstrained decoding methods.
• Paired with the defense CodeShield, the approach highlights the interplay between structured output constraints and safety alignment, enabling safer code modality.

Searching arXiv for CodeSpear and closely related work on grammar-constrained decoding jailbreaks. CodeSpear is a jailbreak attack on LLMs for code generation that exploits grammar-constrained decoding (GCD) to induce the production of malicious code. Rather than relying on adversarial prompt engineering or model access, the attack operates by constraining decoding with an ordinary programming-language grammar, thereby excluding the model’s usual natural-language refusal behavior from the output space and forcing generation in the code modality (Zhang et al., 10 Jun 2026). In the same line of work, CodeSpear is paired with a defense, CodeShield, which aligns models to emit semantically harmless yet structurally diverse “honeypot code” under GCD while preserving ordinary natural-language refusals when unconstrained decoding is available (Zhang et al., 10 Jun 2026).

1. Definition and threat model

CodeSpear is introduced as a decoding-time jailbreak for safety-aligned LLMs used in code generation (Zhang et al., 10 Jun 2026). Its central claim is that GCD, a mechanism widely adopted to improve syntactic validity and structured-output reliability, can itself create a safety vulnerability. The attack requires only that an interface support grammar-constrained or structured decoding and allow the attacker to specify a benign programming-language grammar such as Python; it does not require gradient access, parameter modification, fine-tuning access, or adversarial grammar synthesis (Zhang et al., 10 Jun 2026).

The threat model is intentionally narrow and practical. The attacker submits a malicious code-generation prompt to a model that is ordinarily safety aligned and constrains decoding with a standard code grammar. The relevant deployment settings include local inference stacks such as vLLM and SGLang as well as API settings exposing structured outputs, including OpenAI and Fireworks AI interfaces (Zhang et al., 10 Jun 2026). In this setting, success means that the model emits code that partially or fully implements the malicious request rather than refusing or responding harmlessly (Zhang et al., 10 Jun 2026).

A common misconception is that jailbreaks of this kind must use adversarially engineered grammars. The reported attack does not depend on such a construction. The paper’s core finding is that a benign off-the-shelf code grammar is already sufficient to suppress natural-language refusals and substantially increase malicious code generation rates (Zhang et al., 10 Jun 2026).

2. Mechanism: output-space restriction as jailbreak surface

The attack is grounded in a formal account of constrained decoding. Unconstrained autoregressive generation is written as

$$y_t \sim P_M(\cdot \mid p, y_{<t}), \quad y_{<t} = (y_1, \ldots, y_{t-1}).$$

Under a grammar $G$, the set of valid next tokens becomes

$$\mathcal{V}_G(y_{<t}) = \left\{\, v \in \mathcal{V} \;\middle|\; \exists\, y_{>t} \text{ s.t. } (y_{<t}, v, y_{>t}) \in \mathcal{L}(G) \,\right\},$$

and decoding is renormalized over that set:

$$P_M^{G}(y_t \mid p, y_{<t}) = \frac{ P_M(y_t \mid p, y_{<t}) \cdot \mathbb{I}[y_t \in \mathcal{V}_G(y_{<t})] }{ \sum_{v \in \mathcal{V}_G(y_{<t})} P_M(v \mid p, y_{<t}) }.$$

This yields

$$P_M^{G}(y \mid p) = \mathbb{I}[y \in \mathcal{L}(G)] \prod_{t=1}^{|y|} P_M^{G}(y_t \mid p, y_{<t})$$

and therefore changes the support of generation from the full token language to the grammar language $\mathcal{L}(G)$ (Zhang et al., 10 Jun 2026).

The paper’s central insight is that contemporary harmful-code alignment is largely learned in the natural-language modality. Safety-aligned models are trained to produce refusals such as “I cannot assist with that,” but these strings are usually not elements of a code grammar. The paper states this as

$$\mathcal{R}_{\text{refuse}} \cap \mathcal{L}(G) = \emptyset, \qquad \Pr_{y \sim P_M^{G}(\cdot \mid p)} \left[y \in \mathcal{R}_{\text{refuse}}\right] = 0 .$$

In other words, GCD can delete the model’s preferred safe behavior from the admissible output space (Zhang et al., 10 Jun 2026).

This suggests that CodeSpear is not best understood as altering model preferences directly. A more precise characterization is that it performs support manipulation: the attack makes the refusal channel unavailable, thereby forcing the model to continue in code space, where its learned competence may dominate and its safety alignment may be weaker (Zhang et al., 10 Jun 2026).

3. Attack procedure

Operationally, CodeSpear is deliberately simple. Given a model $M$, malicious prompt $p$, benign code grammar $G$, and maximum generation length $G$0, the attack repeatedly computes the valid token set under the grammar, samples from the constrained next-token distribution, appends the sampled token to the prefix, and stops at end-of-sequence (Zhang et al., 10 Jun 2026). The paper describes this as querying

$G$1

The assumptions required by the procedure are modest:

• Prompt access: the attacker can submit harmful code-generation requests.
• Constraint access: the interface supports grammar-constrained decoding.
• Grammar choice: the attacker can supply an ordinary programming-language grammar.

No internal model state is required, and no adversarial search over grammars is necessary (Zhang et al., 10 Jun 2026). This distinguishes CodeSpear from prior structured-output jailbreaks such as APT, which rely on crafting request-specific adversarial grammars (Zhang et al., 10 Jun 2026).

The implementation reported in the paper uses llguidance with off-the-shelf Python grammars. Across attacks, the experiments use temperature $G$2 and top-$G$3 $G$4, repeat runs three times, and report averages (Zhang et al., 10 Jun 2026).

4. Empirical evaluation

The evaluation spans 10 models across 4 benchmarks (Zhang et al., 10 Jun 2026). The local models are Qwen2.5-Coder-7B, Qwen2.5-Coder-32B, Qwen2.5-7B, Qwen2.5-32B, and LLaMA3-8B. The API-based models are GPT-5, GPT-5-mini, GPT-OSS-120B, MiniMax-M2.5, and MiniMax-M2.7 (Zhang et al., 10 Jun 2026). Safety evaluation uses the code-generation subset of RMCBench, specifically Level 1 and Level 2 with 182 malicious requests, and the original malicious subset of MalwareBench with 320 malicious requests (Zhang et al., 10 Jun 2026). Utility evaluation uses HumanEval with 164 tasks and MBPP with 974 tasks (Zhang et al., 10 Jun 2026).

The safety metrics are Attack Success Rate (ASR), defined as the fraction of malicious requests where the response is harmful, and Malicious Rate (MR), defined as the fraction where the generated code functionally realizes the malicious intent. Both are judged by DeepSeek-V4-Flash. A reliability check on 100 manually inspected samples reports human–LLM agreement of 87% for ASR and 85% for MR (Zhang et al., 10 Jun 2026).

Local-model results

For local models, CodeSpear reaches 81.82% ASR and 54.58% MR on average, compared with 54.92% ASR and 36.61% MR for Vanilla. This corresponds to gains of +26.90 percentage points ASR and +17.98 points MR (Zhang et al., 10 Jun 2026). The strongest baseline is CodeJailbreaker at 71.25% ASR and 48.45% MR, still below CodeSpear (Zhang et al., 10 Jun 2026).

Representative reported cases include the following.

Model and benchmark	Vanilla ASR/MR	CodeSpear ASR/MR
Qwen2.5-Coder-7B, RMCBench	26.92 / 23.26	82.78 / 62.09
Qwen2.5-Coder-32B, RMCBench	59.89 / 45.60	92.86 / 75.64
Qwen2.5-Coder-7B, MalwareBench	29.79 / 20.83	83.44 / 46.15
Qwen2.5-32B, MalwareBench	53.23 / 33.33	84.69 / 54.58

These outcomes indicate that the attack is not confined to a single architecture family or safety-training regime (Zhang et al., 10 Jun 2026).

API-model results

For API-based models, CodeSpear reaches 67.39% ASR and 36.62% MR on average, compared with 22.00% ASR and 11.04% MR for Vanilla. This is a gain of +45.39 points ASR and +25.59 points MR (Zhang et al., 10 Jun 2026). The best baseline average is CodeJailbreaker at 39.95% ASR and 20.40% MR, still substantially below CodeSpear (Zhang et al., 10 Jun 2026).

Representative cases include GPT-5 on RMCBench, where Vanilla scores 32.05/26.01 and CodeSpear 55.49/43.96, and MiniMax-M2.7 on RMCBench, where Vanilla scores 20.33/17.40 and CodeSpear 85.53/64.29 (Zhang et al., 10 Jun 2026). The strength of the results on API systems is notable because these platforms are generally expected to have stronger alignment and deployment-time safeguards.

5. Baselines, ablations, and behavioral interpretation

The evaluation compares CodeSpear with Vanilla, Vanilla-T, DAN, LRL, PAIR, CodeJailbreaker, and APT (Zhang et al., 10 Jun 2026). The paper explicitly excludes gradient-based attacks that do not fit the adopted threat model (Zhang et al., 10 Jun 2026). Across the reported averages, CodeSpear outperforms these representative jailbreak baselines (Zhang et al., 10 Jun 2026).

A particularly revealing analysis concerns fixed safe code patterns. On GPT-5 and GPT-5-mini, the model under GCD often emits a trivial pass statement in response to malicious prompts. While superficially safe, that pattern is brittle. When the grammar is tightened to disallow pass, attack success increases further. For GPT-5 on RMCBench, deepred/CodeSpear rises from 55.49/43.96 to 70.30/53.65; for GPT-5-mini on RMCBench, from 53.48/39.56 to 63.75/44.58 (Zhang et al., 10 Jun 2026). Similar increases are reported on MalwareBench (Zhang et al., 10 Jun 2026).

This matters for two reasons. First, it shows that code-modality “safety” anchored to a narrow stub such as pass is not robust. Second, it clarifies that CodeSpear exploits a structural weakness in the safety policy rather than merely surfacing preexisting unsafe responses. The attack remains effective even when the model attempts a code-form safe fallback, because small grammar adjustments can remove that fallback from the output space (Zhang et al., 10 Jun 2026).

The paper also reports that the attack remains strong across Python, C++, and Java grammars. On Qwen2.5-Coder-7B, ASR is said to remain above 70% once GCD is applied, regardless of grammar, while remaining below 40% without GCD (Zhang et al., 10 Jun 2026). This suggests that the vulnerability is not tied to idiosyncrasies of a single grammar formalism.

6. CodeShield: code-modality alignment defense

The defense paired with CodeSpear is CodeShield, named deepgreen in the body of the paper (Zhang et al., 10 Jun 2026). Its premise is that if natural-language refusals are suppressed by GCD, the model must still retain safe behaviors inside the code modality itself. CodeShield therefore aligns models to emit “honeypot code” under constrained decoding (Zhang et al., 10 Jun 2026).

The paper defines honeypot code by two properties:

• Semantically harmless: it does not implement the malicious request.
• Structurally diverse: it varies enough that an attacker cannot easily eliminate it through tighter grammars without also constraining away much of the expressive space needed for malicious programs.

This design explicitly rejects fixed-template safe stubs such as pass or canned refusal comments (Zhang et al., 10 Jun 2026).

The intended preference hierarchy is modality dependent. Under unconstrained decoding, natural-language refusal should be preferred over honeypot code. Under constrained decoding, honeypot code should be preferred over harmful code (Zhang et al., 10 Jun 2026). Training uses a DPO-based objective over pairwise preferences. The paper prints the standard DPO form

$G$5

with the implicit reward defined against a reference model, though the TeX rendering is partially corrupted in the supplied text (Zhang et al., 10 Jun 2026).

The preference dataset is constructed from malicious prompts by pairing each refusal with multiple honeypot code samples and pairing each honeypot code sample with the harmful response generated under GCD (Zhang et al., 10 Jun 2026). The data pipeline filters PKU-RLHF with Qwen3-32B to obtain 744 seed malicious code-generation prompts, augments them with DeepSeek-V4-Pro to 2,000 total malicious prompts, generates natural-language refusals with DeepSeek-V4-Pro, obtains harmful responses by querying the target model under GCD, and samples honeypot snippets from OpenCodeInstruct (Zhang et al., 10 Jun 2026). Training uses $G$6 honeypot samples, learning rate $G$7, one epoch, and an additional 40k general code-generation tasks from OpenCodeInstruct for utility preservation (Zhang et al., 10 Jun 2026).

Defense results are strong. On Qwen2.5-Coder-7B under attack, Vanilla averages 83.11/54.12 and Safe-DPO 77.39/45.03, whereas deepgreen/CodeShield reduces this to 5.57/2.78 (Zhang et al., 10 Jun 2026). Similar reductions are reported for Qwen2.5-7B and LLaMA3-8B (Zhang et al., 10 Jun 2026). The comparison with Safe-DPO is especially significant because it isolates the need for code-modality alignment: natural-language refusal training alone does not withstand CodeSpear once decoding is forced into code (Zhang et al., 10 Jun 2026).

Utility degradation is reported as modest. On Qwen2.5-Coder-7B, average HumanEval/MBPP pass@1 and pass@3 change from 68.60/83.51 for Vanilla to 66.94/80.88 for deepgreen (Zhang et al., 10 Jun 2026). On Qwen2.5-7B, deepgreen even improves the reported average over Vanilla, from 49.29/68.46 to 51.53/71.74 (Zhang et al., 10 Jun 2026). A plausible implication is that utility-preservation SFT contributes materially to stabilizing general code performance.

7. Significance, limitations, and relation to adjacent code-generation work

The significance of CodeSpear lies in identifying GCD as a security-relevant intervention rather than a purely reliability-oriented one (Zhang et al., 10 Jun 2026). Prior repository-scale code generation work such as CodeS decomposes generation into sketches to improve structural coherence (Zan et al., 2024), and lexical retrieval work such as SpareCodeSearch argues that lightweight keyword-based repository context retrieval can be sufficient for practical completion systems (Nguyen, 14 Oct 2025). Those systems address capability, structure, and deployment efficiency. CodeSpear addresses a distinct axis: the interaction between output constraints and safety alignment. This suggests that advances in structured decoding and repository-aware generation do not automatically transfer to safe deployment under adversarial prompting.

The paper is careful about scope. It notes that absolute attack success rates may vary across GCD engines and APIs, and that the benchmark suite, while broad, does not cover every malicious coding scenario (Zhang et al., 10 Jun 2026). It also reports ethical restrictions: CodeSpear source is released only to researchers for controlled use, whereas CodeShield is fully released (Zhang et al., 10 Jun 2026).

Several limitations remain. First, the vulnerability analysis is strongest for models whose safety policy is predominantly implemented as natural-language refusal. This suggests a research direction toward genuinely cross-modality safety policies rather than modality-local refusals. Second, the defense depends on the availability and quality of structurally diverse harmless code. Third, adaptive grammar tightening can increase ASR in some defended settings, even though the paper reports no catastrophic collapse of CodeShield under a 10-round adaptive attack (Zhang et al., 10 Jun 2026).

A further misconception is that GCD itself is inherently unsafe. The paper’s actual claim is narrower: GCD becomes an attack surface when alignment is modality specific and the constrained language excludes the model’s learned refusal behavior (Zhang et al., 10 Jun 2026). This suggests that the vulnerability is not a property of syntax constraints per se but of the interaction between constraint-induced support shifts and alignment objectives.

In summary, CodeSpear establishes that grammar-constrained decoding can jailbreak safety-aligned code LLMs by removing natural-language refusal from the reachable output space and forcing continuation in code. Its empirical results show strong improvements over prompt-based and structured-output jailbreak baselines across local and API models, while the accompanying CodeShield defense demonstrates that aligning safe behavior directly in the code modality can largely restore robustness under constrained decoding (Zhang et al., 10 Jun 2026).