Eliciting Harmful Capabilities by Fine-Tuning On Safeguarded Outputs

Abstract: Model developers implement safeguards in frontier models to prevent misuse, for example, by employing classifiers to filter dangerous outputs. In this work, we demonstrate that even robustly safeguarded models can be used to elicit harmful capabilities in open-source models through elicitation attacks. Our elicitation attacks consist of three stages: (i) constructing prompts in adjacent domains to a target harmful task that do not request dangerous information; (ii) obtaining responses to these prompts from safeguarded frontier models; (iii) fine-tuning open-source models on these prompt-output pairs. Since the requested prompts cannot be used to directly cause harm, they are not refused by frontier model safeguards. We evaluate these elicitation attacks within the domain of hazardous chemical synthesis and processing, and demonstrate that our attacks recover approximately 40% of the capability gap between the base open-source model and an unrestricted frontier model. We then show that the efficacy of elicitation attacks scales with the capability of the frontier model and the amount of generated fine-tuning data. Our work demonstrates the challenge of mitigating ecosystem level risks with output-level safeguards.

• The paper demonstrates that fine-tuning open-source LLMs on benign outputs can recover up to 40% of harmful capability despite conventional safeguards.
• It introduces an anchored comparison evaluation that more accurately detects technical errors than traditional rubric grading, achieving ~88% expert agreement.
• The findings highlight that as frontier models improve, the risk of transferring dangerous capabilities increases, urging novel defense strategies beyond output filtering.

Elicitation Attacks: Transferring Harmful Capabilities via Safeguarded Outputs

Introduction and Threat Model

This work introduces and systematically analyzes elicitation attacks, a category of attacks in which adversaries fine-tune open-source LLMs using ostensibly benign outputs from strongly safeguarded frontier models to recover dangerous capabilities, specifically in the domain of chemical weapon synthesis and processing. The attack pipeline bypasses typical output-level safeguards by (1) generating prompts in domains adjacent to harmful use cases, (2) collecting safeguarded model responses, and (3) fine-tuning open-source models on these benign prompt-response pairs.

Ecosystem-level risk is central: while frontier models are subject to robust deployment safeguards, attackers can leverage their benign outputs to circumvent intended safety controls, resulting in open-source models that demonstrate substantially improved performance on conventionally “refused” tasks. This attack paradigm, unlike task decomposition or inference-time model chaining, facilitates dangerous capability transfer directly into open-source models, enabling offline, unmonitored use.

Figure 1: Schematic of the elicitation attack. Prompts in benign adjacent domains are run on a safeguarded model to generate outputs, which are then used to fine-tune an open-source model; the fine-tuned open-source model partially recovers dangerous capabilities.

Evaluation Methodology: Anchored Comparison vs. Rubric

Classical evaluation via rubric grading—scoring outputs by counting technical keywords—was found to insufficiently penalize critical but subtle procedural mistakes, leading to inflated apparent uplift. A new anchored comparison evaluation is introduced: judge LLMs (typically jailbroken Gemini 2.5 Pro) compare tested outputs against high-quality anchor responses across subgoals, accounting for not just keyword presence but logical ordering, technical accuracy, and the avoidance of catastrophic errors.

Figure 2: Demonstration of the difference between anchored comparison (left) and rubric (right) evaluations; only anchored comparisons reliably penalize critical errors outside rubric coverage.

Validation via human expert labeling demonstrates that anchored comparisons align more closely with expert judgment (~88% agreement) than rubrics (~75% agreement). Furthermore, the anchored comparison approach is approximately five times more sensitive to introduced mistakes than rubric grading.

Empirical Assessment of Elicitation Attacks

The core empirical claim is that elicitation attacks can recover up to 40% of the performance gap between baseline abliterated open-source models and unrestricted, jailbroken frontier models on eight chemical weapons tasks—an average performance gap recovered (APGR) of ~39% for Llama 3.3 70B using anchored comparisons, and markedly higher values under rubric evaluation due to the previously mentioned limitations of that metric.

Baseline methods using only open-source model self-generated data or textbook-derived data exhibit negligible or negative uplift, specifically implicating the unique value of benign outputs from strong models.

Figure 3: Across various weak models and evaluation methods, elicitation attacks using safeguarded frontier outputs effect the largest increase in harmful capability as measured by APGR.

Strong numerical results support the scalability of elicitation attacks with respect to both frontier model capability and size of the fine-tuning dataset. Successive releases of Anthropic and OpenAI models progressively enhance the ability to recover harmful capability in the same open-source target, indicating defenders should account for temporal increases in ecosystem risk.

Figure 4: (Left) As frontier models improve, APGR from elicitation attacks increases. (Right) Attack efficacy continues to scale with the number of benign training samples, saturating only for select tasks.

A breakdown by task and subgoal demonstrates that some facets (e.g., synthesis tasks) are less susceptible to this attack, while others see uplift approaching or exceeding 100% relative to the strongest anchor.

Figure 5: Detailed per-task APGR for Llama 3.3 70B fine-tuned on safeguarded Claude 3.5 Sonnet outputs.

Failure Modes of Safeguards and Domain Dependence

The examination of a classifier-guarded frontier (constitutionally filtered) system reveals that even aggressive refusals do not reliably prevent the collection of sufficiently rich benign data to enable uplift. Domains such as soap or cheese production, which appear benign, can still leak transferable knowledge.

Transfer experiments reveal rapid dropoff in attack efficacy as the domain of the benign outputs diverges from the target harmful domain. Notably, fine-tuning on unrelated science or inorganic chemistry yields less than 12% uplift; organic chemistry domains provide up to ~34% APGR.

Contrastively, training on harmful data directly achieves 50.9% APGR versus 33.7% for benign chemical synthesis (a one-third reduction), but this margin is eclipsed by improvements from more powerful frontier models (e.g., Claude 4 Opus benign data achieves 71.1% APGR against a Claude 3.5 Sonnet upper bound).

Qualitative Analysis and Validator Consistency

A thorough analysis determined that the anchored comparison methodology is both internally consistent across repeated runs and robustly identifies introduced technical mistakes, especially in key subgoals associated with chemical procedural logic. Human expert comparison further validates the method, with judges rating the LLM-based evaluations as recognizably accurate and useful in the majority of cases.

Figure 6: Anchored comparison self-consistency for Gemini 2.5 Pro and Llama 4 Maverick evaluators across repeated resamples.

Figure 7: Gemini 2.5 Pro outperforms other judge models and rubric evaluation in recognizing deliberate, technically subtle errors introduced into model completions.

Implications and Recommendations

Elicitation attacks present a structural challenge to output-level safeguard paradigms. Since only harmless inputs and outputs are ever observed, adversaries can exploit scientific knowledge embedded in protected models without tripping detection. Practical mitigation would require either suppressing entire scientific domains or restricting access to strong frontier models at the API/data level—both of which have severe usability implications.

As the state-of-the-art frontier model capabilities advance, the ability to transfer harmful procedural or scientific competence will increase at a rate outpacing the margin gained by output filtering and refusal strategies.

Future Outlook

1. Enhanced Attacks: By optimizing prompt distributions, leveraging better response curation, or adopting RL methods for demonstration selection, elicitation attacks could yield even closer parity with unrestricted frontier models.
2. Defense Directions: Mitigations may require vetting scientific outputs, dynamic necessity-of-use review (analogous to “Know Your Customer”), adversarial capability audits before model release, or fundamentally new model design paradigms beyond output-based filtering.
3. Transfer Limitation via Domain Restriction: Domain-targeted capability suppression may help, but adversaries could develop pretext tasks that bridge the gap between benign and malicious end-use, as suggested by strong scaling effects in dataset quantity and frontier model strength.

Conclusion

The study demonstrates that frontier model safeguards focused on output refusal and filtering are not sufficient to prevent ecosystem-level risks. Elicitation attacks exploit the presence of powerful, widely accessible models to transfer restricted knowledge into open-source systems via fine-tuning on only benign procedural or adjacent scientific data. As such, deployment and regulatory frameworks that assume per-model safety, rather than ecosystem interaction, risk underestimating real-world risk profiles (2601.13528).