Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysis

Abstract: LLM-assisted software security operates at a difficult boundary: the vulnerability-analysis terminology needed for legitimate code review, triage, and repair can closely resemble terminology associated with misuse. Existing safety and cybersecurity evaluations are difficult to interpret in this setting because they often compare unrelated model families, thereby conflating safety behavior with differences in architecture, scale, training data, and deployment. To isolate this factor, we study safety state: whether refusal behavior remains intact (Aligned) or has been refusal-ablated (Abliterated) within same-lineage models. We ask how this safety state affects defensive utility across software-security workflows. We compare aligned instruction-tuned models with publicly released refusal-ablated descendants from two model families, Gemma and Qwen. We evaluate Aligned and Abliterated states on vulnerability detection, CWE attribution, vulnerable-line localization, root-cause localization, and executable patch validation. We further treat prompt wording as a controlled framing dimension: prompts begin with neutral code-review language, add authorization context, and vary the density of cybersecurity terminology. In a Gemma-based Java/Vul4J repair-validation study, Abliterated achieves higher early-stage validation rates, with 67.8%, 65.0%, and 32.8% of patches judged usable, successfully applied, and successfully compiled, respectively, compared with 29.9%, 24.9%, and 9.0% for Aligned. In the Qwen pair, Abliterated improves localization performance, increasing line-level F1 from 2.08% to 3.91% and Top-1 accuracy from 4.10% to 6.95%. These findings suggest that evaluations of LLM-based security assistants should jointly measure whether models respond, whether their usable responses are correct, and whether their outputs remain actionable across the engineering workflow.

• The paper demonstrates that safety-state modifications significantly alter LLM performance on vulnerability detection and patch validation.
• It introduces a same-lineage experimental design comparing aligned and abliterated models across various task depths and security prompt framings.
• Results reveal that abliterated models yield higher actionable outputs under explicit security cues, highlighting trade-offs with diagnostic accuracy.

Beyond Refusal: A Same-Lineage Analysis of Aligned and Abliterated LLMs for Vulnerability Analysis

Introduction

The paper "Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysis" (2607.05842) presents a systematic investigation of how safety-state interventions (alignment versus refusal ablation) impact the practical utility of LLMs for legitimate software vulnerability analysis. The motivation is drawn from the observation that much of the existing discussion and benchmarking on LLM safety in cybersecurity conflate model architecture, data, and safety layers, and mostly only measure outright refusal events. This work employs a rigorous same-lineage setup, comparing models that differ solely in safety-state but are otherwise matched in architecture and training history, in order to disambiguate the origin of observed behavioral differences. The study probes a task-depth spectrum from detection and labeling to executable patch validation and considers not only outcomes of refusal but fine-grained differences in actionable utility under manipulated prompt conditions.

Experimental Design and Methodology

The primary methodological innovation is the strict control of experimental confounders via a same-lineage model comparison. The Aligned models are instruction-tuned to standard alignment baselines, while the Abliterated variants undergo explicit vector surgery to remove directions in activation or weight space mediating refusal (leveraging techniques from [arditi2024refusal, wang2024surgical]). This design enables precise attribution of behavioral differences to safety-state. Experiments are conducted on two model families: Gemma (gemma-4-31B-it versus gemma-4-31B-CRACK) and Qwen (Qwen3.6-27B-MXFP4 vs. Qwen3.6-27B-MXFP4-CRACK).

Evaluator prompts are systematically constructed by crossing task depth (from vulnerability detection and CWE attribution through vulnerable-line and root-cause localization to repair with executable validation) with prompt framing. Framing manipulations include neutral code review language, explicit authorization context, and increased density of professional security/cyber terminology.

Figure 1: Study design crossing task depth and prompt framing; and the patch-validation funnel delineating sub-gates in the repair process.

The analysis decomposes utility into: (1) Coverage (model answers at all and is parseable), (2) Quality (task-specific correctness conditional on answer), and (3) End-to-end utility (the actionable outcome rate across the validation pipeline). Multiple behavioral endpoints are probed, including output stability under prompt perturbations.

Results

Task-Depth Effects and Refusal is Not the Full Story

A central claim demonstrated by the data is that the practical, defender-side cost of safety alignment is non-monotonic and context-dependent. Simple focus on refusal rate is inadequate: both Aligned and Abliterated states answer most legitimate requests, but differ strongly in actionability, localization quality, and patch validation rates. The relative advantage flips across task depths and with prompt context.

Vulnerability Detection and Attribution

On high-level detection and CWE attribution tasks under neutral code-review framing, Aligned models often show higher correctness. For example, on the Gemma pair, the Aligned model outperforms Abliterated by several points in both binary vulnerability detection and CWE label assignment under neutral language, but this advantage shrinks or reverses with more security-explicit prompts. Qwen displays the reverse pattern under some prompt conditions, highlighting the model-dependence of high-level task results.

Code-Grounded Localization

As tasks become more code-grounded, e.g., identifying vulnerable lines or causal regions, Abliterated models consistently outperform Aligned models on actionable localization metrics. For both the Gemma and Qwen pairs, mean line-level F1 and Top-1 hit rates for vulnerable-line identification favor Abliterated by clear margins across all prompt framings. This pattern holds in both balanced and realistic, scenario-driven datasets.

Root-Cause Localization and Patch Validation

When targeting causal diagnosis and executable repair, nuanced trade-offs emerge. In root-cause localization, Aligned models have marginally higher ranked precision in the neutral frame, but Abliterated achieves higher mean line-F1 in frames using explicit security language.

Figure 2: Root-cause localization accuracy and differential analysis across prompt frames and safety-states.

For executable repair (e.g., patching Java vulnerabilities and validating against project-level proof-of-vulnerability triggers), Abliterated models far surpass Aligned in early-stage executable throughput. In the Vul4J Java benchmark, Abliterated produces patches successfully used, applied, and compiled in 67.8%, 65.0%, and 32.8% of cases, versus 29.9%, 24.9%, and 9.0% for Aligned. However, the ultimate repair success at the strictest validation gate sometimes favors Aligned under neutral prompting. The Abliterated advantage is most pronounced under explicit authorization and cyber terminology.

Figure 3: Patch-validation funnel showing stagewise pass rates, demonstrating Abliterated's early-stage actionability advantage.

The Role of Prompt Framing

The prompt-framing analysis makes explicit that professional (i.e., security-specific) language, scenario terms, and authorization cues materially alter both model utility and output stability, particularly in the Aligned condition. Actionable localization and repair performance for Aligned suffers notably as the lexical intensity (e.g., mention of exploit, attacker, scenario) increases, while Abliterated maintains consistently higher throughput.

Figure 4: Prompt-intensity utility. Increased cyber-related framing narrows or reverses Aligned's advantage in classification; Abliterated dominates for localization.

Prompt-driven output instability is also highlighted: Aligned models, and to a lesser but still significant extent Abliterated, exhibit substantial label and output drift as a function of benign changes in evaluation prompt security wording—even with task, code, and output schema held constant.

Figure 5: Output drift induced by prompt wording: significant shifts in labeling and localization as a function of lexical-level modifications, especially for Aligned.

Implications and Discussion

Theoretical Impact

This study challenges the adequacy of refusal rate as the principal safety metric in cybersecurity LLM deployment, arguing that correct, localized, and actionable outcomes are security-critical properties alongside refusal. Aligned models, while better at coarse detection under neutral settings, are prone to silent degradation in output actionability as prompts are made more representative of actual defender workflows or made to satisfy policy-conscious logging/auditing requirements. The fact that refusals can be surgically ablated without other major model changes emphasizes that safety layers are mechanistically separable, but often interact in complex, prompt-contingent ways with underlying task competence.

Practical Implications

The robust finding that refusal ablation increases actionable output, especially under more security-realistic prompts, has immediate implications for the design and deployment of LLM-based vulnerability analysis tools for defenders. Alignment interventions can unintentionally impair diagnostics and repair, causing models to produce plausible but incorrect or mislocalized recommendations, or to inappropriately modulate their outputs in response to innocuous authorization context. Evaluation pipelines should thus move beyond surface refusal to encompass staged, code-grounded, and patch-validated metrics.

Guardrails and alignment procedures must be further refined to distinguish misuse from benign use in high-stakes dual-use domains. There is a clear need for more granular, context-aware safety controls, potentially leveraging external validation or explainability overlays.

Future Directions

The results suggest several research trajectories: (1) design of safety mechanisms that condition on explicit authorization flows with cryptographic or institutional guarantees; (2) development of mechanistic interpretability and modification tools for safety-related directions that do not negatively impact downstream actionable utility; (3) deeper investigation of prompt engineering and context serialization for security-relevant tasks; (4) extension of these analyses to additional programming substrates and adversarial prompt attacks, and integration in multi-agent and tool-augmented settings.

Conclusion

Through a rigorously controlled, same-lineage analysis of aligned and abliterated LLMs, this work demonstrates that defender-side utility in vulnerability analysis is non-trivially modulated by alignment state. Refusal ablation enhances model actionability, particularly at deeper task depths and under realistic defensive prompt framing, but may sometimes trade-off with coarse diagnostic accuracy in less specialized contexts. Safety and utility must be jointly and contextually monitored—future alignment policy and evaluation regimes should incorporate metrics reflecting not only the absence of refusal or harmful compliance, but the nuanced and actionable needs of legitimate software defenders.