Gabliteration: Adaptive Multi-Directional Neural Weight Modification for Selective Behavioral Alteration in Large Language Models (2512.18901v1)

Abstract: We present Gabliteration, a novel neural weight modification technique that advances beyond traditional abliteration methods by implementing adaptive multi-directional projections with regularized layer selection. Our approach addresses the fundamental limitation of existing methods that compromise model quality while attempting to modify specific behavioral patterns. Through dynamic layer optimization, regularized projection matrices, and adaptive scaling mechanisms, we achieve theoretically superior weight modification while minimizing quality degradation in unrelated domains. We validate our method through the gabliterated-v1 model series (0.6B to 4B parameters) available on Hugging Face, demonstrating practical applicability across multiple model scales.

• The paper introduces Gabliteration, an adaptive framework applying multi-directional SVD-based weight projections to selectively alter LLM behaviors.
• It employs dynamic layer selection and ridge-regularized projections to mitigate undesired responses while maintaining overall task performance.
• Empirical results demonstrate a reduction in refusal rates (Δρ ≈ -0.87) with less than 1.2% MMLU performance loss across multiple LLM architectures.

Gabliteration: Adaptive Multi-Directional Neural Weight Modification for Selective Behavioral Alteration in LLMs

Introduction and Motivation

The paper presents Gabliteration, an extension of the abliteration paradigm for behavioral modification in LLMs. Unlike prior single-direction interventions, Gabliteration employs regularized, multi-directional weight projections informed by data-driven layer selection and adaptive scaling. This framework is explicitly designed to mitigate the characteristic tradeoff of traditional abliteration: while targeted weight modifications can suppress undesired behaviors (e.g., refusal), they often degrade task performance outside the targeted behavioral domain. Gabliteration systematizes layer impact assessment, applies data-guided multi-directional SVD, and utilizes partial, regularized projections with adaptive per-layer scaling to reconcile behavioral elimination with minimal collateral performance loss.

Gabliteration Framework: Methodological Advances

Dynamic Layer Selection

Layer selection is driven by a separation metric between “harmful” and “harmless” prompt activations. For each layer $\ell$, the $L^2$-distance $S_{\ell} = \|\mu_h^{(\ell)} - \mu_n^{(\ell)}\|_2$ between the class means is computed, and the most separable layers are chosen as candidates for weight modification. Theoretically, the separability-maximizing subset minimizes the expected loss relative to the targeted behavior, provided per-layer contributions are monotonic in $S_\ell$.

Multi-Directional SVD-Based Extraction

Whereas prior work such as “Refusal in LLMs Is Mediated by a Single Direction” (Arditi et al., 17 Jun 2024) employs a single vector, Gabliteration generalizes to a $k$-dimensional refusal subspace, extracted via SVD on paired differences of harmful/harmless activations. Empirically, SVD-based extraction achieves refusal reduction on par with Fisher LDA but at lower computational cost and greater stability, and outperforms logistic probe-based extraction regarding both generalization and overfitting.

Ridge-Regularized Projections

To circumvent instability and rank-deficiency issues inherent in high-dimensional subspace projections, Gabliteration utilizes a ridge-regularized projection $$\mathbf{P} = \mathbf{R}(\mathbf{R}^T \mathbf{R} + \lambda I_k)^{-1}\mathbf{R}^T$$ (with $\mathbf{R}$ the $d \times k$ matrix of SVD right vectors). The regularization parameter $\lambda$ ensures numerically stable inversion regardless of $d$ and $k$. Theoretical error bounds show the regularized projection closely approximates an orthogonal projection for practical $\lambda$, with the approximation error controlled at $O(\lambda/\sigma_{\min}^2)$.

Adaptive Layerwise Scaling

Scaling factors $\alpha_\ell$ are modulated across layers via an affine function of normalized layer index, ensuring maximal modification in high-separability (typically central) layers while reducing boundary perturbations that impact input/output representations. This adaptive scaling yields significantly improved behavioral modification efficacy with negligible general-domain performance impact.

Single-Pass Modification

Unlike iterative or reinforcement-based interventions, Gabliteration makes one weight update per layer, providing a direct upper bound on the modification magnitude and simplifying theoretical analysis of performance preservation.

Theoretical Guarantees

Performance Preservation Bounds

Let $\mathcal{T}$ be the downstream-task subspace and $\mathcal{R}$ the extracted refusal subspace. The key result is a cosine-based performance preservation bound: post-modification, the task-relevant weights satisfy

$$\|\mathbf{W}_\mathcal{T} - \mathbf{W}_\mathcal{T}'\|_F \leq \epsilon_\ell \cos\theta \cdot \frac{\|\mathbf{R}\|_F^2}{\|\mathbf{R}\|_F^2 + \lambda}$$

where $\theta$ is the principal angle between the task and refusal subspaces. Notably, if these are nearly orthogonal ($\cos\theta \ll 1$), task performance is nearly preserved.

Regularization and Conditioning

Ridge regularization strictly limits the condition number of the Gram matrix to $$\kappa(\mathbf{R}^T\mathbf{R} + \lambda I) \leq (\sigma_1^2 + \lambda)/\lambda$$. This ensures stability and avoids the catastrophic performance collapse observed with exact orthogonalization-based methods.

Statistical Robustness

The SVD-based pairing is robust in high dimensions under standard sample concentration inequalities. Refusal and performance benchmarks in the ablation studies are statistically significant ($p < 0.001$ across runs), corroborating methodological robustness.

Empirical Results

Gabliteration is validated across multiple LLM architectures (Qwen2.5, Qwen3, Llama3, GPT-oss, sizes ranging 0.6B-32B). The technique achieves strong reduction in model refusal rates ($\Delta\rho \approx -0.87$ with $k=2,\alpha=0.3,\lambda=0.1$), with less than 1.2% mean MMLU performance loss—an order of magnitude improvement over unregularized orthogonalization baselines, which can incur >6% accuracy degradation. Adaptive scaling outperforms uniform scaling by 12–18% in refusal suppression at fixed performance cost.

Ablation studies show that SVD-based pairing achieves near-optimal behavioral suppression while maintaining computational advantages over Fisher LDA and logistic probe alternatives. Regularized projection methods prevent the performance collapse associated with bruteforce orthogonalization in the presence of subspace overlap.

Practical Implications

Gabliteration offers a practical, efficiently computable, and easily implemented mechanism for precise, minimal-impact behavioral interventions in large-scale transformer models. By regularizing over both directionality ($k$) and projection ($\lambda, \alpha$), it is robust against both subspace misalignment and ill-conditioning, even in multi-billion parameter models. This structural framework is especially important for practitioners seeking to repair or neuter specific forms of undesirable behavior (refusal, bias, jailbreak triggers) without sacrificing task capabilities.

The methodology is general and easily extensible. Core architectural choices (SVD vs. LDA, adaptive scaling, layer selection) are modular, and the implementation is straightforward for use with HuggingFace-Transformers-style models. The released model weights (JOSIEfied-Gabliterated and others) empirically demonstrate the scalability and durability of the approach.

Limitations and Future Directions

Identified issues include: computational cost for extremely large models, hyperparameter sensitivity, current focus on pure text (not multimodal), and lack of iterative refinement. The SVD-based extraction does not exploit within-class variance (cf. LDA), nor does it learn discriminative boundaries as with probes or adversarial techniques. Theoretical bounds are currently tight only in the small-$\lambda$ regime; unifying the approximate and exact projection analyses remains open.

Future work directions include: automated hyperparameter selection, iterative adaptive modifications, extension to other modalities (e.g., vision-language transformers), tighter theoretical understanding of regularization effects, and large-scale benchmarking against RLACE, activation steering, and other ablation/steering methods. Investigation into advanced discriminative extraction (CCA, adversarial, nonlinear) is expected to yield further improvement for heterogeneous datasets.

Conclusion

Gabliteration demonstrates that adaptive, regularized, multi-directional weight projection with data-driven layer control can reliably produce targeted behavioral modifications in LLMs while suppressing task-interfering side-effects. This architecture-agnostic intervention approach provides a theoretically grounded, empirically validated, and computationally efficient alternative to earlier abliteration and orthogonalization techniques. The methodology is broadly applicable as a practical tool for model alignment, safety testing, or behavioral research in neural sequence models.

Reference: "Gabliteration: Adaptive Multi-Directional Neural Weight Modification for Selective Behavioral Alteration in LLMs" (2512.18901)