Rank-One Safety Injection (ROSI)

• The paper introduces a novel rank-one update method that robustly increases refusal behavior in LLMs while preserving performance on benign tasks.
• ROSI leverages mechanistic interpretability by extracting a one-dimensional refusal-mediating subspace to compute safety-aligned weight modifications.
• Empirical findings show that ROSI increases harmful-prompt refusal rates by up to 18 points and reduces jailbreak success by over 50% with minimal impact on standard utility.

Rank-One Safety Injection (ROSI) is a white-box, fine-tuning-free method for amplifying safety alignment in LLMs. ROSI operates by permanently steering a model’s internal activations toward the refusal-mediating subspace, leveraging mechanistic interpretability to robustly enforce refusal behavior on harmful instructions. This is achieved with a simple rank-one weight modification applied to all residual stream write matrices, computed from a small set of harmful and harmless instruction pairs. Empirical findings show that ROSI substantially increases refusal rates to harmful requests—resistant to jailbreak attacks—while preserving utility on standard knowledge and reasoning tasks. ROSI can re-align uncensored models, enabling effective last-mile safety procedures, and is exceptionally computationally efficient compared to conventional fine-tuning paradigms (Shairah et al., 28 Aug 2025).

1. Safety Alignment in LLMs and ROSI’s Approach

Safety alignment in LLM deployment requires that models reliably refuse unsafe or disallowed requests, such as instructions for prohibited activities. Conventional mechanisms include supervised fine-tuning and reinforcement learning from human feedback (RLHF), which incur significant resource demands and can be susceptible to narrow fine-tunes that bypass global safety measures. Inference-time steering techniques inject safety features dynamically, but these methods introduce computational overhead and possible interference with model utility.

ROSI occupies a distinct niche: drawing on mechanistic interpretability insights (e.g., Arditi et al., 2024), which revealed refusal behavior is governed by a one-dimensional subspace within the residual stream. Classical jailbreaks ablate this feature, disabling refusal. ROSI inverts this vulnerability, permanently amplifying the refusal-mediating direction in every residual-stream write matrix. This intervention makes models significantly more robust against jailbreak techniques without degrading their knowledge, reasoning, or benign compliance capabilities.

2. Mathematical Formulation of the Rank-One Injection

Refusal-Mediating Subspace

Let $\mathcal{D}_{harm}$ denote a small set of harmful prompts and $\mathcal{D}_{harmless}$ denote benign prompts. For a given layer $\ell$ and last token position $i$ per prompt $t$, gather residual-stream activation vectors $x^{(\ell)}_i(t) \in \mathbb{R}^{d_{model}}$. Compute

$$\mu^{(\ell)} = \frac{1}{|\mathcal{D}_{harm}|} \sum_{t \in \mathcal{D}_{harm}} x^{(\ell)}_i(t), \quad \nu^{(\ell)} = \frac{1}{|\mathcal{D}_{harmless}|} \sum_{t \in \mathcal{D}_{harmless}} x^{(\ell)}_i(t)$$

and define the raw safety direction

$s^{(\ell)} = \mu^{(\ell)} - \nu^{(\ell)}$

Select the layer $\ell^*$ maximizing refusal signal (via validation against known refusal direction or ablation), then normalize: $\hat{s} = s^{(\ell^*)} / \|s^{(\ell^*)}\|_2$.

Rank-One Weight Update Mechanism

In decoder-only Transformers, every attention output projection $W_O^{(\ell)} \in \mathbb{R}^{d_{model} \times d_k}$ and MLP output $$W_{out}^{(\ell)} \in \mathbb{R}^{d_{model} \times d_{mlp}}$$ writes into the residual stream. ROSI modifies each such matrix $W$ by adding a rank-one safety bump:

$\Delta W = \alpha \, u v^T$

with

• $u = \hat{s} \in \mathbb{R}^{d_{model}}$ (unit safety direction),
• $v \in \mathbb{R}^{d_{in}}$, obtained as the mean of $W$’s row vectors,
• $\alpha > 0$ is the injection strength hyperparameter.

Set $v = \frac{1}{d_{in}} \sum_{j=1}^{d_{in}} w_j$, and update $W \leftarrow W + \alpha \hat{s} v^T$. This operation introduces a component along the refusal axis $\hat{s}$ at every residual write.

Representative Pseudo-Code

Input: Model M, harmful prompts D_harm, harmless prompts D_harmless Hyperparams: chosen layer indices l_1...l_K, injection strength α 1. For each layer l in {l_1...l_K}: • Run M forward on D_harm and collect x_i^(l) for last token i • Compute μ^(l) and ν^(l); form s^(l) = μ^(l) − ν^(l) • Evaluate refusal signal strength of s^(l) 2. Select l* maximizing signal; set ŝ = s^(l*) / ‖s^(l*)‖_2 3. For each residual-stream write matrix W in M: • Compute v = mean(rows(W)) • Update W ← W + α ŝ v^T 4. Return edited model M_ROSI

3. Empirical Performance and Refusal Enhancement

ROSI’s injection of the learned safety direction at $\alpha \approx 51 \times 10^{-6}$ yields robust improvements:

• On seven open-source chat and instruction models (0.5B–14B parameters), harmful-prompt refusal rates (measured by Llama Guard 3 on CatQA) increase by up to +18 points, frequently elevating refusal rates to near 100%.
• Adversarial jailbreak success rates on challenging benchmarks (DAN, HarmBench, WildGuardTest, WildJailbreak) drop by over 50%, indicating substantially enhanced resilience.
• Standard utility metrics (MMLU, HellaSwag, ARC Easy/Challenge, BoolQ, TruthfulQA) exhibit minimal shifts (average score change $< \pm 0.5\%$), with benign compliance remaining $>95\%$.
• ROSI does not induce excessive refusal, maintaining appropriate model responsiveness to benign queries.

4. Re-Alignment and Latent Safety Elicitation in Uncensored Models

Uncensored “Dolphin” variants, which have undergone fine-tuning to suppress refusal, present a null refusal direction such that direct difference-of-means extraction of $\hat{s}$ fails. ROSI addresses this by prefixing each prompt during safety direction computation with a “safety system prompt” (e.g., instructing the model to refuse unsafe requests). This primes the activation space to generate a coherent refusal subspace. Subsequent rank-one injection amplifies this direction throughout the model’s weights, restoring robust refusal behavior without requiring continued prompt conditioning at inference time.

Ablating the safety prompt during $\hat{s}$ computation (“162” ablation) markedly diminishes ROSI’s efficacy, underscoring the necessity of this elicitation step for fully uncensored models. Empirically, harmful refusal rates in these models increase by 20–34 points (e.g., from 50% to 86%), with jailbreak success rates sharply reduced and unaffected utility.

5. Computational Efficiency, Model Agnosticism, and Practical Concerns

ROSI’s computational footprint is minimal:

• Extraction of $\hat{s}$ requires only two batches (∼50 harmful, 50 harmless prompts) of forward passes per target layer.
• The weight injection is a single scan over all relevant matrices ($W_O$, $W_{out}$), each receiving a dense rank-one outer product update.
• Application to large models ($\sim$70B parameters) completes within minutes on a single GPU.
• ROSI’s method is model-agnostic as long as the relevant matrices can be identified and edited; $\alpha$ is the sole tunable hyperparameter, validated on a small safety set.

6. Extensions, Limitations, and Research Directions

Several extensions and open questions result from ROSI’s interpretability-driven design:

• Multi-dimensional safety subspaces: Possibility of stacking multiple principal refusal axes.
• Layer-wise injection strength $\alpha$: Exploration of differential effects and optimal layer targeting.
• Generalization to other conceptual directions: Honesty, bias mitigation, and domain-specific safety requirements.
• Adversarial robustness: Analysis of potential attack vectors specifically targeting the injected refusal direction.
• Dynamic conditioning: Complementarity between permanent rank-one injection and inference-time steering for rare cases.

A plausible implication is that “surgical” model edits driven by mechanistic understanding deliver durable, low-cost safety alignment, and that the paradigm may generalize to additional alignment facets beyond refusal mediation. This suggests that ROSI can serve as both a primary and complementary component in contemporary LLM safety toolkits (Shairah et al., 28 Aug 2025).