SAE Interventions are Unreliable: Post-Intervention Recovery of Suppressed Behavior

Abstract: Sparse Autoencoders (SAEs) decompose residual-stream activations into interpretable features. Recent latent-space defenses increasingly rely on these decompositions, assuming that identified "unsafe" SAE features serve as actionable handles for monitoring and intervention. In this paradigm, clamping a specific harmful feature is expected to reliably prevent model misbehavior. However, we show that this success may hide a recoverable failure mode: the clamp may block one visible route to a behavior without eliminating the behavior itself. We formulate this vulnerability as post-intervention recovery, a constrained residual-space optimization problem. Starting from the post-intervention residual state, we optimize residual perturbations to recover the pre-intervention behavior while preserving the post-intervention values of the targeted SAE features. Even under a strong threat model where the intervention remains active throughout optimization and generation, recovery remains possible. To rule out that recovery simply undoes the intervention, we use encoder-orthogonal updates for single-layer interventions and the corresponding feature-map Jacobian in the cross-layer setting. Across TPP, unlearning, IOI, and refusal steering experiments, this stress test reveals recoverable behavior despite successful feature-level intervention. Especially in the safety-critical refusal-steering setting, we achieve a 95.8% recovery rate on valid samples while keeping defended-feature relative drift to 0.131, substantially below suffix-based baselines. A recovery-path attribution analysis further localizes this recovery to the SAE reconstruction residual, the component left unexplained by the SAE. These results expose a gap between feature-level control and behavioral completeness: SAE features can support causal intervention, but controlling them does not guarantee control over the underlying behavior.

• The paper demonstrates that SAE interventions do not fully block behaviors due to the recovery mechanism through the residual stream.
• The authors employ constrained optimization with orthogonality projections to measure recovery across latent, output, and circuit levels.
• Empirical results reveal that even strict feature clamping permits alternative computational routes, challenging existing SAE safety assumptions.

SAE Interventions Are Unreliable: Post-Intervention Recovery of Suppressed Behavior

Introduction

This work rigorously examines the effectiveness of Sparse Autoencoder (SAE)-mediated interventions in transformer LLMs, a prevailing paradigm for interpretable latent-space editing and safety-aligned behavior modification. While existing literature contends that actionable SAE features can be leveraged for robust monitoring and control—especially through feature clamping or ablation to suppress unsafe outputs—this paper establishes that such interventions do not provide mechanistically complete control even under idealized conditions where clamp actions are perfectly deployed. Empirically, the analysis demonstrates that suppressed behaviors can be systematically recovered via constrained optimization pathways that leave the targeted SAE features strictly unchanged. This research fundamentally differentiates between SAE feature sets as causal handles versus complete bottlenecks for targeted behavior, with substantial implications for the reliability of representation-based safety mechanisms.

Post-Intervention Recovery: Framework and Methodology

The central diagnostic introduced is post-intervention recovery, formulated as a constrained optimization problem within the residual-stream of the transformer. Starting from the defended residual state—i.e., the post-clamp activation, where select SAE features have been set to defended values and the SAE reconstruction error is preserved—the method seeks a perturbation $\delta_x$ such that the target behavior re-emerges, but all monitored SAE features remain at their post-clamp values.

Figure 1: Post-intervention recovery framework, optimizing a residual perturbation from the defended state while keeping SAE feature clamps enforced.

Concretely, constraints are enforced to prevent trivial bypasses:

• C1: Perturbations are projected onto the nullspace of defended SAE encoders (single layer) or the row-space complement of the defended cross-layer feature map Jacobian (multiple layers).
• C2, C3: Activation and decode stability constraints ensure defended features do not drift from their clamp values.
• C4: Perturbation budget constraints preclude degenerate or wholesale state re-initializations.

Projected gradient methods are deployed to maintain orthogonality constraints throughout recovery optimization, with post-hoc analysis to quantify clamp-feature drift and any reactivation.

Empirical Results Across Tasks

Latent-Level (TPP) Recovery

Using the official layer-5 TPP benchmark, encoder-projected recovery (i.e., updating residuals orthogonal to the selected SAE encoders) achieves 0.749 valid-flip recovery, compared to 0.819 for unconstrained optimization. Critically, this is accomplished with a drastic reduction in defended-feature reactivation (mean reactivation $0.002$ vs. $0.013$) and activation drift. A striking 0.680 zero-reactivation recovery fraction confirms behavioral persistence independent of selected SAE latents.

Figure 2: TPP recovery-reactivation trade-off at layer 5; encoder projection significantly reduces SAE feature reactivation with modest sacrifice in behavioral recovery.

Output-Level Recovery (Unlearning)

In the WMDP-Bio unlearning context, where answer accuracy is suppressed by actively clamping knowledge-associated SAE features, encoder-projected recovery restores 90/91 answer-choice flips with zero clamp-feature drift, while unconstrained recovery achieves 91/91 with substantial drift. The recovery holds across all answer permutations, indicating that SAE feature clamping does not disable alternative computational routes to the suppressed task-relevant behavior.

Figure 3: WMDP-Bio unlearning recovery; encoder-projected path matches baseline recovery without post-hoc SAE feature movement.

Circuit-Level Recovery (IOI)

Under IOI circuits, which provide a transparent readout for logit-level interventions, both unconstrained and encoder-projected recovery restore the original indirect-object behavior. Encoder-projected recovery uniquely achieves this with minimal activation and decode drift, demonstrating behaviorally sufficient pathways exist orthogonal to clamp targets.

Figure 4: IOI recovery under fixed SAE clamp; encoder projection avoids activation drift and feature reopening.

Refusal Steering: Safety-Critical Evaluation

The most safety-relevant finding emerges in SAE-based refusal control. Utilizing the cross-layer Jacobian projection to maintain all defended refusal SAEs invariant, recovery restores 23/24 harmful completions on strict valid AdvBench prompts, with a mean defended-feature drift of only 0.131, compared to much higher drift for soft suffix baselines or unconstrained residual recovery.

Figure 5: Refusal recovery-preservation trade-off; Jacobian projection achieves near-perfect recovery with negligible defended-feature state movement.

Recovery-path attribution experiments demonstrate that the SAE reconstruction residual—the component orthogonal to all monitored SAE features—carries nearly all the recoverable behavior, as opposed to compensation via non-clamped features or trivial clamp reactivation.

Figure 6: Recovery is predominantly carried by the SAE reconstruction residual, not by reopening visible SAE latents.

Sweep and Robustness Analyses

Supplementary sweeps over feature set size and alternative datasets confirm that even as the number of clamped SAE features increases, non-refusal recovery remains high within practical clamp budgets. Degeneration only arises when clamp sets reach impractical breadth, wherein both base-answer fidelity declines and side-effects dominate the output.

Figure 7: Non-refusal recovery persists across feature-set sizes, but excessive clamping induces capability loss rather than eliminating recovery.

Theoretical and Practical Implications

The central implication is that SAE-latent interventions, while offering interpretable and locally causal handles, do not reliably serve as sufficient bottlenecks for behavioral control. This undermines the mechanistic assumptions that currently guide latent-space safety and editability approaches, especially in safety-critical domains such as refusal control, content filtering, or knowledge unlearning.

The finding that the SAE reconstruction residual serves as a computationally useful “escape hatch” for suppressed behaviors is particularly salient. Even when clamp-induced feature states remain invariant, the orthogonal complement in the residual stream—the part “unexplained” by the SAE dictionary—can effect complete recovery of complex behaviors, including otherwise harmful outputs. This result is robust to white-box constraints and strong clamp deployments and is not an artifact of trivial reparametrization or feature selection.

Therefore, safety protocols that rely solely on SAE-mediated interventions will be insufficient unless (a) SAE decompositions are driven to near-complete coverage of the residual stream, which is currently unattainable at scale, or (b) post-clamp recovery is empirically and theoretically ruled out via more global / multi-layer or trajectory-level constraints.

Future Directions

Potential lines of further inquiry include:

• Developing SAE objectives that minimize the behavioral expressivity of the reconstruction residual;
• Jointly optimizing dictionaries and intervention mechanisms with explicit post-intervention recovery adversaries;
• Exploring non-SAE-based latent representation frameworks for intervention completeness;
• Formal characterization of what class of behaviors are “unavoidably” present in the SAE residual and how this relates to feature superposition, absorption, and hedging phenomena identified in the sparse interpretability literature.

Conclusion

This study establishes that, contrary to prevailing assumptions, SAE interventions are not robust behavioral bottlenecks: suppressed behaviors can be systematically and reliably recovered via perturbation trajectories orthogonal to all monitored features, with the SAE residual acting as the principal carrier. Behavioral suppression through SAE feature control should not be conflated with elimination, and future interpretability-driven safety efforts must adapt theoretical and empirical frameworks to address this gap in mechanistic completeness.

(2606.18322)