---
title: Obfuscated Activations in Neural Networks
url: https://www.emergentmind.com/topics/obfuscated-activations
type: topic
---

# Obfuscated Activations in Neural Networks

Searching arXiv for recent and foundational papers on obfuscated activations, SAP, latent-space defenses, and activation-space interventions.
arxiv.search(query="obfuscated activations SAP BPDA EOT latent-space defenses activation surgery", max_results=10, sort_by="relevance")
Obfuscated activations are internal neural states that preserve a target behavior while defeating the mechanism used to inspect, constrain, or optimize those states. In adversarial robustness, the term denotes activation patterns whose stochasticity or non-differentiability causes gradients to become noisy, zeroed, or misaligned; in backdoor research, it denotes abnormal neuron activations induced by stealthy triggers; in large language models, it denotes latent trajectories that appear benign to latent-space monitors or are actively rewritten to suppress refusal without altering the prompt [2010.00071]. Across these settings, the common structure is not a single architecture or threat model, but a mismatch between external behavior and internal signatures: harmful, jailbroken, or adversarial behavior is realized through latent states that either conceal the relevant computation or redirect it into forms the defense does not reliably capture [2412.09565].

## 1. Conceptual scope and formal characterization

A precise formulation appears in latent-space monitoring work: for an adversarial input $x_{\mathrm{adv}}$, let $h(x_{\mathrm{adv}})$ denote a model’s activations and $D(h)$ a scalar monitor score. The activations are obfuscated if $D(h(x_{\mathrm{adv}}))$ is low even though the target concept, such as harmfulness or a backdoor behavior, is present in the input or generation [2412.09565]. This definition treats obfuscation as a property of the relation between behavior and a detector, rather than of the activations in isolation.

Two additional formulations refine the idea in distinct ways. In Stochastic Activation Pruning, obfuscation arises because the defended inference path contains random hard masks and evaluation-time averaging, so naïve backpropagation does not reflect the underlying model sensitivity [2010.00071]. In activation-space backdoor defense, obfuscated activations are neuron activation patterns subtly altered by backdoor triggers so they do not rely on obvious surface artifacts yet still steer the model toward the attacker’s target; the defense therefore models clean activation distributions and repairs deviations in activation space rather than in word space [2405.11227]. In white-box LLM jailbreaks, activation surgery constructs obfuscated activations by replacing portions of the illicit run’s residual stream with states from a benign “twin” run, so that downstream safety computations receive representations closer to a licit trajectory even though the prompt is unchanged [2603.14278].

A useful synthesis is that obfuscated activations can be either incidental or deliberate. They are incidental when a defense mechanism itself induces gradient or activation masking, as with stochastic pruning. They are deliberate when an attacker optimizes or overwrites latent states to evade a monitor, implant a backdoor, or suppress refusal. This suggests that the term names a broader failure mode of latent observability: the network’s internal representation is sufficiently malleable that surface-level or latent-level inspection no longer tracks the behavior of interest.

## 2. Stochastic pruning, gradient shattering, and the SAP erratum

Stochastic Activation Pruning is a stochastic, magnitude-guided sparsification of intermediate activations during inference. For a feedforward network with layer activations
$$
h_\ell = \phi(W_\ell h_{\ell-1}), \qquad \ell=1,\dots,d,
$$
SAP applies a stochastic transformation $T$ at each non-linear layer,
$$
h'_\ell = T(h_\ell) = M_\ell \odot h_\ell \oslash q_\ell,
$$
where $M_\ell$ is a sampled mask and $q_\ell$ stores retention probabilities chosen from activation magnitudes [2010.00071]. Neurons are sampled with probabilities proportional to absolute activation magnitude,
$$
p_{\ell,j} = \frac{|h_{\ell,j}|}{\sum_k |h_{\ell,k}|},
$$
with retention probability
$$
q_{\ell,j} = 1 - (1 - p_{\ell,j})^{r_\ell}.
$$
The recommended parameter is $r_\ell = m_\ell$ for all non-linear layers, and defended inference averages multiple stochastic forward passes, typically $N=100$.

The erratum identifies a flaw in the re-implementation used in the earlier attack on SAP: the required expectation at evaluation time was omitted. To keep clean accuracy close to the reported $\sim 83\%$ on CIFAR-10, the re-implementation increased the sampling parameter to $r_\ell = 2m_\ell$, which weakened pruning and altered the defense’s behavior. When SAP is evaluated correctly, with $r_\ell = m_\ell$ and averaging over 100 forward passes, the original re-implemented targeted attack remains below $50\%$ success at $\epsilon = 0.031$ even after more than 10,000 iterations of gradient ascent, while a transfer attack generated on the undefended model attains a targeted success rate of approximately $70\%$ [2010.00071]. That mismatch is presented as a diagnostic of obfuscation: direct optimization fails, but transfer succeeds.

The corrected attack uses Backward Pass Differentiable Approximation. For a non-differentiable or stochastic transform $T$, BPDA replaces the backward pass by a differentiable surrogate, typically the identity:
$$
\nabla_x L(f(T(x)), y) \approx \nabla_x L(f(g(x)), y), \qquad g(x)=x.
$$
For stochastic defenses, Expectation over Transformation is the corresponding principle for the attack objective,
$$
\nabla_x \mathbb{E}_T[L(f(T(x)), y)] \approx \frac{1}{N}\sum_{i=1}^N \nabla_x L(f(T_i(x)), y).
$$
In practice, the erratum reports that retaining SAP’s stochastic pruning in the forward pass while removing neuron-dropping in the backward pass and computing gradients on the vanilla network is sufficient to reduce CIFAR-10 test accuracy under attack to $0.1\% \; (\pm 0.05\%)$ at $\epsilon = 0.031$ [2010.00071].

The significance of the SAP case is methodological. It shows that obfuscated activations need not imply any genuine change in the decision boundary. Random hard masking can make gradient-based attacks appear ineffective even when principled attack methodology recovers near-total failure. It also shows that implementation fidelity is itself part of robustness evaluation: omitting the specified stochastic averaging led to compensatory hyperparameter changes and to an incorrect conclusion about the defense.

## 3. Activation-space purification and backdoor-triggered abnormal activations

BadActs treats obfuscated activations as abnormal activation distributions induced by backdoor triggers, especially feature-space triggers that do not correspond to a removable token. The defended model is a Transformer classifier with embedding block $\mathrm{Emd}(\cdot)$ and $L$ Transformer blocks, with class-$c$ logit
$$
g_c(x)=\mathbf{w}_c^\top\big(\sigma^{(L)}\circ\cdots\circ\sigma^{(1)}\big(\mathrm{Emd}(x)\big)+\mathbf{b}_c.
$$
For each neuron $i$ across all block outputs, the method estimates clean mean and standard deviation on a clean validation set,
$$
\mu_X^i = \frac{1}{|X|}\sum_{x\in X} r_i(x), \qquad
\sigma_X^i = \sqrt{\frac{1}{|X|}\sum_{x\in X}(r_i(x)-\mu_X^i)^2},
$$
and defines a clean activation distribution interval
$$
I_X^i(k) = [\mu_X^i-k\sigma_X^i,\; \mu_X^i+k\sigma_X^i].
$$
A Gaussian approximation per neuron is assumed, and $k\in\{3,4\}$ performs best empirically [2405.11227].

Detection is based on the NAS score, which counts the fraction of neuron activations that remain within their clean intervals:
$$
\Phi_X^i(r_i(x))=\mathbf{1}_{[\mu_X^i-k\sigma_X^i,\;\mu_X^i+k\sigma_X^i]}(r_i(x)),
$$
$$
\mathrm{NAS}(x;X)=\frac{1}{L\cdot d}\sum_{i=1}^{L\cdot d}\Phi_X^i(r_i(x)).
$$
With a threshold $\lambda$ chosen from a desired validation false rejection rate, the sample is declared clean if $\mathrm{NAS}(x;X)\ge \lambda$ and poisoned otherwise. Poisoned samples are purified by clamping activations at each Transformer block output to learned lower and upper bounds. The bounded activation operator is an elementwise clamp,
$$
\bar{\sigma}^{(l)}(\mathbf{a};\mathbf{z}_l^{\mathrm{low}},\mathbf{z}_l^{\mathrm{up}})
=
\max\{\min\{\mathbf{a},\mathbf{z}_l^{\mathrm{up}}\},\mathbf{z}_l^{\mathrm{low}}\},
$$
and the bounds are optimized by shrinking interval widths while enforcing a minimum validation accuracy constraint [2405.11227].

The method is motivated by the observation that modern backdoor attacks increasingly operate in feature space by modifying style and syntax rather than inserting a rare word. BadActs therefore aims to capture both surface-level and higher-level semantic deviations. The paper evaluates word-level triggers, sentence-level triggers, feature-space style triggers, and feature-space syntax triggers, including badnets rare tokens, addsent, stylebkd in Bible style, and synbkd with syntax template $S(SBAR)(,)(NP)(VP)(.)$ [2405.11227].

| Measure | Result | Comparator |
|---|---:|---|
| Detection AUROC average | 95.80 | STRIP 51.89, RAP 62.01, DAN 86.09 |
| Purification CACC / PACC / ASR average | 93.10 / 73.71 / 23.90 | ONION 88.09 / 24.40 / 75.14 |
| FRR trade-off, 20% | 93.10 / 73.71 / 23.90 | 10%: 93.33 / 70.50 / 27.19 |

These results are accompanied by an explicit trade-off: increasing the false rejection rate flags more samples for purification, improving defense metrics but slightly reducing clean accuracy. Without detection, clamping all samples maximizes defense but hurts clean accuracy most [2405.11227]. The broader implication is that obfuscated activations are not only an attack primitive; they can also be the object of defense, provided the defense models activation statistics directly and limits intervention to anomalous samples.

## 4. White-box LLM jailbreaks by sequential activation substitution

Activation surgery reframes obfuscated activations as a runtime intervention on a white-box autoregressive transformer. Let $f_\theta$ be a transformer with $L$ blocks and residual dimension $d$, and let $h_l(t^\star)$ denote the residual stream at the final prompt position $t^\star = T$. For an illicit prompt $x$ and a nearby benign twin $x'$, the method collects $h_l^{\mathrm{ill}}(t^\star)$ and $h_l^{\mathrm{twin}}(t^\star)$ at every layer. It then defines a thresholded coordinate mask
$$
m_l(\tau)=\mathbf{1}[|h_l^{\mathrm{ill}}(t^\star)-h_l^{\mathrm{twin}}(t^\star)|>\tau],
$$
an interpolated target
$$
h_l^{(\gamma)}(t^\star)=(1-\gamma)h_l^{\mathrm{twin}}(t^\star)+\gamma h_l^{\mathrm{ill}}(t^\star),
$$
and a single-layer patching operator
$$
\Phi_l(h_l^{\mathrm{ill}}(t^\star);\tau,\gamma)
=
(1-m_l(\tau))\odot h_l^{\mathrm{ill}}(t^\star)+m_l(\tau)\odot h_l^{(\gamma)}(t^\star).
$$
The intervention is applied sequentially, layer by layer, using a patch-then-compute schedule so that each subsequent block receives inputs produced from the already modified state [2603.14278].

The twin prompt is hand-constructed to differ by as few tokens as possible while remaining licit and eliciting a substantive benign answer; the paper gives the example “How to make a bomb?” $\rightarrow$ “How to make a book?” Minimal edits preserve grammar, topic structure, and token alignment. Conceptually, the method prevents refusal signals from propagating across layers by overwriting the illicit run’s residual stream with a benign counterfactual trajectory. Because the prompt tokens remain unchanged, prompt-level audits still observe the illicit request, while the internal residual stream at key layers becomes donor-like rather than prompt-faithful [2603.14278].

The primary empirical setting is Llama-4-Scout-17B-16E-Instruct, with 48 layers and $d=5120$, evaluated on AdvBench harmful prompts with $N=520$. The prompts are categorized as weapons/chemical/murder (11%), cyberattacks/safety bypass (45%), and misinformation/discrimination (44%). Hyperparameters are swept over $\tau \in \{0.4,0.5,\dots,1.1\}$ and $\gamma \in \{0.0,0.2,0.4,0.6\}$, with greedy decoding and interventions only at the final prompt token across all layers [2603.14278].

The evaluation pipeline uses an LLM judge to classify outputs as Refusal, Unrelated, or Actionable, followed by Elo-based pairwise ranking within the Actionable set and manual inspection of top-ranked outputs. Under the actionability criterion, activation surgery yields $32.8\%$ actionable answers versus $7.1\%$ without surgery. By category, the paper reports $24.1\%$ versus $0.0\%$ for weapons/chemical/murder, $39.8\%$ versus $8.0\%$ for cyberattacks, and $27.9\%$ versus $8.0\%$ for misinformation/discrimination. With up to three twin-prompt attempts for cyberattacks, success reaches $69.1\%$ [2603.14278].

Mechanistically, the illicit and twin activations are reported to remain similar in early layers and diverge at intermediate depths, where refusal pathways activate. Smaller $\gamma$ increases refusal suppression but risks drift toward the benign topic; larger $\gamma$ preserves illicit semantics but can allow refusal to re-emerge. This suggests that obfuscated activations in LLM safety are not merely a monitor-evasion artifact but can be used as a direct control channel on generation dynamics.

## 5. Latent-space monitoring, monitor evasion, and representational malleability

“Obfuscated Activations Bypass LLM Latent-Space Defenses” studies the strongest general form of the concept: latent states that produce harmful or target behavior while appearing benign to a broad class of latent-space defenses. The targeted defenses include supervised probes, Gaussian Mahalanobis OOD detectors, VAE-based detectors, sparse autoencoder features, and logistic regression over SAE latents. Monitors operate either on the last prompt token or on generation tokens across layers, typically aggregating by mean over layers and max or mean over tokens [2412.09565].

The generic obfuscation objective is multi-objective:
$$
\min \; L_{\mathrm{beh}}(M(x)) + \lambda_{\mathrm{obf}} L_{\mathrm{obf}}(h(x)).
$$
For jailbreaks, the paper optimizes universal or per-input suffixes appended to token embeddings or hard tokens; for backdoors, it studies finetuning with explicit obfuscation losses and black-box data poisoning. Gaussian OOD monitors use a normalized squared Mahalanobis score,
$$
D(h)=\frac{1}{d_{\mathrm{res}}}(h-\mu)^\top \Sigma_\alpha^{-1}(h-\mu),
$$
and GemmaScope SAEs use JumpReLU features of the form
$$
f(x)=\sigma(W_{\mathrm{enc}}x+b_{\mathrm{enc}}), \qquad \sigma(z)= z\odot H(z-\theta).
$$
The preferred metric is recall at $1\%$ false positive rate, chosen to emphasize the low-FPR regime relevant to deployment [2412.09565].

A central empirical finding is that standard jailbreaks do not obfuscate by default: generation-based logistic and MLP probes generalize to a variety of jailbreak families with high recall, often approximately $95$–$100\%$ at $1\%$ FPR. However, optimized obfuscation attacks can drive recall to near zero while preserving behavior. The abstract gives a representative result: against probes trained to classify harmfulness, attacks can reduce recall from $100\%$ to $0\%$ while retaining a $90\%$ jailbreaking rate [2412.09565].

The paper reports several transfer asymmetries. Attacks optimized against OOD detectors often transfer to input-based supervised probes, and attacks optimized against generation-based probes transfer to input-based probes. The reverse direction is weaker: attacks optimized against probes do not reliably transfer to OOD detectors. This is interpreted geometrically as moving harmful activations into the benign manifold, which frequently crosses supervised decision boundaries as a side effect [2412.09565].

The same phenomenon appears in backdoor settings. Models finetuned with explicit obfuscation losses against Gaussian and VAE OOD detectors plus supervised probes can fool all of them while maintaining backdoor performance, with backdoor ASR approximately $95\%$ and false trigger approximately $0.6$–$1.0\%$ under StrongREJECT grading. In black-box poisoning, complex triggers and poison-rate choices can reduce OOD recall while maintaining more than $50\%$ trigger success and less than $0.5\%$ false triggers in some regimes [2412.09565].

Obfuscation does have limits. On SQL generation, raw SAE feature monitors and SAE-latent probes can be obfuscated so that recall at $1$–$2\%$ FPR drops toward zero even when the model still emits SQL tokens, but correctness measured by SQL execution degrades as more layers are obfuscated. The reported Pearson correlations between number of layers obfuscated and SQLEval accuracy are strongly negative, for example $-0.88$ or stronger. The paper also reports an orthogonality limit: making activations nearly orthogonal to natural ones while preserving behavior required KL divergence greater than $100$ between original and perturbed logits, which qualitatively breaks generation [2412.09565]. These results constrain the notion that representational malleability is unlimited: some tasks require latent structure that cannot be hidden arbitrarily without performance loss.

## 6. Evaluation principles, misconceptions, and security implications

A recurrent misconception is that low attack success against a defended model, or low monitor score for a harmful run, is direct evidence of robustness or safety. The SAP erratum shows why this fails: direct optimization can be defeated by stochastic masking while transfer attacks and BPDA reveal vulnerability [2010.00071]. The latent-defense results show an analogous failure mode in LLMs: high low-FPR recall against standard jailbreaks does not imply robustness to optimized obfuscation, and adversarial retraining of monitors did not converge to robustness in more than 70 rounds [2412.09565].

A second misconception is that input-space inspection is sufficient. BadActs is motivated precisely by attacks whose triggers reside in style, syntax, or other feature-space properties rather than a single token, and activation surgery bypasses prompt-level auditing entirely because the prompt is unchanged while only the residual stream is modified [2405.11227]. This suggests that obfuscated activations are a cross-domain challenge to any defense that assumes harmful intent must remain legible either in surface form or in a small set of latent directions.

Several evaluation principles emerge repeatedly. Defenses should be reproduced exactly, including evaluation-time stochastic averaging when specified. Non-differentiable or stochastic components should be attacked with BPDA and, where appropriate, EOT. Transfer attacks, black-box attacks, and loss-surface checks remain important diagnostics of masking [2010.00071]. In LLM monitoring, generation-based and multi-layer monitoring is more robust in practice than input-only monitoring, but the latent-defense study concludes that latent monitors alone are not reliable and recommends behavior-grounded verification, low-FPR evaluation, and continuous red-teaming [2412.09565]. Activation surgery adds a distinct security implication: open-weight or instrumented inference deployments expose an execution-integrity attack surface, motivating attested inference, restrictions on hooks and tensor hot-patching, activation-integrity monitoring, and strict runtime controls [2603.14278].

The broader significance of obfuscated activations is therefore diagnostic as much as adversarial. In one line of work, they reveal that a defense’s apparent robustness is an artifact of gradient or activation masking. In another, they are the substrate on which a defense operates, as in activation-space purification of backdoors. In the most recent LLM work, they become a design target for adversaries who reshape or overwrite latent trajectories to evade monitors or suppress refusal. Taken together, these results suggest that internal activations are simultaneously informative and highly manipulable: they support mechanistic analysis, but they cannot be assumed to furnish a stable or privileged basis for security unless the attack surface includes the latent space itself.

Source: https://www.emergentmind.com/topics/obfuscated-activations