Obfuscated Activations in Neural Networks

• Obfuscated activations are neural internal states that conceal harmful or adversarial behaviors by generating noisy or misaligned gradients.
• Techniques like stochastic activation pruning, BPDA, and EOT illustrate how defenses can be misled through randomization and surrogate gradients.
• Advanced strategies such as activation surgery and latent-space monitoring demonstrate both the vulnerabilities and potential countermeasures in modern neural architectures.

Searching arXiv for recent and foundational papers on obfuscated activations, SAP, latent-space defenses, and activation-space interventions. arxiv.search(query="obfuscated activations SAP BPDA EOT latent-space defenses activation surgery", max_results=10, sort_by="relevance") Obfuscated activations are internal neural states that preserve a target behavior while defeating the mechanism used to inspect, constrain, or optimize those states. In adversarial robustness, the term denotes activation patterns whose stochasticity or non-differentiability causes gradients to become noisy, zeroed, or misaligned; in backdoor research, it denotes abnormal neuron activations induced by stealthy triggers; in LLMs, it denotes latent trajectories that appear benign to latent-space monitors or are actively rewritten to suppress refusal without altering the prompt (Dhillon et al., 2020). Across these settings, the common structure is not a single architecture or threat model, but a mismatch between external behavior and internal signatures: harmful, jailbroken, or adversarial behavior is realized through latent states that either conceal the relevant computation or redirect it into forms the defense does not reliably capture (Bailey et al., 2024).

1. Conceptual scope and formal characterization

A precise formulation appears in latent-space monitoring work: for an adversarial input $x_{\mathrm{adv}}$, let $h(x_{\mathrm{adv}})$ denote a model’s activations and $D(h)$ a scalar monitor score. The activations are obfuscated if $D(h(x_{\mathrm{adv}}))$ is low even though the target concept, such as harmfulness or a backdoor behavior, is present in the input or generation (Bailey et al., 2024). This definition treats obfuscation as a property of the relation between behavior and a detector, rather than of the activations in isolation.

Two additional formulations refine the idea in distinct ways. In Stochastic Activation Pruning, obfuscation arises because the defended inference path contains random hard masks and evaluation-time averaging, so naïve backpropagation does not reflect the underlying model sensitivity (Dhillon et al., 2020). In activation-space backdoor defense, obfuscated activations are neuron activation patterns subtly altered by backdoor triggers so they do not rely on obvious surface artifacts yet still steer the model toward the attacker’s target; the defense therefore models clean activation distributions and repairs deviations in activation space rather than in word space (Yi et al., 2024). In white-box LLM jailbreaks, activation surgery constructs obfuscated activations by replacing portions of the illicit run’s residual stream with states from a benign “twin” run, so that downstream safety computations receive representations closer to a licit trajectory even though the prompt is unchanged (Jenny et al., 15 Mar 2026).

A useful synthesis is that obfuscated activations can be either incidental or deliberate. They are incidental when a defense mechanism itself induces gradient or activation masking, as with stochastic pruning. They are deliberate when an attacker optimizes or overwrites latent states to evade a monitor, implant a backdoor, or suppress refusal. This suggests that the term names a broader failure mode of latent observability: the network’s internal representation is sufficiently malleable that surface-level or latent-level inspection no longer tracks the behavior of interest.

2. Stochastic pruning, gradient shattering, and the SAP erratum

Stochastic Activation Pruning is a stochastic, magnitude-guided sparsification of intermediate activations during inference. For a feedforward network with layer activations

$$h_\ell = \phi(W_\ell h_{\ell-1}), \qquad \ell=1,\dots,d,$$

SAP applies a stochastic transformation $T$ at each non-linear layer,

$$h'_\ell = T(h_\ell) = M_\ell \odot h_\ell \oslash q_\ell,$$

where $M_\ell$ is a sampled mask and $q_\ell$ stores retention probabilities chosen from activation magnitudes (Dhillon et al., 2020). Neurons are sampled with probabilities proportional to absolute activation magnitude,

$$p_{\ell,j} = \frac{|h_{\ell,j}|}{\sum_k |h_{\ell,k}|},$$

with retention probability

$h(x_{\mathrm{adv}})$0

The recommended parameter is $h(x_{\mathrm{adv}})$1 for all non-linear layers, and defended inference averages multiple stochastic forward passes, typically $h(x_{\mathrm{adv}})$2.

The erratum identifies a flaw in the re-implementation used in the earlier attack on SAP: the required expectation at evaluation time was omitted. To keep clean accuracy close to the reported $h(x_{\mathrm{adv}})$3 on CIFAR-10, the re-implementation increased the sampling parameter to $h(x_{\mathrm{adv}})$4, which weakened pruning and altered the defense’s behavior. When SAP is evaluated correctly, with $h(x_{\mathrm{adv}})$5 and averaging over 100 forward passes, the original re-implemented targeted attack remains below $h(x_{\mathrm{adv}})$6 success at $h(x_{\mathrm{adv}})$7 even after more than 10,000 iterations of gradient ascent, while a transfer attack generated on the undefended model attains a targeted success rate of approximately $h(x_{\mathrm{adv}})$8 (Dhillon et al., 2020). That mismatch is presented as a diagnostic of obfuscation: direct optimization fails, but transfer succeeds.

The corrected attack uses Backward Pass Differentiable Approximation. For a non-differentiable or stochastic transform $h(x_{\mathrm{adv}})$9, BPDA replaces the backward pass by a differentiable surrogate, typically the identity:

$D(h)$0

For stochastic defenses, Expectation over Transformation is the corresponding principle for the attack objective,

$D(h)$1

In practice, the erratum reports that retaining SAP’s stochastic pruning in the forward pass while removing neuron-dropping in the backward pass and computing gradients on the vanilla network is sufficient to reduce CIFAR-10 test accuracy under attack to $D(h)$2 at $D(h)$3 (Dhillon et al., 2020).

The significance of the SAP case is methodological. It shows that obfuscated activations need not imply any genuine change in the decision boundary. Random hard masking can make gradient-based attacks appear ineffective even when principled attack methodology recovers near-total failure. It also shows that implementation fidelity is itself part of robustness evaluation: omitting the specified stochastic averaging led to compensatory hyperparameter changes and to an incorrect conclusion about the defense.

3. Activation-space purification and backdoor-triggered abnormal activations

BadActs treats obfuscated activations as abnormal activation distributions induced by backdoor triggers, especially feature-space triggers that do not correspond to a removable token. The defended model is a Transformer classifier with embedding block $D(h)$4 and $D(h)$5 Transformer blocks, with class-$D(h)$6 logit

$D(h)$7

For each neuron $D(h)$8 across all block outputs, the method estimates clean mean and standard deviation on a clean validation set,

$D(h)$9

and defines a clean activation distribution interval

$D(h(x_{\mathrm{adv}}))$0

A Gaussian approximation per neuron is assumed, and $D(h(x_{\mathrm{adv}}))$1 performs best empirically (Yi et al., 2024).

Detection is based on the NAS score, which counts the fraction of neuron activations that remain within their clean intervals:

$D(h(x_{\mathrm{adv}}))$2

$D(h(x_{\mathrm{adv}}))$3

With a threshold $D(h(x_{\mathrm{adv}}))$4 chosen from a desired validation false rejection rate, the sample is declared clean if $D(h(x_{\mathrm{adv}}))$5 and poisoned otherwise. Poisoned samples are purified by clamping activations at each Transformer block output to learned lower and upper bounds. The bounded activation operator is an elementwise clamp,

$D(h(x_{\mathrm{adv}}))$6

and the bounds are optimized by shrinking interval widths while enforcing a minimum validation accuracy constraint (Yi et al., 2024).

The method is motivated by the observation that modern backdoor attacks increasingly operate in feature space by modifying style and syntax rather than inserting a rare word. BadActs therefore aims to capture both surface-level and higher-level semantic deviations. The paper evaluates word-level triggers, sentence-level triggers, feature-space style triggers, and feature-space syntax triggers, including badnets rare tokens, addsent, stylebkd in Bible style, and synbkd with syntax template $D(h(x_{\mathrm{adv}}))$7 (Yi et al., 2024).

Measure	Result	Comparator
Detection AUROC average	95.80	STRIP 51.89, RAP 62.01, DAN 86.09
Purification CACC / PACC / ASR average	93.10 / 73.71 / 23.90	ONION 88.09 / 24.40 / 75.14
FRR trade-off, 20%	93.10 / 73.71 / 23.90	10%: 93.33 / 70.50 / 27.19

These results are accompanied by an explicit trade-off: increasing the false rejection rate flags more samples for purification, improving defense metrics but slightly reducing clean accuracy. Without detection, clamping all samples maximizes defense but hurts clean accuracy most (Yi et al., 2024). The broader implication is that obfuscated activations are not only an attack primitive; they can also be the object of defense, provided the defense models activation statistics directly and limits intervention to anomalous samples.

4. White-box LLM jailbreaks by sequential activation substitution

Activation surgery reframes obfuscated activations as a runtime intervention on a white-box autoregressive transformer. Let $D(h(x_{\mathrm{adv}}))$8 be a transformer with $D(h(x_{\mathrm{adv}}))$9 blocks and residual dimension $$h_\ell = \phi(W_\ell h_{\ell-1}), \qquad \ell=1,\dots,d,$$0, and let $$h_\ell = \phi(W_\ell h_{\ell-1}), \qquad \ell=1,\dots,d,$$1 denote the residual stream at the final prompt position $$h_\ell = \phi(W_\ell h_{\ell-1}), \qquad \ell=1,\dots,d,$$2. For an illicit prompt $$h_\ell = \phi(W_\ell h_{\ell-1}), \qquad \ell=1,\dots,d,$$3 and a nearby benign twin $$h_\ell = \phi(W_\ell h_{\ell-1}), \qquad \ell=1,\dots,d,$$4, the method collects $$h_\ell = \phi(W_\ell h_{\ell-1}), \qquad \ell=1,\dots,d,$$5 and $$h_\ell = \phi(W_\ell h_{\ell-1}), \qquad \ell=1,\dots,d,$$6 at every layer. It then defines a thresholded coordinate mask

$$h_\ell = \phi(W_\ell h_{\ell-1}), \qquad \ell=1,\dots,d,$$7

an interpolated target

$$h_\ell = \phi(W_\ell h_{\ell-1}), \qquad \ell=1,\dots,d,$$8

and a single-layer patching operator

$$h_\ell = \phi(W_\ell h_{\ell-1}), \qquad \ell=1,\dots,d,$$9

The intervention is applied sequentially, layer by layer, using a patch-then-compute schedule so that each subsequent block receives inputs produced from the already modified state (Jenny et al., 15 Mar 2026).

The twin prompt is hand-constructed to differ by as few tokens as possible while remaining licit and eliciting a substantive benign answer; the paper gives the example “How to make a bomb?” $T$0 “How to make a book?” Minimal edits preserve grammar, topic structure, and token alignment. Conceptually, the method prevents refusal signals from propagating across layers by overwriting the illicit run’s residual stream with a benign counterfactual trajectory. Because the prompt tokens remain unchanged, prompt-level audits still observe the illicit request, while the internal residual stream at key layers becomes donor-like rather than prompt-faithful (Jenny et al., 15 Mar 2026).

The primary empirical setting is Llama-4-Scout-17B-16E-Instruct, with 48 layers and $T$1, evaluated on AdvBench harmful prompts with $T$2. The prompts are categorized as weapons/chemical/murder (11%), cyberattacks/safety bypass (45%), and misinformation/discrimination (44%). Hyperparameters are swept over $T$3 and $T$4, with greedy decoding and interventions only at the final prompt token across all layers (Jenny et al., 15 Mar 2026).

The evaluation pipeline uses an LLM judge to classify outputs as Refusal, Unrelated, or Actionable, followed by Elo-based pairwise ranking within the Actionable set and manual inspection of top-ranked outputs. Under the actionability criterion, activation surgery yields $T$5 actionable answers versus $T$6 without surgery. By category, the paper reports $T$7 versus $T$8 for weapons/chemical/murder, $T$9 versus $$h'_\ell = T(h_\ell) = M_\ell \odot h_\ell \oslash q_\ell,$$0 for cyberattacks, and $$h'_\ell = T(h_\ell) = M_\ell \odot h_\ell \oslash q_\ell,$$1 versus $$h'_\ell = T(h_\ell) = M_\ell \odot h_\ell \oslash q_\ell,$$2 for misinformation/discrimination. With up to three twin-prompt attempts for cyberattacks, success reaches $$h'_\ell = T(h_\ell) = M_\ell \odot h_\ell \oslash q_\ell,$$3 (Jenny et al., 15 Mar 2026).

Mechanistically, the illicit and twin activations are reported to remain similar in early layers and diverge at intermediate depths, where refusal pathways activate. Smaller $$h'_\ell = T(h_\ell) = M_\ell \odot h_\ell \oslash q_\ell,$$4 increases refusal suppression but risks drift toward the benign topic; larger $$h'_\ell = T(h_\ell) = M_\ell \odot h_\ell \oslash q_\ell,$$5 preserves illicit semantics but can allow refusal to re-emerge. This suggests that obfuscated activations in LLM safety are not merely a monitor-evasion artifact but can be used as a direct control channel on generation dynamics.

5. Latent-space monitoring, monitor evasion, and representational malleability

“Obfuscated Activations Bypass LLM Latent-Space Defenses” studies the strongest general form of the concept: latent states that produce harmful or target behavior while appearing benign to a broad class of latent-space defenses. The targeted defenses include supervised probes, Gaussian Mahalanobis OOD detectors, VAE-based detectors, sparse autoencoder features, and logistic regression over SAE latents. Monitors operate either on the last prompt token or on generation tokens across layers, typically aggregating by mean over layers and max or mean over tokens (Bailey et al., 2024).

The generic obfuscation objective is multi-objective:

$$h'_\ell = T(h_\ell) = M_\ell \odot h_\ell \oslash q_\ell,$$6

For jailbreaks, the paper optimizes universal or per-input suffixes appended to token embeddings or hard tokens; for backdoors, it studies finetuning with explicit obfuscation losses and black-box data poisoning. Gaussian OOD monitors use a normalized squared Mahalanobis score,

$$h'_\ell = T(h_\ell) = M_\ell \odot h_\ell \oslash q_\ell,$$7

and GemmaScope SAEs use JumpReLU features of the form

$$h'_\ell = T(h_\ell) = M_\ell \odot h_\ell \oslash q_\ell,$$8

The preferred metric is recall at $$h'_\ell = T(h_\ell) = M_\ell \odot h_\ell \oslash q_\ell,$$9 false positive rate, chosen to emphasize the low-FPR regime relevant to deployment (Bailey et al., 2024).

A central empirical finding is that standard jailbreaks do not obfuscate by default: generation-based logistic and MLP probes generalize to a variety of jailbreak families with high recall, often approximately $M_\ell$0–$M_\ell$1 at $M_\ell$2 FPR. However, optimized obfuscation attacks can drive recall to near zero while preserving behavior. The abstract gives a representative result: against probes trained to classify harmfulness, attacks can reduce recall from $M_\ell$3 to $M_\ell$4 while retaining a $M_\ell$5 jailbreaking rate (Bailey et al., 2024).

The paper reports several transfer asymmetries. Attacks optimized against OOD detectors often transfer to input-based supervised probes, and attacks optimized against generation-based probes transfer to input-based probes. The reverse direction is weaker: attacks optimized against probes do not reliably transfer to OOD detectors. This is interpreted geometrically as moving harmful activations into the benign manifold, which frequently crosses supervised decision boundaries as a side effect (Bailey et al., 2024).

The same phenomenon appears in backdoor settings. Models finetuned with explicit obfuscation losses against Gaussian and VAE OOD detectors plus supervised probes can fool all of them while maintaining backdoor performance, with backdoor ASR approximately $M_\ell$6 and false trigger approximately $M_\ell$7–$M_\ell$8 under StrongREJECT grading. In black-box poisoning, complex triggers and poison-rate choices can reduce OOD recall while maintaining more than $M_\ell$9 trigger success and less than $q_\ell$0 false triggers in some regimes (Bailey et al., 2024).

Obfuscation does have limits. On SQL generation, raw SAE feature monitors and SAE-latent probes can be obfuscated so that recall at $q_\ell$1–$q_\ell$2 FPR drops toward zero even when the model still emits SQL tokens, but correctness measured by SQL execution degrades as more layers are obfuscated. The reported Pearson correlations between number of layers obfuscated and SQLEval accuracy are strongly negative, for example $q_\ell$3 or stronger. The paper also reports an orthogonality limit: making activations nearly orthogonal to natural ones while preserving behavior required KL divergence greater than $q_\ell$4 between original and perturbed logits, which qualitatively breaks generation (Bailey et al., 2024). These results constrain the notion that representational malleability is unlimited: some tasks require latent structure that cannot be hidden arbitrarily without performance loss.

6. Evaluation principles, misconceptions, and security implications

A recurrent misconception is that low attack success against a defended model, or low monitor score for a harmful run, is direct evidence of robustness or safety. The SAP erratum shows why this fails: direct optimization can be defeated by stochastic masking while transfer attacks and BPDA reveal vulnerability (Dhillon et al., 2020). The latent-defense results show an analogous failure mode in LLMs: high low-FPR recall against standard jailbreaks does not imply robustness to optimized obfuscation, and adversarial retraining of monitors did not converge to robustness in more than 70 rounds (Bailey et al., 2024).

A second misconception is that input-space inspection is sufficient. BadActs is motivated precisely by attacks whose triggers reside in style, syntax, or other feature-space properties rather than a single token, and activation surgery bypasses prompt-level auditing entirely because the prompt is unchanged while only the residual stream is modified (Yi et al., 2024). This suggests that obfuscated activations are a cross-domain challenge to any defense that assumes harmful intent must remain legible either in surface form or in a small set of latent directions.

Several evaluation principles emerge repeatedly. Defenses should be reproduced exactly, including evaluation-time stochastic averaging when specified. Non-differentiable or stochastic components should be attacked with BPDA and, where appropriate, EOT. Transfer attacks, black-box attacks, and loss-surface checks remain important diagnostics of masking (Dhillon et al., 2020). In LLM monitoring, generation-based and multi-layer monitoring is more robust in practice than input-only monitoring, but the latent-defense study concludes that latent monitors alone are not reliable and recommends behavior-grounded verification, low-FPR evaluation, and continuous red-teaming (Bailey et al., 2024). Activation surgery adds a distinct security implication: open-weight or instrumented inference deployments expose an execution-integrity attack surface, motivating attested inference, restrictions on hooks and tensor hot-patching, activation-integrity monitoring, and strict runtime controls (Jenny et al., 15 Mar 2026).

The broader significance of obfuscated activations is therefore diagnostic as much as adversarial. In one line of work, they reveal that a defense’s apparent robustness is an artifact of gradient or activation masking. In another, they are the substrate on which a defense operates, as in activation-space purification of backdoors. In the most recent LLM work, they become a design target for adversaries who reshape or overwrite latent trajectories to evade monitors or suppress refusal. Taken together, these results suggest that internal activations are simultaneously informative and highly manipulable: they support mechanistic analysis, but they cannot be assumed to furnish a stable or privileged basis for security unless the attack surface includes the latent space itself.