Activation Space Manipulation

• Activation space manipulation is the process of directly modifying intermediate neural activations to adjust model behavior without full retraining.
• It leverages geometric, statistical, and information-theoretic techniques to steer features, enable unlearning, and defend against adversarial and backdoor attacks.
• Applications span LLM control, image classifier defense, and DRL policy safety, implemented via methods like Angular Steering, FALCON, BadActs, and IMPACT.

Activation space manipulation refers to the direct analysis and alteration of a neural network's intermediate representations (activations), with the aim of modifying model behavior, improving interpretability, enforcing safety constraints, or achieving efficient compression. Unlike traditional parameter-space interventions, activation space manipulations operate on the forward-pass signals propagated through neural layers, enabling targeted control of behaviors, robust sanitation against adversarial or backdoor triggers, and mechanisms for machine unlearning. This paradigm leverages the geometric, statistical, and information-theoretic structure of neural activations, and is implemented in LLMs, vision networks, and deep reinforcement learning (DRL) agents.

1. Theoretical Foundations and Formal Definitions

Neural network activation space is defined for each layer as the set of vectors $a^{(l)} \in \mathbb{R}^{d_l}$ resulting from layer-$l$ transformations on input $x$. The collection $V^{(l)} = \{a^{(l)}(x) \mid x \in \mathcal{X}\}$ defines the $l$-th layer's activation space. These spaces are high-dimensional and typically exhibit geometric structure, including low-rankness, nontrivial topologies, and feature disentanglement.

Activation space can be analyzed via:

• Vector geometric techniques: mapping, projecting, or rotating activations in subspaces defined by contrasting dataset statistics or principle components.
• Statistical hypothesis testing and divergence measures: e.g., Mahalanobis distance, quantile thresholding, or KL divergence between activation distributions across clean and perturbed data (Vyas et al., 21 Jul 2024, Yi et al., 18 May 2024).
• Topological data analysis: persistent homology of activation graphs, quantifying task-relevant substructures and providing semantic signatures of classes (Gebhart et al., 2019).
• Information-theoretic measures: entropy, mutual information (MI), and principal offset vectors to disentangle overlapping feature representations for tasks like unlearning (Hu et al., 3 Feb 2025).

2. Methodologies for Manipulation and Control

Activation space manipulation enables fine-grained modification of neural behaviors without retraining the full model. Key methodologies include:

• Angular Steering: This method geometrically rotates the activation vector $h \in \mathbb{R}^{d_m}$ within a two-dimensional plane $P$ spanned by a feature direction $\nu$ (extracted from contrastive dataset means) and an orthogonal axis $\mu$ (from PCA across candidate $\nu$'s). Angular Steering applies the map

$h' = h + (R(\theta)-I_2)P(h)$

where $R(\theta)$ is the 2D rotation matrix and $\theta$ is the steering angle. $\theta \approx 0^\circ$ enhances the feature, $\theta \approx 90^\circ$ ablates it, and $\theta > 180^\circ$ inverts it. Adaptive Angular Steering further masks rotations to only act on positively aligned activations, improving stability and avoiding unintended effects (Vu et al., 30 Oct 2025).

• Contrastive Orthogonal Unalignment (FALCON): For unlearning, FALCON selects the layer with minimal MI between "forget" and "retain" activation distributions, then pushes forget activations away from dominant subspaces via singular value decomposition (SVD), guided by contrastive losses and orthogonal gradient projection to balance forgetting and retention in parameter updates (Hu et al., 3 Feb 2025).
• Activation Interval Clipping (BadActs): For backdoor defense, neuron-wise clean activation intervals are learned as tight as possible while maintaining clean accuracy. During inference, activations are clipped into these intervals if a detector finds abnormal activations, thus neutralizing backdoor triggers and minimizing distortion to benign content (Yi et al., 18 May 2024).
• Importance-Aware Subspace Compression (IMPACT): Rather than uniform approximation, IMPACT weights activation coordinates by gradient sensitivity, and reconstructs activations in a low-rank subspace chosen by the top eigenvectors of the importance-weighted covariance. This preserves model utility under aggressive compression, since task-relevant directions are retained at higher fidelity (Chowdhury et al., 4 Jul 2025).

3. Empirical Results and Benchmarks

Empirical studies demonstrate the effectiveness of activation space manipulation for safety, interpretability, compression, and robustness across diverse domains:

Method	Task	Key Metrics
Angular Steering (Vu et al., 30 Oct 2025)	LLM behavior control	HarmBench↑ (0.875), LlamaGuard3↑ (1.00), SubstringRef↓ (0.0); steerable refusal/jailbreak arc via θ sweep
FALCON (Hu et al., 3 Feb 2025)	LLM unlearning	WMDP-Bio ↓ (65%→27%), MMLU utility preserved (≤1pt drop), Recovery resistance maintained
BadActs (Yi et al., 18 May 2024)	DNN backdoor defense	Detection AUROC (95.8%), CACC (93.1%), PACC (73.7%), Attack Success Rate↓ (23.9%)
IMPACT (Chowdhury et al., 4 Jul 2025)	LLM compression	Up to 48.6% more model size reduction, ≤1pt accuracy loss, PPL maintained
DRL Activation Detector (Vyas et al., 21 Jul 2024)	DRL backdoor	Recall 0.92, F1 0.94, AUROC ≈ 0.98 on MiniGrid imitation

Empirically, interventions in activation space yield precise control and robust defense without substantial degradation to general task performance. For example, Angular Steering's adaptive masking preserves perplexity and accuracy even under large rotations, and quantile-based detectors in DRL policies detect in-distribution triggers at low false positive rates (Vu et al., 30 Oct 2025, Vyas et al., 21 Jul 2024).

4. Applications Across Architectures and Modalities

Activation space manipulation has broad applicability:

• LLMs: Behavior control (e.g., refusal/compliance), content unlearning, and compression.
• Image classifiers: Adversarial detection/tracking via spatial trajectory in activation space; topological characterization of activation graphs explains adversarial sparsity and enables class-guided sculpting (Gebhart et al., 2019, Katzir et al., 2018).
• Reinforcement learning: Detection and defense against policy backdoors via outlier detection in penultimate layer activations (Vyas et al., 21 Jul 2024).
• Text classifiers: Universal backdoor purification and statistical detection in the activation domain, robust against sophisticated feature-space triggers (Yi et al., 18 May 2024).

The manipulation tools are made architecture-agnostic by operating on normalized activations and leveraging intrinsic activation statistics.

5. Detection and Defense via Activation Space

Several recent defense strategies operate in activation space rather than input or parameter space:

• Backdoor defense: BadActs first computes a Neuron Activation State (NAS) score per input by counting fraction of neurons within a three-sigma safety interval, then purifies only abnormally-activated samples, yielding a 95.8% AUROC, substantially outperforming word-level or attribution baselines (Yi et al., 18 May 2024).
• Adversarial attack detection: Layer-wise k-NN voting on PCA-projected activations reveals that adversarial examples exhibit anomalous class-switching patterns, especially in later layers, supporting a general trajectory-based detection framework (Katzir et al., 2018).
• Reinforcement learning policy safety: Lightweight, unsupervised detectors based on per-neuron quantiles in DRL agents robustly detect and block backdoor-triggered actions, even for in-distribution triggers intended to evade pixel-level scrutiny (Vyas et al., 21 Jul 2024).

6. Limitations and Future Directions

Current limitations include:

• The need for heuristic subspace selection in geometric manipulation frameworks (the choice of steering plane $\{\nu, \mu\}$ may not generalize across behaviors or architectures) (Vu et al., 30 Oct 2025).
• Most methods focus on single-layer interventions or activation-domain statistics without exploiting full multi-layer or temporal dependencies (Hu et al., 3 Feb 2025, Vyas et al., 21 Jul 2024).
• Hyperparameter tuning (e.g., quantile levels, MI weights, mask thresholds) is necessary per architecture and use-case.

Future directions include:

• Automated or supervised discovery of optimal activation subspaces for targeted manipulation.
• Extension to multi-modal and multi-layer interventions with cross-modal activation correlation modeling.
• Integration of topological signatures and trajectory-based detection for more effective and generalizable adversarial and backdoor defenses (Gebhart et al., 2019, Katzir et al., 2018).

A plausible implication is that as model architectures diversify, activation space manipulation frameworks with robust theoretical grounding and architecture-agnostic components will become central for post-deployment model control, interpretability, and security.