Activation Transport Operators (ATOs)

• Activation Transport Operators (ATOs) are parametric maps that steer neural activation flows to enforce safety, enhance interpretability, and support dynamical analyses.
• They employ linear, affine, and nonlinear formulations—using methods like ridge regression, SVD truncation, and MLPs—to precisely map upstream activations to downstream states.
• Empirical studies show that ATOs achieve high transport efficiency (e.g., R² > 0.95 for small offsets) and effective safety intervention in generative models with minimal output fidelity loss.

Activation Transport Operators (ATOs) are a class of parametric transformations designed to manipulate or analyze the flow of neural activations within deep networks, with applications spanning interpretability, safety intervention, and dynamical analysis. ATOs formalize the mapping of upstream activations onto downstream states, typically under constraints or with specific geometric or safety objectives. They have been instantiated as both inference-time steering mechanisms in generative models and as analyzers of signal propagation in transformer residual streams, with further connections to stochastic transport processes in mathematical physics.

1. Formal Definitions and Core Mechanisms

ATO definitions vary to match context—modern neural architectures or stochastic particle systems—but they adhere to a common principle: an ATO is a parametric map $T_\theta$ operating on latent feature representations at a specific layer or location, optionally coupled to a gating or conditioning function $C$ which determines the region of action.

In transformer models, consider activations $v_{\ell,i} \in \mathbb{R}^{d_m}$ at layer $\ell$ and token $i$. An ATO of rank $r$ between layers $\ell$ and $\ell+k$ is a rank-restricted affine map $(W, b)$, with $W \in \mathbb{R}^{d_m \times d_m}$, $C$0, that minimizes the projected squared error between downstream residuals and their ATO-mapped upstream counterparts after projection into a feature space defined by Sparse Autoencoder (SAE) decoders (Szablewski et al., 24 Aug 2025):

$C$1

Here, $C$2 projects onto a set of decoder directions for selected SAE features.

In generative vision models, as presented in Conditioned Activation Transport (CAT), the ATO at layer $C$3 is a pair $C$4 acting on the mean-pooled activation vector $C$5:

$C$6

where $C$7 is trained to transport unsafe activations onto a learned safe submanifold, and $C$8 gates this action via learned geometric criteria (Chrabąszcz et al., 3 Mar 2026).

2. Mathematical Formulation and Variants

ATO formulations encompass linear, affine, and nonlinear maps, with associated regularization and control strategies.

• Linear/affine ATOs: $C$9, trained by mean-squared error (MSE) regression over paired activations, or determined by second-order moment-matching transport costs for optimal linear re-alignment.
• ActAdd (Activation Addition): $v_{\ell,i} \in \mathbb{R}^{d_m}$0, using empirical centroids of safe and unsafe activation sets.
• MLP-based ATOs: $v_{\ell,i} \in \mathbb{R}^{d_m}$1, with a single hidden layer, RMSNorm, GELU activation, and zero-initialized final linear projection to ensure identity at initialization.
• Conditioning functions: For selective intervention, $v_{\ell,i} \in \mathbb{R}^{d_m}$2 may use
  • Min-Max gating (bounding-box in feature space)
  • Gaussian Discriminant Analysis gating (ellipsoidal/linear boundaries)
  • Mahalanobis Out-Of-Distribution gating (ellipsoidal within empirical quantiles of unsafe samples)

This enables ATOs to precisely control when and where transport is applied, minimizing adverse side effects on benign activations.

3. Construction, Training, and Algorithmic Application

ATO construction involves data-driven model fitting and explicit algorithmic steps:

• Data collection: For transport analysis, paired upstream/downstream activations from large corpora (e.g., 250K tokens from SlimPajama) (Szablewski et al., 24 Aug 2025). For safety steering, semantically aligned prompt pairs with safe/unsafe activations (SafeSteerDataset, 2300 pairs spanning 23 subcategories) (Chrabąszcz et al., 3 Mar 2026).
• Operator learning: Linear ATOs are fitted by ridge regression (with $v_{\ell,i} \in \mathbb{R}^{d_m}$3 penalty), followed by rank truncation using SVD. MLP-based ATOs train using regularized contrastive objectives:

$v_{\ell,i} \in \mathbb{R}^{d_m}$4

The first term aligns unsafe-to-safe transport, the second penalizes perturbation of already-safe inputs, promoting identity behavior in the safe region.

• Inference-time deployment: Algorithmic application iterates over model layers, pooling activations, gating, and updating token representations with the learned displacement (full pseudocode in (Chrabąszcz et al., 3 Mar 2026)).
• Compute considerations: ATO fitting requires no fine-tuning of the base model and remains computationally lightweight (<50 GPU-hours for transformer ATO analysis; negligible overhead for CAT intervention).

4. Empirical Properties and Performance

Empirical studies confirm that ATOs reveal and exploit significant linear and nonlinear structure in deep models' activation dynamics:

• Transformer ATOs: For small layer offsets ($v_{\ell,i} \in \mathbb{R}^{d_m}$5), most features exhibit high linear transportability ($v_{\ell,i} \in \mathbb{R}^{d_m}$6) (Szablewski et al., 24 Aug 2025). For larger $v_{\ell,i} \in \mathbb{R}^{d_m}$7, transport decays but remains significant for a substantial subspace (estimated by Canonical Correlation Analysis as $v_{\ell,i} \in \mathbb{R}^{d_m}$8 for $v_{\ell,i} \in \mathbb{R}^{d_m}$9; declines with increasing $\ell$0).
• Transport efficiency: $\ell$1 quantifies how much of the maximal explainable variance is captured by the learned operator, saturating at the effective subspace dimensionality.
• Safety steering in T2I: CAT (a nonlinear MLP ATO with Mahalanobis OOD gating) reduces Attack Success Rate (ASR) from 33.9% to $\ell$2 on Z-Image, with minimal CLIP score degradation ($\ell$3) (Chrabąszcz et al., 3 Mar 2026). Linear ATOs force lower ASR but at the cost of severe fidelity loss (CLIP drops to $\ell$4). Ablations reveal the intrinsically nonlinear nature of the “toxicity manifold” in activation space.

Table: Representative ATO Evaluation on Z-Image (from (Chrabąszcz et al., 3 Mar 2026))

Method	Conditioning	ASR (%)	CLIP Score
No steering	–	33.91	0.35
ActAdd	min-max	9.57	0.34
Linear-ACT	min-max	2.61	0.22
Affine	none	8.70	0.25
CAT (MLP)	none	6.96	0.33

5. Interpretability, Safety, and Diagnostic Applications

ATO frameworks open multiple avenues for model control, interpretability, and robust deployment:

• Safety and jailbreak prevention: By precisely mapping the flow of potentially dangerous features, ATOs can gate or suppress risky signals before they propagate nonlinearly (Szablewski et al., 24 Aug 2025).
• Error detection and causal intervention: Low transport success for a desired feature signals synthesis or erasure in intermediate layers, providing actionable signals for error correction by direct re-injection or targeted transport.
• Interpretability: High per-feature $\ell$5 identifies a “linear highway” in models' residual streams, while divergences spotlight circuit instantiation or feature creation.
• Vision-generation safety steering: CAT demonstrates inference-time, model-agnostic intervention: precise suppression of unsafe features with minimal collateral distortion of benign outputs (Chrabąszcz et al., 3 Mar 2026).
• Tooling recommendations: Stable ATO estimation with ridge regression and SVD truncation; calibration of operator rank to effective subspace dimensionality to avoid overfitting; extensibility to cross-token or attention-driven transport in future work.

6. Connections to Stochastic Transport and Renewal Dynamics

Beyond neural networks, activation and transport operators arise in dynamical systems and discrete stochastic processes, as in the Harmonic Activation and Transport (HAT) operator on $\ell$6 particle sets (Calvert et al., 2021):

• HAT operator $\ell$7: Samples a particle according to harmonic measure from infinity, removes it, runs a random walk until it reenters the configuration, and reinstates it at the last outside position. This generates configuration flows with well-characterized collapse, stationarity, and renewal properties.
• Collapse phenomenon: The diameter of the configuration decreases from $\ell$8 to $\ell$9 in $i$0 steps (Theorem 1.5), establishing exponential tightness of the stationary distribution's support.
• Renewal structure and diffusion: Partitioning into i.i.d. excursions yields convergence of the rescaled center of mass to Brownian motion.
• Extremal transport estimates: Lower bounds for harmonic measure and escape probability are established, controlling for rare but critical escape events that structure the collapse process.

7. Limitations and Frontiers

• Mean-pooling caveats: In conditioning mechanisms, mean-pooling may miss spatially localized anomalies; region-level gating could offer finer control (Chrabąszcz et al., 3 Mar 2026).
• Inference-time spectrum: ATOs act at inference and do not alter underlying unsafe capabilities—white-box adversaries could circumvent or invert interventions.
• Linear vs. nonlinear boundaries: Nonlinearity is essential for manifold boundaries such as those found in “toxicity space;” linear ATOs may introduce collateral damage in model outputs.
• Interpretation of low transport: Low $i$1 does not always disentangle synthesis from destruction; further granular analysis is needed.
• Open technical directions: Attention-guided transport operators, multi-source/multi-token mapping, and integration with higher-order geometric statistics remain open.

Activation Transport Operators, across neural and physical systems, provide principled, efficient mechanisms for probing, steering, or realizing targeted manipulation of high-dimensional activation dynamics, grounded in explicit mathematical formalism and validated by robust empirical methodology (Szablewski et al., 24 Aug 2025, Chrabąszcz et al., 3 Mar 2026, Calvert et al., 2021).