Sample-Wise Activation Patterns (SWAP)

• Sample-Wise Activation Patterns (SWAP) are a zero-shot evaluation metric that measures the diversity of thresholded activation responses across a mini-batch in neural networks.
• It computes a SWAP-Score by counting unique activation-position codes, effectively transposing the traditional sample-indexed approach to overcome saturation issues.
• SWAP has shown strong predictive correlations in NAS studies for CNNs and Transformers and is enhanced by regularization to adjust for varying model sizes.

Sample-Wise Activation Patterns (SWAP) denotes a training-free representation and zero-shot proxy in which a neural network at random initialization is characterized by the diversity of its thresholded activation responses across a mini-batch. The central move is to represent each intermediate activation position by its sign-pattern over samples, rather than representing each sample by its sign-pattern over activation positions. The resulting scalar, the SWAP-Score, is the number of distinct such activation-position codes. In current usage, SWAP is primarily a zero-shot neural network evaluation method for ranking architectures without training, especially in NAS, but it also sits within a broader literature on activation-region geometry, activation-signature monitoring, and conditional activation mechanisms (Peng et al., 8 May 2026).

1. Definition and representational shift

The immediate precursor to SWAP is the standard activation-pattern construction for ReLU networks. For a network $\mathcal N$ with fixed random parameters $\theta$, $S$ input samples, and $V$ intermediate values feeding into activation functions, the standard activation-pattern set is

$\mathbb{A}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(s)} : \mathbf{p}^{(s)} = \mathds{1}(p_v^{(s)})_{v=1}^{V},~ s \in \{1, \ldots, S\} \right\}.$

This representation attaches one binary code to each sample. Its cardinality is therefore upper-bounded by $S$, and for modern networks with $V \gg S$ it tends to saturate: almost every sample obtains its own distinct code, so the count loses discriminatory power across architectures (Peng et al., 2024).

SWAP reverses that orientation. Instead of asking how samples differ across activation positions, it asks how activation positions differ across samples. The sample-wise activation pattern set is

$\mathbb{\hat{A}}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(v)} : \mathbf{p}^{(v)} = \mathds{1}(p_s^{(v)})_{s=1}^{S},~ v \in \{1, \ldots, V\} \right\}.$

Each $\mathbf p^{(v)}$ is thus a length-$S$ code attached to one activation position $\theta$0, and the maximal number of distinct patterns is tied to $\theta$1, not $\theta$2. The literature explicitly characterizes this as a pattern-diversity measure over neurons or features across samples, rather than over samples across neurons (Peng et al., 8 May 2026).

This transpose is the defining conceptual contribution of SWAP. It preserves the activation-pattern perspective associated with ReLU expressivity, but avoids the low ceiling that afflicts sample-indexed counting. A plausible implication is that SWAP is best viewed not as a direct count of sample codes, but as a count of how many distinct sample-partitioning behaviors the network’s internal activation sites realize on a batch.

2. Formalism, scoring, and computation

The SWAP-Score is the cardinality of the sample-wise activation pattern set: $\theta$3 Operationally, computing it requires a randomly initialized architecture, a mini-batch of unlabeled inputs, one forward pass, extraction of post-activation values, sign-thresholding, and uniqueness counting; it does not require labels, backpropagation, gradients, training, or loss evaluation (Peng et al., 8 May 2026).

The indicator is instantiated as the Signum function: $\theta$4 For ReLU networks, this effectively reduces to a $\theta$5 code because activations are nonnegative. For GELU networks, it produces a ternary code in $\theta$6. This is one reason the later universal formulation of SWAP is applicable across both ReLU CNNs and GELU Transformers (Peng et al., 8 May 2026).

The raw score can be regularized by parameter count: $\theta$7 where $\theta$8 is the total number of network parameters, $\theta$9 specifies a preferred model-size region, and $S$0 controls its width. This regularized SWAP-Score was introduced for NAS, where unregularized zero-shot metrics often favor larger models (Peng et al., 2024).

The number of intermediate activation positions grows rapidly with architecture size. The universal SWAP formulation gives

$S$1

This growth is precisely why the activation-wise representation has a much larger discrimination capacity than the sample-wise baseline when $S$2 (Peng et al., 8 May 2026).

The empirical sensitivity profile is also part of the method’s practical definition. On 1000 DARTS CNNs on CIFAR-10, Spearman correlation is $S$3 at batch size $S$4 and decreases to $S$5 at batch size $S$6. For 500 BERT-like Transformers on GLUE, the correlation stays roughly $S$7 across batch sizes. The same work reports that larger image dimensions improve correlation when using real task data, whereas increasing dimension with Gaussian noise hurts correlation (Peng et al., 8 May 2026).

3. Geometric and theoretical context

SWAP inherits its theoretical backdrop from work on activation patterns and activation regions in ReLU networks. In that literature, an activation pattern is a sign assignment over neurons,

$S$8

and the corresponding activation region is

$S$9

A fixed input sample therefore induces a binary sign vector, while a fixed pattern determines the subset of input space on which that vector is constant (Hanin et al., 2019).

This matters for SWAP because any distinct sample-wise activation code observed on a finite dataset must correspond to some realized activation region. The principal average-case result for feed-forward ReLU networks at initialization is that, for $V$0 and input dimension $V$1, the expected density of non-empty activation regions in a cube is bounded by

$V$2

The dominant term depends on input dimension and total neurons, not explicitly on depth, and the paper reports that this depth-independent scaling is tight both at initialization and during training on the probed slices (Hanin et al., 2019). This constrains any interpretation of SWAP as a practical expressivity proxy: realized activation-pattern diversity is much smaller than worst-case exponential-depth bounds suggest.

Topological work on ReLU activation signatures adds a second layer of interpretation. There, a sample $V$3 is assigned a binary vector

$V$4

where each bit records whether a hidden-neuron preactivation is positive. The resulting sign patterns induce a polyhedral decomposition of input space, can be organized by a dual graph whose vertices are activation patterns and whose edges connect Hamming-distance-$V$5 neighbors, and admit spectral and homological analysis. The weighted Fiedler partition of this dual graph appears to correlate with the decision boundary in binary classification, while cell counts and Betti curves change during training and track structural reorganization (Bosca et al., 14 Oct 2025). This suggests that distinct-pattern counting is only the coarsest summary of sample-wise activation structure.

4. Zero-shot evaluation and NAS

SWAP is used primarily as a zero-shot proxy for neural architecture evaluation. The 2026 formulation reports strong predictive performance across both computer vision and natural language processing tasks, emphasizing that the metric is training-free, label-independent, and broadly applicable across CNNs and Transformers. The abstract highlights Spearman’s correlation coefficient of $V$6 between SWAP-Score and CIFAR-10 validation accuracy for DARTS CNNs, and $V$7 for FlexiBERT Transformers on GLUE tasks. On FlexiBERT/GLUE, the reported table gives SWAP-Score a Spearman value of $V$8, exceeding Attention Confidence at $V$9, #Params at $\mathbb{A}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(s)} : \mathbf{p}^{(s)} = \mathds{1}(p_v^{(s)})_{v=1}^{V},~ s \in \{1, \ldots, S\} \right\}.$0, and #FLOPs at $\mathbb{A}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(s)} : \mathbf{p}^{(s)} = \mathds{1}(p_v^{(s)})_{v=1}^{V},~ s \in \{1, \ldots, S\} \right\}.$1 (Peng et al., 8 May 2026).

In vision benchmarks, the universal SWAP study states that SWAP-Score and regularized SWAP outperform existing zero-shot metrics in the majority of evaluations. For NAS-Bench-201, the reported values include SWAP $\mathbb{A}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(s)} : \mathbf{p}^{(s)} = \mathds{1}(p_v^{(s)})_{v=1}^{V},~ s \in \{1, \ldots, S\} \right\}.$2 and Reg. SWAP $\mathbb{A}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(s)} : \mathbf{p}^{(s)} = \mathds{1}(p_v^{(s)})_{v=1}^{V},~ s \in \{1, \ldots, S\} \right\}.$3 on CIFAR-10, SWAP $\mathbb{A}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(s)} : \mathbf{p}^{(s)} = \mathds{1}(p_v^{(s)})_{v=1}^{V},~ s \in \{1, \ldots, S\} \right\}.$4 and Reg. SWAP $\mathbb{A}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(s)} : \mathbf{p}^{(s)} = \mathds{1}(p_v^{(s)})_{v=1}^{V},~ s \in \{1, \ldots, S\} \right\}.$5 on CIFAR-100, and SWAP $\mathbb{A}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(s)} : \mathbf{p}^{(s)} = \mathds{1}(p_v^{(s)})_{v=1}^{V},~ s \in \{1, \ldots, S\} \right\}.$6 and Reg. SWAP $\mathbb{A}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(s)} : \mathbf{p}^{(s)} = \mathds{1}(p_v^{(s)})_{v=1}^{V},~ s \in \{1, \ldots, S\} \right\}.$7 on ImageNet16-120. On NAS-Bench-301, the reported values are SWAP $\mathbb{A}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(s)} : \mathbf{p}^{(s)} = \mathds{1}(p_v^{(s)})_{v=1}^{V},~ s \in \{1, \ldots, S\} \right\}.$8 and Reg. SWAP $\mathbb{A}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(s)} : \mathbf{p}^{(s)} = \mathds{1}(p_v^{(s)})_{v=1}^{V},~ s \in \{1, \ldots, S\} \right\}.$9. The same study also notes a weaker setting on GLUE: SWAP correlates strongly with most tasks except RTE, which it attributes to task instability rather than to the metric itself (Peng et al., 8 May 2026).

The original SWAP-NAS paper positioned SWAP inside an evolutionary NAS pipeline. It reports that SWAP-NAS on the DARTS space requires $S$0 GPU days, about $S$1 minutes, on CIFAR-10, with three reported operating points: SWAP-NAS-A at $S$2 test error and $S$3M parameters, SWAP-NAS-B at $S$4 and $S$5M, and SWAP-NAS-C at $S$6 and $S$7M. For direct ImageNet search, the same line of work reports $S$8 GPU days, about $S$9 minutes, and $V \gg S$0 top-1/top-5 test error with $V \gg S$1M parameters (Peng et al., 2024).

A recurring empirical theme is that SWAP’s predictive value is not reducible to model size alone. The regularization experiments explicitly compare Reg. SWAP with regularized #Params and #FLOPs; on NAS-Bench-101 CIFAR-10, regularized #Params and regularized #FLOPs each reach $V \gg S$2, while SWAP is $V \gg S$3 and Reg. SWAP is $V \gg S$4. On NAS-Bench-201 and NAS-Bench-301, however, regularized baselines improve less or not at all, and Reg. SWAP remains strongest overall (Peng et al., 8 May 2026).

5. Related formulations and adjacent uses

Several later frameworks reinterpret or extend SWAP rather than replacing it. L-SWAG, “Layer-Sample Wise Activation with Gradients information,” keeps the activation-pattern core but computes it layer by layer and multiplies it by a gradient-based trainability term: $V \gg S$5 Here $V \gg S$6 is the cardinality of the layer-wise sample-wise activation pattern set, and $V \gg S$7 is a logarithmic inverse-gradient-variance statistic. The work explicitly states that it deploys the cardinality of activation patterns of ReLU and, for the first time, of GeLU networks on a layer-wise partition, and reports an average $V \gg S$8 across benchmarks, compared with $V \gg S$9 for the second-best nontrivial proxy in its summary figure (Casarin et al., 12 May 2025).

A different but closely related direction is conditional activation. “Switchable Activation Networks” introduces input-dependent binary gates

$\mathbb{\hat{A}}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(v)} : \mathbf{p}^{(v)} = \mathds{1}(p_s^{(v)})_{s=1}^{S},~ v \in \{1, \ldots, V\} \right\}.$0

so that each sample induces its own binary activation mask $\mathbb{\hat{A}}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(v)} : \mathbf{p}^{(v)} = \mathds{1}(p_s^{(v)})_{s=1}^{S},~ v \in \{1, \ldots, V\} \right\}.$1. The paper is explicit that this is highly relevant to the broader idea of Sample-Wise Activation Patterns even though it does not use the term SWAP. Its emphasis, however, is on learning activation control for efficient inference and model compression, not on measuring or analyzing the resulting patterns as an independent object (Ale et al., 17 Feb 2026).

Runtime-monitoring work provides an earlier deployment-oriented use of sample-specific activation signatures. “Runtime Monitoring Neuron Activation Patterns” constructs, for each class, a set of binary ReLU patterns extracted from correctly classified training samples and expands that set by Hamming distance into a $\mathbb{\hat{A}}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(v)} : \mathbf{p}^{(v)} = \mathds{1}(p_s^{(v)})_{s=1}^{S},~ v \in \{1, \ldots, V\} \right\}.$2-comfort zone: $\mathbb{\hat{A}}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(v)} : \mathbf{p}^{(v)} = \mathds{1}(p_s^{(v)})_{s=1}^{S},~ v \in \{1, \ldots, V\} \right\}.$3 At runtime, a sample is warned if its activation pattern is not contained in the comfort zone of its predicted class. The robust extension later integrates symbolic perturbation reasoning before abstraction, yielding a monitor that records activation patterns with worst-case guarantees under bounded feature perturbations (Cheng et al., 2018, Cheng, 2020). These works are not SWAP metrics, but they show that sample-wise activation signatures can support runtime support checks and robust acceptance regions.

6. Limitations, misconceptions, and open directions

A common misconception is to treat SWAP as a direct count of sample codes across neurons. In fact, the defining construction counts unique activation-position codes across samples. That distinction is not cosmetic; it is the entire reason the score avoids saturation in the $\mathbb{\hat{A}}_{\mathcal{N},\theta} = \left\{ \mathbf{p}^{(v)} : \mathbf{p}^{(v)} = \mathds{1}(p_s^{(v)})_{s=1}^{S},~ v \in \{1, \ldots, V\} \right\}.$4 regime (Peng et al., 8 May 2026).

A second misconception is to equate high SWAP with unqualified expressive superiority. The activation-region literature gives a more constrained picture: practical activation-pattern diversity at initialization and during training is governed mainly by total neurons and input dimension, and does not exhibit the worst-case depth explosion often associated with ReLU expressivity theory (Hanin et al., 2019). SWAP is therefore better understood as a practical zero-shot signature of realized activation diversity than as a direct measure of worst-case combinatorial capacity.

The empirical limitations are also specific. Raw SWAP can favor large models, which motivates the Gaussian regularizer in NAS. Small CNNs are sensitive to large batch sizes, and Gaussian noise inputs are less informative than real task data. The GLUE task RTE is a reported weak case. These constraints indicate that SWAP is not a wholly architecture- or dataset-invariant statistic, even though it is broader than many earlier zero-shot proxies (Peng et al., 8 May 2026).

Related work also indicates where SWAP may be insufficient on its own. L-SWAG argues that SWAP’s rank consistency can degrade on harder search spaces and on ViTs unless complemented by gradient information and layer selection (Casarin et al., 12 May 2025). SWAN shows that sample-wise activation patterns can also be optimized as operational control signals for efficiency rather than merely measured (Ale et al., 17 Feb 2026). Monitoring and topological analyses suggest richer pattern-level objects than a single cardinality: class-conditional occupancy, Hamming-neighborhood structure, robust interval abstractions, dual-graph spectra, and homology (Cheng, 2020, Bosca et al., 14 Oct 2025).

Related work suggests several open directions. These include layer-wise or token-/head-/block-level variants for larger architectures, stronger class- and domain-conditional analyses of pattern organization, more robust multi-bit abstractions of activation signatures, and extensions beyond current vision and language settings to video and other modalities (Casarin et al., 12 May 2025, Ale et al., 17 Feb 2026). In that broader landscape, SWAP remains the canonical minimal formulation: a label-free, training-free count of distinct thresholded activation responses across samples, designed to expose architecture-dependent expressivity before optimization.