Activation Maximization in Neural Networks

• Activation Maximization is a set of optimization techniques that synthesize inputs to maximally activate selected neurons, revealing each unit's preferred stimulus.
• It employs regularizers and parameterizations, including Fourier transforms and generator-based priors, to produce human-interpretable images and control adversarial artifacts.
• The method is pivotal for model diagnosis, comparing architecture biases, and extending interpretability to domains like medical imaging, time series, and point clouds.

Activation Maximization (AM), also called Feature Visualization via Optimization, is a family of methods for synthesizing an input that maximizes the response of a selected neuron, channel, logit, or feature in a fixed neural network. In its most direct form, AM asks what input pattern a trained model “prefers,” and interprets the resulting synthetic input as the preferred stimulus of the target unit. The method is rooted in an analogy to neuroscience, where preferred stimuli are used to probe biological neurons, and in deep learning it has become a standard instrument for feature visualization, model diagnosis, and representational analysis. Across the recent literature, AM appears both as a pixel-space gradient-ascent procedure and as a broader optimization paradigm that can incorporate priors, generators, latent-space search, Fourier parameterizations, and black-box optimization, with applications extending from CNNs and Vision Transformers to point clouds, time series, spiking neural networks, medical imaging, and DNN-based brain encoding models (Zhu et al., 10 Aug 2025, Nguyen et al., 2019).

1. Canonical objective and optimization problem

The canonical AM problem fixes a pretrained network and asks for an input $x$ that maximizes a scalar activation. With $f_i(x)$ denoting the activation of unit $i$, or $f^\ell_t(x)$ the activation of unit $t$ in layer $\ell$, the basic objective is

$x^* = \arg\max_x f^\ell_t(x),$

or, with an explicit regularizer,

$$x^* = \arg\max_x \bigl[f_i(x) - \lambda R(x)\bigr].$$

Equivalently, one may write a minimization objective $L(x) = -f^\ell_t(x) + \lambda R(x)$ and perform gradient descent on $L$, which is the same as gradient-ascent on the activation with a penalty term. In the unconstrained case, a standard update is

$f_i(x)$0

and with a differentiable regularizer one obtains

$f_i(x)$1

If the procedure uses non-differentiable operators such as Gaussian blur, the update may instead take the form $f_i(x)$2. Stopping criteria are typically a fixed number of iterations or a pre-set activation threshold (Nguyen et al., 2019).

This formulation is non-convex. The literature therefore treats AM not as a guaranteed global optimizer but as a local search method whose output depends on initialization, parameterization, the chosen target, and the imposed prior. A closely related distinction is between optimization-based AM, which directly synthesizes an input, and dataset-driven AM, which ranks held-out examples by activation and inspects the top-$f_i(x)$3 natural inputs for a neuron or channel. The optimization-based branch is the canonical form for feature visualization, while the dataset-driven branch is often used for comparison, validation, or attack analysis (Nanfack et al., 2023).

2. Priors, parameterizations, and probabilistic interpretations

Unregularized AM is well known to produce high-frequency “fooling” or “rubbish” patterns that humans cannot interpret. The central technical problem is therefore not merely maximizing activation, but doing so while constraining the search toward a human-interpretable image manifold. The most common hand-designed regularizers are $f_i(x)$4 decay, total variation, Gaussian or bilateral blurring, jitter, random rotation or scaling, and patch- or color-statistics matching. A standard regularizer is

$f_i(x)$5

with

$f_i(x)$6

The $f_i(x)$7 term keeps pixel values from growing unbounded, while total variation penalizes rapid intensity changes and reduces checkerboard artifacts (Zhu et al., 10 Aug 2025, Nguyen et al., 2019).

A major line of development replaces direct pixel optimization with alternative parameterizations. In frequency-domain AM, the input is represented as $f_i(x)$8, where $f_i(x)$9 is a complex spectrum. Optimizing $i$0 rather than pixels suppresses high-frequency noise and encourages smooth, coherent images. The 2025 CNN/ViT framework gives an explicit Fourier-domain algorithm: initialize a complex spectrum $i$1, repeatedly form $i$2 using random jittering, scaling, or rotation, compute the target activation, and update $i$3 by gradient ascent (Zhu et al., 10 Aug 2025). A more restrictive alternative is Deep Generator Network AM, which optimizes a latent code $i$4 of a generator $i$5:

$i$6

This shifts the prior from hand-designed pixel regularization to a learned generative manifold, and subsequent Plug and Play Generative Networks add a learned prior $i$7 via a denoising autoencoder and injected noise to improve diversity (Nguyen et al., 2019).

The survey literature also gives a probabilistic interpretation: AM can be viewed as approximate sampling from the posterior $i$8, where $i$9 means that neuron $f^\ell_t(x)$0 is selected. Under Bayes’ rule,

$f^\ell_t(x)$1

and a Metropolis-adjusted Langevin style update combines three forces: an activation term, a prior term, and injected noise. Within that framework, using $f^\ell_t(x)$2 is the exact gradient of $f^\ell_t(x)$3 and is reported to work robustly in practice. This interpretation unifies no-prior AM, Gaussian-prior AM, hand-designed priors, and generator-latent AM under a single posterior-sampling view (Nguyen et al., 2019).

3. Target units, hierarchical structure, and architecture-specific formulations

AM was initially emphasized for output-layer neurons in CNNs, typically by maximizing a pre-softmax logit $f^\ell_t(x)$4 for class $f^\ell_t(x)$5, but more recent work extends it to intermediate layers and to transformer architectures. For intermediate convolutional layers, a channel $f^\ell_t(x)$6 may be treated as a neuron by spatially averaging its feature map,

$f^\ell_t(x)$7

For Vision Transformers, which process images as token sequences, a hidden dimension $f^\ell_t(x)$8 at layer $f^\ell_t(x)$9 can be visualized by averaging the hidden-state coordinate across tokens,

$t$0

The 2025 framework explicitly presents this as a unified feature-visualization pipeline for both CNNs and ViTs, with the same AM machinery applied after architecture-specific target extraction (Zhu et al., 10 Aug 2025).

Intermediate-layer AM makes hierarchical structure visually explicit. In MobileNet, early layers yield Gabor-like edges, mid-level layers yield periodic textures such as honeycombs and grids, and deeper layers yield object-part prototypes. In ResNet50V2, deeper blocks show coherent shape primitives and compositional object-part forms, a behavior attributed in the paper to residual connections. In ViTs, early layers yield fine textures, middle layers form token-group patterns, and deeper shallow layers already show holistic, global abstractions (Zhu et al., 10 Aug 2025).

Output-layer visualizations also expose architectural differences. For class-specific logits such as ladybug, butterfly, or pineapple, CNNs produce more localized, prototypical object parts, whereas ViT produces diffuse, abstract patterns. In the reported examples, the ladybug neuron produces red-black spotted patterns, butterfly produces orange-black wing textures, and pineapple produces clustered yellow-green motifs across MobileNet, InceptionV3, ResNet50V2, and ViT-B/16. This suggests that AM is not merely a way to inspect single neurons, but also a tool for comparing the inductive biases of distinct architecture families under a common optimization objective (Zhu et al., 10 Aug 2025).

4. Extensions beyond standard image CNNs

AM now spans settings in which gradients are available, unavailable, unreliable, or where the input domain is not an image. In black-box biological or proprietary systems, gradients may be inaccessible. XDream addresses this by combining a trained generator $t$1 with a genetic algorithm over latent codes. Candidate codes are evaluated by the scalar response $t$2, the top-performing codes are selected, recombined by crossover, and mutated by Gaussian noise. In silico tests on ConvNet units showed that XDream is applicable across network layers, architectures, and training sets; that its convergence curves rise rapidly over the first $t$3 generations and saturate by 300–500 generations; and that it is robust under simulated Poisson-like variability, outperforming finite-difference gradient approximation and natural evolution strategies (Xiao et al., 2019).

In spiking neural networks, Pospelov et al. formulate AM over the latent space of SN-GAN or VQ-VAE generators and optimize a low-rank Tensor Train approximation of the discretized activation map. Their PROTES-based approach is explicitly gradient-free, uses only black-box spike-count responses, and was reported to find near-optimal latent codes in 2–4× fewer queries than a strong evolutionary benchmark. Early-layer most-exciting images show simple color and stripe patterns, while deeper layers become semantically richer and more variable across runs (Pospelov et al., 2023).

For univariate time series, Sequence Dreaming adapts AM to sequential inputs by combining a fidelity term to a target activation with an $t$4 penalty, total variation, and sequence-wide smoothing. The framework supports two modes: centered AM, which targets the mean activation of a class and produces a “typical” class prototype, and border AM, which targets a larger-than-observed activation and pushes a sequence toward the decision boundary. The reported regularization toolbox includes Gaussian smoothing, exponential smoothing, and random restarts to keep synthesized sequences realistic (Schlegel et al., 2024).

For point clouds, Flow AM avoids both image-style regularizers and external generative models. It defines class-specific activation flows as layerwise activation centroids and then adds a latent-alignment loss that matches a generated point cloud’s intermediate activations to the class flow while also imposing continuity-and-smoothness and legal-restriction losses. The paper argues that generative-model AM for point clouds fails sanity checks because silhouettes remain intact even when 50% of classifier weights are randomly dropped, whereas Flow AM collapses into incoherence under such dropout, indicating stronger dependence on the classifier itself (Tan, 2024).

AM has also been adapted to brain encoding and medical imaging. Voxel-weighted AM applies activation maximization to DNN-based fMRI encoding models by optimizing an image against a voxel-specific linear readout over multi-layer Inception V3 features, regularized by total variation, image-gradient smoothness, and Fourier-spectrum priors. ROI-level AM images were then shown to human participants in an fMRI study, where target-region responses were significantly higher for matching AM images than for nonmatching ones in several ROIs (Shinkle et al., 4 Jun 2025). In medical imaging, Cycle-Consistent Activation Maximization uses paired generators and discriminators, cycle-consistency, similarity losses, and a classifier-driven AM loss to produce high-quality decision explanations on LIDC-IDRI and BreastMNIST, outperforming DeepSHAP and LRP in a user study on the medical datasets while ranking second in the natural-image setting (Katzmann et al., 2020).

5. Diagnostic, adversarial, and control uses

Although AM is primarily an interpretability method, the same optimization machinery can be repurposed to expose vulnerabilities and decision boundaries. One recent formulation searches for an adversarial image $t$5 near an original image $t$6 by maximizing a target logit under an $t$7 constraint and a total-variation penalty on the perturbation:

$t$8

The corresponding projected gradient-ascent loop alternates an ascent step on the objective with clipping to both the valid pixel range and the $t$9-ball around the original input. In reported experiments, perturbations were visually different across model families—grid-like for ViT and more texture-like for CNN—and targeted misclassifications included examples such as ladybug $\ell$0 butterfly (Zhu et al., 10 Aug 2025).

The older survey literature already framed AM as a debugging and explanation tool. Applications include revealing output-class preferences, hidden-unit selectivity for edges, textures, or object parts, uncovering multifaceted units through pairwise-unit activation or clustering-based initialization, discovering adversarial or fooling patterns, diagnosing color-space bugs such as RGB $\ell$1 BRG swaps, and surfacing learned biases, as in a captioning network that always “sees a bird on a branch” even when none is present. The same framework has also been used to monitor feature evolution during training and to condition synthesis on captions or segmentation maps (Nguyen et al., 2019).

AM can moreover be used for direct control rather than passive visualization. Voxel-weighted AM does not merely visualize cortical selectivity; it generates synthetic stimuli that reliably drive activity in targeted visual ROIs across both low- and high-level visual areas and across subjects. This suggests a transition from “what does a unit prefer?” to “what input can modulate a desired response?” within the same mathematical framework (Shinkle et al., 4 Jun 2025).

6. Reliability, manipulation, limitations, and open problems

A recurring misconception is that an AM image is a transparent readout of a neuron’s “true concept.” The literature instead presents AM as an optimization-dependent probe whose outputs can be distorted by priors, optimization failures, off-distribution effects, or adversarial model modification. The 2024 Gradient Slingshots work shows that feature visualizations can be manipulated without modifying the model architecture or significantly degrading performance. By shaping a manipulated activation landscape in a slingshot zone around the initialization and a landing zone around an attacker-chosen target, AM can be coerced to converge to a predefined visualization. The same paper proposes defenses that modify the AM procedure rather than the model: gradient clipping, transformation robustness, Adam or AdamW instead of vanilla ascent, and natural AM signals based on top-$\ell$2 real images (Bareeva et al., 2024).

A separate line of work studies adversarial attacks on AM interpretations by fine-tuning network parameters so that overall predictive behavior remains almost unchanged while the top-$\ell$3 examples or synthetic AM outputs drift. For dataset-driven AM, this drift is quantified using Kendall-$\ell$4 between ranked activation lists and a CLIP-$\ell$5 measure of semantic displacement. Reported attacks include push-down attacks that erase original top-$\ell$6 exemplars, push-up attacks that make decoy images such as goldfish dominate top-$\ell$7 sets, and fairwashing attacks that reduce apparent feature-visualization bias while leaving downstream bias measures nearly unchanged. A notable empirical observation is that synthetic AM and natural top-$\ell$8 AM can be largely decoupled (Nanfack et al., 2023).

There are also optimization-pathology objections internal to AM itself. Linse et al. argue that standard AM can fail even for simple functions containing ReLUs or Leaky ReLUs, because zero or small negative slopes create sparse gradients and shallow traps. Their proposed ProxyGrad algorithm keeps the original forward network but computes gradients through a proxy copy with a larger backward negative slope, thereby producing denser gradients and higher maxima. On ImageNet-pretrained ResNet-18, the reported mean class-score after 10 iterations increases from $\ell$9 for a ReLU baseline to $x^* = \arg\max_x f^\ell_t(x),$0 for ReLU+ProxyGrad, and after 500 iterations from $x^* = \arg\max_x f^\ell_t(x),$1 to $x^* = \arg\max_x f^\ell_t(x),$2 (Linse et al., 2024).

Beyond attacks and optimization failure, several broader limitations recur. Large ViTs present more complex optimization landscapes, with gradients that may be less informative and AM that converges more slowly or to less-interpretable images. Even well-regularized visualizations can be hard for humans to interpret, creating a semantic gap between synthesized stimuli and verbalizable concepts. AM also probes units in isolation, whereas many modern networks encode concepts across populations of units; this motivates work on compositionality, higher-order interactions, and distributed representations. The survey literature further identifies open problems in evaluating fidelity, generalizing AM across architectures and datasets without biased priors, combining AM with attribution heatmaps or concept activation vectors, and extending AM to richer input spaces such as 3D renderers, video, and language (Zhu et al., 10 Aug 2025, Nguyen et al., 2019).

Taken together, these results position Activation Maximization as both a powerful and a fragile interpretability technique. It is powerful because it offers a direct, differentiable, and increasingly general route to preferred-stimulus synthesis across modalities and model classes. It is fragile because its products depend on priors, parameterizations, optimizers, and model integrity, and because faithful-looking visualizations can be fabricated. The present research trajectory therefore treats AM not as a standalone oracle, but as a flexible optimization framework whose interpretive value depends on regularization design, sanity checks, robustness analysis, and complementary evidence from other interpretability methods (Tan, 2024, Bareeva et al., 2024).