Salience-Affected Neural Networks (SANN)

• SANNs are neural-network formulations where a salience signal explicitly biases learning, representation, or inference, with early models using neuromodulatory tagging and later versions employing human-derived saliency masks.
• They differ from standard attention mechanisms by applying global, non-local modifications to neuron thresholds and weights rather than selectively reweighting features.
• Empirical studies indicate that SANNs enable rapid one-time learning with minimal computational overhead, offering applications in anomaly detection, robotics, and active learning.

Searching arXiv for the cited SANN papers and related formulations. Salience-Affected Neural Networks (SANNs) are neural-network formulations in which a salience signal explicitly alters learning, representation, or inference. In the arXiv literature, the term has been used for two technically distinct families. The earlier family models diffuse neuromodulatory broadcast from limbic or ascending systems to cortex by superimposing a global salience signal on a standard ANN, typically through threshold shifts, activation-function changes, or salience-dependent weight strengthening, and by reading out a reverse salience response during recall (Remmelzwaal et al., 2010, Remmelzwaal et al., 2019). A later usage applies the same label to vision systems trained with human visual saliency masks so that model CAMs align with human-relevant image regions, including an active-learning setting that reduces human saliency annotation while preserving interpretability and accuracy (Boyd et al., 2024). The shared theme is that salience is not treated as an ordinary input feature but as a factor that biases what the network learns, retains, or highlights.

1. Terminological scope and historical development

The 2010 formulation introduced SANN as a simple neural network model combining a locally connected feedforward structure with a layer of undifferentiated connections intended to model diffuse projections from the human limbic system to the cortex. In that setting, the salience signal is global, non-local, and broadcast-like rather than synapse-specific; it shifts effective neuronal thresholds during training and later supports a reverse salience signal at test time (Remmelzwaal et al., 2010). The 2019 work retained this neuromodulatory interpretation, but made the mechanism more explicit by adding a single salience dimension that is applied diffusely to currently active nodes, allowing both activation-function modulation and multiplicative weight strengthening, together with an inference-time salience response and a derived “desire to act” scalar (Remmelzwaal et al., 2019).

The 2024 usage broadens the label substantially. There, salience is not a global affective broadcast but “the regions within an image that humans use to make a classification decision,” represented as per-image saliency masks and enforced through a saliency-alignment loss on CAMs. The network remains “salience-affected” because its internal focus is trained to align with human visual salience, and this salience-guided training is embedded in an active-learning loop (Boyd et al., 2024). A common source of confusion is therefore terminological: in the 2010 and 2019 papers, salience is a scalar neuromodulatory tag; in the 2024 paper, salience is a spatial supervision signal derived from human visual attention.

This dual usage also clarifies what SANN is not. In the neuromodulatory line, it is not ordinary feature attention, because the signal is applied globally to all neurons and alters intrinsic excitability or synaptic strength rather than routing information selectively. In the 2024 line, it is not a new architectural block, because the paper states that no architectural changes are required and that saliency is injected purely via the loss and the CAM derived from the last convolutional layer (Boyd et al., 2024).

2. Diffuse neuromodulatory architectures

In the 2010 model, the base network is a fully connected multilayer perceptron with 49 input units, a single hidden layer of 10 units, and a single scalar output, where the target is the average grayscale of the input image. The 49-dimensional input is derived from parts-based Non-negative Matrix Factorization, using the 49 weights from the mixing matrix $W$ as the MLP inputs. On top of this conventional 49-10-1 MLP, the model adds an undifferentiated salience pathway: a global scalar salience signal $s$ is supplied to every neuron during training, but it is not connected through trainable weight matrices. Instead, it modulates each neuron’s activation threshold, with threshold limits enforced to ensure stability. At test time, each neuron emits a nodal reverse salience signal, and these are summed into a global reverse salience signal that is orthogonal to the task output (Remmelzwaal et al., 2010).

The 2019 architecture follows a different experimental setup but the same biological rationale. It uses a two-stage system: an Encoder compresses $28 \times 28$ inputs to a $4 \times 4$ latent representation, yielding a 16-dimensional vector; this feeds a fully connected SANN with three layers sized 16-16-15. The 15 sigmoid outputs comprise 3 class labels and 12 individual image labels, so salience effects can be measured at both class and individual levels. Salience is represented by a per-node scalar $S_i \in [-1,1]$, initialized to 0, and delivered diffusely across the active pattern. During one-time salience training, the network can strengthen incoming weights in proportion to salience and current activation, modulate activation-function parameters, or both. During inference, the network returns not only class predictions and confidences but also a salience response $R$ and the derived quantity $D$, the “desire to act” (Remmelzwaal et al., 2019).

The biological motivation is explicit in both papers. The 2010 paper grounds the design in widespread diffuse projections from limbic nuclei and monoamine or ascending systems, including dopamine and norepinephrine, that bias neuronal response probabilities and plasticity across large populations, and explicitly relates the mechanism to emotionally tagged memories in the sense of Damásio; it also notes a conceptual relation to GasNets (Remmelzwaal et al., 2010). The 2019 paper makes the mapping more direct by associating positive salience with dopamine and negative salience with noradrenaline, framing one-time learning as strengthening the entire currently active pattern “at one go,” analogous to synaptic tagging and capture and LTP/LTD (Remmelzwaal et al., 2019).

3. Mathematical mechanisms of salience injection and readout

The 2010 SANN defines salience as threshold modulation. For layer $l$, with weight matrix $W^{(l)}$, bias $b^{(l)}$, and threshold vector $s$0, the pre-activation is

$s$1

and the activation is

$s$2

During training, $s$3 is updated according to salience and current activation; during testing it remains fixed as the stored memory trace. The direction term is

$s$4

and the threshold update is clarified as

$s$5

The paper states that “the magnitude of the adjustment factor used in this research was 20% of the distance between the threshold and the threshold limit, in the appropriate direction.” The inference-time nodal reverse salience signal is

$s$6

with global reverse salience

$s$7

This construction makes reverse salience a side-channel readout rather than a replacement for the supervised output (Remmelzwaal et al., 2010).

The 2019 model uses a related but not identical formalism. Its one-time tagging rule updates per-node salience as

$s$8

where $s$9 is the activation of node $28 \times 28$0 when the salient example is presented. Weight strengthening is

$28 \times 28$1

The base activation is the sigmoid

$28 \times 28$2

and the paper explores three salience-induced variants: horizontal offset,

$28 \times 28$3

as well as gradient-change and amplitude-change variants reproduced verbatim in the paper despite incomplete printed expressions. At inference, the salience response is

$28 \times 28$4

and the desire-to-act is

$28 \times 28$5

The paper explicitly states that propagation and bias functions are not affected by salience training (Remmelzwaal et al., 2019).

The 2024 visual-saliency formulation dispenses with threshold or weight tagging and instead inserts salience through a multi-objective loss. For batch size $28 \times 28$6, human saliency map $28 \times 28$7, and model saliency $28 \times 28$8 obtained from CAM for the ground-truth class, the total objective is

$28 \times 28$9

The classification loss is categorical cross-entropy,

$4 \times 4$0

and the saliency alignment term is

$4 \times 4$1

Model saliency for class $4 \times 4$2 is computed as

$4 \times 4$3

then upsampled to input resolution and min-max normalized to $4 \times 4$4. For the “accuracy” model $4 \times 4$5, $4 \times 4$6; for the “interpretability” model $4 \times 4$7, $4 \times 4$8 (Boyd et al., 2024).

4. Training regimes and empirical behavior in biologically inspired SANNs

The 2010 experiments used 200 unique face images from the CBCL dataset, parts-based NMF features, and standard backpropagation for 200 iterations in the 49-10-1 MLP. Hidden-layer size was explored from 2 to 18 units, with 10–13 units reported as fastest to reach $4 \times 4$9 of initial error, while networks smaller than 9 were slow and networks larger than 14 over-specialized. In the multiple-trial setting, selected images such as 9, 10, and 11 received positive salience $S_i \in [-1,1]$0, while other images were neutral with $S_i \in [-1,1]$1. The paper reports that the presence of salience “retards the speed of response,” slightly deforming the learning curve; fastest learning occurs when $S_i \in [-1,1]$2, but the overall shape is not significantly altered. At the same time, similar faces that were not directly tagged, such as images 2 and 3 of the same person, produced elevated reverse salience values, indicating generalization of the salience tag based on feature similarity (Remmelzwaal et al., 2010).

A central empirical claim of the 2010 paper is that reverse salience is effectively independent of the supervised task output. Two networks trained under different salience conditions produced nearly identical outputs, with correlation 0.9998, while the reverse salience side-channel differed. The paper also reports a residual reverse salience baseline even for a control input with all inputs set to 0.5; this baseline is highly correlated with the task output, with correlation coefficient approximately 0.985, so reverse salience must be interpreted relatively rather than absolutely. For one-time learning, the network was first pre-trained with neutral salience, then given a single salient training iteration on selected images. An amplification factor $S_i \in [-1,1]$3 scaled the salience effect. The resulting reverse salience profile after a single salient iteration closely matched the multiple-trial profile, especially as amplification increased, with Figure 1 showing correlation rising from roughly 0.85 toward roughly 0.95 across the tested range (Remmelzwaal et al., 2010).

The 2019 study evaluates a different notion of salience effect: improvement in classification confidence and retrieval-time salience response after a one-time salience tag. The Encoder was trained for 200 epochs and achieved reconstruction accuracy 93.81%, training loss 0.0036, and validation loss 0.0035. The SANN was trained for 500 epochs and achieved 100% classification accuracy from epoch 355, with validation error reported as 0.33695. One-time salience tagging improved classification confidence for the salience-tagged image and for other images in the same class, and in several cases rivaled or exceeded the improvement obtained by continuing standard training from epoch 355 to epoch 500. The paper further reports that increasing salience intensity to $S_i \in [-1,1]$4 and $S_i \in [-1,1]$5 produced stronger confidence improvements across the network, that negative salience produced a symmetrical effect with negative $S_i \in [-1,1]$6, and that sequential positive and negative tagging on different images yielded persistent mixed salience tags (Remmelzwaal et al., 2019).

The computational overhead in the 2019 model is explicitly measured. For 1200 classifications, median inference time was 4000 $S_i \in [-1,1]$7 without salience response and 4001 $S_i \in [-1,1]$8 with salience response; mean inference time was 4171 $S_i \in [-1,1]$9 without $R$0 and 4351 $R$1 with $R$2, corresponding to approximately 4.3% mean overhead. The paper therefore characterizes the computational impact of producing a salience response as minimal. The 2010 formulation makes a similar complexity claim at the algorithmic level: the salience mechanism adds $R$3 per training sample to compute $R$4, $R$5, and threshold updates, which is negligible relative to forward and backward passes (Remmelzwaal et al., 2019, Remmelzwaal et al., 2010).

5. Visual-saliency SANNs and the SAL framework

The 2024 formulation defines visual salience as the regions within an image that humans use to make a classification decision and represents it as a ground-truth binary or continuous saliency mask $R$6 per image. During training, the network’s CAM is aligned to this mask by the saliency loss; during inference, no saliency is supplied, and interpretability is evaluated post hoc by comparing the model CAM to ground-truth saliency on the test set. Because the mechanism is loss-based, no architectural changes are required for either ResNet50 or SwinTransformer, and supplementary experiments report similar trends when CAM is replaced by Grad-CAM, Grad-CAM++, or HiResCAM (Boyd et al., 2024).

The paper operationalizes this formulation in SAL, a dual-model active-learning loop. The model $R$7 is trained with $R$8 so the objective is classification-dominant, while $R$9 is trained with $D$0 so the objective is saliency-dominant. The active-learning parameters are fixed as follows: query size per iteration is 5% of the total training set, total iterations $D$1, change point $D$2, and initial labeled set size 5–10% with at least one sample per class. Before the change point, queried samples receive labels and human saliency; after the change point, they receive labels only, and $D$3 generates saliency masks for the newly selected samples. Six acquisition criteria are studied: Random Sampling, Least Confidence, Entropy Sampling, Margin Sampling, BADGE, and Core-Set (Boyd et al., 2024).

The empirical study spans five public datasets with per-image saliency masks: CUB-200-2011, Flowers102, ImageNet-S, HAM1000, and Food201. Interpretability is measured using the Dice similarity coefficient between binary masks derived from model CAMs and ground-truth saliency, with CAM binarized by taking the top-$D$4 pixels where $D$5 equals the number of positives in the ground-truth mask. Area-under-learning-curve metrics $D$6 and $D$7 summarize performance over active-learning iterations, and each curve is the mean of 8 runs with shaded bands denoting $D$8. The paper reports that saliency incorporation improves interpretability by up to 30% over no-saliency baselines, that SAL matches the interpretability and accuracy of full saliency incorporation using approximately 80% fewer saliency annotations, and that the trends are consistent across all five datasets and six acquisition criteria. It further notes that SwinTransformer starts with lower interpretability than ResNet50 under no saliency, but saliency incorporation via CYBORG boosts interpretability substantially (Boyd et al., 2024).

Several ablations sharpen the interpretation. Baseline B1, which uses classification loss only, has the lowest interpretability; B2, which uses human saliency for all images, functions as an upper bound. TAIT, which trains a teacher model once on human saliency and then freezes it to synthesize saliency for unannotated samples, is outperformed by SAL, which continually updates $D$9 in the active-learning loop. A simplified SAL variant that uses $l$0 itself to generate AI saliency is inferior to the two-model SAL design, and the “No AI Saliency” variant, which stops collecting saliency after the initial 20%, degrades interpretability considerably. Segment Anything masks improve interpretability over the no-saliency baseline but underperform SAL with a small human-annotation budget. The trade-off is computational: the dual-model approach roughly doubles training cost compared with a single model, although the paper reports that SAL remains practical on modern servers and cites hardware consisting of $l$1 NVIDIA RTX A6000 GPUs (Boyd et al., 2024).

6. Relationship to adjacent methods, applications, and unresolved issues

Across the three papers, SANN is repeatedly positioned as distinct from neighboring paradigms. The 2010 and 2019 neuromodulatory models are contrasted with attention mechanisms because attention typically reweights inputs or features through normalized or learned attention maps, whereas salience in these models is a global scalar applied non-locally and affects intrinsic excitability or weight strength rather than routing signals. The 2010 paper further distinguishes SANN from FiLM-like modulation, meta-learning, and memory-augmented networks: FiLM learns feature-wise affine transforms, meta-learning optimizes for rapid parameter updates or episodic reuse, and memory-augmented networks rely on explicit external memory, whereas SANN stores affective tags intrinsically in thresholds. Both neuromodulatory papers compare SANN with GasNets, emphasizing that GasNets model non-local modulation but do not demonstrate explicit salience training and testing with one-time learning in the same way. The 2019 paper also contrasts SANN with Emotional Neural Networks and EMANN, arguing that SANN embeds salience in existing neurons and performs a single-shot post-training update rather than co-training separate emotional components (Remmelzwaal et al., 2010, Remmelzwaal et al., 2019).

The 2024 paper repositions the concept again. There, salience alignment is closer to interpretability supervision than to affective tagging, but the authors still frame the result as a SANN because training and subsequent inference are explicitly affected by salience. This suggests that the term has become an umbrella for architectures or training schemes in which salience exerts a structured influence outside the standard task objective. A plausible implication is that contemporary usage now spans two research programs: biologically inspired non-local modulation and human-saliency-guided interpretability optimization (Boyd et al., 2024).

The application space follows from those different emphases. The 2010 work identifies rapid learning from rare events, anomaly detection and event tagging, memory consolidation markers, reinforcement-learning-like global reward signals, and human-computer interaction as natural settings for one-time affective tagging and reverse salience recall. The 2019 paper points toward robotics and cognitive architectures through the “desire to act” scalar and explicitly mentions NEUCOGAR as a possible integration target. The 2024 paper extends the framework beyond vision, proposing that in text salience could be human rationales or token highlights and in audio it could be expert-provided salient time-frequency regions, with the general template remaining $l$2 once suitable saliency probes are defined (Remmelzwaal et al., 2010, Remmelzwaal et al., 2019, Boyd et al., 2024).

The unresolved issues are likewise formulation-specific. For the 2010 threshold-based SANN, broader tasks and larger architectures such as CNNs and RNNs remain to be investigated; threshold modulation is only a simplified proxy for neuromodulatory biology; and the residual reverse-salience baseline, correlated with task output, complicates absolute calibration. The same paper notes that high amplification could over-commit thresholds to particular instances, although bounded limits mitigate drift. The 2019 model does not specify how salience is triggered, includes no explicit decay term, and raises the possibility that overlapping positive and negative salience tags could conflict; it is also demonstrated only on a small fully connected network and simple datasets. The 2024 visual-saliency framework fixes the change point at 20% of the training data, increases compute by training two models, depends on the quality and representativeness of saliency masks, and reports no formal statistical significance tests despite showing means and $l$3 over 8 runs. Taken together, these limitations indicate that SANN remains a heterogeneous research area unified more by the idea of salience-conditioned learning than by a single canonical architecture (Remmelzwaal et al., 2010, Remmelzwaal et al., 2019, Boyd et al., 2024).