Memory Aware Synapses (MAS) for Lifelong Learning

• Memory Aware Synapses (MAS) is a continual learning method that quantifies neural parameter sensitivity to preserve key knowledge and reduce catastrophic forgetting.
• It computes importance measures online from unsupervised data by analyzing gradients of the output function, enabling updates without label dependency.
• MAS integrates a regularization term during new task learning to selectively protect important parameters, showing state-of-the-art performance in experiments.

Searching arXiv for the MAS paper and closely related continual-learning context. arXiv search query: (Aljundi et al., 2017) Memory Aware Synapses (MAS) is a novel approach for lifelong learning that computes the importance of the parameters of a neural network in an unsupervised and online manner. Given a new sample fed to the network, MAS accumulates an importance measure for each parameter based on how sensitive the predicted output function is to a change in that parameter. When learning a new task, changes to important parameters are penalized, effectively preventing important knowledge related to previous tasks from being overwritten. The method is motivated by the claim that, given limited model capacity and unlimited new information to be learned, knowledge has to be preserved or erased selectively rather than only accumulated across tasks (Aljundi et al., 2017).

1. Problem setting and rationale

MAS is framed around continuous learning under finite capacity. The motivating premise is that humans can learn in a continuous manner: old rarely utilized knowledge can be overwritten by new incoming information, while important, frequently used knowledge is prevented from being erased. In artificial learning systems, lifelong learning had focused mainly on accumulating knowledge over tasks and overcoming catastrophic forgetting. MAS argues for a complementary requirement: selective preservation and selective erasure, conditioned by how much the learned function depends on individual parameters (Aljundi et al., 2017).

The central design choice is to measure importance through the learned function itself rather than through labels. In MAS, the importance measure is derived from sensitivity of the output function to parameter perturbations. Because this sensitivity does not require supervision, the method can update parameter importance on any stream of inputs. This is the basis for the paper’s claim that MAS can adapt importance toward what the network needs not to forget, including under test conditions that differ from the original training conditions (Aljundi et al., 2017).

2. Derivation of the parameter-importance measure

Let $\theta \in \mathbb{R}^P$ be the vector of all network parameters $\{\theta_i\}$. After finishing task $T$, the network has learned a function

$F(x;\theta) \approx \bar{F}(x).$

MAS asks how sensitive the output of $F$ is to a small change $\delta$ in one parameter $\theta_i$. Using a first-order Taylor expansion for one input $x_k$,

$$F(x_k;\theta+\delta) - F(x_k;\theta) \simeq \sum_{i=1}^P g_i(x_k)\cdot \delta_i,$$

where

$$g_i(x_k) = \frac{\partial F(x_k;\theta)}{\partial \theta_i}.$$

To quantify how much a small change $\{\theta_i\}$0 affects $\{\theta_i\}$1 over all observed inputs $\{\theta_i\}$2, MAS defines

$\{\theta_i\}$3

In practice, $\{\theta_i\}$4 often has vector output. Rather than summing one $\{\theta_i\}$5 per output dimension and back-propagating once per output, MAS uses the gradient of the squared $\{\theta_i\}$6-norm of the output:

$\{\theta_i\}$7

Then $\{\theta_i\}$8 remains as in (1), with a single backward pass per $\{\theta_i\}$9 (Aljundi et al., 2017).

The interpretation given in the paper is direct: large $T$0 means $T$1 depends strongly on $T$2, hence $T$3 is important to preserve. A common misunderstanding is to treat MAS importance as a property of the supervised objective; in the formulation above, the quantity is tied to the learned function’s sensitivity.

3. Online and unsupervised accumulation

Because $T$4 does not depend on any labels or loss gradients, MAS can accumulate $T$5 on any stream of inputs, including training, validation, test, and unlabeled data. The online accumulation procedure is:

$\{\theta_i\}$06

When finishing task $T$6, one may store the current model $T$7 and its $T$8. When proceeding to $T$9, one can add new contributions to $F(x;\theta) \approx \bar{F}(x).$0 from additional unlabeled data, such as test-time data, and keep a running total

$F(x;\theta) \approx \bar{F}(x).$1

This online accumulation is one of the method’s distinctive features: importance can be revised by exposure to new input streams without requiring new labels (Aljundi et al., 2017).

4. Regularization during learning of a new task

Suppose a new task $F(x;\theta) \approx \bar{F}(x).$2 is given with supervised loss $F(x;\theta) \approx \bar{F}(x).$3, for example cross-entropy. To avoid overwriting parameters important to previous tasks, MAS uses the regularized objective

$F(x;\theta) \approx \bar{F}(x).$4

Here $F(x;\theta) \approx \bar{F}(x).$5 denotes the parameters at the end of $F(x;\theta) \approx \bar{F}(x).$6, and $F(x;\theta) \approx \bar{F}(x).$7 trades off plasticity versus stability (Aljundi et al., 2017).

The role of the quadratic term is parameter-specific. A large $F(x;\theta) \approx \bar{F}(x).$8 penalizes deviation of $F(x;\theta) \approx \bar{F}(x).$9 from $F$0 and therefore protects that parameter. A small $F$1 leaves more freedom for adaptation. In this sense, MAS instantiates selective preservation rather than uniform freezing. This is the mechanism through which the method operationalizes the claim that some knowledge should be preserved and some knowledge may be overwritten.

5. Local MAS and the connection to Hebb’s rule

The paper also describes a local version of MAS. Instead of measuring sensitivity of the global function $F$2, the same idea can be applied per layer. Consider layer $F$3 with parameters $F$4 connecting neuron $F$5 in layer $F$6 with activation $F$7 to neuron $F$8 in layer $F$9 with activation $\delta$0. Let

$\delta$1

The sensitivity of $\delta$2 to a change in $\delta$3 is written as

$\delta$4

For ReLU activations, and ignoring the non-differentiable kink at zero by subgradient, one obtains

$\delta$5

Averaging over $\delta$6 samples gives

$\delta$7

Up to the constant $\delta$8, this is exactly a Hebbian rule: strengthen connections between neurons whose activations are highly correlated (Aljundi et al., 2017).

This connection is important conceptually. MAS is not presented only as a global sensitivity measure; it also admits a layer-local form with an explicit correlation structure. The paper uses this to relate the method to neuroplasticity.

6. Experimental results

The reported experiments cover a sequence of object recognition tasks and the problem of learning an embedding for predicting $\delta$9subject, predicate, object$\theta_i$0 triplets. The paper reports state-of-the-art performance and emphasizes the ability to adapt parameter importance from unlabeled data (Aljundi et al., 2017).

For object recognition, the model is AlexNet pretrained on ImageNet. On two-task sequences—Scene$\theta_i$1Birds, Birds$\theta_i$2Scene, Flower$\theta_i$3Birds, and Flower$\theta_i$4Scene—the paper reports the following Task 1 retention and Task 2 accuracy:

Method	Task 1 Acc (drop %)	Task 2 Acc
FineTune	45.2 (-8.0)	57.8
LwF	51.7 (-2.0)	55.6
EWC	52.2 (-1.4)	55.7
SI	52.6 (-1.0)	55.9
MAS (ours)	53.2 (-0.4)	55.0

The reported interpretation is that MAS has by far the smallest forgetting, with $\theta_i$5 drop versus $\theta_i$6--$\theta_i$7 for the listed alternatives.

On the eight-task sequence Flower$\theta_i$8Scenes$\theta_i$9Birds$x_k$0Cars$x_k$1Aircraft$x_k$2Actions$x_k$3Letters$x_k$4SVHN, the average end-of-sequence accuracy is:

Method	Average end-of-sequence accuracy
FineTune	32.7%
LwF	49.5%
IMM	43.4%
SI	50.5%
MAS	52.7%

For the same eight-task sequence, average forgetting is reported as MAS $x_k$5 drop versus SI $x_k$6--$x_k$7 drop. The memory cost statement is also explicit: MAS only stores one $x_k$8-matrix of same size as $x_k$9 (constant per task), whereas methods like IMM/EWC store one matrix per task.

For fact learning, the model is VGG-16 and the setting is a four-task random split of 186 facts from the 6DS dataset. The metric is mean-average-precision (mAP) for retrieving images of each task at end of sequence:

Method	Evaluation	mAP
FineTune	$$F(x_k;\theta+\delta) - F(x_k;\theta) \simeq \sum_{i=1}^P g_i(x_k)\cdot \delta_i,$$0	0.19
FineTune	$$F(x_k;\theta+\delta) - F(x_k;\theta) \simeq \sum_{i=1}^P g_i(x_k)\cdot \delta_i,$$1	0.19
FineTune	$$F(x_k;\theta+\delta) - F(x_k;\theta) \simeq \sum_{i=1}^P g_i(x_k)\cdot \delta_i,$$2	0.28
FineTune	$$F(x_k;\theta+\delta) - F(x_k;\theta) \simeq \sum_{i=1}^P g_i(x_k)\cdot \delta_i,$$3	0.71
FineTune	overall	0.18
SI	$$F(x_k;\theta+\delta) - F(x_k;\theta) \simeq \sum_{i=1}^P g_i(x_k)\cdot \delta_i,$$4	0.36
SI	$$F(x_k;\theta+\delta) - F(x_k;\theta) \simeq \sum_{i=1}^P g_i(x_k)\cdot \delta_i,$$5	0.32
SI	$$F(x_k;\theta+\delta) - F(x_k;\theta) \simeq \sum_{i=1}^P g_i(x_k)\cdot \delta_i,$$6	0.38
SI	$$F(x_k;\theta+\delta) - F(x_k;\theta) \simeq \sum_{i=1}^P g_i(x_k)\cdot \delta_i,$$7	0.68
SI	overall	0.25
MAS (ours)	$$F(x_k;\theta+\delta) - F(x_k;\theta) \simeq \sum_{i=1}^P g_i(x_k)\cdot \delta_i,$$8	0.42
MAS (ours)	$$F(x_k;\theta+\delta) - F(x_k;\theta) \simeq \sum_{i=1}^P g_i(x_k)\cdot \delta_i,$$9	0.37
MAS (ours)	$$g_i(x_k) = \frac{\partial F(x_k;\theta)}{\partial \theta_i}.$$0	0.41
MAS (ours)	$$g_i(x_k) = \frac{\partial F(x_k;\theta)}{\partial \theta_i}.$$1	0.65
MAS (ours)	overall	0.29

The paper summarizes these numbers by stating that MAS improves $$g_i(x_k) = \frac{\partial F(x_k;\theta)}{\partial \theta_i}.$$2 mAP over SI ($$g_i(x_k) = \frac{\partial F(x_k;\theta)}{\partial \theta_i}.$$3 versus $$g_i(x_k) = \frac{\partial F(x_k;\theta)}{\partial \theta_i}.$$4) and $$g_i(x_k) = \frac{\partial F(x_k;\theta)}{\partial \theta_i}.$$5 over FineTune.

A specialization and adaptation test is also reported. A small “sports” subset of facts from Task 1 is defined, and $$g_i(x_k) = \frac{\partial F(x_k;\theta)}{\partial \theta_i}.$$6 is computed only on that subset, using unlabeled data. At the end of the four-task sequence, mAP on the sports subset is approximately $$g_i(x_k) = \frac{\partial F(x_k;\theta)}{\partial \theta_i}.$$7 for FineTune, approximately $$g_i(x_k) = \frac{\partial F(x_k;\theta)}{\partial \theta_i}.$$8 for SI, and approximately $$g_i(x_k) = \frac{\partial F(x_k;\theta)}{\partial \theta_i}.$$9 for MAS(adaptive). The paper describes this as MAS learning what not to forget for that subset, boosting mAP from $\{\theta_i\}$00 to $\{\theta_i\}$01 (Aljundi et al., 2017).

7. Clarifications, scope, and implications

Several points in the paper address likely misunderstandings. First, MAS importance is not tied to labels: $\{\theta_i\}$02 does not depend on any labels or loss gradients, and $\{\theta_i\}$03 can be accumulated on any stream of inputs, including unlabeled data. Second, computing $\{\theta_i\}$04 on test images versus train versus train+test gives nearly identical forgetting on the two-task object-recognition setting, which the paper presents as confirmation that MAS can use any unlabeled data. Third, the local form is not only loosely inspired by Hebbian learning; up to a constant factor of $\{\theta_i\}$05, equation (3) is exactly a Hebbian correlation rule (Aljundi et al., 2017).

The reported significance is twofold. Empirically, the method is presented as yielding state-of-the-art continual-learning performance on the reported object-recognition and fact-learning experiments. Methodologically, it is presented as the first demonstration of adapting the importance of parameters based on unlabeled data toward what the network needs not to forget, with the added observation that this target may vary depending on test conditions. A plausible implication is that MAS is particularly suited to settings where the input distribution available after training is informative about which previously learned behavior should remain stable.