--- title: Memory Aware Synapses (MAS) for Lifelong Learning url: https://www.emergentmind.com/topics/memory-aware-synapses-mas type: topic --- # Memory Aware Synapses (MAS) for Lifelong Learning Searching arXiv for the MAS paper and closely related continual-learning context. arXiv search query: 1711.09601 Memory Aware Synapses (MAS) is a novel approach for lifelong learning that computes the importance of the parameters of a neural network in an unsupervised and online manner. Given a new sample fed to the network, MAS accumulates an importance measure for each parameter based on how sensitive the predicted output function is to a change in that parameter. When learning a new task, changes to important parameters are penalized, effectively preventing important knowledge related to previous tasks from being overwritten. The method is motivated by the claim that, given limited model capacity and unlimited new information to be learned, knowledge has to be preserved or erased selectively rather than only accumulated across tasks [1711.09601]. ## 1. Problem setting and rationale MAS is framed around continuous learning under finite capacity. The motivating premise is that humans can learn in a continuous manner: old rarely utilized knowledge can be overwritten by new incoming information, while important, frequently used knowledge is prevented from being erased. In artificial learning systems, lifelong learning had focused mainly on accumulating knowledge over tasks and overcoming catastrophic forgetting. MAS argues for a complementary requirement: selective preservation and selective erasure, conditioned by how much the learned function depends on individual parameters [1711.09601]. The central design choice is to measure importance through the learned function itself rather than through labels. In MAS, the importance measure is derived from sensitivity of the output function to parameter perturbations. Because this sensitivity does not require supervision, the method can update parameter importance on any stream of inputs. This is the basis for the paper’s claim that MAS can adapt importance toward what the network needs not to forget, including under test conditions that differ from the original training conditions [1711.09601]. ## 2. Derivation of the parameter-importance measure Let $\theta \in \mathbb{R}^P$ be the vector of all network parameters $\{\theta_i\}$. After finishing task $T$, the network has learned a function $$ F(x;\theta) \approx \bar{F}(x). $$ MAS asks how sensitive the output of $F$ is to a small change $\delta$ in one parameter $\theta_i$. Using a first-order Taylor expansion for one input $x_k$, $$ F(x_k;\theta+\delta) - F(x_k;\theta) \simeq \sum_{i=1}^P g_i(x_k)\cdot \delta_i, $$ where $$ g_i(x_k) = \frac{\partial F(x_k;\theta)}{\partial \theta_i}. $$ To quantify how much a small change $\delta_i$ affects $F$ over all observed inputs $\{x_k\}$, MAS defines $$ \Omega_i = \frac{1}{N}\sum_{k=1}^N \|g_i(x_k)\|. \tag{1} $$ In practice, $F(x)$ often has vector output. Rather than summing one $g_i$ per output dimension and back-propagating once per output, MAS uses the gradient of the squared $\ell_2$-norm of the output: $$ g_i(x_k) = \frac{\partial \|F(x_k;\theta)\|_2^2}{\partial \theta_i} = \frac{\partial \left(\sum_j [F_j(x_k;\theta)]^2\right)}{\partial \theta_i}. $$ Then $\Omega_i$ remains as in (1), with a single backward pass per $x_k$ [1711.09601]. The interpretation given in the paper is direct: large $|g_i|$ means $F$ depends strongly on $\theta_i$, hence $\theta_i$ is important to preserve. A common misunderstanding is to treat MAS importance as a property of the supervised objective; in the formulation above, the quantity is tied to the learned function’s sensitivity. ## 3. Online and unsupervised accumulation Because $g_i(x)$ does not depend on any labels or loss gradients, MAS can accumulate $\Omega_i$ on any stream of inputs, including training, validation, test, and unlabeled data. The online accumulation procedure is: ```text Initialize Ωᵢ ← 0, count ← 0 For each new sample x: count ← count + 1 Compute y = F(x;θ) Compute g = ∂‖y‖₂²/∂θ via one backward pass For each parameter i: Ωᵢ ← Ωᵢ + |gᵢ| End At any point, final Ωᵢ ← Ωᵢ / count ``` When finishing task $T_n$, one may store the current model $\theta^*$ and its $\Omega^{T_n}$. When proceeding to $T_{n+1}$, one can add new contributions to $\Omega$ from additional unlabeled data, such as test-time data, and keep a running total $$ \Omega_{\text{total}} \leftarrow \Omega^{ 0$ trades off plasticity versus stability [1711.09601]. The role of the quadratic term is parameter-specific. A large $\Omega_i$ penalizes deviation of $\theta_i$ from $\theta_i^*$ and therefore protects that parameter. A small $\Omega_i$ leaves more freedom for adaptation. In this sense, MAS instantiates selective preservation rather than uniform freezing. This is the mechanism through which the method operationalizes the claim that some knowledge should be preserved and some knowledge may be overwritten. ## 5. Local MAS and the connection to Hebb’s rule The paper also describes a local version of MAS. Instead of measuring sensitivity of the global function $F$, the same idea can be applied per layer. Consider layer $\ell$ with parameters $\theta_{ij}$ connecting neuron $i$ in layer $\ell-1$ with activation $y_i$ to neuron $j$ in layer $\ell$ with activation $y_j$. Let $$ y = \operatorname{ReLU}(W y_{\text{prev}}). $$ The sensitivity of $\|y\|_2^2$ to a change in $W_{ij}$ is written as $$ \Delta\|y\|_2^2 \simeq \sum_{i,j} g_{ij}\delta_{ij}, \qquad g_{ij} = \frac{\partial \|y\|_2^2}{\partial W_{ij}}. $$ For ReLU activations, and ignoring the non-differentiable kink at zero by subgradient, one obtains $$ g_{ij}(x) = 2\, y_i^{\ell-1}(x)\cdot y_j^{\ell}(x). $$ Averaging over $N$ samples gives $$ \Omega_{ij} = \frac{1}{N}\sum_{k=1}^N g_{ij}(x_k) = 2\cdot \frac{1}{N}\sum_{k=1}^N y_i^{\ell-1}(x_k)\,y_j^{\ell}(x_k). \tag{3} $$ Up to the constant $2$, this is exactly a Hebbian rule: strengthen connections between neurons whose activations are highly correlated [1711.09601]. This connection is important conceptually. MAS is not presented only as a global sensitivity measure; it also admits a layer-local form with an explicit correlation structure. The paper uses this to relate the method to neuroplasticity. ## 6. Experimental results The reported experiments cover a sequence of object recognition tasks and the problem of learning an embedding for predicting $<$subject, predicate, object$>$ triplets. The paper reports state-of-the-art performance and emphasizes the ability to adapt parameter importance from unlabeled data [1711.09601]. For object recognition, the model is AlexNet pretrained on ImageNet. On two-task sequences—Scene$\rightarrow$Birds, Birds$\rightarrow$Scene, Flower$\rightarrow$Birds, and Flower$\rightarrow$Scene—the paper reports the following Task 1 retention and Task 2 accuracy: | Method | Task 1 Acc (drop %) | Task 2 Acc | |---|---:|---:| | FineTune | 45.2 (-8.0) | 57.8 | | LwF | 51.7 (-2.0) | 55.6 | | EWC | 52.2 (-1.4) | 55.7 | | SI | 52.6 (-1.0) | 55.9 | | MAS (ours) | 53.2 (-0.4) | 55.0 | The reported interpretation is that MAS has by far the smallest forgetting, with $0.4\%$ drop versus $1.0$--$8.0\%$ for the listed alternatives. On the eight-task sequence Flower$\rightarrow$Scenes$\rightarrow$Birds$\rightarrow$Cars$\rightarrow$Aircraft$\rightarrow$Actions$\rightarrow$Letters$\rightarrow$SVHN, the average end-of-sequence accuracy is: | Method | Average end-of-sequence accuracy | |---|---:| | FineTune | 32.7% | | LwF | 49.5% | | IMM | 43.4% | | SI | 50.5% | | MAS | 52.7% | For the same eight-task sequence, average forgetting is reported as MAS $\simeq 0.5\%$ drop versus SI $\simeq 2$--$3\%$ drop. The memory cost statement is also explicit: MAS only stores one $\Omega$-matrix of same size as $\theta$ (constant per task), whereas methods like IMM/EWC store one matrix per task. For fact learning, the model is VGG-16 and the setting is a four-task random split of 186 facts from the 6DS dataset. The metric is mean-average-precision (mAP) for retrieving images of each task at end of sequence: | Method | Evaluation | mAP | |---|---|---:| | FineTune | $T_1$ | 0.19 | | FineTune | $T_2$ | 0.19 | | FineTune | $T_3$ | 0.28 | | FineTune | $T_4$ | 0.71 | | FineTune | overall | 0.18 | | SI | $T_1$ | 0.36 | | SI | $T_2$ | 0.32 | | SI | $T_3$ | 0.38 | | SI | $T_4$ | 0.68 | | SI | overall | 0.25 | | MAS (ours) | $T_1$ | 0.42 | | MAS (ours) | $T_2$ | 0.37 | | MAS (ours) | $T_3$ | 0.41 | | MAS (ours) | $T_4$ | 0.65 | | MAS (ours) | overall | 0.29 | The paper summarizes these numbers by stating that MAS improves $\sim 4\%$ mAP over SI ($0.29$ versus $0.25$) and $\sim 11\%$ over FineTune. A specialization and adaptation test is also reported. A small “sports” subset of facts from Task 1 is defined, and $\Omega$ is computed only on that subset, using unlabeled data. At the end of the four-task sequence, mAP on the sports subset is approximately $0.20$ for FineTune, approximately $0.20$ for SI, and approximately $0.50$ for MAS(adaptive). The paper describes this as MAS learning what not to forget for that subset, boosting mAP from $0.20$ to $0.50$ [1711.09601]. ## 7. Clarifications, scope, and implications Several points in the paper address likely misunderstandings. First, MAS importance is not tied to labels: $g_i(x)$ does not depend on any labels or loss gradients, and $\Omega_i$ can be accumulated on any stream of inputs, including unlabeled data. Second, computing $\Omega$ on test images versus train versus train+test gives nearly identical forgetting on the two-task object-recognition setting, which the paper presents as confirmation that MAS can use any unlabeled data. Third, the local form is not only loosely inspired by Hebbian learning; up to a constant factor of $2$, equation (3) is exactly a Hebbian correlation rule [1711.09601]. The reported significance is twofold. Empirically, the method is presented as yielding state-of-the-art continual-learning performance on the reported object-recognition and fact-learning experiments. Methodologically, it is presented as the first demonstration of adapting the importance of parameters based on unlabeled data toward what the network needs not to forget, with the added observation that this target may vary depending on test conditions. A plausible implication is that MAS is particularly suited to settings where the input distribution available after training is informative about which previously learned behavior should remain stable. Source: https://www.emergentmind.com/topics/memory-aware-synapses-mas