Metaplastic Binarized Backbones

• Metaplastic binarized backbones are neural architectures that combine binary synapses with metaplasticity, enabling synapse-specific adaptation and efficient computation.
• They employ a dual-weight scheme where full-precision hidden weights modulate binary operational weights to prevent catastrophic forgetting while maintaining low-resource usage.
• Innovations like quantization-aware NAS and stochastic engram gating enhance continual learning performance and optimize gradient flow in constrained environments.

A metaplastic binarized backbone is a neural network architecture in which synaptic plasticity is modulated by the historical strength of each synapse, implemented explicitly within binarized neural networks (BNNs). Here, "metaplasticity" denotes a synapse-local adaptation of plasticity in response to prior activity, inspired by biological memory mechanisms. This framework connects computational neuroscience and efficient deep learning by harnessing both low-precision computation and dynamic, synapse-specific consolidation to mitigate catastrophic forgetting while maintaining resource efficiency. Architectures can incorporate algorithmic metaplasticity in weight updates, as well as binary-specific architectural motifs for optimal gradient flow and feature utilization (Laborieux et al., 2020, Laborieux et al., 2021, Aguilar et al., 27 Mar 2025, Kim et al., 2020).

1. Metaplastic Weight Dynamics in Binarized Backbones

At the heart of a metaplastic binarized backbone lies the dual-weight scheme per synapse: each synapse maintains both a full-precision hidden weight ($H$ or $W_h \in \mathbb{R}$) and a binary operational weight ($B$ or $W_b \in \{\pm1\}$). The binary weight is used exclusively during forward and backward passes, while the full-precision hidden weight serves as a latent state controlling both functional output and plasticity (Laborieux et al., 2020, Laborieux et al., 2021, Aguilar et al., 27 Mar 2025):

• Binarization: $B_k = \mathrm{sign}(H_k)$ determines the operational state, updated before each batch.
• Forward computation: Binary weights enable efficient XNOR+popcount arithmetic, while activations are often also binarized; batch normalization is applied post-aggregation.
• Plasticity modulation: During training, the update magnitude to $H_k$ is modulated via a metaplastic factor, reducing changes for consolidated (large $|H_k|$) synapses.

This dual-state structure naturally aligns with Fusi et al.'s synaptic metaplasticity models, wherein the ease of synaptic modification decays as a function of usage and past potentiation events (Laborieux et al., 2021).

2. Mathematical Formulation and Update Rule

The metaplasticity update is formalized by defining a plasticity modulation function $f_{\mathrm{meta}}$ which exponentially reduces plasticity as $|H|$ increases:

$f_{\mathrm{meta}}(m, H) = 1 - \tanh^2(m H)$

• Update Condition: If the proposed weight update would reduce the absolute value of $W_h \in \mathbb{R}$0 or flip its sign (i.e., $W_h \in \mathbb{R}$1), the step is scaled by $W_h \in \mathbb{R}$2; otherwise, a full step is taken.
• Update Rule:
  • If $W_h \in \mathbb{R}$3:

  $W_h \in \mathbb{R}$4 - Else:

  $W_h \in \mathbb{R}$5

Where $W_h \in \mathbb{R}$6 is the learning rate, $W_h \in \mathbb{R}$7 controls metaplastic sharpness, and $W_h \in \mathbb{R}$8 is the gradient estimate (e.g., via Adam) (Laborieux et al., 2020, Laborieux et al., 2021, Aguilar et al., 27 Mar 2025). Backpropagation through the sign function leverages a straight-through estimator: $W_h \in \mathbb{R}$9.

This scheme enforces slow, exponential consolidation for large-magnitude synapses, yielding stability against catastrophic forgetting without the need for explicit task-awareness.

3. Architectural Innovations: Quantization-Aware NAS and Sparse Gating

Metaplastic binarized backbones are enhanced by architectural co-design. Two complementary techniques have been developed:

• Binary Neural Architecture Search (BNAS): BNAS optimizes binary backbones by searching over a quantization-aware space including 3×3/5×5 bin-convolutions, dilated binary convolutions, pooling, and the "Zeroise" layer, which outputs zero for reduced quantization error (Kim et al., 2020).

  • Cell Template: Four-node cells with explicit inter-cell skip connections provide "float-like" gradient highways, mitigating signal attenuation caused by repeated binarization.
  • Diversity Regularization: An entropy-based regularizer ensures broad exploration of both learnable and parameter-free ops, improving early convergence and final accuracy.
• Stochastic Engram Gating: To further reduce interference among tasks, a Bernoulli-gated mask is applied per hidden layer. Individual gates are sampled per neuron and task from a sigmoid-parameterized subnetwork, creating sparse, dynamically allocated subnetworks ("engrams") in the style of biological memory traces (Aguilar et al., 27 Mar 2025). Gradients are propagated through gate samples using the straight-through estimator.

These techniques are orthogonal and can be combined, as shown in recent continual learning research.

4. Theoretical Properties and Synaptic Importance

Theoretical analysis on simplified quadratic binary objectives shows that the hidden weight $B$0 functions as a local synaptic "importance factor." In a convex quadratic loss

$B$1

where $B$2 is the curvature matrix and $B$3 denotes the optimum, it is shown that:

• If $B$4 then $B$5 grows unbounded and linearly over time, protecting the binary state from flipping.
• The penalty for flipping a consolidated synapse grows as $B$6, meaning signal retention is proportional to synaptic consolidation (Laborieux et al., 2020).

This separation of timescales—fast adaptation near $B$7, slow drift for large $B$8—implements synaptic memory consolidation in a directly interpretable manner.

5. Empirical Evaluation: Continual and Stream Learning

Metaplastic binarized backbones have been rigorously evaluated under continual, multitask, and stream learning protocols (Laborieux et al., 2020, Laborieux et al., 2021, Aguilar et al., 27 Mar 2025):

• Multitask (Permuted-MNIST): For a two-hidden-layer BNN (4096 units per layer), standard BNNs (no metaplasticity) degrade to ∼10% accuracy on early tasks. With metaplasticity ($B$9), final accuracy remains ∼97% across all six tasks—matching Elastic Weight Consolidation (EWC) but without task boundaries or explicit penalty terms.
• Stream Learning: On Fashion-MNIST and CIFAR-10, a metaplastic BNN matches or exceeds "offline" full-dataset trained models, with no reliance on replay data or external memory.
• Class-incremental Scenarios with Stochastic Engrams: Incorporating context-dependent gating (engrams), accuracy for split MNIST increases to $W_b \in \{\pm1\}$0 in class-incremental and $W_b \in \{\pm1\}$1 in domain-incremental (CORe50-NI) scenarios, compared to baseline BNNs ($W_b \in \{\pm1\}$2 and $W_b \in \{\pm1\}$3 respectively). Negative backward transfer (BWT) is sharply reduced, indicating effective continual learning (Aguilar et al., 27 Mar 2025).

Resource utilization is also reduced: for split MNIST, peak GPU/RAM usage of the metaplastic engram BNN is $W_b \in \{\pm1\}$4 and $W_b \in \{\pm1\}$5, respectively.

6. Optimization and Training Protocols

Metaplastic binarized backbones retain standard optimization recipes with modifications only to the per-weight update:

• Weight Initialization: Full-precision $W_b \in \{\pm1\}$6/hidden weights initialized from uniform or normal distributions, typically within $W_b \in \{\pm1\}$7 or $W_b \in \{\pm1\}$8.
• Batch Normalization: Applied post-binarization in each hidden layer to stabilize training dynamics.
• Surrogate Gradient: The straight-through estimator is used in all cases for non-differentiable steps (sign function, engram Bernoulli sampling).
• Hyperparameters:
  • Metaplasticity factor $W_b \in \{\pm1\}$9 (suggested defaults $B_k = \mathrm{sign}(H_k)$0 for vision, up to $B_k = \mathrm{sign}(H_k)$1 on high-variability incremental CORe50 tasks).
  • Learning rate $B_k = \mathrm{sign}(H_k)$2 (e.g., $B_k = \mathrm{sign}(H_k)$3 with Adam).
• NAS/BNAS-specific: SGD optimizer (e.g., momentum $B_k = \mathrm{sign}(H_k)$4, weight decay $B_k = \mathrm{sign}(H_k)$5), cosine annealing schedules, diversity regularization decay parameter $B_k = \mathrm{sign}(H_k)$6 (Kim et al., 2020).

Practical guidelines include routine monitoring of $B_k = \mathrm{sign}(H_k)$7 distribution to detect over-consolidation, and reduction of $B_k = \mathrm{sign}(H_k)$8 to restore plasticity if needed.

7. Comparative Analysis and Impact

Metaplastic binarized backbones achieve several unique advantages compared to standard BNNs and full-precision continual learning baselines:

• Mitigation of Catastrophic Forgetting: Synapse-local consolidation outperforms replay-free and task-ID-free approaches, matching EWC-like methods in continual and streaming learning (Laborieux et al., 2020, Laborieux et al., 2021).
• Quantization-Aware Structural Robustness: BNAS architectures (including inter-cell skips and Zeroise layers) consistently outperform binarized floating-point CNNs on CIFAR-10 and ImageNet, achieving up to 94.43% and 63.51% Top-1 accuracy at competitive FLOPs (Kim et al., 2020).
• Resource Efficiency: Peak GPU and memory usage for metaplastic engram BNNs is sharply reduced relative to both vanilla BNNs and full-precision models, supporting deployment in embedded and neuromorphic hardware settings (Aguilar et al., 27 Mar 2025).
• Stability–Plasticity Trade-Off: The combination of metaplastic update rules and stochastic engram gating enhances retention (stability) while permitting adaptation (plasticity), as quantified by backward and forward transfer metrics.

These advances have significant implications for robust continual learning in highly constrained environments and inform the design of scalable, biologically inspired deep learning systems.