Complementary Anchor Replay (CAR)

• CAR is a continual learning algorithm that augments standard experience replay with learned anchor points to mitigate catastrophic forgetting in neural networks.
• It leverages a bilevel optimization framework that combines traditional replay loss with an anchoring penalty to preserve predictions on key data points.
• Empirical results show significant accuracy improvements and reduced forgetting on benchmarks like Permuted MNIST, Split CIFAR-100, and miniImageNet.

Complementary Anchor Replay (CAR), also referred to as Hindsight Anchor Learning (HAL), is a continual learning algorithm designed to mitigate catastrophic forgetting in neural networks exposed to non-stationary data streams. CAR augments standard experience replay strategies by incorporating a learned set of anchor points, optimized to identify and preserve predictions most susceptible to forgetting upon subsequent learning phases. The method leverages bilevel optimization, episodic memory, and a bespoke anchoring loss to ensure retention of decision boundaries crucial to prior tasks while accommodating new information, offering consistent performance improvements across standard lifelong learning benchmarks (Chaudhry et al., 2020).

1. Objective and Core Formulation

The primary objective of CAR is to enable deep models to retain knowledge acquired from previous tasks while learning new ones, particularly in supervised continual learning scenarios. Conventional experience replay minimizes the standard supervised loss over the union of a current minibatch $D_\text{curr}$ and a sampled subset $B_M$ of episodic memory $M$:

$\ell(\theta; D_\text{curr} \cup B_M)$

where $\theta \in \mathbb{R}^P$ denotes the model parameters.

CAR augments this with an anchoring penalty, added to each SGD update, that constrains the change in model predictions on a set of learned anchor points $\{ e_i \}$. The two-step procedure for every gradient descent step is:

1. Temporary update: $$\tilde{\theta} \leftarrow \theta - \alpha \, \nabla_\theta \ell(\theta; D_\text{curr} \cup B_M)$$
2. CAR update:

$$\theta \leftarrow \theta - \alpha \nabla_\theta \left[ \ell(\theta; D_\text{curr} \cup B_M) + \lambda \sum_i \left\| f_\theta(e_i,t_i) - f_{\tilde{\theta}}(e_i,t_i) \right\|^2 \right]$$

where $f_\theta(\cdot, t_i)$ is the task-conditioned output, $\lambda > 0$ modulates the strength of anchoring, and the summation is over anchor points for all previous tasks. This combined loss penalizes deviation of the network’s predictions on anchors after a small future ("look-ahead") update, effectively preserving information most at risk of being lost.

2. Bilevel Optimization in CAR

CAR is formulated as a bilevel optimization problem in both the parameter update and anchor selection phases.

• Parameter update as bilevel problem:
  • Outer: $B_M$0
  • where $B_M$1 and $B_M$2
  • Inner: $B_M$3
• Anchor selection as bilevel problem:
  • Inner: $B_M$7 (fine-tuning on memory)
  • Outer:

$B_M$8

Here, $B_M$9 is the feature embedding, $M$0 a running mean feature for task $M$1, and $M$2 regularizes the anchor’s proximity to the data manifold.

3. Loss Functions

The CAR methodology involves three principal loss functions:

• Classification loss: For any set $M$3,

$M$4

where $M$5 is the cross-entropy loss.

• Anchoring loss: On learned anchors,

$M$6

• Forgetting loss (anchor selection):

$M$7

4. Anchor Selection and Update Procedure

Anchor points are learned at the completion of each task using gradient ascent to maximize the proxy forgetting loss:

1. Initialize $M$8 for each class $M$9 in task $\ell(\theta; D_\text{curr} \cup B_M)$0 (random initialization or mean-embedding).
2. Update via $\ell(\theta; D_\text{curr} \cup B_M)$1 gradient-ascent steps:

$\ell(\theta; D_\text{curr} \cup B_M)$2

1. Freeze $\ell(\theta; D_\text{curr} \cup B_M)$3 for use as an anchor in subsequent iterations.

The feature extractor $\ell(\theta; D_\text{curr} \cup B_M)$4 is defined as the penultimate layer of the network; $\ell(\theta; D_\text{curr} \cup B_M)$5 is tracked online with exponential moving average: $\ell(\theta; D_\text{curr} \cup B_M)$6, with decay $\ell(\theta; D_\text{curr} \cup B_M)$7.

5. Algorithmic Workflow

The CAR/HAL workflow is structured as follows:

• Maintain an episodic memory $\ell(\theta; D_\text{curr} \cup B_M)$8 with a per-class ring buffer for storage of past examples.
• After each new task is learned:
  1. Update $\ell(\theta; D_\text{curr} \cup B_M)$9 with examples from the task.
  2. Compute $\theta \in \mathbb{R}^P$0 (feature mean) online.
  3. After completing the task, fine-tune parameters on $\theta \in \mathbb{R}^P$1 to obtain $\theta \in \mathbb{R}^P$2.
  4. For each class, optimize the anchor $\theta \in \mathbb{R}^P$3 via $\theta \in \mathbb{R}^P$4 steps of gradient ascent on the forgetting objective.

• During training on each minibatch:

  1. Sample replay batch $\theta \in \mathbb{R}^P$5 from $\theta \in \mathbb{R}^P$6.
  2. Compute look-ahead parameters $\theta \in \mathbb{R}^P$7.
  3. Update $\theta \in \mathbb{R}^P$8 using the combined loss: standard replay plus anchoring on all current anchors.

Pseudocode

$f_\theta(\cdot, t_i)$2

6. Empirical Results and Comparative Analysis

CAR/HAL was evaluated on multiple supervised continual learning benchmarks: Permuted MNIST (23 tasks × 10 classes), Rotated MNIST (23 × 10), Split CIFAR-100 (20 × 5), and Split miniImageNet (20 × 5), across varying episodic memory budgets—1, 3, or 5 examples per class per task. Baselines included Finetune, EWC, VCL (with/without random replay), ICARL, AGEM, MER, ER-Ring, MIR, and oracle strategies.

Performance metrics comprised final average accuracy $\theta \in \mathbb{R}^P$9 and final forgetting $\{ e_i \}$0. With 1 example per class memory:

• On Permuted MNIST: HAL achieved $\{ e_i \}$1 vs. MIR $\{ e_i \}$2, ER-Ring $\{ e_i \}$3, MER $\{ e_i \}$4

• On Rotated MNIST: HAL $\{ e_i \}$5, ER-Ring $\{ e_i \}$6
• On Split CIFAR: HAL $\{ e_i \}$7, MIR $\{ e_i \}$8, ER-Ring $\{ e_i \}$9
• On Split miniImageNet: HAL $$\tilde{\theta} \leftarrow \theta - \alpha \, \nabla_\theta \ell(\theta; D_\text{curr} \cup B_M)$$0, MIR $$\tilde{\theta} \leftarrow \theta - \alpha \, \nabla_\theta \ell(\theta; D_\text{curr} \cup B_M)$$1, ER-Ring $$\tilde{\theta} \leftarrow \theta - \alpha \, \nabla_\theta \ell(\theta; D_\text{curr} \cup B_M)$$2

Relative forgetting was consistently reduced by 15–25%. Increasing memory to 5 examples per class further increased HAL’s accuracy—approximately $$\tilde{\theta} \leftarrow \theta - \alpha \, \nabla_\theta \ell(\theta; D_\text{curr} \cup B_M)$$3 on MNIST tasks, $$\tilde{\theta} \leftarrow \theta - \alpha \, \nabla_\theta \ell(\theta; D_\text{curr} \cup B_M)$$4 on Split CIFAR—always outperforming the compared replay or meta-learning methods.

7. Hyperparameters and Implementation Considerations

Key hyperparameters and implementation aspects include:

• Learning rate $$\tilde{\theta} \leftarrow \theta - \alpha \, \nabla_\theta \ell(\theta; D_\text{curr} \cup B_M)$$5: $$\tilde{\theta} \leftarrow \theta - \alpha \, \nabla_\theta \ell(\theta; D_\text{curr} \cup B_M)$$6; selected as $$\tilde{\theta} \leftarrow \theta - \alpha \, \nabla_\theta \ell(\theta; D_\text{curr} \cup B_M)$$7 for MNIST, $$\tilde{\theta} \leftarrow \theta - \alpha \, \nabla_\theta \ell(\theta; D_\text{curr} \cup B_M)$$8 for CIFAR/ImageNet.
• Anchor-loss weight $$\tilde{\theta} \leftarrow \theta - \alpha \, \nabla_\theta \ell(\theta; D_\text{curr} \cup B_M)$$9: same grid, set as $$\theta \leftarrow \theta - \alpha \nabla_\theta \left[ \ell(\theta; D_\text{curr} \cup B_M) + \lambda \sum_i \left\| f_\theta(e_i,t_i) - f_{\tilde{\theta}}(e_i,t_i) \right\|^2 \right]$$0 for MNIST, $$\theta \leftarrow \theta - \alpha \nabla_\theta \left[ \ell(\theta; D_\text{curr} \cup B_M) + \lambda \sum_i \left\| f_\theta(e_i,t_i) - f_{\tilde{\theta}}(e_i,t_i) \right\|^2 \right]$$1 for CIFAR, $$\theta \leftarrow \theta - \alpha \nabla_\theta \left[ \ell(\theta; D_\text{curr} \cup B_M) + \lambda \sum_i \left\| f_\theta(e_i,t_i) - f_{\tilde{\theta}}(e_i,t_i) \right\|^2 \right]$$2 for miniImageNet.
• Mean-embedding regularizer $$\theta \leftarrow \theta - \alpha \nabla_\theta \left[ \ell(\theta; D_\text{curr} \cup B_M) + \lambda \sum_i \left\| f_\theta(e_i,t_i) - f_{\tilde{\theta}}(e_i,t_i) \right\|^2 \right]$$3: typically $$\theta \leftarrow \theta - \alpha \nabla_\theta \left[ \ell(\theta; D_\text{curr} \cup B_M) + \lambda \sum_i \left\| f_\theta(e_i,t_i) - f_{\tilde{\theta}}(e_i,t_i) \right\|^2 \right]$$4.
• Decay $$\theta \leftarrow \theta - \alpha \nabla_\theta \left[ \ell(\theta; D_\text{curr} \cup B_M) + \lambda \sum_i \left\| f_\theta(e_i,t_i) - f_{\tilde{\theta}}(e_i,t_i) \right\|^2 \right]$$5 for running mean-embedding: $$\theta \leftarrow \theta - \alpha \nabla_\theta \left[ \ell(\theta; D_\text{curr} \cup B_M) + \lambda \sum_i \left\| f_\theta(e_i,t_i) - f_{\tilde{\theta}}(e_i,t_i) \right\|^2 \right]$$6.
• Anchor-learning steps $$\theta \leftarrow \theta - \alpha \nabla_\theta \left[ \ell(\theta; D_\text{curr} \cup B_M) + \lambda \sum_i \left\| f_\theta(e_i,t_i) - f_{\tilde{\theta}}(e_i,t_i) \right\|^2 \right]$$7: $$\theta \leftarrow \theta - \alpha \nabla_\theta \left[ \ell(\theta; D_\text{curr} \cup B_M) + \lambda \sum_i \left\| f_\theta(e_i,t_i) - f_{\tilde{\theta}}(e_i,t_i) \right\|^2 \right]$$8 per anchor, anchor step size $$\theta \leftarrow \theta - \alpha \nabla_\theta \left[ \ell(\theta; D_\text{curr} \cup B_M) + \lambda \sum_i \left\| f_\theta(e_i,t_i) - f_{\tilde{\theta}}(e_i,t_i) \right\|^2 \right]$$9.
• Feature extractor $f_\theta(\cdot, t_i)$0: penultimate network activation.
• Episodic memory: per-class ring buffer of size $f_\theta(\cdot, t_i)$1; sampling is uniform.
• Computational cost: CAR approximately doubles computation time over vanilla replay and remains substantially faster than meta-experience replay (MER).
• Anchor initialization: random Gaussian initialization in input space is effective; mean-embedding initialization is a possible alternative.

A plausible implication is that the computational efficiency, combined with modest memory usage and scalable performance, makes CAR suitable for practical large-scale continual learning.

---

CAR/HAL thus functions as a principled extension to experience replay, leveraging trajectory-sensitive anchor points—discovered via explicit maximization of forgetting—to maintain essential decision boundaries. The method demonstrates consistent gains in both accuracy and forgetting metrics relative to multiple strong baselines, with transparent and reproducible optimization and implementation strategies (Chaudhry et al., 2020).