ILRec: Incremental Latent Rectification

• ILRec is a continual learning method that employs small rectifier units to map current deep neural network representations back to earlier latent spaces, reducing catastrophic forgetting.
• It decouples new-task learning from forgetting mitigation through an alignment loss that corrects representation drift without relying heavily on replay buffers.
• Empirical results on CIFAR-100 and Tiny ImageNet show that ILRec achieves competitive accuracy and resource efficiency by using a fixed-size alignment set with linear parameter growth.

Searching arXiv for the specified paper and closely related continual learning context. Searching arXiv for (Nguyen et al., 2024). ILRec is the continual learning method introduced in "Forget but Recall: Incremental Latent Rectification in Continual Learning" (Nguyen et al., 2024). It investigates an unexplored continual learning direction for incremental learning called Incremental Latent Rectification, in which a model learns to propagate with correction, or rectify, the representation from the current trained DNN backward to the representation space of an old task, where performing predictive decisions is easier. Rather than relying on exemplar replay, regularization, or dedicated capacity allocation alone, ILRec uses a chain of small representation mapping networks, called rectifier units, to mitigate catastrophic forgetting while preserving plasticity on incoming tasks (Nguyen et al., 2024).

1. Problem setting and central idea

Continual learning in DNNs is concerned with continuously learning a changing data stream. The central difficulty is catastrophic forgetting: after sequentially learning tasks $1,\dots,N$, the feature extractor drifts, so that for an old task $t' < t$ one has $f_t(x) \neq f_{t'}(x)$. In ILRec, this drift is not prevented directly at the representation level. Instead, the method learns to map the current latent representation back into an earlier latent space associated with a previous task (Nguyen et al., 2024).

The framework denotes by $f_t$ the feature extractor after learning task $t$, and by $w_t$ the task-specific classifier head for task $t$. For a sample $x$ and current latent $z_t = f_t(x)$, a rectifier unit $r_t$ is trained so that

$t' < t$0

This makes the old task decision process operate in a representation space that was previously learned for that task, rather than forcing the final feature extractor to preserve all earlier latent geometries exactly (Nguyen et al., 2024).

A plausible implication is that ILRec reframes forgetting mitigation as a post hoc latent-space correction problem. The paper states this explicitly as an advantage: it decouples new-task learning (plasticity) from forgetting mitigation (stability) (Nguyen et al., 2024).

2. Rectifier units and latent rectification

A rectifier unit is a small, trainable mapping network that pushes back a current latent into the previous latent space. Formally, the paper writes the rectifier as

$t' < t$1

with

$t' < t$2

Using the shorthand $t' < t$3, the goal is

$t' < t$4

By chaining $t' < t$5, ILRec can at inference time recover an old representation $t' < t$6 from the final extractor $t' < t$7 (Nguyen et al., 2024).

Each rectifier unit consists of three architectural components:

1. a weak feature extractor $t' < t$8, described as two conv layers plus pooling, which distills a low-dim embedding of $t' < t$9;
2. a linear compress layer $f_t(x) \neq f_{t'}(x)$0 that projects $f_t(x) \neq f_{t'}(x)$1 into a low-dim bottleneck;
3. a linear combine layer $f_t(x) \neq f_{t'}(x)$2 that concatenates $f_t(x) \neq f_{t'}(x)$3 and outputs the rectified vector (Nguyen et al., 2024).

This construction is intentionally lightweight. The method does not alter the loss or architecture of the main model for new tasks; instead, it attaches auxiliary rectification modules whose role is specifically to recover prior latent spaces (Nguyen et al., 2024).

3. Objectives and incremental training procedure

ILRec uses a standard cross-entropy objective for the current task and a dedicated alignment loss for each rectifier. The alignment loss is

$f_t(x) \neq f_{t'}(x)$4

where $f_t(x) \neq f_{t'}(x)$5 weights the cosine-similarity term and $f_t(x) \neq f_{t'}(x)$6 is an alignment set (Nguyen et al., 2024).

The paper also allows an optional regularization term on $f_t(x) \neq f_{t'}(x)$7 using old-task latents,

$f_t(x) \neq f_{t'}(x)$8

with $f_t(x) \neq f_{t'}(x)$9 small or zero if one only wants to train $f_t$0 (Nguyen et al., 2024).

The incremental procedure is organized task by task:

• receive new task data $f_t$1;
• train $f_t$2 by minimizing the cross-entropy loss, optionally with the additional alignment term;
• distill $f_t$3 from $f_t$4 so that $f_t$5 on task $f_t$6;
• if $f_t$7, freeze $f_t$8 and train $f_t$9 on the alignment set $t$0 (Nguyen et al., 2024).

The alignment set can be chosen as the previous task’s training data $t$1, the current task’s data $t$2 together with a kept copy of $t$3, or samples from a small generative model $t$4 (Nguyen et al., 2024). This choice determines how data access and memory are traded against alignment fidelity.

4. Inference, task structure, and continual learning regimes

In the task-incremental setting, ILRec uses the known task identity to determine which rectifier chain must be applied to the current latent. This dependence on task identity is a structural property of the method rather than an incidental implementation detail. The limitations section states that ILRec relies on known task identity for chaining (Nguyen et al., 2024).

For class-incremental learning, where task identity is unavailable, the paper reports that ILRec forms an ensemble of rectifications over all domains. On S-CIFAR100, this achieves approximately $t$5 average accuracy with $t$6 alignment samples, and is described as competitive with DER++ and ER-ACE (Nguyen et al., 2024). The same section notes that this class-incremental approach is an ensemble hack which may suffer from miscalibrated confidences (Nguyen et al., 2024).

This suggests that ILRec is most naturally aligned with task-incremental protocols, where the chain structure is explicit and predictable. Its extension to class-incremental learning is functional, but the paper treats it as a less direct instantiation of the rectification principle.

5. Empirical performance and resource profile

The paper evaluates ILRec on CIFAR-10, CIFAR-100, and Tiny ImageNet in the task-incremental setting after all tasks. The following table reproduces the reported average accuracies.

Method	CIFAR-10 AA	CIFAR-100 AA	TinyImageNet AA
Joint (upper)	98.46%	86.37%	81.86%
Finetune (lower)	64.16%	24.01%	13.79%
EWC (online)	69.60%	36.61%	15.67%
LwF (multiclass)	60.96%	41.00%	23.24%
AGEM (500 buf)	90.37%	63.35%	37.14%
ER (500 buf)	94.24%	67.41%	46.07%
DER++ (500 buf)	92.49%	68.52%	50.84%
ER-ACE (500 buf)	94.52%	67.26%	47.72%
TAMiL (500 buf)	94.89%	76.39%	64.24%
CLS-ER (500 buf)	95.35%	77.03%	54.69%
ILRec (500 align)	86.28%	74.59%	59.78%
ILRec (5000 align)	94.84%	82.05%	72.50%

With a moderate alignment set of $t$7 samples, ILRec matches or exceeds most rehearsal-based methods on CIFAR-100 and TinyImageNet. The paper also emphasizes that ILRec uses no ongoing replay buffer, only a fixed-size alignment set per task (Nguyen et al., 2024).

The memory overhead per new task is reported as follows:

Component	Parameters per task
Weak extractor $t$8	$t$9
Compress layer $w_t$0	$w_t$1
Combine layer $w_t$2	$w_t$3

These values sum to approximately $w_t$4 parameters per task. By $w_t$5 tasks, ILRec’s total overhead is reported as approximately $w_t$6, versus TAMiL at approximately $w_t$7 or CLS-ER at approximately $w_t$8 (Nguyen et al., 2024).

The empirical profile therefore combines strong performance on CIFAR-100 and Tiny ImageNet with bounded but linear growth in task-specific parameters.

6. Advantages, limitations, and prospective extensions

The paper lists three principal advantages. First, ILRec decouples new-task learning from forgetting mitigation. Second, it requires no large replay buffer of all old tasks, only per-task rectifiers plus a small alignment set or a generator. Third, it is plug-and-play in the sense that it does not alter the loss or architecture of the main model for new tasks (Nguyen et al., 2024).

The limitations are equally explicit. Parameter growth is linear in the number of tasks, although lighter than many expansion methods. Inference may be slower for long chains of rectifiers, because there is one rectifier per past task. Best results require access to task $w_t$9 data; pure data-free operation through generators is slightly weaker. Finally, the method relies on known task identity for chaining, and its class-incremental ensemble formulation may suffer from miscalibrated confidences (Nguyen et al., 2024).

The future directions proposed in the paper follow directly from these constraints. They include skip-chain rectifiers every $t$0 tasks to reduce chain length, better synthetic alignment data through more powerful continual generative models, jointly learning task-ID inference to enable true class-incremental rectification, and adaptive vector-quantization or shared sub-space across rectifiers to reduce memory (Nguyen et al., 2024).

Taken together, these properties place ILRec within the family of continual learning methods that do not attempt to freeze representation drift completely. Instead, they learn a reversible path through successive latent spaces. In the terminology of the paper, the method aims to forget in the current feature extractor while still recalling by rectification, which is the defining idea of incremental latent rectification (Nguyen et al., 2024).