Increment Vector Transformation in CIL

• Increment Vector Transformation (IVT) is a framework for class incremental learning that transforms standard parameter updates to align with oracle solutions using curvature information.
• It leverages Fisher-weighted transformations to maintain low-loss linear connectivity between current and previous task optima, thereby reducing catastrophic forgetting.
• The plug-and-play approach is compatible with various CIL methods and has demonstrated robust performance improvements across datasets like CIFAR-100 and ImageNet.

Searching arXiv for the specified IVT paper and a possible acronym disambiguation paper. arxiv_search.query{"search_query":"ti:\"Closing the Oracle Gap: Increment Vector Transformation for Class Incremental Learning\" OR id:(Qiu et al., 26 Sep 2025)","start":0} arxiv_search.query{"search_query":"ti:\"Application of Integral Value Transformation (IVT) in a Specialized Computer Network Design\" OR id:(Naskar et al., 2012)","start":0} Increment Vector Transformation (IVT) is a framework for class incremental learning (CIL) that is designed to mitigate catastrophic forgetting by transforming the parameter increment produced by a standard CIL update into a curvature-aware increment that better matches the geometry of an oracle solution. In the formulation introduced in "Closing the Oracle Gap: Increment Vector Transformation for Class Incremental Learning," IVT is motivated by the observation that CIL oracle solutions typically preserve low-loss linear connections to the optimum of previous tasks, whereas standard incremental updates often move away from that low-loss manifold. IVT therefore acts as a plug-and-play intervention on the training trajectory: rather than directly following ordinary CIL updates, it periodically "teleports" model parameters to transformed solutions intended to preserve linear connectivity to previous task optima (Qiu et al., 26 Sep 2025).

1. Problem setting and conceptual definition

Class Incremental Learning aims to sequentially acquire knowledge of new classes without forgetting previously learned ones. The setting considered for IVT is the standard CIL regime in which a model is exposed to tasks over time and does not retain full access to historical data, so performance on old classes can degrade as training proceeds on new classes. The paper positions the central difficulty as the gap between practical CIL methods and their oracle counterparts, where the oracle denotes a model trained incrementally with full access to historical data (Qiu et al., 26 Sep 2025).

Within that setting, IVT is defined as a framework that transforms the parameter update rather than replacing the underlying CIL method. The approach is explicitly described as plug-and-play, applicable in both exemplar-free and exemplar-based scenarios, and compatible with various initialization strategies. The stated objective is to preserve stable performance on previously learned tasks by maintaining low-loss behavior along linear paths that connect the current solution to previous task optima (Qiu et al., 26 Sep 2025).

The underlying contrast is between two trajectories in parameter space. Standard CIL follows ordinary SGD-based or regularized/distilled updates for the current task. IVT-augmented CIL instead periodically reprojects the evolving parameter vector through a Fisher-weighted transformation of the task increment, with the aim of aligning the update with the geometry observed in oracle solutions. This suggests a shift from viewing forgetting solely as a penalty-balancing problem toward viewing it as a trajectory-shaping problem.

2. Geometric motivation: oracle solutions and linear mode connectivity

The geometric premise of IVT is the paper’s empirical observation that CIL oracles often exhibit linear mode connectivity (LMC) with previous task optima. In this context, LMC refers to the existence of low-loss linear paths in parameter space connecting solutions for different tasks. The reported finding is that oracle solutions maintain low loss on previous tasks not only at the endpoint optimum, but also along the straight line connecting current and previous optima (Qiu et al., 26 Sep 2025).

Standard CIL models are reported not to share this property. The path between their incrementally trained weights and previous optima is not generally a low-loss path, and this lack of connectivity is associated with greater forgetting. IVT is motivated precisely by that discrepancy: if oracle solutions preserve low-loss linear connectivity and practical CIL solutions do not, then transforming the incremental update to better approximate the oracle increment may reduce forgetting (Qiu et al., 26 Sep 2025).

The paper frames this as a way of "closing the oracle gap." The gap is not presented merely as a deficit in access to old data, but as a geometric divergence between the solution manifold followed by oracle training and the one reached by ordinary incremental optimization. A plausible implication is that IVT’s value depends not only on replay or distillation strength, but also on how closely the transformed trajectory remains within the same low-loss basin as earlier tasks.

3. Mathematical formulation

The formulation introduces the following quantities for task $t$:

• $\theta_{t-1}^*$: optimum parameters after task $t-1$
• $\theta_t$: parameters after standard CIL update on task $t$
• $\theta_t^*$: oracle parameters after task $t$
• $V_t = \theta_t - \theta_{t-1}^*$: increment vector for standard CIL
• $V_t^* = \theta_t^* - \theta_{t-1}^*$: increment vector for oracle

The stated goal is to find a transformation $S_t$ such that

$\theta_{t-1}^*$0

thereby mapping the standard CIL increment to the direction taken by the oracle (Qiu et al., 26 Sep 2025).

The key theoretical result reported in the paper is an approximation for the oracle solution: $\theta_{t-1}^*$1 where $\theta_{t-1}^*$2, with $\theta_{t-1}^*$3 the Hessian of $\theta_{t-1}^*$4. The interpretation given in the source is that the oracle increment vector is a transformed version of the incremental update, weighted by curvature information from the loss landscape (Qiu et al., 26 Sep 2025).

This formulation is the conceptual core of IVT. Rather than treating the observed CIL update as final, IVT treats it as a raw increment that should be corrected by a transformation derived from local curvature. In that sense, the "increment vector transformation" nomenclature is literal: the method operates on $\theta_{t-1}^*$5, not on the training objective alone. This suggests an intermediate standpoint between regularization-based continual learning and explicit oracle imitation.

4. Fisher approximation and training procedure

Because full Hessian computation is infeasible for deep models, the practical implementation uses the Fisher Information Matrix (FIM) as a proxy for the expected Hessian. The paper specifically employs a diagonal approximation: $\theta_{t-1}^*$6 The diagonal, per-parameter entries are accumulated during training (Qiu et al., 26 Sep 2025).

The resulting IVT update rule is

$\theta_{t-1}^*$7

where $\theta_{t-1}^*$8, $\theta_{t-1}^*$9, and $t-1$0 denotes the IVT-corrected parameter at task $t-1$1 (Qiu et al., 26 Sep 2025).

The described algorithmic pattern is periodic rather than one-shot. For each incremental task $t-1$2, training begins from $t-1$3. During each training epoch, the model is updated according to the normal CIL method and the diagonal FIM is accumulated. Every $t-1$4 epochs, the IVT correction is applied: $t-1$5 After training, $t-1$6 is updated and $t-1$7 is set to the final task parameter (Qiu et al., 26 Sep 2025).

The source characterizes this procedure as lightweight, requiring only per-parameter operations and negligible extra memory. It also states that IVT is invoked periodically to keep the trajectory close to regions where the local quadratic approximation is valid. This suggests that the interval parameter $t-1$8 mediates a trade-off between adherence to the curvature model and the natural drift of the underlying optimizer.

5. Relationship to standard CIL methods

IVT does not replace standard CIL baselines; it augments them. Standard CIL is described as proceeding with typical SGD-based weight updates for the new task, often constrained by regularization, distillation, or replay, but without explicitly considering the geometric trajectory or LMC. By contrast, IVT-augmented CIL periodically "teleports" the model to a new parameter point through a Fisher-weighted transformation of the parameter increment, aligning the update with the direction and magnitude suggested by the oracle’s curvature-aware increment vector (Qiu et al., 26 Sep 2025).

This distinction matters because IVT is not framed as another replay policy or another distillation term. Its stated function is to actively shape the training trajectory toward regions that support low-loss connectivity for all encountered tasks, rather than merely penalizing deviations from previous function values or feature representations. In the paper’s terms, this is how IVT seeks to improve the stability-plasticity trade-off (Qiu et al., 26 Sep 2025).

The framework is also presented as broad in scope. It is reported to work with both exemplar-based and exemplar-free methods, as well as from-scratch and pretrained models. The data block further notes compatibility with CLIP-pretrained settings, including CLIP ViT-B/16, and with various initialization strategies. A plausible implication is that IVT is intended as a geometry-level correction that is largely orthogonal to the particular baseline used for continual adaptation.

6. Empirical results, analytical findings, and scope

The reported evaluation covers CIFAR-100, FGVCAircraft, ImageNet-Subset (100 classes), and ImageNet-Full (1000 classes), with Average Accuracy (AA), Last Accuracy (LA), and Forgetting Measure (FM) as metrics. Across these settings, the paper states that IVT consistently enhances the performance of strong CIL baselines (Qiu et al., 26 Sep 2025).

Setting	Reported result	Metric(s)
CIFAR-100, PASS baseline	improves the last accuracy by +5.12% and reduces forgetting by 2.54%	LA, FM
FGVCAircraft, CLIP-pre-trained SLCA baseline	yields gains of +14.93% in average accuracy and +21.95% in last accuracy	AA, LA
ImageNet-Subset and ImageNet-Full	consistently enhances the performance of strong CIL baselines	not numerically specified here

Beyond endpoint metrics, the paper reports several analytical findings. Linear connectivity visualizations show that, after IVT, the model solutions lie along linear, low-loss paths connecting previous and current optima, matching the oracle geometry and avoiding the sharp accuracy drops seen in standard CIL. Stability-plasticity trade-off plots are said to show improvements close to oracle solutions. Loss landscape plots indicate that IVT-corrected weights remain within the same low-loss basin as the previous task, whereas standard CIL does not. The paper also reports that IVT is insensitive to the interval parameter, making it robust in practice (Qiu et al., 26 Sep 2025).

The strengths stated in the source are that IVT is general and plug-and-play, efficient because it requires only diagonal Fisher tracking, empirically robust across data and methods, and theoretically informed by the connection between oracle geometry and LMC. The listed limitations are equally specific: the method relies on a local quadratic approximation, so efficacy may be reduced if parameters move too far before an IVT update; diagonal Fisher is a practical approximation to a fuller curvature model; and the oracle gap may persist because IVT does not generally recover full access to past data (Qiu et al., 26 Sep 2025).

A terminological caution is warranted. The acronym "IVT" also appears in a distinct literature as "Integral Value Transformation," a family of transformations from $t-1$9 to $\theta_t$0 in p-adic systems with an application to specialized computer network design (Naskar et al., 2012). That usage is unrelated to Increment Vector Transformation in class incremental learning.