GESCL: Group & Exclusive Sparsity in CL

• The paper introduces a dual-regularization approach integrating group and exclusive sparsity to balance stability and plasticity in fixed-capacity CNNs.
• It employs a proximal gradient descent algorithm with filter importance estimation to effectively mitigate catastrophic forgetting during sequential task learning.
• Empirical evaluations on benchmarks like CIFAR-10/100 and SVHN demonstrate that GESCL maintains high accuracy while efficiently managing network capacity.

Group and Exclusive Sparsity based Continual Learning (GESCL) is a regularization-based continual learning methodology designed for fixed-capacity convolutional neural networks (CNNs), aiming to resolve the stability–plasticity dilemma inherent to lifelong learning systems. GESCL achieves this by integrating two sparsity-inducing regularizers: group sparsity, which prunes whole filters, and exclusive sparsity, which encourages diversity of filter usage, in the context of sequential task learning without network expansion or exemplar storage. The framework further stabilizes performance by penalizing deviations in filters deemed important for previous tasks, thus preventing catastrophic forgetting while facilitating network plasticity for new tasks (Tousside et al., 7 Jan 2026).

1. Continual Learning in Fixed-Capacity CNNs

In the continual learning (CL) setting, models experience a sequence of $T$ classification tasks $$\{\mathcal{T}_1, \mathcal{T}_2, \ldots, \mathcal{T}_T\}$$, each with its training set $D^t_{\rm train}$. After training on task $t$, previous task data are not accessible. The central challenges are:

• Stability: preserving performance on previously learned tasks in the absence of past data.
• Plasticity: enabling adaptation to new tasks within a fixed parameter budget.

Simple naive fine-tuning causes catastrophic forgetting, while rigid parameter freezing harms adaptability. GESCL introduces a dual-regularizer approach atop the standard cross-entropy loss, explicitly separating filters important for old tasks from those available for reuse, thereby balancing stability and plasticity (Tousside et al., 7 Jan 2026).

2. Mathematical Formulation and Regularizers

The training objective in GESCL is as follows:

$$\mathcal{L}^t(\Theta^t) = \frac{1}{N^t} \sum_{n=1}^{N^t} \ell(f(x_n^t;\Theta^t), y_n^t) + \mu_S\,\mathcal{R}_S(\Theta^t) + \mu_P\,\mathcal{R}_P(\Theta^t)$$

where:

• $\ell(\cdot,\cdot)$ is cross-entropy loss,
• $\mathcal{R}_S$ is the stability regularizer,
• $\mathcal{R}_P$ is the plasticity regularizer,
• $\mu_S, \mu_P > 0$ balance these terms.

Stability Regularizer ($\mathcal{R}_S$):

Penalizes modifications to filters with high task-importance (as measured post-training), enforcing:

$$\{\mathcal{T}_1, \mathcal{T}_2, \ldots, \mathcal{T}_T\}$$0

$$\{\mathcal{T}_1, \mathcal{T}_2, \ldots, \mathcal{T}_T\}$$1 denotes the set of important filters, $$\{\mathcal{T}_1, \mathcal{T}_2, \ldots, \mathcal{T}_T\}$$2 their importance, $$\{\mathcal{T}_1, \mathcal{T}_2, \ldots, \mathcal{T}_T\}$$3 the current filter, and $$\{\mathcal{T}_1, \mathcal{T}_2, \ldots, \mathcal{T}_T\}$$4 the frozen previous-task filter.

Plasticity Regularizer ($$\{\mathcal{T}_1, \mathcal{T}_2, \ldots, \mathcal{T}_T\}$$5):

Applies to filters not important for prior tasks, leveraging a weighted combination of group and exclusive sparsity:

$$\{\mathcal{T}_1, \mathcal{T}_2, \ldots, \mathcal{T}_T\}$$6

where $$\{\mathcal{T}_1, \mathcal{T}_2, \ldots, \mathcal{T}_T\}$$7 interpolates between group sparsity (lower layers) and exclusive sparsity (higher layers). This mechanism promotes removal of redundant filters and differentiation in feature representations (Tousside et al., 7 Jan 2026).

3. Mechanisms of Group and Exclusive Sparsity

Group Sparsity refers to penalizing the $$\{\mathcal{T}_1, \mathcal{T}_2, \ldots, \mathcal{T}_T\}$$8-norm across filters in a layer, encouraging many entire filters to shrink to zero:

$$\{\mathcal{T}_1, \mathcal{T}_2, \ldots, \mathcal{T}_T\}$$9

Exclusive Sparsity (squared $D^t_{\rm train}$0-norm) penalizes shared activation across filters:

$D^t_{\rm train}$1

This induces competition among filters, driving disjoint activity patterns and higher feature diversity. By modulating the balance ($D^t_{\rm train}$2) across layers, GESCL allows greater filter-sharing in early layers (promoting robust low-level representations) and increased exclusivity in later layers (encouraging specialization for task-specific features) (Tousside et al., 7 Jan 2026).

A comparative summary is provided below:

Sparsity Type	Formulation	Effect on Filters
Group Sparsity	$D^t_{\rm train}$3	Prunes entire filters
Exclusive Sparsity	$D^t_{\rm train}$4	Diversifies filter usage

4. Filter Importance Estimation and Stability Integration

After each task, the importance of every convolutional filter is measured via post-activation statistics:

$D^t_{\rm train}$5

with $D^t_{\rm train}$6 denoting the per-pixel standard deviation of the feature map $D^t_{\rm train}$7 over the task's data. These are recursively accumulated across tasks:

$D^t_{\rm train}$8

where $D^t_{\rm train}$9 is a decay parameter. A filter is considered important if $t$0, and otherwise is re-initialized and made available for future tasks. This separation of filters underpins the distinct application of the stability and plasticity regularizers (Tousside et al., 7 Jan 2026).

5. Optimization Algorithm and Proximal Updates

GESCL employs a proximal gradient descent scheme at each epoch. The iterative process features:

• Gradient step: updates on the differentiable cross-entropy loss.
• Proximal step: closed-form updates for the non-smooth stability and plasticity terms, decoupled across filters.

Specifically:

• For important filters:

  $t$1

• For unimportant filters:

  $t$2

  $t$3

After training on a task, filters reclassified as unimportant are re-initialized, and their downstream input channels are zeroed (Tousside et al., 7 Jan 2026).

6. Computational Complexity and Performance Evaluation

GESCL operates with a constant parameter count $t$4, avoiding the model expansion seen in dynamic-capacity approaches. The per-epoch computational overhead is modest, consisting of:

• Activation variance computations: $t$5 per task, dominated by regular stochastic gradient descent operations.
• Proximal filter-wise updates: $t$6 due to their independence.

Empirical results highlight that GESCL sustains near-perfect retention of early tasks and superior average accuracy across all tasks on SVHN, CIFAR-10/100, and ImageNet-50. For instance, on CIFAR-10/100, average accuracy $t$7, forgetting $t$8, significantly outperforming strong baselines such as HAT and AGS (Tousside et al., 7 Jan 2026).

7. Strengths, Limitations, and Comparisons

GESCL provides a principled methodology for continual learning under parameter constraints, directly aligning regularization mechanics with the structure of CNNs. Its main advantages include:

• Parameter efficiency and computation scalability,
• Direct mitigation of catastrophic forgetting through dual-regularization,
• Applicability to memory- and compute-bounded regimes.

Limitations include the need for dataset-specific hyperparameter tuning ($t$9, $$\mathcal{L}^t(\Theta^t) = \frac{1}{N^t} \sum_{n=1}^{N^t} \ell(f(x_n^t;\Theta^t), y_n^t) + \mu_S\,\mathcal{R}_S(\Theta^t) + \mu_P\,\mathcal{R}_P(\Theta^t)$$0, $$\mathcal{L}^t(\Theta^t) = \frac{1}{N^t} \sum_{n=1}^{N^t} \ell(f(x_n^t;\Theta^t), y_n^t) + \mu_S\,\mathcal{R}_S(\Theta^t) + \mu_P\,\mathcal{R}_P(\Theta^t)$$1) and potential network capacity exhaustion on highly heterogeneous or numerous tasks.

In summary, GESCL advances continual learning by fusing importance-based filter stability with systematic group and exclusive sparsity for adaptive plasticity, offering a fixed-capacity alternative to dynamic network expansion and rehearsal-based methods (Tousside et al., 7 Jan 2026).