Two-Way Is Better Than One: Bidirectional Alignment with Cycle Consistency for Exemplar-Free Class-Incremental Learning

Abstract: Continual learning (CL) seeks models that acquire new skills without erasing prior knowledge. In exemplar-free class-incremental learning (EFCIL), this challenge is amplified because past data cannot be stored, making representation drift for old classes particularly harmful. Prototype-based EFCIL is attractive for its efficiency, yet prototypes drift as the embedding space evolves; therefore, projection-based drift compensation has become a popular remedy. We show, however, that existing one-directional projections introduce systematic bias: they either retroactively distort the current feature geometry or align past classes only locally, leaving cycle inconsistencies that accumulate across tasks. We introduce BiCyc, a bidirectional projector alignment approach with a cycle-consistency objective. BiCyc jointly optimizes two maps, old-to-new and new-to-old, with stop-gradient gating so that transport and representation co-evolve. Analytically, we show that the cycle loss contracts the singular spectrum toward unity in whitened space, and that improved transport of class means and covariances yields smaller perturbations of classification log-odds, preserving old-class decisions and mitigating catastrophic forgetting. Empirically, across standard EFCIL benchmarks, BiCyc substantially reduces forgetting and improves accuracy in from-scratch settings, while remaining competitive in the pretrained fine-grained regime.

• The paper introduces BiCyc, a bidirectional projector framework with cycle consistency to effectively reduce prototype drift and mitigate catastrophic forgetting.
• It demonstrates enhanced accuracy and retention on benchmarks like CIFAR-100 and TinyImageNet, outperforming previous one-way projection methods.
• The method leverages joint learnable projectors and a cycle loss that guarantees near-isometric transport with minimal extra parameters.

Bidirectional Cycle Consistency for Exemplar-Free Class-Incremental Learning

Introduction and Context

Exemplar-Free Class-Incremental Learning (EFCIL) addresses continual learning (CL) under a strictly memory-constrained regime, where no raw samples from previous tasks may be stored. Prototype-based approaches, storing only class-wise means and covariances, offer an efficient solution but suffer severely from representation drift as the backbone network adapts with each new task. Projection-based compensation for prototype drift has emerged as a key strategy, yet the one-directional nature of prior work (applying only old-to-new or new-to-old projections) introduces systematic errors and cumulative inconsistencies, ultimately degrading long-term retention and exacerbating catastrophic forgetting.

The paper "Two-Way Is Better Than One: Bidirectional Alignment with Cycle Consistency for Exemplar-Free Class-Incremental Learning" (2606.05675) formalizes and overcomes these limitations by introducing BiCyc, a bidirectional projector alignment framework with a cycle-consistency constraint. This approach not only establishes rigorous theoretical properties for drift compensation but empirically demonstrates across standard CL benchmarks that joint bidirectional transport yields stronger retention, lower forgetting, and favorable plasticity-stability tradeoffs compared to the state-of-the-art.

Technical Formulation

Prototype Drift and Prior Compensation Paradigms

In exemplar-free regimes, class prototypes (means/covariances computed under feature extractor $f_{t-1}$) become stale after the backbone adapts to a new task $t$. The standard two-stage drift compensation paradigm regularizes the representation with distillation and then trains, post hoc, a transport projector mapping cached old prototypes to the new space. However, this induces two critical problems:

1. Asymmetry: One-way transport accumulates mismatches across tasks, with geometric inconsistencies never corrected.
2. Temporal Misalignment: Post-hoc adapters do not co-evolve with the main representation, so drift and correction are always "out of phase."

Bidirectional Cycle Consistency: BiCyc

The BiCyc method replaces the two-stage, one-way pipeline with a nearly single-stage, bidirectional approach. Two learnable projectors are parameterized and jointly optimized during main task training:

• $A$: Old $\to$ New (prototype adapter)
• $D$: New $\to$ Old (feature-level distiller)

The losses are:

• Bidirectional Alignment: Enforces $D(f_t(x)) \approx f_{t-1}(x)$ and $A(f_{t-1}(x)) \approx \text{stopgrad}[f_{t}(x)]$, ensuring projection consistency without impeding backbone plasticity.
• Cycle Consistency: $A(D(z_{new})) \approx z_{new}$ and $D(A(z_{old})) \approx z_{old}$, regularizing $t$0 and $t$1 as near-inverses on the data support. All targets use stop-gradient gating to avoid adverse gradient flows.

The method proceeds with a short adapter fine-tuning phase, then infers with a Gaussian Bayes classifier combining transported old and new class distributions.

Theoretical Analysis

The paper provides rigorous analysis:

• In the whitened feature space, the cycle-consistency loss contracts the singular values of $t$2 towards 1. This ensures near-isometry of transport and prevents rank deficiencies or energy loss.
• The alignment and cycle error jointly bound the induced perturbation on classification log-odds. This yields a direct connection between prototype transport fidelity and decision stability, quantifying the direct mitigation of catastrophic forgetting.

Empirical Results

Main Benchmarks

BiCyc is evaluated on CIFAR-100, TinyImageNet, ImageNet-100, and CUB-200, comparing to an extensive set of EFCIL baselines including AdaGauss, LDC, ADC, EFC, FeCAM, and others. Both from-scratch and pre-trained initialization regimes are studied.

Key findings:

• On CIFAR-100 and TinyImageNet, BiCyc consistently outperforms AdaGauss, LDC, ADC, and other recent methods in terms of both last-task average accuracy and average incremental accuracy across 10- and 20-task splits. E.g., CIFAR-100 (T=10): BiCyc achieves 50.6\% (Alast) versus 46.8\% for AdaGauss.
• Forgetting (last-task average) is substantially reduced: e.g., 13.5\% (CIFAR-100 T=10) versus AdaGauss (16.7\%) and LDC (21.7\%).
• Gains are most pronounced for earlier tasks, indicating greatly improved long-term retention and direct mitigation of catastrophic forgetting.
• On CUB-200 (fine-grained recognition with pre-trained encoder, where drift is constrained), improvements are attenuated but performance remains fully competitive.

Architectural and Ablation Studies

• Bidirectional losses: Both the alignment and cycle components yield complementary improvements; combining them delivers maximal gains in accuracy and forgetting reduction.
• Adapter architecture: While all tested nonlinearities (MLP, cross-attention, MoE) outperform a simple linear projector, further improvements (trade-offs in stability/accuracy) are possible using content-dependent and nonlinear models.
• Adapter fine-tuning: A brief post-training consolidation of the adapter provides additional benefit, especially with more incremental steps.
• Parameter overhead: The extra bidirectional projector adds less than 2.5\% to total model parameters, maintaining computational and memory efficiency.

Prototype Geometry and Robustness

• BiCyc produces significantly lower drift between stored and ground-truth prototypes (means/covariances) and maintains tighter, near-isometric geometry under sequential updates. Diagnostics on task-0 clusters after many tasks show more stable and Gaussian-like class-wise distributions as compared to AdaGauss and LDC.
• The robust anti-collapse objective on covariances, incorporating symmetrization, shrinkage, and Cholesky-failure fallback, secures numerical stability even in small-batch, high-dimensional settings.

Practical and Theoretical Implications

Theoretical Implications:

• The approach establishes a rigorous link between drift-minimizing cycle consistency and preservation of decision margins, using prototype transport as the principal mechanism for catastrophic forgetting mitigation under strict memory constraints.
• The spectral contraction property of the cycle loss motivates future advances in other forms of space-bridging regularization in lifelong learning.

Practical Implications:

• The method enables state-of-the-art performance on challenging benchmarks without storing any exemplars, establishing bidirectional transport as a new best practice for EFCIL.
• Its marginal computational overhead and architectural flexibility facilitate integration with emerging backbone architectures and real-world deployment settings demanding privacy or strict storage constraints.

Future Directions:

• Extending the framework to uncertainty-aware, class-imbalance-robust prototype transport.
• Derivation of non-asymptotic generalization bounds beyond the Gaussian prototype regime.
• Integration with test-time adaptation and multi-modal continual learning under memory/performance trade-offs.

Conclusion

The BiCyc method reframes prototype transport for exemplar-free class-incremental learning through joint bidirectional and cycle-consistent alignment, theoretically ensuring stable and invertible inter-task mappings and empirically reducing forgetting beyond prior art. The framework's combination of formal guarantees, robust empirical performance, and low overhead make it a compelling advance for scalable, privacy-constrained continual learning scenarios. Future developments will likely generalize these principles beyond the Gaussian/linear setting and further close the gap to memory-augmented rehearsal-based methods.