Continual Latent Alignment (CLA) in Online SSL

• CLA is a self-supervised online continual learning strategy that aligns current and past latent representations to reduce forgetting and enable fast adaptation.
• The method leverages an EMA teacher network and alignment regularizers, with variants like CLA-E and CLA-R using replay buffers to maintain latent stability.
• Empirical results on Split CIFAR-100 and Split ImageNet100 show that CLA achieves faster convergence and superior accuracy under fixed computational budgets compared to standard SSL methods.

Searching arXiv for the cited CLA-related papers and closely related continual-learning latent-alignment work. Continual Latent Alignment (CLA) is a self-supervised learning strategy for online continual learning in which the representations learned by the current model are aligned with past representations to mitigate forgetting under a one-pass, non-stationary stream with small minibatches, no task-boundary information, and a fixed computational budget (Cignoni et al., 14 Jul 2025). In its canonical formulation, CLA is an online continual self-supervised learning method rather than a supervised classifier, a generative replay model, or a task-conditioned expert policy. At the same time, the broader continual-learning literature contains earlier latent-space alignment programs—most notably translator-based consolidation in generative models and latent-geometry regularization—that clarify how “latent alignment” functions as a mechanism for stability, transfer, and representation reuse across sequential data distributions (Deja et al., 2023, Deja et al., 2021, Frascaroli et al., 2023).

1. Problem setting and conceptual motivation

The setting studied by CLA is online continual self-supervised learning (OCSSL): data arrive as a one-pass stream, minibatches are small, task boundaries are not available, and the learner may do multiple gradient passes over a current minibatch before moving on, but cannot revisit past minibatches once they are gone (Cignoni et al., 14 Jul 2025). In this regime, the objective is not only to avoid forgetting, but also to learn useful representations quickly, because the model may be queried at any time during training. The paper therefore treats fast adaptation as a first-class requirement rather than a secondary optimization concern.

Standard SSL methods such as SimSiam, SimCLR, and BYOL usually assume jointly available data, sufficiently large minibatches, long optimization schedules, and stationary i.i.d. sampling. In OCSSL, these assumptions break down because data are non-stationary, minibatches can be as small as a streaming minibatch size $b_s = 10$, and the computational budget is fixed (Cignoni et al., 14 Jul 2025). The resulting failure modes are representation drift, forgetting, and slow convergence under limited compute. CLA addresses these by retaining the SSL objective on the current batch while adding an alignment regularizer that keeps current latent representations close to a target representation from the past.

The basic representational setup is an encoder

$\theta : \mathcal{X} \to \mathcal{Z},$

with two augmented views $x_1, x_2$ of the same sample mapped to

$z_1 = \theta(x_1), \qquad z_2 = \theta(x_2).$

The generic continual SSL formulation used to motivate CLA is

$$\mathcal{L} = \mathcal{L}_{SSL}(z_1, z_2) + \omega \mathcal{L}_{reg},$$

with

$$\mathcal{L}_{reg} = \frac{\mathcal{L}_{alg}(a_\phi(z_1), z_1^{t-1})}{2} + \frac{\mathcal{L}_{alg}(a_\phi(z_2), z_2^{t-1})}{2}.$$

Here $a_\phi$ is an extra alignment projector, $z^{t-1}$ are target features from a past network, $\omega$ controls alignment strength, and $\mathcal{L}_{alg}$ is the alignment loss (Cignoni et al., 14 Jul 2025).

2. Core mechanism and CLA variants

CLA replaces the frozen previous-task teacher used by task-boundary-dependent methods with an EMA network $\theta : \mathcal{X} \to \mathcal{Z},$0, updated continuously after each minibatch as

$\theta : \mathcal{X} \to \mathcal{Z},$1

where $\theta : \mathcal{X} \to \mathcal{Z},$2 is the EMA decay parameter (Cignoni et al., 14 Jul 2025). This yields the base variant, CLA-b, in which the current model is trained to match the latent geometry of a slowly moving teacher. The regularizer is

$\theta : \mathcal{X} \to \mathcal{Z},$3

where

$\theta : \mathcal{X} \to \mathcal{Z},$4

The total loss is

$\theta : \mathcal{X} \to \mathcal{Z},$5

The method is then extended with replay to address the fact that small streaming minibatches are a serious bottleneck in OCSSL. Replay samples $\theta : \mathcal{X} \to \mathcal{Z},$6 are drawn from memory buffer $\theta : \mathcal{X} \to \mathcal{Z},$7, and each training step uses a combined minibatch $\theta : \mathcal{X} \to \mathcal{Z},$8. After augmentation, the SSL loss is computed on the full combined batch

$\theta : \mathcal{X} \to \mathcal{Z},$9

but the alignment loss is applied only to replay features, not current stream features, in order to preserve plasticity for new data (Cignoni et al., 14 Jul 2025).

Variant	Alignment target	Replay
CLA-b	EMA teacher network $x_1, x_2$0	No
CLA-E	EMA features for replayed samples	Yes
CLA-R	Stored past target representations $x_1, x_2$1	Yes

In CLA-E, the replay targets come from the EMA network; in CLA-R, the buffer stores exemplars together with their past target representations $x_1, x_2$2 (Cignoni et al., 14 Jul 2025). For CLA-R, the regularizer is

$x_1, x_2$3

and the total objective is

$x_1, x_2$4

After each training step, the stored feature for replay samples is updated as

$x_1, x_2$5

The design choice that alignment is applied only to replay features is central: it is intended to retain past knowledge while leaving stream samples freer to adapt (Cignoni et al., 14 Jul 2025).

3. Computational budget, memory design, and evaluation protocol

A major contribution of CLA is the explicit treatment of equal-compute comparison through Cumulative Backward Passes (CBP), defined as

$x_1, x_2$6

where $x_1, x_2$7 is the number of views required by the SSL method, $x_1, x_2$8 is the total number of training steps, and $x_1, x_2$9 is the minibatch size used during optimization (Cignoni et al., 14 Jul 2025). With

$z_1 = \theta(x_1), \qquad z_2 = \theta(x_2).$0

CBP serves as an abstract computational proxy for OCSSL, where performance depends strongly on how quickly the model converges. For i.i.d. training, the equivalent number of epochs under the same budget is

$z_1 = \theta(x_1), \qquad z_2 = \theta(x_2).$1

This makes the i.i.d. comparison an equal-budget baseline rather than a looser upper bound.

CLA-E and CLA-R use a FIFO buffer of size 2000. The paper explicitly compares FIFO with reservoir and MinRed-style buffers and argues that FIFO is better for OCSSL because it gives a more uniform number of updates per sample, improves convergence under a one-pass streaming protocol, and emphasizes recent data, which is helpful in fast adaptation (Cignoni et al., 14 Jul 2025).

The reported experimental configuration uses Split CIFAR-100 and Split ImageNet100, both in 20 experiences, with ResNet-18 as backbone. Main experiments use SimSiam; appendix experiments repeat the study with SimCLR. Optimization uses SGD with momentum $z_1 = \theta(x_1), \qquad z_2 = \theta(x_2).$2, weight decay $z_1 = \theta(x_1), \qquad z_2 = \theta(x_2).$3, and an EMA parameter $z_1 = \theta(x_1), \qquad z_2 = \theta(x_2).$4. Evaluation is by linear probing, and results are reported over 3 runs with different seeds (Cignoni et al., 14 Jul 2025).

4. Empirical properties and observed effects

The central empirical claim is that CLA is state of the art in OCSSL under the same computational budget: CLA-E usually achieves the best Final Accuracy, CLA-R often gives the best or near-best Average Accuracy, and both outperform prior OCSSL and CSSL baselines on Split CIFAR-100 and Split ImageNet100 (Cignoni et al., 14 Jul 2025). The emphasis on Average Accuracy is methodologically important because OCSSL models may be queried throughout the stream rather than only after the final experience.

A second finding is that CLA converges faster. Its advantage is especially visible under limited CBP, and when methods are compared at the same CBP, CLA generally outperforms strong baselines. Simple replay remains a strong baseline, but CLA usually does better or is more stable (Cignoni et al., 14 Jul 2025). The paper also reports a more surprising result: CLA can match or surpass i.i.d. training at the same computational budget, especially in the early phase of training. This is one reason the method is also presented as a pretraining protocol. If pretraining begins with CLA in the early stage and then continues with i.i.d. training, the final performance can be better than training fully i.i.d. from scratch.

Downstream transfer is also improved. Encoders pretrained on CIFAR-100 are probed on SVHN, and encoders pretrained on ImageNet100 are probed on Stanford Cars; CLA variants, especially CLA-R and sometimes CLA-E, outperform other methods and often the i.i.d. baseline (Cignoni et al., 14 Jul 2025).

The ablations support the main design decisions. FIFO is best for both CLA-E and CLA-R; varying $z_1 = \theta(x_1), \qquad z_2 = \theta(x_2).$5 has only a limited effect; CLA-R is less sensitive to $z_1 = \theta(x_1), \qquad z_2 = \theta(x_2).$6, whereas CLA-E prefers smaller $z_1 = \theta(x_1), \qquad z_2 = \theta(x_2).$7, especially on CIFAR-100; and increasing the number of passes per minibatch $z_1 = \theta(x_1), \qquad z_2 = \theta(x_2).$8 helps only up to a point, after which performance saturates (Cignoni et al., 14 Jul 2025). This suggests that in OCSSL, larger compute budgets do not automatically translate into better representations unless batch diversity or memory design also improves.

5. Relation to earlier latent-space alignment in continual learning

Although the term Continual Latent Alignment is attached specifically to the OCSSL method of 2025, its broader intellectual lineage lies in earlier continual-learning work that treated latent-space organization as the primary object of consolidation rather than merely a by-product of replay (Deja et al., 2023, Deja et al., 2021). In Adapt & Align, each new task is first learned by a local generative model and then consolidated by a global model through a translator network

$z_1 = \theta(x_1), \qquad z_2 = \theta(x_2).$9

which maps task-specific latent variables into a common latent space (Deja et al., 2023). The same two-stage logic appears in Multiband VAE, where a local latent band is learned for the current task and then aligned into a shared global latent space $$\mathcal{L} = \mathcal{L}_{SSL}(z_1, z_2) + \omega \mathcal{L}_{reg},$$0, with controlled forgetting based on cosine similarity and a threshold $$\mathcal{L} = \mathcal{L}_{SSL}(z_1, z_2) + \omega \mathcal{L}_{reg},$$1 (Deja et al., 2021). Those works position latent alignment as a mechanism for knowledge consolidation, forward transfer, and backward transfer in generative continual learning.

A separate but related trajectory appears in Latent Spectral Regularization for Continual Learning, which does not propose latent alignment in the translator or teacher-student sense. Instead, CaSpeR-IL constructs a latent geometry graph over replay features, computes the normalized Laplacian, and regularizes its spectrum with

$$\mathcal{L} = \mathcal{L}_{SSL}(z_1, z_2) + \omega \mathcal{L}_{reg},$$2

to promote a partitioning behavior in the latent space (Frascaroli et al., 2023). Here the concern is not alignment to a past representation per se, but the preservation of class separation and latent structural consistency over time.

Approach	Alignment object	Continual-learning setting
CLA	Current model representations aligned with past representations	Online continual self-supervised learning (Cignoni et al., 14 Jul 2025)
Adapt & Align / Multiband VAE	Task-specific latent codes aligned into a shared global latent space $$\mathcal{L} = \mathcal{L}_{SSL}(z_1, z_2) + \omega \mathcal{L}_{reg},$$3	Generative continual learning (Deja et al., 2023, Deja et al., 2021)
CaSpeR-IL	Latent geometry regularized through Laplacian spectrum	Rehearsal-based Class-IL (Frascaroli et al., 2023)

This comparison clarifies that “latent alignment” is not a single algorithmic template. In CLA, alignment is online, boundary-free, and compute-budget-aware; in Adapt & Align and Multiband VAE, alignment is translator-based and generative; in CaSpeR-IL, the organizing principle is spectral graph geometry rather than explicit feature matching. A plausible implication is that continual latent alignment is best understood as a family of stability mechanisms defined by the use of a structured latent reference—teacher features, shared latent coordinates, or graph partitions—to counteract representation drift.

6. Terminological scope and neighboring usages

The acronym CLA is overloaded across contemporary machine learning. In multilingual LLM research, CLA usually refers to Cross-lingual Alignment, where the goal is to align multilingual representations for transfer across languages (Han et al., 29 Oct 2025). That literature introduces a transfer-localization plane, studies cultural erasure, and proposes Surgical Steering; it does not study “Continual Latent Alignment” in the OCSSL sense. In vision-language navigation, LCLA denotes Language-Conditioned Latent Alignment, where raw visual-language observations are mapped into the latent space of a privileged expert policy through a lightweight adapter, again without a continual-learning formulation (Subedi et al., 7 Feb 2026).

Within continual learning itself, the phrase “latent alignment” also appears in work on knowledge consolidation with generative models, where the shared latent space $$\mathcal{L} = \mathcal{L}_{SSL}(z_1, z_2) + \omega \mathcal{L}_{reg},$$4, translator networks, and controlled forgetting rules make task relations explicit (Deja et al., 2023, Deja et al., 2021). By contrast, the 2025 CLA paper fixes its scope narrowly: online continual self-supervised learning without task boundaries, under a fixed computational budget, with latent alignment implemented as a regularizer between current and past representations (Cignoni et al., 14 Jul 2025).

A common misconception is therefore to treat CLA as only an anti-forgetting method. The reported findings are broader: CLA is designed not only to reduce forgetting, but also to speed up convergence in the online scenario and even to improve the final performance of later i.i.d. pretraining when used in the early stages (Cignoni et al., 14 Jul 2025). This suggests that, in the OCSSL regime, continual latent alignment is as much a method for training efficiency and early representation formation as it is a method for retaining past knowledge.