Lifelong Learning in Vision-Language Models: Enhanced EWC with Cross-Modal Knowledge Retention

Abstract: Large language-vision models (LVLMs) such as CLIP, Flamingo, and BLIP have revolutionized AI by enabling understanding across textual and visual modalities. These models excel at tasks like image captioning, visual question answering, and cross-modal retrieval. However, they face catastrophic forgetting when learning new tasks sequentially, particularly challenging in multi-modal settings where preserving cross-modal alignments adds complexity to the learning process. This paper presents a comprehensive continual learning framework for LVLMs that combines enhanced Elastic Weight Consolidation (EWC) with parameter-efficient fine-tuning techniques. We integrate multi-modal Fisher Information Matrix calculation, consistency preservation across modalities, and adaptive regularization that considers dependencies across visual and textual encoders. The framework achieves a 78% reduction in forgetting rates relative to naive sequential training approaches through extensive evaluation testing. The framework also preserves alignment between modalities during sequential learning with only 15% additional computational cost. This work advances the state of the art in lifelong learning for multi-modal AI systems, with direct applications to autonomous driving, intelligent robotic assistants, and adaptive robotic systems that must continuously learn in dynamic real-world environments.

• The paper introduces modality-specific EWC, cross-modal consistency preservation, and parameter-efficient adaptation to reduce catastrophic forgetting in sequential vision-language learning.
• The proposed method achieves a 0.08 forgetting rate, 0.82 average accuracy, and 0.09 forward transfer, outperforming naive training, standard EWC, and replay across four image-text tasks.
• Preserving cross-modal alignment is the most influential component, limiting similarity degradation to 2% and retaining 95% of cross-modal retrieval performance with 15% additional compute.

Overview

This paper addresses catastrophic forgetting in large language-vision models (LVLMs) during sequential task learning. The authors propose a continual learning framework built on a CLIP-style dual-encoder backbone, combining three components: an enhanced Elastic Weight Consolidation (EWC) scheme with modality-specific Fisher Information Matrices (FIMs), an explicit cross-modal consistency preservation objective, and parameter-efficient fine-tuning inspired by LoRA. The central claim is that preserving cross-modal alignment—rather than merely protecting per-modality performance—is the decisive factor in multi-modal continual learning. The reported headline results are a 74–78% reduction in forgetting rate relative to naive sequential training, an average accuracy of 0.82 versus 0.68 for naive training, and only 15% additional computational overhead (2605.12789).

Motivation and related work

The paper situates itself against three established continual learning families. Regularization-based methods (EWC, SI, MAS) penalize changes to parameters deemed important for prior tasks; replay-based methods interleave stored samples from earlier tasks; architectural methods allocate dedicated capacity per task. The authors argue that all three were developed for single-modality, small-scale models and do not address two LVLM-specific difficulties: the need to preserve fragile cross-modal correspondences established during initial pretraining, and the computational infeasibility of applying these methods to billion-parameter architectures. Existing multi-modal continual learning work, they contend, uses small datasets and simple metrics that do not capture alignment degradation, leaving a theory-practice gap. The related-work section also surveys the LVLM lineage from CLIP through Flamingo, BLIP, and GPT-4V, and notes recent efficiency-oriented work such as SmolVLM.

Methodology

The problem is formulated as sequential learning over tasks $\mathcal{T}_1, \dots, \mathcal{T}_n$, where each task consists of image-text pairs, with access restricted to current-task data only, subject to maintaining performance $\mathcal{P}(T_i) \geq \epsilon$ on all prior tasks.

Modality-specific EWC. Rather than computing a single FIM over all parameters, the framework estimates separate Fisher matrices for the visual encoder parameters $\theta_v$, textual encoder parameters $\theta_t$, and cross-modal projection parameters $\theta_c$, each with its own adaptive weighting factor $\lambda_v, \lambda_t, \lambda_c$ that shifts with task complexity and modality significance. This departs from uniform-perturbation EWC and allows regularization strength to differ across functional parameter groups.

Cross-modal consistency preservation. A dedicated loss penalizes divergence between the cosine similarity structure of image-text pairs under current encoders $\phi_v, \phi_t$ and that under encoders from previous tasks $f_v, f_t$. The intent is to keep the relational geometry of the joint embedding space stable across tasks, addressing alignment drift that per-modality retention alone does not prevent.

Parameter-efficient adaptation. Updates are restricted via low-rank factorization with a hierarchical selection policy: lower-level parameters remain frozen to preserve foundational capabilities, while higher-level parameters responsible for task-specific reasoning and alignment are adapted. The authors note this serves both as a computational optimization and as implicit regularization.

Experimental setup

Evaluation uses four sequential tasks: 10,000 MSCOCO pairs (Task A), 8,000 Flickr30K pairs (Task B), 12,000 Visual Genome pairs (Task C), and 15,000 Conceptual Captions pairs (Task D), increasing in linguistic and compositional complexity. Baselines comprise naive sequential fine-tuning, traditional EWC applied uniformly to the full model, replay with a 10%-of-past-data memory buffer, and L2 weight decay.

Results

Method	Backward transfer	Forward transfer	Forgetting rate	Average accuracy
Naive sequential	-0.23	0.02	0.31	0.68
EWC	-0.12	0.04	0.18	0.74
Replay	-0.08	0.06	0.14	0.76
Proposed method	-0.05	0.09	0.08	0.82

The proposed method outperforms all baselines on all four metrics, achieving a backward transfer of -0.05 (only a 5% average performance decrease on prior tasks) and a forgetting rate of 0.08. Forward transfer of 0.09 indicates that retained knowledge facilitates new-task acquisition rather than competing with it.

Ablations. Removing cross-modal consistency preservation raises the forgetting rate from 0.08 to 0.14—a 75% increase—which the authors present as evidence that this component is the most critical element of the framework. Multi-modal FIM computation yields 5% higher average accuracy than standard EWC (0.82 vs. 0.78). Parameter-efficient adaptation reduces computational cost substantially (the paper states a "300%" reduction relative to full fine-tuning, presumably meaning cost is reduced to roughly one third) while maintaining comparable performance.

Alignment and efficiency. Cosine similarity between image-text pairs degrades only 2% after all four tasks, versus 15% for naive training; cross-modal retrieval retains 95% of original performance versus 78%. Total overhead comprises 15% additional compute, roughly 10% memory growth, and 20% longer training time. Scalability analysis reports constant per-task overhead across the task sequence.

Representation analysis. Fisher information histograms are sharply peaked near 0.0075, indicating that importance is concentrated in a sparse parameter subset—an observation supporting selective protection over uniform regularization. PCA projections show task-specific clustering along the first principal component, and cosine similarity matrices between Task A and Task B embeddings exhibit a block-diagonal structure, indicating task-distinct but partially shared representations.

Key insights and limitations

The paper's principal insight is that cross-modal consistency preservation, not per-modality knowledge retention, dominates multi-modal continual learning outcomes; approaches that treat modalities independently leave alignment drift unaddressed. The sparse Fisher distribution further suggests that importance-aware, selective regularization is the efficient regime for large models.

The authors state several limitations explicitly. Evaluation uses datasets far smaller than those behind frontier systems such as GPT-4V, and scalability to models with hundreds of billions of parameters is unvalidated. The evaluation protocol covers image-text matching rather than complex multi-modal reasoning, and the framework assumes clearly delineated task boundaries, which real-world deployment rarely provides. The abstract's 78% forgetting reduction and the results section's 74% figure are not reconciled in the text, and the consistency loss formulation as written compares similarity matrices over potentially different index sets across tasks, which merits closer scrutiny. Open questions include extension to audio-visual-textual architectures, evaluation under ambiguous task boundaries, and federated multi-modal settings.

Conclusion

The paper presents a modality-aware extension of EWC, an explicit cross-modal alignment preservation objective, and LoRA-style parameter-efficient adaptation, evaluated on a four-task image-text benchmark. It reports consistent improvements over naive, EWC, and replay baselines—most notably a 0.08 forgetting rate and 95% retention of cross-modal retrieval performance—at modest computational cost. Its most consequential finding is the ablation showing that alignment preservation accounts for the majority of the forgetting reduction. The results are, however, established at small scale and under idealized task segmentation, and validating the framework on frontier-scale models and reasoning-oriented benchmarks remains unresolved.