TRACER: Persistent Regularization for Robust Multimodal Finetuning

Abstract: Mainstream strategies for finetuning pretrained multimodal models often degrade out-of-distribution (OOD) robustness, a phenomenon known as catastrophic forgetting. In this paper, we develop a theoretical framework for multimodal contrastive finetuning, yielding closed-form solutions and a geometric decomposition for each strategy. This framework shows that self-distillation is more effective than other regularization approaches to retain the knowledge of the pretrained model. Our analysis reveals a largely overlooked limitation: standard Exponential Moving Average (EMA) teachers, widely used in robust finetuning, suffer from collapse. To solve this, we prove that a Weighted Moving Average (WMA) teacher maintains a persistent regularizing force over finite horizons and yields bias-free convergence in the task subspace while preserving orthogonal knowledge. These insights motivate TRACER (Trajectory-Robust Anchoring for Contrastive Encoder Regularization), which combines contrastive learning with WMA-guided multi-perspective distillation. Extensive experiments on CLIP finetuning demonstrate consistent OOD accuracy and calibration gains across three backbone architectures, and comprehensive ablations confirm that TRACER is both principled and robust to hyperparameter choices. Code is available at https://github.com/HesamAsad/TRACER.

• The paper presents TRACER, a method using a WMA teacher to enforce persistent regularization and robust knowledge retention during finetuning.
• It offers a linearized analysis recasting contrastive finetuning as a matrix least-squares problem, clarifying the geometric preservation of pretrained features.
• Empirical tests on benchmarks like ImageNet and ObjectNet demonstrate significant OOD accuracy gains and reduced catastrophic forgetting compared to baseline methods.

Persistent Regularization for Robust Multimodal Finetuning: A Technical Perspective on TRACER

Motivation and Theoretical Grounding

Finetuning large-scale pretrained multimodal models such as CLIP frequently leads to catastrophic forgetting and degradation of out-of-distribution (OOD) robustness, as models adapt too specifically to new data at the expense of generalizable representations. Existing regularization approaches, especially those based on Exponential Moving Average (EMA) teachers, fail to maintain persistent regularization—the teacher collapses towards the student as finetuning progresses, leading to insufficient constraint on the student's drift and thereby poor OOD performance.

This paper introduces a rigorous linearized analysis of multimodal contrastive finetuning and establishes closed-form solutions demonstrating the geometric mechanism by which knowledge is preserved or forgotten across various strategies, including $L_2$ regularization and self-distillation. It identifies and formalizes a crucial limitation of EMA-based teacher--student schemes—their regularization vanishes asymptotically—and shows that the key to robustness lies in maintaining a persistent regularizing force through a Weighted Moving Average (WMA) teacher, which accumulates information from both the pretrained initialization and later training trajectory.

Contrastive Target Matrix and Solution Geometry

The authors linearize the contrastive learning (CL) objective for the vision-to-language alignment task and recast it as a matrix least-squares problem. This is achieved via a reformulation with a "contrastive target matrix" $Y_{\mathrm{FT}}$ (derived from frozen text encoder features and batch statistics), converting the central finetuning optimization into

$$\min_{W_I} \frac{1}{2} \|W_I X_I - Y_{\mathrm{FT}}\|_F^2.$$

This yields closed-form solutions for multiple finetuning paradigms:

• Direct Finetuning discards pretrained components in the task subspace and fully replaces them with new task-optimal components, while strictly preserving orthogonal information.
• $L_2$-SP Regularization blends the pretrained and finetuned solutions uniformly along all parameter directions, with no clean separation between shared and task-specific subspaces.
• Static Self-Distillation retains the pretrained solution in directions orthogonal to current data, and forms a convex mixture of pretrained and finetuned solutions in the task subspace. However, this introduces a persistent anchor bias: the final solution cannot fully align with the new optimal unless the regularization vanishes, which in turn erodes orthogonal preservation.

Dynamic Self-Distillation with WMA Teacher

The analysis exposes the inefficacy of static and EMA-based teachers for robust knowledge preservation. Static teachers enforce a fixed bias, while EMA teachers collapse to the current student, extinguishing their regularizing effect as the student converges. In contrast, the proposed WMA teacher is constructed via an explicit weighted average over the student's trajectory, using a kernel on normalized training time (e.g., $\mathrm{Beta}(0.5,0.5)$—arcsine kernel—which emphasizes both endpoint contributions).

Theoretical results show that, under this WMA formulation:

• The regularization signal from the teacher remains non-vanishing throughout finite training horizons.
• The solution converges to the exact minimum-norm task-optimum in the finetuning subspace (bias-free convergence) while preserving pretrained knowledge in the orthogonal subspace.

In summary, the WMA teacher achieves both persistent regularization (maintaining generalization during the entire finetuning process) and bias elimination (no residual anchor to the pretrained initialization within the task subspace).

TRACER Methodology

The above analysis motivates TRACER (Trajectory-Robust Anchoring for Contrastive Encoder Regularization). This method implements contrastive finetuning using:

• Symmetric InfoNCE loss for image--text matching.
• Multi-perspective self-distillation loss, derived from a WMA teacher, incorporating:
  • Feature Distillation (FD);
  • Contrastive Relational Distillation (CRD);
  • Interactive Contrastive Learning (ICL);
  • Cross-Knowledge Distillation (Cross-KD).
• Teacher weights are computed as a weighted average of past student weights along the training trajectory, with WMA kernels ensuring persistent regularization.

Crucially, TRACER is robust to hyperparameter settings (teacher update schedules, strength, kernel shape), and its computational complexity scales favorably—its main cost arises from batch relational terms, not expensive eigendecomposition operations.

Experimental Results and Empirical Validation

Experiments include ImageNet and standard OOD benchmarks (IN-V2, IN-R, IN-A, IN-S, ObjectNet) across backbone architectures (ViT-B/16, ViT-L/14, RN50). Key findings:

• TRACER consistently achieves the strongest or co-strongest OOD accuracy, especially in challenging domains (e.g., ObjectNet, IN-A), outperforming state-of-the-art like CaRot and other baselines by notable margins (up to +5.9% in OOD average accuracy).
• Catastrophic forgetting is greatly reduced—dynamic self-distillation (as operationalized by TRACER) retains nearly all original task accuracy in ablation setups, compared to severe degradation for direct finetuning and $L_2$.
• Calibration improvements are observed as measured by ECE, emphasizing both reliable confidence estimation and recognition accuracy under distribution shift.
• Ablations confirm that each distillation perspective (notably FD and CRD) contributes, but all four together are optimal for combined accuracy/calibration.
• TRACER achieves computational efficiency superior to methods such as CaRot, avoiding cubic costs in representation dimension.
• Representational similarity analysis (e.g., CKA) shows TRACER preserves the geometry of pretrained features across all layers, in direct contrast to standard finetuning that causes progressive representational drift.

The persistent knowledge gap between WMA teacher and student is empirically validated: the regularization term stays non-trivial throughout training, unlike EMA teachers whose signal collapses.

Implications and Future Directions

TRACER’s geometric foundation and teacher design clarify the limitations of conventional regularization in robust finetuning. By explicitly anchoring to the trajectory—with non-trivial endpoint mass via the WMA kernel—the method avoids the classic “collapse” of EMA-based teachers and achieves bias-free adaptation in the task subspace. Beyond practical applications for robust multimodal adaptation and safe deployment in distributionally unstable environments, the theory suggests extensions to continual learning, parameter-efficient finetuning, and larger foundation model backbones.

Future work should investigate:

• Extension of the theory from the linearized setting to Neural Tangent Kernel and random-feature regimes.
• Integration with PEFT and prompt-based adaptation, given TRACER’s modular regularization structure.
• Empirical evaluation on broader modalities (video, audio, multimodal LLMs) and adaptation scenarios such as continual learning and adverse environments.

Conclusion

TRACER provides a mathematically principled, efficient, and empirically validated approach for robust multimodal finetuning, overcoming the persistent pitfalls of OOD degradation, anchor bias, and teacher collapse. Its WMA-guided distillation framework operationalizes geometric theory into practical methodology, yielding strong gains in real-world OOD benchmarks while preserving pretrained knowledge and calibration (2605.29380).