Recyclable Tuning for Continual Pre-training (2305.08702v1)

Abstract: Continual pre-training is the paradigm where pre-trained LLMs (PLMs) continually acquire fresh knowledge from growing data and gradually get upgraded. Before an upgraded PLM is released, we may have tuned the original PLM for various tasks and stored the adapted weights. However, when tuning the upgraded PLM, these outdated adapted weights will typically be ignored and discarded, causing a potential waste of resources. We bring this issue to the forefront and contend that proper algorithms for recycling outdated adapted weights should be developed. To this end, we formulate the task of recyclable tuning for continual pre-training. In pilot studies, we find that after continual pre-training, the upgraded PLM remains compatible with the outdated adapted weights to some extent. Motivated by this finding, we analyze the connection between continually pre-trained PLMs from two novel aspects, i.e., mode connectivity, and functional similarity. Based on the corresponding findings, we propose both an initialization-based method and a distillation-based method for our task. We demonstrate their feasibility in improving the convergence and performance for tuning the upgraded PLM. We also show that both methods can be combined to achieve better performance. The source codes are publicly available at https://github.com/thunlp/RecyclableTuning.

• The paper demonstrates that recyclable tuning preserves task-specific knowledge by reusing outdated weights in continual pre-training.
• The study shows that adapted models maintain close parametric connectivity and functional similarity, supporting weight recycling.
• The research combines initialization-based and distillation-based methods to offer a sustainable, efficient approach for updating LLMs.

Recyclable Tuning for Continual Pre-Training of PLMs: An Insightful Overview

Introduction

LLMs have become a cornerstone in the field of NLP, significantly due to their ability to be adapted for a wide variety of downstream tasks effectively. A prevalent method to leverage these pre-trained models is through continual pre-training, a process that allows models to integrate emerging knowledge from constantly updated datasets. However, the continual evolution of pre-trained models introduces a challenge: adapted weights from earlier versions of the models might be disregarded, posing a waste of computational resources and valuable task-specific knowledge.

Problem Formulation

The work at hand explores the concept of recyclable tuning for continual pre-training, aiming to tackle the aforementioned challenge. The researchers propose a system where outdated adapted weights are reused (recycled) when tuning updated models. This approach not only holds the promise of preserving task-specific knowledge encapsulated in these weights but also suggests a reduction in computational overhead.

Empirical Analysis

To understand the underlying connections across versions of continually pre-trained models, the paper explores two main aspects:

1. Mode Connectivity: It is shown that adapted models on an identical task demonstrate a close parametric relationship, evidenced by the existence of a low-loss path between them in the parameter space.
2. Functional Similarity: An analysis of the representational similarity suggests that adapted models maintain a degree of functional continuity across updates, further justifying the feasibility of recycling old weights.

Recyclable Tuning Methods

The paper introduces two strategies for recyclable tuning:

• Initialization-Based Method: Leveraging the close parametric connection, this approach uses outdated weights as the starting point for tuning the updated model. The findings support that such initialization accelerates convergence and can enhance performance on the target tasks.
• Distillation-Based Method: By distilling knowledge stored in the outdated weights, this method aims to refine the tuning of the updated model. The results demonstrate the effectiveness of this approach in transferring knowledge and improving model performance.

Discussions

The paper posits several notable implications and future directions:

• The initialization-based method, while efficient, may face practical limitations due to the necessity for parameter compatibility and access to outdated weights. In contrast, the distillation-based method offers a more flexible solution to recycling weights without direct access to the outdated model parameters.
• The combination of both recyclable tuning methods can yield better performance, suggesting a complementary relationship between them.
• The concept of recyclable tuning aligns with the broader objectives of computational efficiency and sustainability in AI research and development. Furthermore, it opens new avenues for maintaining the relevance of adapted weights amidst the rapid evolution of pre-trained models.

Conclusion

Recyclable tuning presents a compelling approach to continual pre-training, emphasizing the preservation and utilisation of previously adapted weights. This research lays the groundwork for more sustainable and efficient methodologies in updating LLMs, ensuring that valuable task-specific adaptations are not lost in the process. Future explorations in this domain are poised to refine these strategies further, potentially broadening the scope of recyclable tuning across various model architectures and adaptation methods.