Time is Encoded in the Weights of Finetuned Language Models

Abstract: We present time vectors, a simple tool to customize LLMs to new time periods. Time vectors are created by finetuning a LLM on data from a single time (e.g., a year or month), and then subtracting the weights of the original pretrained model. This vector specifies a direction in weight space that, as our experiments show, improves performance on text from that time period. Time vectors specialized to adjacent time periods appear to be positioned closer together in a manifold. Using this structure, we interpolate between time vectors to induce new models that perform better on intervening and future time periods, without any additional training. We demonstrate the consistency of our findings across different tasks, domains, model sizes, and time scales. Our results suggest that time is encoded in the weight space of finetuned models.

• The paper demonstrates that fine-tuning language models with time vectors effectively encodes temporal linguistic trends within the weight space.
• The method involves subtracting base model weights from period-specific fine-tuned models to reveal organized temporal directions and mitigate misalignment.
• Applying interpolation and task analogy techniques, the approach improves model performance on intervening and future text without full retraining.

Introduction to Time Vectors

Language evolves over time, and this poses a challenge for LLMs which must adapt to shifts in word usage and context that occur over months and years. This phenomenon, known as temporal misalignment, can affect the performance of a model when the data it was trained on becomes outdated. To address this issue, a new concept of time vectors has been introduced, which allows LLMs to be fine-tuned for specific time periods, improving their performance on text from those periods without the need for continual retraining.

Understanding Temporal Misalignment

Temporal misalignment becomes apparent when variations in training and testing data timelines lead to performance degradation, which is particularly noticeable across different time periods. This misalignment has been analyzed at both yearly and monthly scales, revealing linear degradation patterns in yearly settings and seasonal patterns on a monthly basis. These findings showcase the imperative need for LLMs that can adjust to the temporal variations inherent in language.

Time Vector Mechanics

Time vectors are essentially created by fine-tuning a pre-trained LLM on text from a single period (e.g., a specific year) and then subtracting the weights of the original model. This yields a direction in the weight space that corresponds to the specific time period's linguistic nuances. Researchers found that these vectors possess an organized manifold structure, with vectors for adjacent time periods being closer together, suggesting that time is indeed encoded within the weight space of these models.

Application of Time Vectors

Leveraging the organization of time vectors, various applications have emerged:

1. Interpolating Time Vectors: By interpolating between time vectors corresponding to different periods, models have shown improved performance on intervening months or years.
2. Generalizing to Future Time Periods: A technique called task analogy allows for the improvement of performance on future data by using only unlabeled text from those future time periods, avoiding the need for additional training data.
3. Multi-Time Period Generalization: Multi-year model performance remains a challenge. Interpolating between all time vectors for a task, known as a 'time soup,' has not yet reached the efficiency of models trained on data from all periods at once.

Conclusions and Implications

This research illustrates the potential of time vectors in enabling LLMs to adapt to new time periods. By utilizing weight space interpolation and task analogy, models can be updated to reflect intervening and future linguistic trends. However, creating a model that generalizes well across multiple time periods is complex, suggesting that further sophisticated methods are needed. The release of both the code and the finetuned models marks a step towards more temporally aware and adaptable AI language systems.