Lifelong Language KD (L2KD)

• Lifelong Language Knowledge Distillation (L2KD) is a technique that uses per-task teacher-student setups to address catastrophic forgetting in sequential language tasks like classification and generation.
• It trains a dedicated teacher for each new task and distills its knowledge into a lifelong student model using soft-label objectives and pseudo-replay, narrowing performance gaps to multi-task upper bounds.
• L2KD is practical and efficient, achieving near-oracle performance (e.g., reducing gaps from 2.9% to 0.7% in sequence generation) while outperforming traditional methods such as fine-tuning and regularization.

Lifelong Language Knowledge Distillation (L2KD) denotes a family of methodologies for mitigating catastrophic forgetting in lifelong and continual language learning settings, in which models are exposed to a stream of tasks or languages, and prior task data is not revisited. Unlike joint multi-task learning, which assumes full data access, L2KD leverages per-task knowledge distillation—assigning a teacher model to each new task and transferring knowledge to a student (lifelong model) via soft-label-based objectives. L2KD has been empirically validated for both sequence generation and classification tasks, as well as multilingual neural machine translation, consistently narrowing the performance gap with multi-task upper bounds and outperforming standard regularization or memory-based continual learning approaches (Chuang et al., 2020, Zhao et al., 2022).

1. Problem Formulation and Motivation

The lifelong language learning (LLL) paradigm assumes a sequence of $M$ tasks $\{T_1, ..., T_M\}$, arriving in an online manner; each comes with its own dataset $D_t = \{X_i^t, Y_i^t\}$. The learner must adapt to a new task $T_t$ and update its parameters $\theta^{(t-1)} \rightarrow \theta^{(t)}$, but cannot access data from previous tasks. This setting encapsulates catastrophic forgetting: post adaptation to $D_t$, performance on $T_1, ..., T_{t-1}$ degrades, a phenomenon not present in oracle multi-task training, which optimizes

$$\mathcal{L}_{\mathrm{joint}}(\theta) = \sum_{t=1}^{M} \mathbb{E}_{(x, y)\in D_t} [-\log p_\theta(y|x)].$$

Previous attempts to address this challenge—such as fine-tuning, elastic weight consolidation, and memory replay—leave a significant 2–3% absolute gap to the multi-task oracle on complex language tasks, and perform poorly in sequence generation compared to text classification (Chuang et al., 2020, Zhao et al., 2022).

2. Core Methodology

2.1 Per-Task Teacher-Student Distillation

L2KD operates by interleaving single-task teacher training and student model distillation at each task boundary. For task $T_t$, a fresh teacher $f^t$ is trained solely on $\{T_1, ..., T_M\}$0:

$\{T_1, ..., T_M\}$1

where $\{T_1, ..., T_M\}$2 aggregates relevant ground-truth-based losses (cross-entropy for targets, optionally plus LM losses for generation). The student model $\{T_1, ..., T_M\}$3 (with parameters $\{T_1, ..., T_M\}$4) is updated using a composite objective:

$\{T_1, ..., T_M\}$5

with the distillation term

$\{T_1, ..., T_M\}$6

for temperature $\{T_1, ..., T_M\}$7 and balance parameter $\{T_1, ..., T_M\}$8.

2.2 Pseudo-Replay for Prior Tasks

To preserve knowledge from previous tasks, pseudo-examples are periodically sampled by the current student for past tasks and used to compute a negative log-likelihood (NLL) loss. No additional distillation is applied for these generated samples, efficiently limiting memory overhead (Chuang et al., 2020).

2.3 Training Procedure

A single full L2KD round comprises:

1. Train the task-specific teacher on $\{T_1, ..., T_M\}$9.
2. Generate or sample pseudo-data for replay.
3. Update the student on batches from both $D_t = \{X_i^t, Y_i^t\}$0 (with hard and distillation loss) and pseudo-data (with NLL).
4. Discard the teacher $D_t = \{X_i^t, Y_i^t\}$1; student parameters become $D_t = \{X_i^t, Y_i^t\}$2.

Pseudocode appears as follows, instantiated for $D_t = \{X_i^t, Y_i^t\}$3 tasks:

$T_t$4 (Chuang et al., 2020)

3. Extensions to Multilingual NMT

3.1 One-to-Many Distillation

In multilingual NMT, the goal is to add a new target language $D_t = \{X_i^t, Y_i^t\}$4 without forgetting previously acquired translation directions. L2KD generates pseudo translations by prefixing source sentences in the new dataset $D_t = \{X_i^t, Y_i^t\}$5 with target indicators, then applying beam search on the prior teacher for each old target language:

$D_t = \{X_i^t, Y_i^t\}$6

This multi-distill strategy closely matches joint training without accessing old datasets (Zhao et al., 2022).

3.2 Many-to-One: Partial Distillation and Solutions

Direct distillation fails in many-to-one settings due to mismatch or sparsity in the new source language (tokens mapped to UNK), leading to "extreme partial distillation." Two solutions are described:

• Pseudo-Input Distillation: Map tokens from the new source to each old source vocabulary by frequency ranking, then generate pseudo-inputs for the old teacher. The loss is the sum of log-likelihoods for pseudo-generated targets and new data.
• Reverse Teacher Distillation: Maintain an explicit reverse model from target to all old sources. For each new example, generate pseudo-sources by feeding the target sentence (prefixed with the source indicator) to the reverse teacher, creating artificial source sentences for distillation. Both forward and reverse students are trained to extend coverage.

These remedies recover most of the BLEU score lost to forgetting (Zhao et al., 2022).

4. Empirical Evaluation

4.1 Lifelong Language Tasks

L2KD has been validated on sequence generation (WikiSQL, CNN/DailyMail, MultiWOZ), domain-differentiated NLG (E2E-NLG, RNNLG), and multi-class classification (AGNews, Yelp, Amazon, DBPedia, Yahoo). Across all modalities, L2KD drastically narrows the gap to multi-task training. For instance, on sequence generation, L2KD with soft distillation reduces the performance gap from 2.9% to 0.7% (averaged across tasks and orders), and the variance across task orders falls correspondingly (Chuang et al., 2020).

4.2 Multilingual NMT

On incremental multilingual NMT tasks using Transformer-base (THUMT), case-insensitive BLEU evaluation shows that:

• One-to-Many: Multi-distill achieves 29.09 BLEU (avg), close to the joint training upper bound of 29.83 and sharply reduces catastrophic forgetting observed under fine-tuning (9.23 BLEU).
• Many-to-One: Pseudo-input and reverse-teacher distillation recover BLEU (30.07, 33.02) nearly matching joint training (33.38), with direct distillation performing poorly (10.66 BLEU).

Memory overhead for L2KD variants is minimal, limited primarily to performing $D_t = \{X_i^t, Y_i^t\}$7 teacher decodings per new instance and, in the reverse-teacher case, training an auxiliary model (Zhao et al., 2022).

Method	Sequence Gen. Avg	NMT BLEU (1-many)	NMT BLEU (many-1)
Multi-task (oracle)	57.8	29.83	33.38
L2KD variant	57.1	29.09	33.02
EWC (regularization)	–	14.60	24.56
Fine-tune	23.4	9.23	17.86

5. Analysis and Limitations

L2KD's strength derives from using a per-task teacher trained solely on the current dataset, which enables adaptation to new tasks without destructively overwriting previous knowledge. The use of temperature-calibrated soft targets ($D_t = \{X_i^t, Y_i^t\}$8) stabilizes optimization and smooths distributional shifts, reducing performance variance across different task orderings (Chuang et al., 2020).

However, L2KD requires additional teacher training per task, which introduces moderate training overhead, though it incurs no test-time costs. The optimal hyperparameters (temperature $D_t = \{X_i^t, Y_i^t\}$9, distillation weight $T_t$0) are not fully explored and may benefit from adaptive scheduling mechanisms. Opportunities for future exploration include leveraging older teachers in distillation, further hybridization with parameter-regularization approaches, and scaling to larger pre-trained LMs to completely close the gap with joint training.

In many-to-one multilingual settings, naive direct distillation can worsen forgetting due to partial distillation on incompatible inputs, motivating more sophisticated pseudo-input and reverse-teacher strategies (Zhao et al., 2022).

6. Practical Considerations and Extensions

L2KD is fundamentally a plug-in to existing LLL architectures (e.g., LAMOL, small GPT-2). It is compatible with pseudo-replay and can be adapted to both classification and generation tasks. In multilingual models, the approach avoids retaining any past data, storing only models and compact vocabulary mappings. Distillation is most effective when using beam search for pseudo-reference generation; $T_t$1-best beams have only marginal benefits over $T_t$2-best ($T_t$3 BLEU improvement) (Zhao et al., 2022).

A plausible implication is that L2KD can be readily incorporated into any neural sequence learning system trained in a streaming fashion, striking a practical balance between plasticity on new data and stability on prior tasks, without incurring significant resource burdens.