Dynamic Distillation Coefficient (DDC)

• Dynamic Distillation Coefficient (DDC) is a scalar weight that modulates the influence of distillation losses during training with adaptive scheduling strategies.
• Various DDC formulations employ time-based, cosine annealing, and Gompertz curve schedules to balance teacher supervision with task-driven learning objectives.
• DDC plays a crucial role in mitigating catastrophic forgetting and class imbalance, supporting improved performance in long-tailed and incremental learning scenarios.

Dynamic Distillation Coefficient (DDC) denotes a scalar weight that multiplies one or more distillation losses and is updated during training rather than held fixed. In recent usage, the term refers to mechanisms that adapt the relative influence of teacher-guided supervision as optimization proceeds, with schedules tied to epoch position, accumulated class-distribution entropy, or a Gompertz growth curve (Sakai et al., 5 May 2026, Xie et al., 5 Sep 2025, Yang et al., 24 Oct 2025). The acronym is overloaded, however: in numerical analysis it also denotes defect–deferred correction, which is unrelated to knowledge distillation (Aggul, 2020).

1. DDC as a weighting mechanism in distillation objectives

Across the distillation literature represented here, DDC is introduced at the level of the training objective as a scalar multiplicative factor on the distillation term. In Long-tailed Class Incremental Learning (LT-CIL), the loss is written as

$$\mathcal{L} = \mathcal{L}_{\mathrm{cls}} + \lambda \,\mathcal{L}_{\mathrm{distill}} + \mathcal{L}_{\mathrm{GCR}},$$

where $\lambda$ is the dynamic distillation coefficient, $\mathcal{L}_{\mathrm{cls}}$ is cross-entropy on current task labels, $\mathcal{L}_{\mathrm{distill}}$ is a teacher-student distillation loss, and $\mathcal{L}_{\mathrm{GCR}}$ corresponds to Gradient Consistency Regularization (Sakai et al., 5 May 2026).

In AFD-SLU, the same structural role appears in the objective

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{task}} + \lambda \,\mathcal{L}_{\text{distill}},$$

with $\mathcal{L}_{\text{task}}$ denoting supervised loss for joint intent detection and slot filling, and $\mathcal{L}_{\text{distill}}$ denoting mean-squared error between teacher and adapter-projected student token-level embeddings (Xie et al., 5 Sep 2025). The coefficient therefore controls distillation strength in the precise sense of how much the student’s parameter updates are driven by distillation loss relative to task loss.

The Gompertz-CNN formulation extends the same pattern to a multi-loss distillation block:

$$L_{\text{total}} = L_{\text{classification}} + \beta \cdot \big( L_{\text{Wasserstein}} + L_{\text{grad}} + L_{\text{distill}} \big).$$

Here a single dynamic coefficient $\beta$ weights soft-label KD, feature-level KD via Wasserstein distance, and gradient matching jointly (Yang et al., 24 Oct 2025). This establishes a common abstraction: DDC is not a new loss class, but a scheduling mechanism for the relative influence of existing distillation losses.

2. Mathematical formulations and scheduling strategies

The LT-CIL formulation defines DDC as the product of a time-based schedule and an entropy-based imbalance term:

$\lambda$0

with

$\lambda$1

and

$\lambda$2

Here $\lambda$3 is the current epoch index, $\lambda$4 is the total number of epochs for the task, $\lambda$5 is the number of classes observed so far, and $\lambda$6 is the proportion of samples belonging to class $\lambda$7 in the accumulated data stream. The coefficient is therefore dynamic in two senses: it changes over epochs within a task via $\lambda$8, and it changes across incremental steps as the global class imbalance evolves via $\lambda$9 (Sakai et al., 5 May 2026).

AFD-SLU adopts an epoch-wise cosine annealing schedule:

$\mathcal{L}_{\mathrm{cls}}$0

where $\mathcal{L}_{\mathrm{cls}}$1, $\mathcal{L}_{\mathrm{cls}}$2, the current epoch $\mathcal{L}_{\mathrm{cls}}$3, and the total number of epochs $\mathcal{L}_{\mathrm{cls}}$4 are the only explicit inputs. In the reported implementation, $\mathcal{L}_{\mathrm{cls}}$5, $\mathcal{L}_{\mathrm{cls}}$6, and $\mathcal{L}_{\mathrm{cls}}$7. The method section makes clear that this is a deterministic schedule over epochs, not an online function of measured intent accuracy or slot F1 (Xie et al., 5 Sep 2025).

Gompertz-CNN introduces a stage-aware schedule based on a Gompertz curve:

$\mathcal{L}_{\mathrm{cls}}$8

with $\mathcal{L}_{\mathrm{cls}}$9 and $\mathcal{L}_{\mathrm{distill}}$0. The intended behavior is slow initial growth, rapid mid-phase increase, and late-stage saturation, mirroring a student learning trajectory described as slow $\mathcal{L}_{\mathrm{distill}}$1 fast $\mathcal{L}_{\mathrm{distill}}$2 saturating (Yang et al., 24 Oct 2025).

These formulations represent three distinct scheduling logics. The LT-CIL variant is jointly task- and distribution-aware, the AFD-SLU variant is epoch-scheduled, and the Gompertz-CNN variant is stage-aware. A plausible implication is that recent DDC designs differ less in where the coefficient enters the loss than in which proxy is chosen for the student’s changing capacity or the training distribution.

3. Functional role in different learning regimes

In LT-CIL, DDC is introduced to address a specific pathology of fixed distillation weights under long-tailed incremental data. The paper identifies two intertwined problems: catastrophic forgetting and severe class imbalance. A large fixed $\mathcal{L}_{\mathrm{distill}}$3 over-preserves old majority classes and suppresses learning of new and minority classes; a small fixed $\mathcal{L}_{\mathrm{distill}}$4 permits acquisition of new classes but accelerates forgetting, especially for minority classes. Because the teacher is itself trained on imbalanced data, distillation loss tends to reflect majority bias; a strong fixed weight therefore reinforces that bias (Sakai et al., 5 May 2026).

The AFD-SLU use case is different. The teacher is a frozen GTE model that outputs token-level embeddings $\mathcal{L}_{\mathrm{distill}}$5, while the student embeddings are projected by a dynamic adapter equipped with a Residual Projection Neural Network (RPNN) into the teacher space as $\mathcal{L}_{\mathrm{distill}}$6. The distillation loss is

$\mathcal{L}_{\mathrm{distill}}$7

DDC therefore modulates feature-level alignment between heterogeneous representation spaces rather than logit retention across tasks (Xie et al., 5 Sep 2025).

In Gompertz-CNN, DDC coordinates a broader bundle of teacher signals. The distillation block contains KL-divergence on softened logits, Wasserstein distance between teacher and student feature maps, and gradient matching built from Euclidean distance and cosine similarity after channel remapping with a $\mathcal{L}_{\mathrm{distill}}$8 convolution. The single coefficient $\mathcal{L}_{\mathrm{distill}}$9 scales this entire block globally, so the method treats temporal variation in distillation strength as a system-level control variable rather than a per-component hyperparameter (Yang et al., 24 Oct 2025).

4. Interactions with auxiliary mechanisms

The LT-CIL paper is explicit that DDC does not stand alone. It is paired with Gradient Reweighting (GR), which assigns class-wise weights

$\mathcal{L}_{\mathrm{GCR}}$0

where $\mathcal{L}_{\mathrm{GCR}}$1 is the cumulative gradient norm for class $\mathcal{L}_{\mathrm{GCR}}$2 in the current task, and with Gradient Consistency Regularization (GCR), which smooths gradients across epochs by using an exponential moving average:

$\mathcal{L}_{\mathrm{GCR}}$3

with $\mathcal{L}_{\mathrm{GCR}}$4, and a regularized gradient

$\mathcal{L}_{\mathrm{GCR}}$5

with $\mathcal{L}_{\mathrm{GCR}}$6. The paper separates roles carefully: GR addresses intra-task