VidCLearn: Continual Text-to-Video Generation

• VidCLearn is a continual learning framework for text-to-video generation that incrementally absorbs new text-video pairs using a student–teacher architecture.
• It employs distillation-based generative replay, temporal consistency loss, and retrieval-guided inference to ensure smooth motion and retention of prior knowledge.
• Experimental results demonstrate improved fidelity and transfer metrics over naive methods, reducing computational costs compared to full retraining.

Searching arXiv for the specified paper and a closely related baseline to ground the article. VidCLearn is a continual learning framework for diffusion-based text-to-video generation that is designed to incrementally incorporate new text–video pairs without retraining from scratch, while preserving previously learned knowledge through a student–teacher architecture, distillation-based generative replay, a temporal consistency loss, and retrieval-guided inference (Zanchetta et al., 21 Sep 2025). It is motivated by two limitations identified for text-to-video diffusion models such as Tune-A-Video: static knowledge after training and the retraining cost required to integrate new data. Within this formulation, continual learning is treated as the mechanism by which a model can incrementally absorb new information while preserving prior knowledge, with the specific target domain being evolving video distributions that include novel actions, styles, and scenes (Zanchetta et al., 21 Sep 2025).

1. Problem setting and design objective

VidCLearn addresses text-to-video generation under a sequential update regime in which new text–video pairs arrive over time. The central problem is that conventional text-to-video diffusion systems are described as relying on static knowledge: once trained, their parameters are frozen, and incorporating additional text–video pairs requires re-training on the entire dataset from scratch (Zanchetta et al., 21 Sep 2025). The reported consequence is a large computational burden, expressed in massive GPU hours and energy consumption.

The framework therefore targets continual adaptation rather than one-shot batch training. In the reported formulation, each incoming text–video pair defines a new task, and the model is updated incrementally rather than by revisiting the entire corpus. This is significant because the continual-learning objective is not merely parameter efficiency in the abstract; it is specifically tied to retaining prior generative capability while adapting to newly observed text-conditioned video content (Zanchetta et al., 21 Sep 2025).

A plausible implication is that VidCLearn occupies a distinct position within video continual learning: it is not framed as a classifier or recognizer, but as a diffusion-based generator whose failure mode is catastrophic forgetting in generative alignment, visual quality, and temporal coherence.

2. Student–teacher continual-learning architecture

The core architecture follows a student–teacher paradigm. The student model is a latent-space diffusion network based on the core Tune-A-Video model and is continually fine-tuned on incoming text–video pairs. The teacher model is a frozen copy of the student before the student sees the new pair; its role is to preserve the student’s previous capability and guide the new update so that catastrophic forgetting is reduced (Zanchetta et al., 21 Sep 2025).

The task-level workflow is explicit. At each new task, the current student is duplicated, frozen, and designated as the teacher. The student is then fine-tuned on the new pair while distilling knowledge from the teacher. This converts the usual replay problem into a teacher-guided distillation problem: instead of persisting an archive of past videos, the previous model snapshot functions as the carrier of historical knowledge (Zanchetta et al., 21 Sep 2025).

This architecture is structurally notable because retention and adaptation are decoupled. Adaptation is assigned to the student, whereas retention is induced by the teacher’s preserved outputs. In the reported system, this is supplemented by an inference-time retrieval mechanism rather than being confined to the training loop, so continual learning is reflected both in optimization and in generation-time control.

3. Generative replay and optimization objective

VidCLearn implements generative replay through distillation rather than explicit storage of past videos. Let $D_{\rm new}$ denote the fresh text–video pair distribution, $f_\theta$ the student network, and $t,s$ the vectorized pre-softmax outputs of teacher and student. The new-data fitting term is given as

$$L_{s} = \mathbb{E}_{(x,y)\sim D_{\rm new}}\bigl\|\hat y - y\bigr\|^2,$$

where $\hat y = f_\theta(x)$ (Zanchetta et al., 21 Sep 2025).

The replay component is a softened-output distillation loss,

$$L_{KL} = T^2\;\mathrm{KL}\bigl(\mathrm{Softmax}(t/T)\,\Vert\,\mathrm{Softmax}(s/T)\bigr),$$

with temperature $T$. These terms are combined as

$$L_{\rm distill} = \alpha\,L_{KL} \;+\;(1-\alpha)\,L_{s},$$

where $\alpha\in[0,1]$ balances retention versus adaptation (Zanchetta et al., 21 Sep 2025).

The paper also gives a classical replay-style expression,

$$L_t \;=\;\mathbb{E}_{x,y\sim D_{\rm new}}[\ell(f_\theta(x),y)]\;+\;\lambda\;\mathbb{E}_{x',y'\sim D_{\rm replay}}[\ell(f_\theta(x'),y')],$$

where $f_\theta$0 is implicitly defined by samples from the teacher, and $f_\theta$1 weights the replay term (Zanchetta et al., 21 Sep 2025). In this sense, VidCLearn’s “replay” is model-mediated rather than buffer-mediated.

To improve motion smoothness, VidCLearn introduces a temporal consistency loss,

$f_\theta$2

where $f_\theta$3 is the student’s per-frame noise prediction, $f_\theta$4 is the ground-truth noise schedule, and $f_\theta$5 (Zanchetta et al., 21 Sep 2025). The final loss is

$f_\theta$6

with $f_\theta$7 and $f_\theta$8 controlling the relative importance of distillation and temporal smoothness (Zanchetta et al., 21 Sep 2025).

This optimization structure indicates that VidCLearn is not only a forgetting-mitigation method but also a motion-regularized one. The temporal term does not merely stabilize training; it explicitly penalizes discrepancies between temporal deltas of predicted and target noise.

4. Retrieval-guided inference and computational profile

At inference time, VidCLearn augments the student model with a video retrieval module. From the pool of all past training prompts, it retrieves the prompt most semantically similar to the input prompt. Prompts are embedded via Sentence-BERT, similarity is measured by cosine score, and the retrieved video is subjected to DDIM inversion to obtain a latent noise template (Zanchetta et al., 21 Sep 2025). That latent is then used to guide DDIM sampling of the student model, providing structural priors such as layout and camera motion.

The retrieval module therefore acts as a structural guidance mechanism rather than a learned memory in the strict training sense. The reported ablation distinguishes prompt-based retrieval from simply using the last-seen video, which is important because continual-learning systems can otherwise over-index recent data. A plausible interpretation is that retrieval partially counteracts recency bias by selecting semantically relevant historical structure rather than temporally adjacent structure.

The computational claims are explicit. Each fine-tuning on a new pair takes approximately $f_\theta$9 hour on a single NVIDIA A100 GPU and uses approximately $t,s$0 MiB of VRAM. Inference, including DDIM inversion and sampling, takes approximately $t,s$1 minute and uses approximately $t,s$2 MiB VRAM (Zanchetta et al., 21 Sep 2025). Relative to full retraining, the new-data integration cost is reported to scale linearly in the number of new pairs, avoiding repeated $t,s$3–$t,s$4-step diffusion trainings (Zanchetta et al., 21 Sep 2025).

5. Experimental evaluation and ablations

The experimental evaluation is conducted on DAVIS, with $t,s$5 text–video pairs generated via BLIP-2 captions for training and $t,s$6 for testing. The reported metrics are FVD $t,s$7, FID $t,s$8, IS $t,s$9, CLIP-score $$L_{s} = \mathbb{E}_{(x,y)\sim D_{\rm new}}\bigl\|\hat y - y\bigr\|^2,$$0, Forward Transfer (FWT) $$L_{s} = \mathbb{E}_{(x,y)\sim D_{\rm new}}\bigl\|\hat y - y\bigr\|^2,$$1, and Backward Transfer (BWT) $$L_{s} = \mathbb{E}_{(x,y)\sim D_{\rm new}}\bigl\|\hat y - y\bigr\|^2,$$2. Baselines are Naive sequential fine-tuning without continual-learning regularization and Elastic Weight Consolidation (EWC) (Zanchetta et al., 21 Sep 2025).

After $$L_{s} = \mathbb{E}_{(x,y)\sim D_{\rm new}}\bigl\|\hat y - y\bigr\|^2,$$3 tasks, the reported quantitative comparison is as follows:

Method	90-task metrics	Retention
Naive	FVD 617,230; FID 7,112; IS 6,328; CLIP 26,743	FWT 1.002; BWT 0.538
EWC	FVD 901,032; FID 6,162; IS 8,003; CLIP 28,288	FWT 2.191; BWT 1.013
VidCLearn	FVD 734,320; FID 3,155; IS 12,832; CLIP 32,903	FWT 3.158; BWT 1.392

At $$L_{s} = \mathbb{E}_{(x,y)\sim D_{\rm new}}\bigl\|\hat y - y\bigr\|^2,$$4 tasks, VidCLearn reports FVD $$L_{s} = \mathbb{E}_{(x,y)\sim D_{\rm new}}\bigl\|\hat y - y\bigr\|^2,$$5, FID $$L_{s} = \mathbb{E}_{(x,y)\sim D_{\rm new}}\bigl\|\hat y - y\bigr\|^2,$$6, IS $$L_{s} = \mathbb{E}_{(x,y)\sim D_{\rm new}}\bigl\|\hat y - y\bigr\|^2,$$7, and CLIP $$L_{s} = \mathbb{E}_{(x,y)\sim D_{\rm new}}\bigl\|\hat y - y\bigr\|^2,$$8, compared with Naive at FVD $$L_{s} = \mathbb{E}_{(x,y)\sim D_{\rm new}}\bigl\|\hat y - y\bigr\|^2,$$9, FID $\hat y = f_\theta(x)$0, IS $\hat y = f_\theta(x)$1, CLIP $\hat y = f_\theta(x)$2, and EWC at FVD $\hat y = f_\theta(x)$3, FID $\hat y = f_\theta(x)$4, IS $\hat y = f_\theta(x)$5, CLIP $\hat y = f_\theta(x)$6 (Zanchetta et al., 21 Sep 2025). The qualitative examples described in the paper state that Naive and EWC often hallucinate the last seen video, whereas VidCLearn generates clips more closely resembling the correct ground truth and with smooth motion (Zanchetta et al., 21 Sep 2025).

The ablation studies concentrate on temporal consistency and retrieval guidance. For the temporal consistency weight $\hat y = f_\theta(x)$7, the reported settings are $\hat y = f_\theta(x)$8, $\hat y = f_\theta(x)$9, and $$L_{KL} = T^2\;\mathrm{KL}\bigl(\mathrm{Softmax}(t/T)\,\Vert\,\mathrm{Softmax}(s/T)\bigr),$$0. At $$L_{KL} = T^2\;\mathrm{KL}\bigl(\mathrm{Softmax}(t/T)\,\Vert\,\mathrm{Softmax}(s/T)\bigr),$$1, the system reports FVD $$L_{KL} = T^2\;\mathrm{KL}\bigl(\mathrm{Softmax}(t/T)\,\Vert\,\mathrm{Softmax}(s/T)\bigr),$$2, FID $$L_{KL} = T^2\;\mathrm{KL}\bigl(\mathrm{Softmax}(t/T)\,\Vert\,\mathrm{Softmax}(s/T)\bigr),$$3, IS $$L_{KL} = T^2\;\mathrm{KL}\bigl(\mathrm{Softmax}(t/T)\,\Vert\,\mathrm{Softmax}(s/T)\bigr),$$4, and CLIP $$L_{KL} = T^2\;\mathrm{KL}\bigl(\mathrm{Softmax}(t/T)\,\Vert\,\mathrm{Softmax}(s/T)\bigr),$$5; at $$L_{KL} = T^2\;\mathrm{KL}\bigl(\mathrm{Softmax}(t/T)\,\Vert\,\mathrm{Softmax}(s/T)\bigr),$$6, it reports FVD $$L_{KL} = T^2\;\mathrm{KL}\bigl(\mathrm{Softmax}(t/T)\,\Vert\,\mathrm{Softmax}(s/T)\bigr),$$7, FID $$L_{KL} = T^2\;\mathrm{KL}\bigl(\mathrm{Softmax}(t/T)\,\Vert\,\mathrm{Softmax}(s/T)\bigr),$$8, IS $$L_{KL} = T^2\;\mathrm{KL}\bigl(\mathrm{Softmax}(t/T)\,\Vert\,\mathrm{Softmax}(s/T)\bigr),$$9, and CLIP $T$0 (Zanchetta et al., 21 Sep 2025). The accompanying interpretation in the data is that moderate $T$1 yields best semantic alignment, while too large a $T$2 slightly degrades CLIP-score.

For guidance choice, “Last” yields FVD $T$3, FID $T$4, IS $T$5, and CLIP $T$6, whereas “Retrieval” yields FVD $T$7, FID $T$8, IS $T$9, and CLIP $$L_{\rm distill} = \alpha\,L_{KL} \;+\;(1-\alpha)\,L_{s},$$0 (Zanchetta et al., 21 Sep 2025). The stated conclusion is that prompt-based retrieval consistently improves IS and CLIP at a minor FID cost.

6. Limitations, scope, and nomenclature

The reported limitations are twofold. First, unseen motion patterns remain challenging, and the model may underfit rare dynamics. Second, visual realism can still lag behind large monolithic models (Zanchetta et al., 21 Sep 2025). These limitations are consistent with the system’s design: continual adaptation and low update cost are prioritized, but neither implies exhaustive coverage of rare dynamics nor parity with larger static models.

The future directions listed in the paper include integrating perceptual and adversarial losses, adopting more sophisticated memory-replay schemes such as storing a small video buffer, and applying model compression through pruning and quantization to further reduce training and inference cost without sacrificing continual adaptability (Zanchetta et al., 21 Sep 2025). This suggests an open design space between pure distillation-based replay and hybrid replay-memory strategies.

A nomenclature issue also deserves clarification. VidCLearn in the sense relevant here refers to the continual-learning text-to-video framework introduced in “VidCLearn: A Continual Learning Approach for Text-to-Video Generation” (Zanchetta et al., 21 Sep 2025). It should be distinguished from Collaborative Procedure Alignment (CPA), a weakly supervised framework for procedure-aware instructional video correlation learning that appears separately in the provided material (He et al., 2023). The two works address different tasks: the former is a diffusion-based continual text-to-video generator, whereas the latter is a procedure-aware video correlation framework for instructional video analysis.