Zero-Shot Incremental Learning

• Zero-Shot Incremental Learning is a continual learning paradigm that enables models to integrate new data without retaining real past exemplars by synthesizing or aligning features.
• It incorporates methods such as data-free replay, embedding-space translation, and semantic or open-vocabulary learning to address issues like catastrophic forgetting and semantic drift.
• Empirical studies show competitive benchmark performance with trade-offs in computational cost and memory efficiency, highlighting ongoing challenges in scalability and stability.

Zero-shot incremental learning (ZS-IL) denotes a family of continual-learning settings in which a model must incorporate new data over time while preserving prior competence and maintaining some form of zero-shot capability. Across the literature, the phrase covers several related but non-identical regimes: replaying past knowledge without stored real exemplars, translating or aligning representations across incremental tasks without old-task images, and continually recognizing classes that remain unseen in the visual stream but are available through semantic descriptors or a pre-trained vision-LLM. The unifying technical concerns are catastrophic forgetting, semantic drift between tasks, absence or restriction of rehearsal, and evaluation over an expanding label space that may contain both previously learned and still-unseen classes (Pourkeshavarz et al., 2021, Wei et al., 2020, Kuchibhotla et al., 2022).

1. Conceptual scope and terminology

The literature uses “zero-shot” in several senses rather than a single canonical one. In the original paper titled “ZS-IL: Looking Back on Learned Experiences For Zero-Shot Incremental Learning” (Pourkeshavarz et al., 2021), the term refers to zero real exemplars: past experiences are replayed using synthesized samples generated by querying the learner’s own parameters, with no exemplar memory buffer and no auxiliary/generative network. In “Incremental Embedding Learning via Zero-Shot Translation” (Wei et al., 2020), the zero-shot aspect is zero-shot translation between embedding spaces: previous-task classes are treated as unseen during the current step because old images are unavailable, and the system aligns old and current embeddings using only current-task images. In continual generalized zero-shot learning and open-vocabulary settings, the zero-shot component is closer to classical ZSL or GZSL: unseen classes are represented by attributes, word vectors, prompts, or text embeddings, and must be recognized jointly with seen classes as tasks arrive (Kuchibhotla et al., 2022, Zhu et al., 2023, Panariello et al., 22 Aug 2025).

Paradigm	Zero-shot aspect	Representative works
Data-free replay	No real past samples are stored; past experiences are replayed using synthesized samples generated by the learner itself	(Pourkeshavarz et al., 2021)
Embedding-space incremental learning	No old-task images; embeddings or prototypes are aligned across tasks by a zero-shot translation model	(Wei et al., 2020, Ren et al., 2024)
Continual generalized/open-vocabulary learning	Unseen classes are available through semantic descriptors or VLM priors and are recognized jointly with incrementally learned classes	(Kuchibhotla et al., 2022, Zhu et al., 2023, Panariello et al., 22 Aug 2025)

This terminological plurality is not merely lexical. It changes what is treated as unavailable at task $t$: raw images of old classes, attributes of future classes, open-vocabulary textual labels, or some combination of these. A plausible implication is that ZS-IL is better viewed as a structured subfield of continual learning defined by missing modalities or missing data channels under temporal shift, rather than by one fixed protocol.

2. Formal problem formulations and evaluation regimes

A common embedding-space formulation uses a sequence of tasks $t = 1,2,\dots,T$, each with a disjoint label set $C^t$, and only current-task data $D^t = \{(x_i,y_i)\}_{i=1}^{n^t}$ available at step $t$. The model is an embedding network $F_\theta(x)$, classification is performed by a nearest-class-mean classifier, and the central difficulty is not logit drift but the semantic gap between adjacent embedding spaces: prototypes computed under $F_{\theta^{t-1}}$ do not remain compatible with features from $F_{\theta^t}$. ZSTCI therefore learns two residual translators, $g_{\text{old}}$ and $g_{\text{cur}}$, maps old-space and current-space embeddings into a common embedding space, and optimizes an alignment loss together with a triplet-based unified-representation loss (Wei et al., 2020). In this regime, two standard metrics are the Average Incremental Accuracy,

$t = 1,2,\dots,T$0

and Average Forgetting,

$t = 1,2,\dots,T$1

Continual generalized zero-shot learning broadens the setting. In the proposed Online-CGZSL formulation, tasks arrive sequentially; each task may introduce new seen classes with visual data and new unseen classes with only semantic descriptors, and previously unseen classes may later become seen. At task $t = 1,2,\dots,T$2, training uses current seen-class data plus replayed features, while evaluation is conducted jointly over all seen and unseen classes encountered so far. The standard summary metrics are mean seen accuracy, mean unseen accuracy, mean harmonic accuracy, and mean AUSUC, with the harmonic term defined taskwise over seen and unseen test splits (Kuchibhotla et al., 2022). This is a direct formalization of a ZS-IL regime in which both the semantic space and the visual evidence evolve over time.

Open-vocabulary continual learning with CLIP instantiates another regime. A frozen zero-shot model provides scores over arbitrary textual labels, while an exemplar-based or modular component learns incrementally from arriving labeled images. In the class-incremental scenario of “Continual Learning in Open-vocabulary Classification with Complementary Memory Systems” (Zhu et al., 2023), classes are partitioned incrementally, and evaluation is performed jointly over seen and unseen classes. In composition-incremental learning, the label space itself is structured: attributes $t = 1,2,\dots,T$3, objects $t = 1,2,\dots,T$4, and compositions $t = 1,2,\dots,T$5. The model sees new compositions over tasks, while a static unseen composition pool remains held out; the principal summary metrics become

$t = 1,2,\dots,T$6

which quantify zero-shot generalization over time and forgetting on previously learned compositions (Li et al., 12 Nov 2025).

3. Core methodological families

One family addresses ZS-IL through memory recovery and synthetic replay. The learner network $t = 1,2,\dots,T$7 itself is treated as a memory source: the method samples candidate softmax vectors from a Dirichlet distribution conditioned by a dynamic confusion matrix, filters them with the constraint $t = 1,2,\dots,T$8, and then optimizes random noise images so that the learner outputs those accepted vectors. The resulting on-call transfer set $t = 1,2,\dots,T$9 is used with a combined objective $C^t$0, where the new-task term is ordinary cross-entropy and the old-task term distills logits on synthetic samples (Pourkeshavarz et al., 2021). The same replay logic reappears in later forms as feature-space generation rather than image synthesis.

A second family is prototype translation in embedding space. ZSTCI learns an embedding network with triplet loss, stores class prototypes, and after each task trains two residual MLPs:

$C^t$1

using the alignment loss

$C^t$2

Old and new prototypes are then translated into a common space and classified by nearest-class-mean (Wei et al., 2020). SFDNet retains the same zero-shot translation structure but enriches the feature extractor with a Spatial-Frequency Feature Extraction module, an Attention Feature Alignment module, CADA-VAE-based alignment, and compensation loss. It explicitly targets the semantic gap by combining spatial-domain and frequency-domain cues, SENet and FcaNet attention, and prototype compensation across tasks (Ren et al., 2024).

A third family is semantic or open-vocabulary continual learning. In Online-CGZSL, a feature generator $C^t$3 synthesizes visual features conditioned on attributes, a discriminator/projector $C^t$4 acts as a class prototype in feature space, and the model is trained with a cosine-similarity GAN objective, classification losses, bi-directional incremental alignment, and generative replay (Kuchibhotla et al., 2022). In open-vocabulary CLIP settings, the zero-shot model remains frozen and an incremental memory system supplies fast adaptation. The complementary-memory approach combines CLIP zero-shot probabilities with an exemplar model by estimating $C^t$5 from CLIP itself and then performing adaptive fusion. MoDER moves this one step further by storing one textual LoRA expert per seen class in a foundational hub and recomposing experts for unseen classes via semantic similarity in CLIP’s text space (Zhu et al., 2023, Panariello et al., 22 Aug 2025). SimE instead freezes a CLIP image encoder, trains adapters only on the base task, concatenates the original and adapted features, and handles later tasks by prototype extension rather than further encoder updates (Luo et al., 11 Mar 2026).

4. Representative architectures and specialized variants

SFDNet is a representative architecture for embedding-based ZS-IL. Its spatial branch uses ResNet-12 and global average pooling, while the frequency branch applies a 2D discrete cosine transform, assigns different scaling coefficients to high- and low-frequency regions, reconstructs an original-like image together with high-frequency and low-frequency images, and feeds them through another ResNet-12. The Attention Feature Alignment module applies SENet and FcaNet, then aligns attention-enriched streams through CADA-VAE with

$C^t$6

and the total objective is

$C^t$7

The final embedding is formed by aligning spatial and frequency features and concatenating the aligned outputs, after which nearest-class-mean and zero-shot translation operate in the fused space (Ren et al., 2024).

Zero-shot incremental learning also extends to object detection. Incrementally Zero-Shot Detection defines three simultaneous class sets at each step: old seen classes $C^t$8, new seen classes $C^t$9, and unseen classes $D^t = \{(x_i,y_i)\}_{i=1}^{n^t}$0. IZSD-EVer combines a Faster R-CNN backbone, a semantic embedding branch, an incremental classifier branch, an old–new model for distillation, and an Extreme Value Analyzer that decides whether a proposal should be classified by the seen-class incremental classifier or by the unseen-class semantic classifier. The open-set gate is based on a Generalized Pareto Distribution fitted to class-wise tail distances in semantic space, with the decision rule

$D^t = \{(x_i,y_i)\}_{i=1}^{n^t}$1

The training objective includes bfMSE to handle background–foreground imbalance, projection distance loss $D^t = \{(x_i,y_i)\}_{i=1}^{n^t}$2, RPN feature distillation, and classification distillation (Zheng et al., 2021).

Compositional ZS-IL introduces a structured label space. In CompIL, the model gradually learns new attribute–object compositions, reserves a static unseen composition pool for testing, and uses a pseudo-replay framework with a visual synthesizer and linguistic primitive distillation. The synthesizer is a composition-conditioned VAE in CLIP feature space trained by

$D^t = \{(x_i,y_i)\}_{i=1}^{n^t}$3

while incremental CZSL training uses

$D^t = \{(x_i,y_i)\}_{i=1}^{n^t}$4

Here the distillation term constrains logits over all past compositions so that primitive representations remain aligned across tasks, directly targeting the drift of attribute and object semantics under incremental updates (Li et al., 12 Nov 2025).

5. Empirical findings across benchmark families

The earliest explicit ZS-IL formulation based on memory recovery showed that synthetic replay from the learner’s own parameters can be competitive with or stronger than both data-free and buffer-based baselines. On CIFAR-10, ZS-IL reported 75.34 in Class-IL and 93.12 in Task-IL; on Tiny-ImageNet it reported 33.06 in Class-IL and 67.42 in Task-IL. The same paper reported that increasing the transfer set size from $D^t = \{(x_i,y_i)\}_{i=1}^{n^t}$5 to $D^t = \{(x_i,y_i)\}_{i=1}^{n^t}$6 on CIFAR-10 improved average accuracy from about 19.2% to 75.3%, indicating a strong dependence on replay set size in this data-free regime (Pourkeshavarz et al., 2021).

In embedding-based exemplar-free incremental learning, ZSTCI and SFDNet provide a clear progression. ZSTCI reported final Task 10 average incremental accuracies on CUB-200-2011 of 52.1% for E-FT + ZSTCI, 55.0% for E-LwF + ZSTCI, 58.1% for E-EWC + ZSTCI, and 54.8% for E-MAS + ZSTCI; on CIFAR-100 the corresponding values were 8.5%, 46.1%, 43.3%, and 44.1% (Wei et al., 2020). SFDNet then reported Task 10 average incremental accuracy of 59.9% on CUB-200-2011 and 53.7% on CIFAR-100, with E-SFDNet reaching 60.4% and 54.0% respectively. Its ablation showed a progression from 36.9% to 59.9% on CUB and from 6.6% to 53.7% on CIFAR-100 as SENet, FcaNet, SFE, FFE, and the full spatial-frequency design were added (Ren et al., 2024).

In open-vocabulary and CLIP-based ZS-IL, the empirical emphasis shifts from preserving zero-shot ability to enhancing it. The complementary-memory system with TreeProbe and AIM-Emb reported, on the MTIL benchmark, Transfer 69.3, Avg. 75.9, and Last 85.5, compared with CLIP zero-shot at 69.4 / 65.3 / 65.3 and ZSCL at 68.1 / 75.4 / 83.6 (Zhu et al., 2023). MoDER reported Class-IL CI-Transfer average of 75.3 for the LoRA variant and 75.2 for the VeRA variant, compared with 55.3 for CLIP and 70.3 for CGIL, and a Final Avg. Accuracy of 87.2 compared with 86.2 for CGIL. On MTIL Order I it reported Transfer 69.7 / Avg 76.9 / Last 85.8, slightly improving over the CLIP baseline on Transfer while improving average and final seen-task performance (Panariello et al., 22 Aug 2025). SimE, while framed as class-incremental learning rather than a standalone ZS-IL formalism, reported that it surpasses traditional methods by 9.6% on TinyImageNet and outperforms other CLIP-based methods by 5.3% on CIFAR-100, and with stronger CLIP backbones reached 91.66 average and 86.03 last accuracy on CIFAR-100 in the 10-step setting (Luo et al., 11 Mar 2026).

Compositional ZS-IL exhibits a different empirical profile because zero-shot performance is evaluated on unseen compositions over time rather than only on closed-set incremental classes. On MIT-States-CompIL with CSP, the proposed pseudo-replay and primitive-distillation framework reported $D^t = \{(x_i,y_i)\}_{i=1}^{n^t}$7, $D^t = \{(x_i,y_i)\}_{i=1}^{n^t}$8, $D^t = \{(x_i,y_i)\}_{i=1}^{n^t}$9, $t$0, $t$1, $t$2, compared with vanilla incremental training at $t$3, $t$4, $t$5, $t$6, $t$7, $t$8. On C-GQA-CompIL with CSP it reported $t$9, $F_\theta(x)$0, $F_\theta(x)$1, $F_\theta(x)$2, $F_\theta(x)$3, $F_\theta(x)$4, approaching the joint-training unseen accuracy while improving average zero-shot performance over time (Li et al., 12 Nov 2025).

6. Technical tensions, limitations, and research directions

Several limitations recur across the field. Data-free replay by memory recovery avoids storing real samples but synthesizes each exemplar by iterative optimization, up to 1500 Adam steps per image in the original ZS-IL method, and the recovered images are “patterns” rather than natural-looking samples; the same paper notes challenges in scalability to very large datasets and sensitivity to hyperparameters such as $F_\theta(x)$5, $F_\theta(x)$6, $F_\theta(x)$7, and $F_\theta(x)$8 (Pourkeshavarz et al., 2021). ZSTCI explicitly aligns only adjacent tasks and provides no explicit scalability analysis for very large numbers of tasks or classes, so accumulation of translation error over long horizons remains an open issue (Wei et al., 2020). SFDNet improves representation stability but adds two ResNet-12 backbones, DCT/inverse DCT, dual attention, and VAE components; its own discussion flags computational cost, memory overhead, and limited validation beyond CUB and CIFAR-100 (Ren et al., 2024).

Semantic and VLM-based ZS-IL methods introduce a different set of dependencies. Online-CGZSL requires semantic descriptors and a meaningful similarity function over attributes; the paper explicitly presents this as semantically transductive and points to future work on dynamic feature-based attention mechanisms (Kuchibhotla et al., 2022). MoDER depends on class names and text semantics, on the quality of feature-space diffusion generators, and on a growing hub of class-specific experts, even though each expert is small and the total footprint is substantially below that of ZSCL or MoE-Adapters (Panariello et al., 22 Aug 2025). SimE’s results show strong dependence on the quality of CLIP pre-training and on adapter placement; increasing intra-block adapter complexity can degrade performance in smaller incremental steps, revealing a nonlinear relationship between added capacity and actual continual-learning behavior (Luo et al., 11 Mar 2026). CompIL, finally, relies on CLIP alignment and is designed primarily for a fixed primitive vocabulary; it shows some transfer to primitive-incremental CZSL, but the core formulation remains composition-incremental rather than fully vocabulary-expanding (Li et al., 12 Nov 2025).

A persistent misconception is that ZS-IL is simply classical zero-shot learning performed repeatedly. The surveyed formulations indicate otherwise. In some settings, zero-shot refers to no old exemplars; in others, to no old-task images but available current-task images; in others, to semantic recognition of unseen classes in a genuinely generalized incremental label space. This suggests that the central scientific question in ZS-IL is not merely whether a model can recognize unseen classes, but how semantic priors, replay, prototype memory, or modular parameterizations can be made stable under continual distributional change without collapsing old knowledge or biasing toward newly seen classes.