---
title: Zero-Shot Incremental Learning
url: https://www.emergentmind.com/topics/zero-shot-incremental-learning-zs-il
type: topic
---

# Zero-Shot Incremental Learning

Zero-shot incremental learning (ZS-IL) denotes a family of continual-learning settings in which a model must incorporate new data over time while preserving prior competence and maintaining some form of zero-shot capability. Across the literature, the phrase covers several related but non-identical regimes: replaying past knowledge without stored real exemplars, translating or aligning representations across incremental tasks without old-task images, and continually recognizing classes that remain unseen in the visual stream but are available through semantic descriptors or a pre-trained vision-language model. The unifying technical concerns are catastrophic forgetting, semantic drift between tasks, absence or restriction of rehearsal, and evaluation over an expanding label space that may contain both previously learned and still-unseen classes [2103.12216] [2012.15497] [2203.16517].

## 1. Conceptual scope and terminology

The literature uses “zero-shot” in several senses rather than a single canonical one. In the original paper titled “ZS-IL: Looking Back on Learned Experiences For Zero-Shot Incremental Learning” [2103.12216], the term refers to **zero real exemplars**: past experiences are replayed using synthesized samples generated by querying the learner’s own parameters, with no exemplar memory buffer and no auxiliary/generative network. In “Incremental Embedding Learning via Zero-Shot Translation” [2012.15497], the zero-shot aspect is **zero-shot translation between embedding spaces**: previous-task classes are treated as unseen during the current step because old images are unavailable, and the system aligns old and current embeddings using only current-task images. In continual generalized zero-shot learning and open-vocabulary settings, the zero-shot component is closer to classical ZSL or GZSL: unseen classes are represented by attributes, word vectors, prompts, or text embeddings, and must be recognized jointly with seen classes as tasks arrive [2203.16517] [2307.01430] [2508.16463].

| Paradigm | Zero-shot aspect | Representative works |
|---|---|---|
| Data-free replay | No real past samples are stored; past experiences are replayed using synthesized samples generated by the learner itself | [2103.12216] |
| Embedding-space incremental learning | No old-task images; embeddings or prototypes are aligned across tasks by a zero-shot translation model | [2012.15497], [2402.07216] |
| Continual generalized/open-vocabulary learning | Unseen classes are available through semantic descriptors or VLM priors and are recognized jointly with incrementally learned classes | [2203.16517], [2307.01430], [2508.16463] |

This terminological plurality is not merely lexical. It changes what is treated as unavailable at task \(t\): raw images of old classes, attributes of future classes, open-vocabulary textual labels, or some combination of these. A plausible implication is that ZS-IL is better viewed as a structured subfield of continual learning defined by **missing modalities or missing data channels under temporal shift**, rather than by one fixed protocol.

## 2. Formal problem formulations and evaluation regimes

A common embedding-space formulation uses a sequence of tasks \(t = 1,2,\dots,T\), each with a disjoint label set \(C^t\), and only current-task data \(D^t = \{(x_i,y_i)\}_{i=1}^{n^t}\) available at step \(t\). The model is an embedding network \(F_\theta(x)\), classification is performed by a nearest-class-mean classifier, and the central difficulty is not logit drift but the **semantic gap** between adjacent embedding spaces: prototypes computed under \(F_{\theta^{t-1}}\) do not remain compatible with features from \(F_{\theta^t}\). ZSTCI therefore learns two residual translators, \(g_{\text{old}}\) and \(g_{\text{cur}}\), maps old-space and current-space embeddings into a common embedding space, and optimizes an alignment loss together with a triplet-based unified-representation loss [2012.15497]. In this regime, two standard metrics are the **Average Incremental Accuracy**,
$$
A_k = \frac{1}{k}\sum_{j=1}^{k} a_{k,j},
$$
and **Average Forgetting**,
$$
f_j^k = \max_{l \in \{1,\dots,k-1\}} (a_{l,j} - a_{k,j}), \qquad
F_k = \frac{1}{k-1}\sum_{j=1}^{k-1} f_j^k.
$$

Continual generalized zero-shot learning broadens the setting. In the proposed **Online-CGZSL** formulation, tasks arrive sequentially; each task may introduce new seen classes with visual data and new unseen classes with only semantic descriptors, and previously unseen classes may later become seen. At task \(t\), training uses current seen-class data plus replayed features, while evaluation is conducted jointly over all seen and unseen classes encountered so far. The standard summary metrics are mean seen accuracy, mean unseen accuracy, mean harmonic accuracy, and mean AUSUC, with the harmonic term defined taskwise over seen and unseen test splits [2203.16517]. This is a direct formalization of a ZS-IL regime in which both the semantic space and the visual evidence evolve over time.

Open-vocabulary continual learning with CLIP instantiates another regime. A frozen zero-shot model provides scores over arbitrary textual labels, while an exemplar-based or modular component learns incrementally from arriving labeled images. In the class-incremental scenario of “Continual Learning in Open-vocabulary Classification with Complementary Memory Systems” [2307.01430], classes are partitioned incrementally, and evaluation is performed jointly over seen and unseen classes. In composition-incremental learning, the label space itself is structured: attributes \(\mathcal{A}\), objects \(\mathcal{O}\), and compositions \(c=(a,o)\in\mathcal{A}\times\mathcal{O}\). The model sees new compositions over tasks, while a static unseen composition pool remains held out; the principal summary metrics become
$$
mU = \frac{1}{T}\sum_{i=1}^T U_i, \qquad
mAUC = \frac{1}{T}\sum_{i=1}^T AUC_i, \qquad
fS = \frac{1}{T}\sum_{i=1}^T (S_{i,i} - S_{T,i}),
$$
which quantify zero-shot generalization over time and forgetting on previously learned compositions [2511.09082].

## 3. Core methodological families

One family addresses ZS-IL through **memory recovery and synthetic replay**. The learner network \(\mathcal{L}\) itself is treated as a memory source: the method samples candidate softmax vectors from a Dirichlet distribution conditioned by a dynamic confusion matrix, filters them with the constraint \(\|\mathcal{V}_j^i - CM^i\|_2^2 < \eta\), and then optimizes random noise images so that the learner outputs those accepted vectors. The resulting on-call transfer set \(\mathcal{S}\) is used with a combined objective \(\ell_{Total} = \ell_{CE} + \lambda \ell_{KD}\), where the new-task term is ordinary cross-entropy and the old-task term distills logits on synthetic samples [2103.12216]. The same replay logic reappears in later forms as feature-space generation rather than image synthesis.

A second family is **prototype translation in embedding space**. ZSTCI learns an embedding network with triplet loss, stores class prototypes, and after each task trains two residual MLPs:
$$
\tilde{m}_i^t = \tilde{z}_i^t + g_{\text{old}}(\tilde{z}_i^t), \qquad
m_i^t = z_i^t + g_{\text{cur}}(z_i^t),
$$
using the alignment loss
$$
\mathcal{L}_{\text{align}} = \frac{1}{n^t}\sum_{i=1}^{n^t}\left\|\tilde{m}_i^t - m_i^t\right\|_1.
$$
Old and new prototypes are then translated into a common space and classified by nearest-class-mean [2012.15497]. SFDNet retains the same zero-shot translation structure but enriches the feature extractor with a Spatial-Frequency Feature Extraction module, an Attention Feature Alignment module, CADA-VAE-based alignment, and compensation loss. It explicitly targets the semantic gap by combining spatial-domain and frequency-domain cues, SENet and FcaNet attention, and prototype compensation across tasks [2402.07216].

A third family is **semantic or open-vocabulary continual learning**. In Online-CGZSL, a feature generator \(G_\theta(z,a)\) synthesizes visual features conditioned on attributes, a discriminator/projector \(D_\phi(a)\) acts as a class prototype in feature space, and the model is trained with a cosine-similarity GAN objective, classification losses, bi-directional incremental alignment, and generative replay [2203.16517]. In open-vocabulary CLIP settings, the zero-shot model remains frozen and an incremental memory system supplies fast adaptation. The complementary-memory approach combines CLIP zero-shot probabilities with an exemplar model by estimating \(p(y\in \mathbf{Y}_e\mid I)\) from CLIP itself and then performing adaptive fusion. MoDER moves this one step further by storing one textual LoRA expert per seen class in a foundational hub and recomposing experts for unseen classes via semantic similarity in CLIP’s text space [2307.01430] [2508.16463]. SimE instead freezes a CLIP image encoder, trains adapters only on the base task, concatenates the original and adapted features, and handles later tasks by prototype extension rather than further encoder updates [2603.11211].

## 4. Representative architectures and specialized variants

SFDNet is a representative architecture for embedding-based ZS-IL. Its spatial branch uses ResNet-12 and global average pooling, while the frequency branch applies a 2D discrete cosine transform, assigns different scaling coefficients to high- and low-frequency regions, reconstructs an original-like image together with high-frequency and low-frequency images, and feeds them through another ResNet-12. The Attention Feature Alignment module applies SENet and FcaNet, then aligns attention-enriched streams through CADA-VAE with
$$
{\cal L}_{CADA-VAE} = {\cal L}_{VAE} + \alpha {\cal L}_{CA} + \beta {\cal L}_{DA},
$$
and the total objective is
$$
{\cal L}_{ALL} = {\cal L}_{CADA-VAE} + {\cal L}_{compensation}.
$$
The final embedding is formed by aligning spatial and frequency features and concatenating the aligned outputs, after which nearest-class-mean and zero-shot translation operate in the fused space [2402.07216].

Zero-shot incremental learning also extends to object detection. Incrementally Zero-Shot Detection defines three simultaneous class sets at each step: old seen classes \(\mathcal{O}\), new seen classes \(\mathcal{N}\), and unseen classes \(\mathcal{U}\). IZSD-EVer combines a Faster R-CNN backbone, a semantic embedding branch, an incremental classifier branch, an old–new model for distillation, and an **Extreme Value Analyzer** that decides whether a proposal should be classified by the seen-class incremental classifier or by the unseen-class semantic classifier. The open-set gate is based on a Generalized Pareto Distribution fitted to class-wise tail distances in semantic space, with the decision rule
$$
\hat{y}= \begin{cases}
\arg\max_{i\in\{1,\dots,S\}} \mathbf{p}^{ic}_i, & P_{\min}<\delta,\\[3pt]
\arg\max_{i\in\{1,\dots,U\}} \mathbf{p}^{zsc}_i, & P_{\min}\ge\delta.
\end{cases}
$$
The training objective includes bfMSE to handle background–foreground imbalance, projection distance loss \(\mathcal{L}_{pd}^{zsc} = \|\mathbf{s}_o - \mathbf{s}_n\|_2^2\), RPN feature distillation, and classification distillation [2103.12609].

Compositional ZS-IL introduces a structured label space. In CompIL, the model gradually learns new attribute–object compositions, reserves a static unseen composition pool for testing, and uses a pseudo-replay framework with a visual synthesizer and **linguistic primitive distillation**. The synthesizer is a composition-conditioned VAE in CLIP feature space trained by
$$
\mathcal{L}_{\mathrm{VS}} = \mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{kl}} + \alpha\cdot \mathcal{L}_{\mathrm{sem}},
$$
while incremental CZSL training uses
$$
\mathcal{L} = \mathcal{L}_{\mathrm{ms}} + \beta\cdot \mathcal{L}_{\mathrm{kd}}.
$$
Here the distillation term constrains logits over all past compositions so that primitive representations remain aligned across tasks, directly targeting the drift of attribute and object semantics under incremental updates [2511.09082].

## 5. Empirical findings across benchmark families

The earliest explicit ZS-IL formulation based on memory recovery showed that synthetic replay from the learner’s own parameters can be competitive with or stronger than both data-free and buffer-based baselines. On CIFAR-10, ZS-IL reported **75.34** in Class-IL and **93.12** in Task-IL; on Tiny-ImageNet it reported **33.06** in Class-IL and **67.42** in Task-IL. The same paper reported that increasing the transfer set size from \(K=0\) to \(K=6000\) on CIFAR-10 improved average accuracy from about **19.2%** to **75.3%**, indicating a strong dependence on replay set size in this data-free regime [2103.12216].

In embedding-based exemplar-free incremental learning, ZSTCI and SFDNet provide a clear progression. ZSTCI reported final Task 10 average incremental accuracies on CUB-200-2011 of **52.1%** for E-FT + ZSTCI, **55.0%** for E-LwF + ZSTCI, **58.1%** for E-EWC + ZSTCI, and **54.8%** for E-MAS + ZSTCI; on CIFAR-100 the corresponding values were **8.5%**, **46.1%**, **43.3%**, and **44.1%** [2012.15497]. SFDNet then reported Task 10 average incremental accuracy of **59.9%** on CUB-200-2011 and **53.7%** on CIFAR-100, with **E-SFDNet** reaching **60.4%** and **54.0%** respectively. Its ablation showed a progression from **36.9%** to **59.9%** on CUB and from **6.6%** to **53.7%** on CIFAR-100 as SENet, FcaNet, SFE, FFE, and the full spatial-frequency design were added [2402.07216].

In open-vocabulary and CLIP-based ZS-IL, the empirical emphasis shifts from preserving zero-shot ability to enhancing it. The complementary-memory system with TreeProbe and AIM-Emb reported, on the MTIL benchmark, **Transfer 69.3**, **Avg. 75.9**, and **Last 85.5**, compared with CLIP zero-shot at **69.4 / 65.3 / 65.3** and ZSCL at **68.1 / 75.4 / 83.6** [2307.01430]. MoDER reported Class-IL **CI-Transfer** average of **75.3** for the LoRA variant and **75.2** for the VeRA variant, compared with **55.3** for CLIP and **70.3** for CGIL, and a **Final Avg. Accuracy** of **87.2** compared with **86.2** for CGIL. On MTIL Order I it reported **Transfer 69.7 / Avg 76.9 / Last 85.8**, slightly improving over the CLIP baseline on Transfer while improving average and final seen-task performance [2508.16463]. SimE, while framed as class-incremental learning rather than a standalone ZS-IL formalism, reported that it **surpasses traditional methods by 9.6% on TinyImageNet** and **outperforms other CLIP-based methods by 5.3% on CIFAR-100**, and with stronger CLIP backbones reached **91.66** average and **86.03** last accuracy on CIFAR-100 in the 10-step setting [2603.11211].

Compositional ZS-IL exhibits a different empirical profile because zero-shot performance is evaluated on unseen compositions over time rather than only on closed-set incremental classes. On MIT-States-CompIL with CSP, the proposed pseudo-replay and primitive-distillation framework reported **\(U=49.11\), \(S=41.43\), \(AUC=16.77\), \(mU=47.23\), \(fS=3.28\), \(mAUC=17.99\)**, compared with vanilla incremental training at **\(U=39.78\), \(S=30.55\), \(AUC=9.76\), \(mU=41.41\), \(fS=15.22\), \(mAUC=13.59\)**. On C-GQA-CompIL with CSP it reported **\(U=27.54\), \(S=20.81\), \(AUC=4.60\), \(mU=26.71\), \(fS=5.67\), \(mAUC=5.42\)**, approaching the joint-training unseen accuracy while improving average zero-shot performance over time [2511.09082].

## 6. Technical tensions, limitations, and research directions

Several limitations recur across the field. Data-free replay by memory recovery avoids storing real samples but synthesizes each exemplar by iterative optimization, up to **1500** Adam steps per image in the original ZS-IL method, and the recovered images are “patterns” rather than natural-looking samples; the same paper notes challenges in scalability to very large datasets and sensitivity to hyperparameters such as \(\lambda\), \(\eta\), \(\tau\), and \(K\) [2103.12216]. ZSTCI explicitly aligns only adjacent tasks and provides no explicit scalability analysis for very large numbers of tasks or classes, so accumulation of translation error over long horizons remains an open issue [2012.15497]. SFDNet improves representation stability but adds two ResNet-12 backbones, DCT/inverse DCT, dual attention, and VAE components; its own discussion flags computational cost, memory overhead, and limited validation beyond CUB and CIFAR-100 [2402.07216].

Semantic and VLM-based ZS-IL methods introduce a different set of dependencies. Online-CGZSL requires semantic descriptors and a meaningful similarity function over attributes; the paper explicitly presents this as semantically transductive and points to future work on dynamic feature-based attention mechanisms [2203.16517]. MoDER depends on class names and text semantics, on the quality of feature-space diffusion generators, and on a growing hub of class-specific experts, even though each expert is small and the total footprint is substantially below that of ZSCL or MoE-Adapters [2508.16463]. SimE’s results show strong dependence on the quality of CLIP pre-training and on adapter placement; increasing intra-block adapter complexity can degrade performance in smaller incremental steps, revealing a nonlinear relationship between added capacity and actual continual-learning behavior [2603.11211]. CompIL, finally, relies on CLIP alignment and is designed primarily for a fixed primitive vocabulary; it shows some transfer to primitive-incremental CZSL, but the core formulation remains composition-incremental rather than fully vocabulary-expanding [2511.09082].

A persistent misconception is that ZS-IL is simply classical zero-shot learning performed repeatedly. The surveyed formulations indicate otherwise. In some settings, zero-shot refers to **no old exemplars**; in others, to **no old-task images but available current-task images**; in others, to **semantic recognition of unseen classes** in a genuinely generalized incremental label space. This suggests that the central scientific question in ZS-IL is not merely whether a model can recognize unseen classes, but how semantic priors, replay, prototype memory, or modular parameterizations can be made stable under continual distributional change without collapsing old knowledge or biasing toward newly seen classes.

Source: https://www.emergentmind.com/topics/zero-shot-incremental-learning-zs-il