Zero-Shot Classification: Principles & Techniques

• Zero-shot classification is a method that uses semantic descriptors to predict labels for classes absent in training, enabling transfer to novel domains.
• It employs compatibility functions, metric learning, and generative approaches to bridge the gap between seen and unseen classes, achieving notable improvements on benchmarks.
• Recent advancements incorporate prompt tuning, hyperdimensional computing, and synthetic feature generation to mitigate domain shift and class imbalance challenges.

Zero-Shot Classification (ZSC) is the problem of predicting labels for classes absent from the training set by leveraging auxiliary semantic information that enables transfer to novel concepts. ZSC has become a foundational strategy across vision, language, and audio domains, motivated by real-world data sparsity, long-tail distributions, and the need for scalable, open-set recognition.

1. Formal Problem Definition and Core Principles

Let $\mathcal{X}$ denote the input space (images, text, audio) and $\mathcal{Y}$ the complete label set, partitioned into disjoint "seen" classes $\mathcal{Y}^{\rm tr}$ and "unseen" classes $\mathcal{Y}^{\rm ts}$ such that $$\mathcal{Y}^{\rm tr}\cap \mathcal{Y}^{\rm ts}=\emptyset$$. Training data $$\mathcal{D}^{\rm tr} = \{(x_i, y_i) : x_i\in\mathcal{X}, y_i\in\mathcal{Y}^{\rm tr}\}$$ is available, while the task is to assign labels in $\mathcal{Y}^{\rm ts}$ to new examples. Each class $y\in\mathcal{Y}$ is associated with semantic descriptors $z_y$ such as attribute vectors, word embeddings, or class-level text.

The canonical ZSC prediction rule is

$$\hat y = \arg\max_{y\in\mathcal{Y}^{\rm ts}} F(x, z_y)$$

where $F$ is a compatiblity or similarity function learned end-to-end, or designed via auxiliary models and heuristics (Molina et al., 2021).

Generalized zero-shot classification (GZSC) evaluates the classifier over both seen and unseen classes at test time, further demanding calibrated transfer.

2. Methodological Taxonomy

2.1 Compatibility-based and Metric Learning Approaches

The dominant early paradigm learns a compatibility function $F(x, z)$, often bilinear: $F(x,z) = \phi(x)^\top W z$, where $\phi(x)$ is a high-dimensional input embedding (e.g., deep features) and $z$ is the semantic descriptor (Molina et al., 2021). SJE [Akata et al. 2015] optimizes a structured hinge loss; ESZSL [Romera-Paredes & Torr 2015] uses Frobenius-regularized least-squares with a closed-form solution; both methods perform nearest prototype assignment at inference.

Metric learning extensions enforce margin-based separation between image-attribute pairs, with additional innovations such as hard negative mining—in which negatives close to true attributes are sampled at higher rate to sharpen decision boundaries and increase generalizability (Bucher et al., 2016). Uncertainty- and correlation-based negative sampling combinations can yield gains of up to +9% mean accuracy and 4$\times$ faster convergence on benchmarks such as AwA and CUB.

Graph-based methods, notably the semantic graph with absorbing Markov chains (Fu et al., 2014), construct a k-NN graph over class prototypes in the semantic embedding space. Test examples are connected to seen classes via their posterior outputs; the final predicted label is the unseen class with maximum absorbing probability, computed in closed form.

2.2 Generative and Feature Synthesis Approaches

Generative methods reframe ZSC as feature imputation: a generator $G$ (e.g., GMMN, AC-GAN, cDAE, AAE) trains on seen classes to output synthetic features for unseen classes conditional on semantic embeddings. After generation, standard supervised classifiers (e.g., softmax) trained on the union of real (seen) and synthetic (unseen) data enable both ZSC and GZSC (Bucher et al., 2017). GMMN-based generation with MMD loss achieves state-of-the-art results, with GZSC accuracy for unseen test samples classifying over all classes exceeding 30% on AwA, compared to low single digits for compatibility methods.

Diffusion-based synthetic data generation further generalizes the paradigm: large, diverse synthetic datasets enable training of arbitrary classifier architectures for model-agnostic ZSC (MA-ZSC). Shipard et al. demonstrate that combining prompt variation, style conditioning, and randomized guidance scales in Stable Diffusion v1.4 lifts zero-shot accuracy on CIFAR-10 to 81.0%, surpassing CLIP-ResNet50's 75.6% (Shipard et al., 2023).

2.3 Injection, Prompt, and Hyperdimensional Computing Methods

Post-hoc classifier injection methods construct unseen-class classifier weights via encoder-decoder autoencoding between semantic descriptors and last-layer weights of a pretrained model, without using any images (Christensen et al., 2023). The ICIS approach outperforms prior “image-free” methods by 7–16% zero-shot accuracy depending on the dataset.

Hyperdimensional Computing Zero-Shot Classifiers (HDC-ZSC) encode class attribute vectors using stationary binary codebooks and vector-symbolic binding, coupled with a standard deep encoder and cosine similarity classifier. This minimalist, hardware-friendly scheme achieves $63.8\%$ top-1 accuracy on CUB-200, exceeding non-generative ZSC baselines with less than 2$\times$ parameter cost (Ruffino et al., 2024).

Prompt-tuned LLMs (e.g., RoSPrompt) leverage soft, learnable prompts atop frozen multilingual PLMs and multilingual verbalizers to yield data-efficient, cross-lingual ZSC. RoSPrompt achieves +7–14 points over hard-prompting or soft-prompting baselines in zero-shot topic and intent classification across up to 106 languages (Philippy et al., 25 Mar 2025).

2.4 Domain-Specific Extensions

Zero-shot audio-to-intent classification synthesizes embeddings of “virtual” audio samples by neural TTS from developer-provided text, enabling cosine-based nearest neighbor classification for unseen intents (Elluru et al., 2023).

Zero-shot one-class classification utilizes LLM-generated negative prototypes and vision-language similarities to establish thresholds for inclusion, demonstrating robust binary discrimination from only a label string and outperforming traditional out-of-distribution baselines (Bendou et al., 2024).

3. Data, Evaluation Protocols, and Performance Metrics

Evaluation commonly relies on standardized datasets with disjoint seen/unseen splits, e.g., AwA, CUB, SUN, aPY, and ImageNet (Bucher et al., 2017, Molina et al., 2021).

Metrics are typically:

• Per-class or per-sample Top-1 accuracy on unseen classes (ZSC).
• Generalized ZSC: accuracy on unseen ($Acc_u$), seen ($Acc_s$), and their harmonic mean $H$ to balance between class sets (Christensen et al., 2023).
• Macro-F1, AUC, and domain-specific scores (e.g., intent accuracy for audio classification (Elluru et al., 2023)).

Variability in ZSC performance is substantial across different training/test splits; for instance, ESZSL and SJE report up to $\pm 11\%$ standard deviation in mean accuracy over 22 random partitions on AWA datasets (Molina et al., 2021). Ensemble techniques (bagging over seen class subsets) halve the variance with negligible impact on mean accuracy.

4. Challenges: Class Imbalance, Domain Shift, and Multilinguality

Real-world ZSC is hindered by class imbalance, with rare seen classes poorly covered by batch SGD leading to suboptimal transfer to related unseen classes. Semantics-guided class imbalance learning addresses this by balanced batch construction and representativeness-weighted prototypes. On imbalanced datasets AwA1/2 and aPY, this raises ZSC accuracy on unseen classes by +3–4% and GZSC harmonic mean by up to +5.3% (Ji et al., 2019).

Cross-lingual and low-resource ZSC is constrained by dictionary or verbalizer coverage. Dictionary-based training, using labeled pairs from synonym and translation dictionaries, consistently surpasses NLI-based entailment models: on Luxembourgish datasets, LETZ-SYN yields 52.1–66.1% accuracy versus 17.5–50.7% for NLI-trained mBERT (Philippy et al., 2024). Prompt-based multilingual models further close the gap by tuning soft prompts over curated verbalizer tokens and leveraging few-shot data from high-resource languages (Philippy et al., 25 Mar 2025).

Performance of ZSC models deteriorates under domain shift and when semantic descriptors are inadequate; generative and prompt-tuned approaches display higher robustness, especially with careful penalty regularization, contrastive smoothing, and hard-negative augmentation.

5. Analysis, Limitations, and Future Directions

ZSC relies intrinsically on the appropriateness of semantic descriptors—whether attributes, word vectors, dictionary entries, or hard-coded prompts. The class-conditional discriminativeness and coverage of descriptors govern upper-bound accuracy.

Generative and synthetic approaches reduce “seen-class bias” endemic in compatibility-based schemes by equalizing training distributions (Bucher et al., 2017). However, sample quality and domain gap in synthetic data remain limiting; substantial accuracy gains are realized when diversity-enhancing “tricks” (randomized prompts, guidance scales, domain styles) are systematically applied in synthetic dataset creation (Shipard et al., 2023).

Method-specific limitations include:

• Dependence on dictionary or attribute coverage and quality for languages and rare classes (Philippy et al., 2024, Ruffino et al., 2024).
• Descriptor ambiguity or lack of granularity limiting fine-grained classification (Christensen et al., 2023).
• Computational scalability for very large class sets (SCILM batch size scaling) (Ji et al., 2019).
• Domain- or language-specific thresholds and calibration requirements in zero-shot one-class settings (Bendou et al., 2024).
• Ensemble-based bagging mitigates performance variance but may not address systematic undertransfer to certain semantic regions (Molina et al., 2021).

Emerging directions include combining synthetic generation with domain adaptation, integrating contextualized and multimodal descriptors, advancing few-shot expansion for low-resource classes and domains, and developing hardware-specialized ZSC models via hyperdimensional or binary neural representations (Ruffino et al., 2024).

6. Empirical Performance Summary

Dataset	Method	Top-1 Unseen Acc (%)	GZSC Harmonic Mean (%)
AwA1	SJE (Molina et al., 2021)	~69.7	–
AwA2	ICIS (Christensen et al., 2023)	64.6	51.6
CUB-200	HDC-ZSC (Ruffino et al., 2024)	63.8	–
CIFAR-10	MA-ZSC (Synth+Diversity)	81.0	–
LuxNews	LETZ-SYN (mBERT)	66.1	47.7
SIB-200	RoSPrompt (XLM-R-L)	65–75	56
iNaturalist	ZS-1C (LLM+VLM, ANP+FT)	81.2–85.9 (macro-F1)	–

Performance figures are as reported in the corresponding works (Ruffino et al., 2024, Shipard et al., 2023, Christensen et al., 2023, Philippy et al., 2024, Philippy et al., 25 Mar 2025, Bendou et al., 2024), reflecting state-of-the-art non-generative and generative ZSC across language, vision, and audio modalities.

7. Practical Impact and Recommendations

Zero-shot classification enables robust scaling of classification systems to a rapidly evolving universe of categories and languages. For best practice, report mean ± standard deviation over multiple class splits (Molina et al., 2021), prefer synthetic generation for GZSC scenarios, augment class descriptors with hard or adaptive negatives, and leverage soft or multilingual prompt tuning for low-resource and cross-domain applications (Ji et al., 2019, Philippy et al., 2024, Philippy et al., 25 Mar 2025). Future advances depend on improved class-representative semantics, scalable graph or hyperdimensional encodings, and more comprehensive evaluation on diverse, imbalanced, and multilingual benchmarks.