Generalized Zero-Shot Learning (GZSL)

• Generalized Zero-Shot Learning (GZSL) is a paradigm where models classify both seen and unseen classes, reflecting real-world challenges in object recognition.
• It employs calibrated stacking with a calibration factor to adjust bias toward seen classes, ensuring a balanced decision rule across the union of classes.
• Performance is measured using metrics like the harmonic mean and AUSUC, which evaluate accuracy trade-offs between seen and unseen classes.

Generalized Zero-Shot Learning (GZSL) is a classification paradigm in which a model is required to correctly recognize both seen and unseen classes at test time. Unlike conventional zero-shot learning, where test examples originate exclusively from unseen classes, GZSL mandates that the classifier operate over the union of seen and unseen label spaces, reflecting more realistic and challenging settings for object recognition and related tasks.

1. Problem Formulation and Foundational Metrics

In GZSL, the set of possible class labels comprises seen classes $\mathcal{S}$ (those available during training) and unseen classes $\mathcal{U}$ (not observed during training). At inference, samples may come from the entire set $\mathcal{T} = \mathcal{S} \cup \mathcal{U}$.

Performance metrics in GZSL are distinguished by their labeling of test samples and evaluated label space:

Metric	Description
$A_{\mathcal{U} \rightarrow \mathcal{U}}$	Accuracy: Test examples from $\mathcal{U}$, labeled among $\mathcal{U}$
$A_{\mathcal{S} \rightarrow \mathcal{S}}$	Accuracy: Test examples from $\mathcal{S}$, labeled among $\mathcal{S}$
$A_{\mathcal{S} \rightarrow \mathcal{T}}$	Accuracy: Test examples from $\mathcal{S}$, labeled among $\mathcal{T}$
$A_{\mathcal{U} \rightarrow \mathcal{T}}$	Accuracy: Test examples from $\mathcal{U}$, labeled among $\mathcal{T}$

GZSL typically reports $A_{\mathcal{S} \rightarrow \mathcal{T}}$ and $A_{\mathcal{U} \rightarrow \mathcal{T}}$, alongside their harmonic mean:

$$H = \frac{2 \cdot A_{\mathcal{S} \rightarrow \mathcal{T}} \cdot A_{\mathcal{U} \rightarrow \mathcal{T}}}{A_{\mathcal{S} \rightarrow \mathcal{T}} + A_{\mathcal{U} \rightarrow \mathcal{T}}}$$

This metric penalizes imbalanced performance and is considered essential for comprehensive GZSL evaluation (Chao et al., 2016).

2. The Seen-Class Bias and Calibrated Stacking

A critical empirical finding is the bias of zero-shot classifiers toward seen classes in the GZSL regime. Naïve application of standard zero-shot decision rules,

$\hat{y} = \arg\max_{c \in \mathcal{T}} f_c(x)$

where $f_c(x)$ is a compatibility function or confidence score, results in overwhelming misclassification of unseen examples as seen classes because scores for seen categories tend to dominate.

To correct this, a calibrated stacking method introduces a calibration factor $\gamma$ that penalizes seen class scores:

$$\hat{y} = \arg\max_{c \in \mathcal{T}} \big[ f_c(x) - \gamma \cdot \mathbb{I}[c \in \mathcal{S}] \big]$$

By varying $\gamma$ over a range, one can explicitly control the trade-off between favoring seen and unseen categories:

• $\gamma = 0$: No calibration, bias toward seen classes.
• $\gamma \to +\infty$: Seen class scores are suppressed; behaves as classical ZSL.
• $\gamma \to -\infty$: Model ignores unseen classes (Chao et al., 2016, Cacheux et al., 2018).

Selecting $\gamma$ via cross-validation (using a held-out validation split) to optimize the harmonic mean is empirically shown to yield more balanced and robust GZSL models (Cacheux et al., 2018).

3. Performance Curves and Unified Evaluation Metrics

To quantify the balance between seen and unseen class accuracy across all possible calibrations, the Area Under the Seen-Unseen accuracy Curve (AUSUC) is introduced. By plotting $A_{\mathcal{U} \rightarrow \mathcal{T}}$ against $A_{\mathcal{S} \rightarrow \mathcal{T}}$ as $\gamma$ varies, the AUSUC provides a single scalar summary of the best achievable trade-off for a model (Chao et al., 2016). This measure has since become a standard performance metric for GZSL methods, supporting both fair model comparison and hyperparameter selection.

4. The Role and Limitation of Semantic Embeddings

Semantic representations—such as human-defined attributes or vector embeddings—are vital for linking visual features to label spaces in ZSL and GZSL. Experimental analyses involving "idealized" or "oracle" semantic embeddings, constructed by averaging actual visual features per class (the "G-attr" approach), reveal a substantial and persistent gap between standard semantic embeddings and an upper performance bound (Chao et al., 2016).

Even few-shot improvements in semantic embedding quality substantially close the performance gap, and with "oracle" embeddings, GZSL performance can approach that of conventional fully supervised classifiers. These findings underscore that the quality and discriminativeness of semantic representations form the critical bottleneck in GZSL.

5. Adaptation and Generalization Across Models

The calibrated stacking and re-tuning for GZSL presented above apply broadly across standard ZSL families, including bilinear models (DS, ALE, SJE), compatibility learning, or regression-based methods. The adjustment procedure consists of:

1. Training a ZSL model with seen classes.
2. Applying a calibrated prediction rule penalizing seen classes at test time.
3. Cross-validating both the calibration parameter ($\gamma^*$) and the regularization coefficient ($\lambda^*_{GZSL}$) for the training objective against the harmonic mean on a development set containing both seen and held-out seen examples.
4. Re-training and evaluating using optimized hyperparameters on the union of seen and unseen classes (Cacheux et al., 2018).

Empirical results on datasets such as CUB and AwA2 show that applying calibration and GZSL-specific regularization can increase the harmonic mean performance by over 20 percentage points compared to naïvely transferring standard ZSL models.

6. Closing the Semantic Gap and Future Directions

Across benchmarks and ZSL methods, the persistent underperformance of GZSL systems is directly attributed to the insufficient granularity and discriminative power of off-the-shelf semantic embeddings. Experimentation with "oracle" embeddings demonstrates that this gap is unlikely to be mitigated by classifier architecture alone. Rather, the development of richer, more informative, and learnable semantic representations—potentially by leveraging distributed LLMs, multi-modal cues, or few-shot class descriptions—is identified as the principal route to advancing GZSL.

Recent directions include visually semantic embeddings that better bridge visual and attribute spaces (Zhu et al., 2018), domain-calibrated objective functions, and cross-modal generative feature synthesis. However, the central practical challenge remains: reducing the semantic gap and increasing the alignment between class descriptors and visual instance variation in the GZSL regime.

7. Summary Table: Key GZSL Mechanisms and Metrics

Aspect	Contribution	Reference
Classifier rule	Calibrated stacking with penalty $\gamma$	(Chao et al., 2016)
Key metric	AUSUC (Area Under Seen-Unseen Curve), harmonic mean H	(Chao et al., 2016)
Cross-validation	Hyperparameter tuning (e.g., $\gamma^*$, $\lambda^*_{GZSL}$) on harmonic mean	(Cacheux et al., 2018)
Semantic embedding gap	Measured empirically via G-attr (oracle) embeddings	(Chao et al., 2016)
Main bottleneck	Semantic representation quality	(Chao et al., 2016)

GZSL thus represents both a realistic and critical extension to zero-shot recognition: a paradigm where progress increasingly relies not only on clever learning algorithms and calibration strategies but especially on advances in class semantic representations and their alignment to real visual world structure.