Extreme Meta-Classification for Large-Scale Zero-Shot Retrieval

Abstract: We develop accurate and efficient solutions for large-scale retrieval tasks where novel (zero-shot) items can arrive continuously at a rapid pace. Conventional Siamese-style approaches embed both queries and items through a small encoder and retrieve the items lying closest to the query. While this approach allows efficient addition and retrieval of novel items, the small encoder lacks sufficient capacity for the necessary world knowledge in complex retrieval tasks. The extreme classification approaches have addressed this by learning a separate classifier for each item observed in the training set which significantly increases the representation capacity of the model. Such classifiers outperform Siamese approaches on observed items, but cannot be trained for novel items due to data and latency constraints. To bridge these gaps, this paper develops: (1) A new algorithmic framework, EMMETT, which efficiently synthesizes classifiers on-the-fly for novel items, by relying on the readily available classifiers for observed items; (2) A new algorithm, IRENE, which is a simple and effective instance of EMMETT that is specifically suited for large-scale deployments, and (3) A new theoretical framework for analyzing the generalization performance in large-scale zero-shot retrieval which guides our algorithm and training related design decisions. Comprehensive experiments are conducted on a wide range of retrieval tasks which demonstrate that IRENE improves the zero-shot retrieval accuracy by up to 15% points in Recall@10 when added on top of leading encoders. Additionally, on an online A/B test in a large-scale ad retrieval task in a major search engine, IRENE improved the ad click-through rate by 4.2%. Lastly, we validate our design choices through extensive ablative experiments. The source code for IRENE is available at https://aka.ms/irene.

• The paper presents EMMETT, a meta-classification framework that synthesizes classifiers on-the-fly to enable robust zero-shot retrieval.
• It combines a fast approximate nearest neighbor classifier selector with a transformer-based meta-classifier generator to achieve significant accuracy gains.
• Empirical tests and theoretical analysis demonstrate that freezing base encoders while updating only the generator significantly improves scalability and performance.

Extreme Meta-Classification for Large-Scale Zero-Shot Retrieval

Introduction and Problem Formulation

Large-scale text retrieval, especially as encountered in modern web search, sponsored search advertising, and product recommendation, routinely faces the challenge of operating over item pools of hundreds of millions, with novel items arriving continuously. The core technical bottleneck in rapid and accurate retrieval arises from the need for item representations that are both rich enough to encode complex world knowledge (e.g., entity and context disambiguation) and computationally efficient to update and query in production. While dense retrievers based on Siamese architectures provide rapid insertion and deployment due to the use of small encoders, their limited expressivity leads to degraded retrieval quality on complex queries. In contrast, extreme classification (XC) architectures leverage separate classifiers per item, substantially increasing representational capacity but lacking support for zero-shot (novel item) retrieval, given the impracticality of classifier training for unseen items in massive-scale settings.

The EMMETT Framework: Design and Algorithms

The paper introduces the Extreme Meta-Classification (EMMETT) framework, which generalizes the XC paradigm to zero-shot scenarios by efficiently leveraging the world knowledge distilled in the classifiers of observed items. EMMETT is architected around two modular components: a classifier selector and a meta-classifier generator.

Figure 1: Schematic of EMMETT, showing how a classifier selector retrieves informative observed item classifiers for a novel item and a transformer-based generator produces a meta-classifier on-the-fly.

Given a fixed base encoder and the pre-trained XC classifiers for the observed item set, the classifier selector rapidly extracts a small subset of the most relevant classifiers for a novel item, exploiting fast approximate nearest neighbor search (ANNS) techniques in the encoder space. The meta-classifier generator, implemented as a transformer, ingests both these retrieved classifiers and the novel item's encoder representation to synthesize a meta-classifier for the new item with minimal latency.

This flexible modularity allows for instantiation of various architectural and optimization trade-offs. One prior approach, DEXA, is shown to be a special case of EMMETT employing hard clustering with simplistic classifier aggregation, which is empirically and theoretically suboptimal for zero-shot generalization.

IRENE: Practical Instantiation for Real-World Retrieval

The IRENE algorithm is proposed as a scalable EMMETT instantiation for real-world deployment. IRENE couples an efficient ANNS-based classifier selector with a compact, single-layer transformer as the meta-classifier generator. Notably, while the encoder and base XC classifiers are fixed after pretraining, only the generator is updated for meta-classification over novel items, significantly reducing overfitting and complexity.

Figure 2: The workflow of the IRENE meta-classifier: (a) base encoder and classifier training, (b) classifier selection via ANNS, and (c) transformation and aggregation using a shallow transformer, resulting in the meta-classifier for retrieval.

For a novel item, K nearest classifiers are extracted from the observed set (empirically, $K \approx 3$ yields best generalization), and the generator synthesizes the novel item's classifier from these and the encoder embedding, leveraging type embeddings to condition the transformer. The training protocol simulates zero-shot by constructing meta-classifiers for observed items without access to their own classifiers, thereby emulating the unseen item setting.

Theoretical Analysis and Generalization Bounds

The paper develops a theoretical framework to analyze zero-shot generalization in meta-classifiers, focusing on bounding the generalization gap between empirical and true risk under sample imbalance and capacity considerations. The key insight is the equivalence between dense retrieval with calibrated scoring and binary classification over the query-item space, enabling the translation of classical Rademacher complexity theory to this setting.

It is established that the generalization error decreases with increasing training set size and is tightly controlled by the choice of the generator's complexity and the number of neighbors $K$. Concretely, the analysis yields:

• The Rademacher complexity of the function class of the IRENE generator grows as $\mathcal{O}(B ||M||_2 \sqrt{d \ln (K+1)})$, where $B$ and $M$ are norm bounds for encoder and linear layer parameters, and $d$ the feature dimensionality.
• If XC classifiers are also made trainable, complexity increases proportionally to label set size, highlighting the necessity of freezing the classifier parameters in large-scale settings.

The importance of the one-vs-all loss for variance reduction and robust generalization, along with rigorous guidance for design choices—specifically, keeping both base encoder and observed classifiers fixed—is theoretically justified.

Empirical Results and Ablations

Extensive benchmarking demonstrates the effectiveness of IRENE. Across diverse retrieval datasets (e.g., product recommendation, category annotation, query completion), IRENE achieves up to 15% absolute improvement in Recall@10 and up to 40% relative improvement in P@1 on zero-shot retrieval tasks compared to strong dense retriever baselines. In an online A/B test for sponsored search at web scale, ad click-through rates improved by 4.2% without notable deployment latency increase, validating IRENE's practical utility.

Key empirical findings:

• IRENE substantially outperforms DEXA, SemSup-XC, and state-of-the-art encoders across all tested scenarios.
• In ablation studies, transformer-based generator architectures are consistently superior to naive summation or weighted aggregation of classifiers.
• Increasing the number of selected neighbors $K$ beyond three leads to performance degradation due to increased model complexity, concurring with the theoretical complexity bounds.
• IRENE is robust to increasing proportions of novel items and adapts effectively to few-shot (even one-shot) settings without retraining the base model.

Practical and Theoretical Implications

EMMETT and IRENE highlight a scalable, modular paradigm for zero-shot retrieval at production scale—moving beyond the limitations imposed by the representation capacity of small encoders or the lack of support for novel items in classical XC. By reframing the problem as meta-classification over learned world-knowledge-rich classifiers, new items can be rapidly and accurately embedded with negligible computational overhead.

Practically, these advances suggest that zero-shot retrieval performance and representational fidelity can be improved orthogonally to expensive model scaling or retraining, with immediate applications in continual learning, search advertising, and recommender systems where item churn is high.

Theoretically, the framework provides refined prescription for future meta-classifier design, benchmarking, and analysis: capacity must be controlled, neighbor selection must prioritize informativeness rather than brute force aggregation, and the frozen classifier approach in meta-generation is both statistically and computationally justified.

Conclusion

The proposed EMMETT framework, instantiated as IRENE, establishes a new standard for efficient and accurate zero-shot retrieval in extremely large-scale item pools. By leveraging the collective world knowledge encoded in observed item classifiers and coupling efficient selection with transformer-based meta-classifier synthesis, IRENE achieves significant gains in accuracy, robustness, and deployment efficiency. The work provides a comprehensive theoretical and empirical foundation for future meta-classification research, with actionable guidance for industrial-scale deployment in high-churn, high-volume retrieval systems.