Iterative Definition Refinement for Zero-Shot Classification via LLM-Based Semantic Prototype Optimization

Abstract: Web filtering systems rely on accurate web content classification to block cyber threats, prevent data exfiltration, and ensure compliance. However, classification is increasingly difficult due to the dynamic and rapidly evolving nature of the modern web. Embedding-based zero-shot approaches map content and category descriptions into a shared semantic space, enabling label assignment without labeled training data, but remain highly sensitive to definition quality. Poorly specified or ambiguous definitions create semantic overlap in the embedding space, leading to systematic misclassification. In this paper, we propose a training-free, adaptive iterative definition refinement framework that improves zero-shot web content classification by progressively optimizing category definitions rather than updating model parameters. Using LLMs as feedback-driven definition optimizers, we investigate three refinement strategies namely example-guided, confusion-aware, and history-aware, each refining class descriptions using structured signals from misclassified instances. Furthermore, we introduce a human-labeled benchmark of 10 URL categories with 1,000 samples per class and evaluate across 13 state-of-the-art embedding foundation models. Results demonstrate that iterative definition refinement consistently improves classification performance across diverse architectures, establishing definition quality as a critical and underexplored factor in embedding-based systems. The dataset is available at https://github.com/naeemrehmat/B2MWT-10C.

• The paper proposes an iterative framework where LLMs refine category definitions to enhance embedding-based zero-shot classification.
• It leverages three strategies—example-guided, confusion-aware, and history-aware—to optimize semantic prototypes without retraining the embedding model.
• Empirical evaluations on various datasets show significant Macro F1 improvements, demonstrating the efficacy of adaptive LLM-guided prototype optimization.

Iterative Definition Refinement for Zero-Shot Web Content Classification

Problem Motivation and Semantic Prototype Limitations

Traditional web content classification, pivotal for filtering, threat mitigation, and compliance, historically relies on supervised deep learning. Although CNNs, RNNs, and transformer-based models have delivered robust performance, these approaches exhibit rigidity—requiring large annotated datasets, retraining with category evolution, and lacking zero-shot generalization. Embedding-based zero-shot classifiers offer an alternative: they operationalize category definitions as semantic prototypes in vector space, supporting label assignment via cosine similarity between content embeddings and category definition embeddings. However, classification accuracy in this framework is fundamentally constrained by the quality of category definitions. Poorly specified or semantically overlapping definitions erode prototype discriminability, leading to systematic misclassification that persists regardless of embedding model representational capacity.

Framework: Adaptive Iterative Definition Refinement

The paper introduces a training-free, adaptive iterative definition refinement framework wherein LLMs optimize category definitions using structured feedback from classification outcomes. The embedding function $f$ and the classification pipeline remain frozen throughout; only textual class descriptions—the semantic prototypes—are iteratively updated.

Three refinement strategies are explored:

• Example-Guided Refinement (M1): Updates definitions to better reflect characteristics of real category instances, grounding prototypes in data.
• Confusion-Aware Refinement (M2): Revises definitions to maximize class boundary clarity, targeting pairs with highest confusion as indicated by the confusion matrix.
• History-Aware Refinement (M3): Incorporates definition evolution trajectory, leveraging performance history to preserve beneficial changes and strategically accept updates via a simulated annealing-inspired Monte Carlo rule.

Each LLM receives prompts detailing the current definitions, Macro F1, and context (instances, confusion pairs, or historical trajectory), with strict constraints to maintain category scope and prevent invalid edits.

Experimental Setup and Benchmarking

Evaluation employs two datasets:

• N24News: Multi-class news categorization benchmark (17 classes, balanced sample distribution).
• B2MWT-10C: A new human-labeled dataset (10,000 samples across 10 semantically overlapping URL categories), released as a deployment-ready benchmark.

Thirteen embedding models spanning multilingual, LLM-derived, and encoder architectures are tested, all inference-only (parameters frozen). Four instruction-tuned LLMs are deployed for definition refinement, covering dense and mixture-of-experts paradigms.

Macro F1 is the primary metric. Iterative refinement is run for up to $T_{max}$ iterations, with confusion-aware and history-aware methods further parameterized by the number of confused pairs ($k$) and definition history length ($m$).

Numerical Results and Empirical Insights

Across both datasets and all models, iterative definition refinement consistently yields positive gains over zero-shot baseline. On B2MWT-10C the strongest improvement is $+19.01$ Macro F1 for Embeddinggemma-300m, and $+12.84$ for Voyage-4-nano. On N24News, Multilingual-e5-large achieves $+19.90$, and Llama-nemotron-embed-1b-v2 $+15.78$.

Refinement strategies manifest distinct efficacy:

• Confusion-aware and history-aware refinement (M2, M3) outperform example-guided (M1), especially in resolving boundary ambiguity and persistent prototype overlap.
• Monte Carlo acceptance in M3 prevents premature convergence: allowing temporary acceptance of suboptimal definitions, facilitating the exploration and resolution of semantic local optima.
• LLM and hyperparameter selection interact nontrivially with embedding architecture; Mistral and Gemma LLMs most frequently optimize Macro F1, but best pairing varies across datasets.

Qualitative t-SNE analyses confirm reduction of inter-class semantic overlap—initially confused definitions migrate to more discriminative positions post-refinement, as visualized by distinctly separated test point clusters.

Definition evolution in highly confused classes (see supplementary Table 4) illustrates how iterative editing introduces crisp, domain-specific exclusion rules and transactional granularity, directly translating to improved classification scores.

Practical and Theoretical Implications

Definition quality is established as a critical, previously underexplored factor in embedding-based zero-shot classification. The proposed framework enables deployment of highly adaptive web content classifiers without costly fine-tuning or retraining, supporting rapid evolution of semantic category boundaries in dynamic environments such as compliance systems and cyber-threat detection. This approach also generalizes to other domains where class definitions are complex and ambiguous, with potential to drive foundation model utility in settings where labeled data is scarce or impractical.

From a theoretical perspective, the results underscore the interplay between semantic prototype specification and geometric separation in embedding space—revealing that improving textual prototype representations can yield discriminative gains even when backbone models are fixed. The integration of classical optimization via simulated annealing into LLM-guided refinement further opens avenues for principled exploration of definition space in downstream tasks.

Future Directions

Future studies may extend this iterative refinement methodology to vision-language or multimodal classification, systematically probe hyperparameter sensitivity ($k$, $m$, annealing schedule), and explore automated discovery of optimal refinement strategy/model pairings. The linkage between definition quality and embedding-induced inductive bias warrants deeper investigation, particularly with respect to domain transfer and cross-lingual category adaptation.

Conclusion

Iterative definition refinement via LLM-guided semantic prototype optimization offers a robust mechanism to improve zero-shot classification performance for web content, without retraining embedding models. The approach demonstrates strong quantitative gains across diverse models and datasets, establishing category definition quality as a critical lever for embedding-based classification systems. This research provides a scalable methodology and practical guidance for adaptive deployment in dynamic web filtering and content analysis contexts (2604.27335).