Fine-tuning Small Language Models as Efficient Enterprise Search Relevance Labelers

Abstract: In enterprise search, building high-quality datasets at scale remains a central challenge due to the difficulty of acquiring labeled data. To resolve this challenge, we propose an efficient approach to fine-tune small LLMs (SLMs) for accurate relevance labeling, enabling high-throughput, domain-specific labeling comparable or even better in quality to that of state-of-the-art LLMs. To overcome the lack of high-quality and accessible datasets in the enterprise domain, our method leverages on synthetic data generation. Specifically, we employ an LLM to synthesize realistic enterprise queries from a seed document, apply BM25 to retrieve hard negatives, and use a teacher LLM to assign relevance scores. The resulting dataset is then distilled into an SLM, producing a compact relevance labeler. We evaluate our approach on a high-quality benchmark consisting of 923 enterprise query-document pairs annotated by trained human annotators, and show that the distilled SLM achieves agreement with human judgments on par with or better than the teacher LLM. Furthermore, our fine-tuned labeler substantially improves throughput, achieving 17 times increase while also being 19 times more cost-effective. This approach enables scalable and cost-effective relevance labeling for enterprise-scale retrieval applications, supporting rapid offline evaluation and iteration in real-world settings.

• The paper introduces a distillation paradigm where a small language model is fine-tuned using synthetic data to replace large LLMs in enterprise search relevance labeling.
• It details a multi-stage curriculum employing GPT-4o for query synthesis, BM25 for negative mining, and graded labeling to enhance label quality.
• Results show fine-tuned SLMs achieve 0.953 NDCG and 63.81% pairwise accuracy, offering 17× cost reduction and 7,000 RPM throughput compared to LLM alternatives.

Fine-tuning Small LLMs for High-Throughput, Cost-Efficient Enterprise Search Relevance Labeling

Introduction

The paper "Fine-tuning Small LLMs as Efficient Enterprise Search Relevance Labelers" (2601.03211) systematically addresses the pressing bottleneck in enterprise search: generating high-quality relevance labels at scale under constraints of privacy, domain specificity, and resource efficiency. Unlike traditional web search, enterprise search environments encompass a heterogeneous landscape of internal documents, chats, emails, and knowledge bases, often characterized by ambiguous, context-dependent, and persona-specific queries. The challenge is exacerbated by the lack of large-scale, public enterprise-labeled datasets, and the impracticality of manual annotation. While LLMs, such as GPT-4o, have demonstrated robust performance, their prohibitive cost and limited throughput render them less viable for large-scale, iterative evaluation pipelines integral to real-world information retrieval (IR) system development.

As an alternative, the authors propose a distillation paradigm leveraging Small LLMs (SLMs), specifically Phi-3.5 Mini Instruct, fine-tuned using a synthetic, high-quality dataset generated through a combination of LLM-driven query synthesis, BM25-based negative mining, and LLM-generated graded relevance judgments. This end-to-end automatic pipeline eschews reliance on sensitive enterprise query logs and manual labels, providing an efficient and privacy-compliant framework for scalable relevance labeling.

Synthetic Data Generation and Distillation Pipeline

The core methodological contribution is the construction of a synthetic query-document-label dataset reflecting the complexity and linguistic distribution of enterprise search. The pipeline consists of four major stages:

1. Positive Query Generation: GPT-4o generates diverse user-like queries from curated enterprise documents using templates grounded in metadata and content. The prompt strategy employs explicit keyword extraction and pattern-driven query assembly, with a secondary LLM-based rewriting stage to maximize lexical and semantic diversity.
2. Negative Document Mining: For each synthetic query, BM25 retrieves $k=4$ hard negatives—documents lexically similar to the query but semantically distinct—producing realistic candidate sets encompassing both near-miss cases and true negatives. This step is essential to capture the ambiguous boundaries of relevance prevalent in enterprise corpora.
3. LLM-Based Labeling: Each query-document pair undergoes evaluation via GPT-4o, which assigns a graded relevance score on a discrete 0–4 scale. Post-processing filters address label consistency, discarding ambiguous examples and enhancing dataset fidelity.
4. SLM Fine-Tuning: The synthetic dataset is combined with open-data IR resources (INTERS, TREC-CAsT, and MS MARCO) for multi-task supervised fine-tuning of Phi-3.5 Mini Instruct. This curriculum is designed to improve both robustness and generalization across hybrid lexical/semantic queries characteristic of the enterprise domain.

The model architecture is illustrated in Figure 1.

Figure 1: The distillation architecture: SLM is fine-tuned using a multi-dataset curriculum after synthesis of high-quality, diversified enterprise queries and hard-negative augmentation.

The data augmentation and example generation steps are further detailed in Figure 2.

Figure 2: Synthetic query generation workflow, integrating LLM-based keyword extraction, query templates, and diversity-promoting rewrites.

The supervised fine-tuning (SFT) workflow involving sequential curriculum stages is shown in Figure 3.

Figure 3: Multi-stage fine-tuning: SLM is pre-trained on INTERS for multi-task robustness, followed by fine-tuning leveraging both synthetic enterprise-specific data and diverse open-source IR benchmarks.

Experimental Evaluation

The evaluation protocol employs a proprietary, human-labeled gold standard comprising 923 enterprise query-document pairs. Performance is assessed primarily through full Normalized Discounted Cumulative Gain (NDCG) and pairwise accuracy, benchmarks that directly quantify consistency with expert human judgments.

Figure 4 presents comparative results for the vanilla SLM, fine-tuned SLM, and GPT-4o teacher:

Figure 4: NDCG and pairwise accuracy of vanilla SLM, fine-tuned SLM, and GPT-4o, measured against human labels on a real enterprise search test set.

Key Results and Findings

• Quality Gain: Fine-tuned SLM achieves NDCG of 0.953 and pairwise accuracy of 63.81%, exceeding GPT-4o's scores (NDCG: 0.944, Accuracy: 62.58%) on the same test bed. This demonstrates that, when distilled on well-structured synthetic data, SLMs can not only match but slightly surpass strong LLM baselines in relevance alignment for enterprise queries.
• Throughput and Cost: The fine-tuned SLM attains 873.33 requests per minute (RPM) on a single A100 GPU, extrapolating to nearly 7,000 RPM on an 8-GPU cluster—a $17\times$ increase over LLM alternatives. Cost per 1M input/output tokens is $0.13/$0.52 versus GPT-4o's $2.50/$10.00, representing a $19\times$ cost reduction, with no observable degradation in label quality.
• Ablation Analysis: Multi-task tuning consistently enhances generalization, while increases in dataset size yield diminishing returns beyond 14K examples, emphasizing the primacy of data diversity and quality over quantity. Removal of synthetic enterprise data from the curriculum reverts results to near-vanilla SLM levels, confirming that open-domain IR datasets are insufficient proxies for enterprise settings.

Theoretical and Practical Implications

This work demonstrates that SLMs, when paired with a rigorous distillation pipeline and high-fidelity, task-specific synthetic data, can replace frontier LLMs for most practical offline labeling scenarios in enterprise IR. The strong alignment with human annotators verifies the adequacy of LLM-generated supervision for downstream distillation, even in highly context-dependent and ambiguous search environments.

From a theoretical standpoint, the success of this approach suggests that the primary performance bottleneck for SLMs in high-stakes evaluation tasks is not model capacity, but rather the availability of clean, relevant, and distribution-matched training data. The multi-task fine-tuning strategy further affirms the value of curriculum learning in enhancing generalizability to long-tail, hybrid query types beyond mere lexical matches.

The practicality of this approach is highlighted by the integration of privacy-preserving workflow design (eschewing user logs), scalability (full automation from data synthesis to labeling), and substantial cost-efficiency, all critical for industrial deployment in corporate and regulated domains.

Future Developments

The generalizability of this pipeline to other SLM architectures (e.g., Llama, Gemma) and forthcoming LLM teachers (GPT-5+) is high, as the gains derive fundamentally from the quality and structure of the data pipeline rather than model-specific innovations. Areas for further exploration include:

• Extending synthetic data generation to cover personalized and contextualized queries by introducing user and organizational attributes into the prompt design.
• Incorporating self-training and active learning loops to iteratively improve data quality and model performance using uncertain or edge-case queries mined from production traffic.
• Adapting the pipeline to support explainable relevance labeling, where labelers must generate rationales or diagnostic attributions in addition to scalar scores, thus enabling finer-grained auditability.

Conclusion

The fine-tuning and distillation framework established in this work offers a robust, practical schema for transforming SLMs into high-accuracy, high-throughput relevance labelers tailored for complex enterprise search. By judiciously leveraging LLM capabilities for synthetic data generation and annotation, and optimizing training workflows with curriculum-based multi-dataset fine-tuning, the resulting SLMs deliver cost and speed advantages without sacrificing alignment to human annotation standards. This paradigm sets the foundation for deploying efficient model-based labelers in privacy-sensitive and resource-constrained enterprise environments, and signals a clear path for extensible, scalable, and sustainable development of IR evaluation tools.