Real-Time Hard Negative Sampling via LLM-based Clustering for Large-Scale Two-Tower Retrieval

Abstract: The two-tower model has been widely used for large-scale recommendation systems, particularly in the retrieval stage. Industry standards for training two-tower models typically involve in-batch and/or out-of-batch negative sampling. However, these methods often produce easy negatives that models can quickly learn, failing to sufficiently challenge the model. To address this issue, a novel self-supervised hard negative sampling technique is proposed that leverages a LLM to generate hard negatives from the same cluster during model training. By utilizing the LLM to learn media representations, the proposed approach ensures that the generated negatives are more challenging and informative. This real-time sampling framework is designed for seamless integration into production models, capable of handling billions of training data points with minimal computational complexity. Experiments on public datasets, along with deployment to a large-scale online system, demonstrate that the proposed negative sampling technique outperforms widely used industry methods. Furthermore, analysis in industrial applications reveals that this sampling method can help break inherent feedback loops in recommendations and significantly reduce popularity bias.

• The paper introduces a novel LLM-based clustering method that generates hard negatives to enhance two-tower retrieval, achieving significant HR@50 and CTR improvements.
• It describes GOOBS, a scalable framework with dedicated update and sampling engines that manage online embeddings efficiently in real time.
• The approach mitigates popularity bias and improves long-tail coverage, offering a robust, deployable alternative to traditional negative sampling methods.

Real-Time Hard Negative Sampling via LLM-based Clustering for Large-Scale Two-Tower Retrieval

Introduction

This paper presents a novel hard negative sampling approach for large-scale two-tower recommendation systems, integrating LLM-based clustering into a real-time, scalable training and serving framework named GOOBS (Global Out-of-Batch Sampling). The motivation arises from the inefficacy of standard in-batch and random out-of-batch negative sampling, which tend to produce negatives that are too easy for the model, thereby restricting learning and exacerbating popularity bias. The proposed method utilizes semantic clusters derived from an LLM-based multimodal encoder, allowing for the generation of informative, hard negatives by sampling from the same cluster as the positive item in real time.

LLM-based Cluster Generation

A central component is the generation of semantic clusters using a multimodal embedding model built atop LLMs. The system is pre-trained using LLM backbones for robust cross-modal semantic representation, followed by fine-tuning for domain-specific tasks. Input modalities include text, images, and videos, which are combined via transformer-based multimodal encoders, resulting in high-quality content clusters.

Figure 1: LLM-based cluster generation enables creation of semantic item groups leveraging multimodal content understanding.

The use of LLMs enables modeling of nuanced item relations that surpass shallow clustering approaches based on genre or category, thus supporting effective selection of hard negatives that closely resemble true positives in the latent space, making discrimination challenging for the model. The level of cluster granularity is carefully controlled to avoid excessive false negatives.

GOOBS: Real-Time Cluster-based Hard Negative Sampling Framework

The GOOBS framework supports scalable, real-time negative sampling for two-tower architectures at industrial scale, efficiently managing an online maintained pool of item embeddings for out-of-batch (OOB) negatives.

Figure 2: GOOBS real-time cluster-based negative sampling framework with separate update and sampling engines.

The training process comprises two engines:

• Update Engine: Assigns items to slots in an OOB pool segmented by cluster via a hashing function for efficient parallel storage and retrieval.
• Sampling Engine: At each step, cluster-aligned negatives are sampled from dedicated segments, ensuring that negatives are both relevant and challenging.

The update engine keeps the pool fresh with recent examples, overcoming pool staleness, while the sampling engine exploits cluster assignments to preferentially select hard negatives.

Pool Management and Sampling Algorithms

To maintain computational efficiency at the scale of billions of samples, each item is mapped to a slot within its cluster segment using a deterministic hash. When updating, new in-batch items are inserted or replace slots by cluster, ensuring both diversity and recency.

Figure 3: Cluster-based pool update with deterministic slot management per cluster.

At sampling time, the engine randomly selects a slot within the relevant cluster segment for each anchor. This algorithm provides a constant-time mechanism to yield hard negatives for every in-batch example.

Figure 4: Cluster-based sampling ensures that negatives are drawn from the same semantic segment as the positive.

Cluster sizes are selected to guarantee statistical utility and sufficient hard negatives per cluster, as illustrated in the empirical distribution.

Figure 5: Distribution of cluster sizes guarantees robust pool diversity and scalability for OOB sampling.

Theoretical Justification

The method provides a formal justification leveraging contrastive learning and gradient analysis. Uniform sampling often yields negatives with low similarity, and consequently, low gradient contributions. Sampling from the same cluster—i.e., local neighborhoods in the semantic space—yields higher-gradient, non-trivial negatives that match the optimal estimator's oracle distribution, as motivated by importance-based sampling theory. Empirically, this matches or exceeds the performance of ANN-based hard negative mining methods, without requiring costly periodic index rebuilds.

Experimental Results

Public Datasets

On four datasets (MovieLens-1M, Amazon-Grocery, Amazon-Electronics, Amazon-Home), models using Cluster GOOBS consistently outperform state-of-the-art negative sampling baselines including DNS, CBNS, and ANCE, as measured by HR@50 and HR@100.

• Absolute improvements in HR@50 for Cluster GOOBS over the in-batch baseline range from +7.2% on MovieLens-1M to +55.6% on Amazon-Electronics.
• GOOBS, without clustering, also achieves non-trivial gains, validating the benefit of larger OOB exposure. The maximum effect is observed for the hardest cluster-negatives, which dominate both recency- and ANN-based methods.

Industrial System Deployment

In online A/B tests within a high-traffic industrial recommendation pipeline, Cluster GOOBS delivers a +53% click-through-rate (CTR) lift over standard OOB sampling, with negligible training efficiency degradation ($-1.4\%$ QPS, no inference QPS penalty). Importantly, the method yields substantial reduction in popularity bias—the share of impressions from the top 100 items is reduced from 50% to 32%, and the long-tail item impression coverage increases by 50%.

Practical and Theoretical Implications

Cluster-based hard negative sampling, when driven by strong LLM-derived multimodal representations, is shown to provide:

• Robust, scalable real-time deployment compatibility, crucial for modern recommender serving scenarios.
• Substantial mitigation of feedback loops and popularity bias through more diverse and informative training signals.
• Improved sample efficiency and model generalization attributable to locally hard negatives.
• Simpler operational complexity compared to global ANN-index approaches, as no periodic index maintenance is required.

Future Directions

Further extensions include dynamically adjusting cluster granularity, hybrid user-query-adaptive cluster sampling, or integrating cluster-weighted losses to control for intra-cluster diversity. Moreover, joint training of clustering representations with downstream objectives may enhance long-tail coverage and resistance to adversarial feedback bias.

Conclusion

This paper introduces a scalable and effective hard negative sampling methodology for two-tower recommendation frameworks at industry scale, realized by integrating LLM-based clustering with an efficient real-time update and sampling infrastructure. Extensive experiments validate that cluster-based sampling of negatives delivers consistent gains across diverse benchmarks and practical environments, advancing both retrieval accuracy and debiasing. The architecture provides a directly deployable solution for improving data efficiency and fairness in large-scale retrieval and recommendation systems.