Negative Data Mining for Contrastive Learning in Dense Retrieval at IKEA.com

Abstract: Contrastive learning is a core component of modern retrieval systems, but its effectiveness heavily relies on the quality of negative examples used during training. In this work, we present a systematic approach to improving dense retrieval for IKEA product search through structured negative sampling strategies and scalable LLM-as-a-judge relevance evaluation. Building on IKEA Search Engine's late-interaction retrieval architectures, we introduce two key contributions: (1) structured negative sampling strategies that leverage product hierarchical taxonomy and product attributes to generate semantically challenging negatives, and (2) a comprehensive LLM-based evaluation methodology for generating training data. Rather than relying on sparse human annotations or random sampling, our LLM-based evaluation system allocates a score for all candidate products against each query. Our methodology achieves +2.6\% average category accuracy on offline real user query experiments on the Canada market. However, our A/B test on long-tail queries showed no statistically significant differences in user engagement metrics between the improved and baseline models ($p > 0.05$). We trace this gap to user search behavior: 67\% of popular searches exhibit zero-click rates above 50\%, indicating that a substantial proportion of search sessions result in no product engagement regardless of result ranking. These findings underscore the importance of hard negative mining but also the need for grounding training data and offline evals in real user search behavior -- including query intent distribution and zero-click patterns -- to bridge the gap between offline retrieval quality and online user engagement.

• The paper demonstrates that leveraging structured hard negative mining in dense retrieval significantly boosts offline metrics, with notable gains in Cat@10 and R@10.
• It details an automated three-stage pipeline integrating LLM-generated synthetic queries and categorical query analysis to curate challenging negative samples.
• Despite offline improvements, the study reveals a gap in online user engagement, highlighting the need for training data that reflects realistic user intent.

Negative Data Mining for Contrastive Learning in Dense Retrieval at IKEA.com

Introduction and Research Objectives

The paper presents a rigorous evaluation of hard negative mining (HNM) and LLM-based data generation strategies in dense retrieval systems with a focus on IKEA.com’s product search engine. Dense retrieval architectures supplant classical keyword-based search by leveraging contrastive learning to position semantically similar query-product pairs in feature space while distancing semantically dissimilar ones. The study addresses three primary research questions: (1) How do distinct hard negative mining strategies affect dense retrieval efficacy for product search? (2) What explains the generalization gap between synthetic and real user queries? (3) To what extent do offline retrieval metrics predict online user engagement?

Dataset, Query Taxonomy, and Problem Analysis

The IKEA Canada dataset consists of 24,350 products annotated with hierarchical taxonomy, structured attributes (e.g., color, material), and LLM-generated text descriptions. Query analysis from production logs (204,528 unique queries) exposes a heavy skew toward category-only queries (44.7%), with substantial volume (37.1%) in queries involving specific attributes, highlighting the operational imperative of robust semantic understanding for relevance in retrieval models. Query typology supports empirical investigation by stratifying evaluation and providing granularity in subsequent analyses of model generalization.

Training Data Generation and Negative Sampling Methodology

A sophisticated, three-stage automated pipeline is proposed, integrating LLM-powered attribute extraction, synthetic query generation reflective of real traffic variance, and LLM-based relevance annotation. This process composes training triplets from diverse query and product combinations, implementing three negative types: (i) Attribute-only (same category, wrong attribute), (ii) Cross-category (wrong category, typically with matched attribute), and (iii) Multi-attribute (violation of a single attribute in complex, multi-attribute queries). The pipeline enables structured ablations of hard negative sources.

Figure 1: (a) Training data generation pipeline consisting of three stages and (b) visualization of data mixtures, demonstrating the systematic composition of hard negatives.

The system scales to nearly 9 million annotated triplets, with clear demarcation between random negative sampling (baseline), structured HNM, and incremental LLM-based query expansion (QE) strategies. Data mixtures are engineered to isolate the effect of data quality (presence of challenging negatives) versus quantity (expansive but potentially lower-relevance training data).

Model Architecture and Experimental Design

All models implement the same late-interaction semantic search backbone, computing query-document scores by pooling maximal token-level interactions over contextualized representations. Training is performed with fixed hyperparameters and repeated across random seeds to ensure reproducibility and statistical validity. Evaluation adopts both synthetic benchmarks (in-distribution, LLM-generated queries) and real user queries (out-of-distribution, drawn from live search traffic), complemented by a production A/B test centered on long-tail query segments. LLM-generated relevance labels are validated by human annotation (93% agreement), supporting their deployment for large-scale evaluation.

Offline Evaluation Results and Generalization

Structured HNM yields substantial gains in offline metrics, especially on synthetic queries (Cat@10: +4.3%, R@10: +4.2% over baseline). Gains on real queries are marginal or absent for recall, and only modest for category accuracy, despite the presence of semantically challenging negatives. Models augmented with large volumes of additional QE data do not outperform those trained on more carefully curated hard negatives; indeed, further expansion can dilute real-world performance, reflecting the risk of synthetic distributional mismatch.

The offline/online gap is attributable to a pronounced mismatch in query intent: synthetic queries are narrowly targeted (mean 1.7 product categories covered), whereas real queries span a much wider range (mean 12.5 categories). As a result, models overly tuned to hard negatives within tight semantic bands fail to generalize when confronted with the diffuse, under-constrained nature of user queries.

Online A/B Test: Retrieval Improves Offline, Not Engagement

Despite demonstrating an average +2.6% gain in offline category accuracy, the HNM-enhanced model reveals no statistically significant difference in online user engagement compared to the baseline. Analysis shows that for a large fraction of sessions (notably, 67% of popular searches), zero-click rates exceed 50%, indicating user needs are often satisfied without explicit interaction, consistent with emerging “zero-click” retrieval paradigms. These behavioral patterns fundamentally limit the observable impact of retrieval on conventional engagement metrics.

Furthermore, the long-tail focus of the A/B test and high baseline accuracy in dominant (broad) queries erode sensitivity to improvements in models better adapted to fine-grained intent discrimination—a result consonant with prior work at major e-commerce platforms.

Theoretical and Practical Implications

The study’s findings reinforce two core conclusions:

• Algorithmic optimization through hard negative mining produces strong offline gains in settings where evaluation queries match the negative sampling distribution, but these improvements fail to propagate to measurable gains in user engagement for real, broad-distribution queries.
• The disconnect highlights the necessity for training data and evaluation protocols that reflect actual user intent distributions, not just synthetic, challenging scenarios, especially in late-interaction semantic search for e-commerce.

Practically, increased data quality (well-targeted, semantically plausible negatives) is shown to outweigh indiscriminate data scaling via synthetic expansion, calling for careful curation and taxonomy-aware negative sampling in future dense retrieval pipelines.

Future Directions

Effective dense retrieval for e-commerce must further integrate real traffic characteristics into negative mining and training data generation pipelines, potentially through self-supervised signals such as query reformulations, partial satisfaction metrics (e.g., scroll, filter actions), and other non-click engagement signals. Causal analysis of heterogeneous treatment effects, robust interleaving-based A/B testing, and dynamic adaptation of synthetic data protocols to cover the true intent distribution are immediate avenues for investigation. Additionally, cross-market or multilingual benchmarks may further challenge current generalization limits and support development of universally adaptive retrieval systems.

Conclusion

This paper delivers a detailed, empirically grounded comparison of negative mining methodologies and their downstream effects on dense retrieval for e-commerce product search, revealing that training and evaluation focused on synthetic or narrowly-defined hard negatives yields diminishing practical utility. The consistent offline/online gap observed demonstrates the primacy of intent-realistic data in both model development and evaluation, refocusing efforts on data representativeness, user behavior alignment, and more sensitive online evaluation frameworks. The results advise practitioners to prioritize realistic data curation over scale and to expect only modest business impact from algorithmic improvements when baseline retrieval for broad queries is already strong.