Multilingual E-commerce Search Competition

• Multilingual e-commerce search competition is defined by tasks that evaluate cross-lingual semantic alignment between short queries and rich product metadata under real-world noise.
• The competitions benchmark both graded and binary relevance, emphasizing translation augmentation and robust validation to overcome language imbalance and label noise.
• Dominant solutions utilize joint encoding with multilingual LLM backbones, combining supervised classification and metric-aware post-processing to fine-tune relevance scoring.

Multilingual e-commerce search competition denotes a class of evaluation settings in which systems must model relevance between shopper queries and commercial entities under multilingual, noisy, and operationally realistic conditions. The clearest instances are the Amazon ESCI Challenge at KDD Cup 2022, centered on multilingual query–product ranking and ESCI labeling, and the Alibaba International / CIKM AnalytiCup 2025 competitions, centered on Query-Category and Query-Item relevance for multilingual queries (Zhang et al., 2022). Across these benchmarks, the central technical problem is not merely text matching: it is cross-lingual semantic alignment under label noise, class imbalance, heterogeneous product metadata, partial language coverage, and strict leaderboard-driven evaluation.

1. Competition formats and task definitions

Two competition lineages dominate the published literature. The earlier Amazon line treats multilingual product search as a graded relevance problem over query–product pairs. The later Alibaba line treats multilingual product search as binary relevance classification over query–category and query–item pairs. A broader but adjacent benchmark, Shopping MMLU, includes multilingual query-product relation and ranking tasks, but is explicitly presented as a multi-task shopping-assistant benchmark rather than a pure search benchmark (Jin et al., 2024).

Benchmark	Main tasks	Official metrics
Amazon ESCI / KDD Cup 2022	Query-product ranking; 4-way ESCI classification; substitute identification	NDCG for ranking; Micro-F1 for Tasks 2 and 3
Alibaba International / CIKM AnalytiCup 2025	Query-Category relevance; Query-Item relevance	Positive-class F1 per task; average of QC and QI
Shopping MMLU / KDD Cup 2024	Multilingual query-product relation, ranking, and related shopping tasks within 57 tasks	Accuracy, NDCG, and task-specific metrics

In the Amazon setting, ESCI denotes Exact, Substitute, Complement, and Irrelevant. The ranking task is therefore not a generic binary ranking problem; it is a graded-relevance problem aligned to shopping intent, where substitutes and complements remain commercially meaningful (Zhang et al., 2022). In the Alibaba setting, the competition is framed around two binary tasks: whether a multilingual query matches a category path, and whether it matches an item listing. The official combined score is

$F_{avg} = 0.5 \times F1_{QC} + 0.5 \times F1_{QI},$

with per-task positive-class F1 defined in the standard way (Nguyen-Ho et al., 29 Oct 2025).

A common misconception is that these competitions are about open-ended retrieval over a full catalog. The system reports repeatedly describe the evaluated task as relevance prediction over candidate pairs or bounded candidate lists. This makes the dominant solutions much closer to reranking or pair classification than to first-stage ANN retrieval (Zhang et al., 2022).

2. Data regimes and multilingual asymmetry

The Amazon ESCI benchmark is explicitly multilingual and based on real shopping traffic rather than a toy dataset. One competition report states that the training data contains about 2 million query-product pairs over about 1.2 million products, with language proportions of roughly 54.5% English, 26.5% Japanese, and 19% Spanish; each query is associated with up to 40 candidate products, and each product carries fields such as title, description, bullet points, brand, color, and locale (Zhang et al., 2022). This structure already implies a strong asymmetry: queries are short and sparse, whereas products are long, heterogeneous, and field-rich.

The 2025 Alibaba competition introduces a different asymmetry. Multiple team reports describe multilingual queries paired against English category paths or English item titles, with development and test languages exceeding training-language coverage (R et al., 23 Oct 2025). This yields a characteristic competition condition: cross-lingual semantic matching must generalize to languages absent from training. A data-centric report makes this explicit for both tasks, describing QC as training on six languages with German, Italian, Polish, and Arabic appearing only in dev/test, and QI as additionally facing zero-train languages such as Vietnamese and Indonesian (Yin et al., 24 Oct 2025).

Several reports also note that real-world multilingual noise is central rather than incidental. DcuRAGONs emphasizes noisy multilingual queries, low-resource languages, and even unseen languages at evaluation time; Tredence_AICOE and EAR-MP emphasize incomplete training coverage, short user-generated queries, and the fact that category or item text remains in English while the query changes language (Nguyen-Ho et al., 29 Oct 2025). This suggests that multilingual e-commerce competition data is typically not balanced multilingual paraphrase data; it is structurally asymmetric commercial text with uneven coverage and cross-lingual mismatch.

A second misconception is that multilinguality here is equivalent to translation. The published systems show that translation is often used, but the harder problem is broader: code-mixed and noisy text, locale-specific catalog conventions, missing supervision for some languages, and intent distinctions that are only weakly lexical.

3. Dominant modeling paradigms

The dominant modeling pattern in competition reports is joint encoding of the query and target text, followed by supervised classification. In the Amazon winning system, the central architecture is a cross-encoder over concatenated query and product text, using multilingual backbones such as mDeBERTa, InfoXLM, XLM-RoBERTa, and RemBERT, with DeBERTa-v3-large reserved for English-only examples (Zhang et al., 2022). The method is explicitly classification-driven but metric-aware: models are first trained as 4-class ESCI classifiers, and ranking is then produced from class probabilities using the competition gains,

$s(q,p)=\sum_{c \in \{E,S,C,I\}} w_c\,P(c\mid q,p),$

with

$$w_E=1.0,\quad w_S=0.1,\quad w_C=0.01,\quad w_I=0.0.$$

This is a precise instance of metric-aware post-processing without a direct ranking loss (Zhang et al., 2022).

The 2025 Alibaba systems are usually framed as binary relevance classification. DcuRAGONs concatenates the original query, its English translation, and the target text into a single sequence,

$x = [CLS]\ q_{orig}\ -\ q_{en}\ [SEP]\ t\ [SEP],$

then encodes it with a multilingual LLM and predicts relevance through a classifier head on the final representation (Nguyen-Ho et al., 29 Oct 2025). Tredence_AICOE and DILAB similarly treat both QC and QI as binary relevance decisions over paired inputs, but operationalize them with instruction-tuned models, zero-shot prompt templates, or yes/no outputs rather than explicit pairwise ranking heads (R et al., 23 Oct 2025).

Across reports, several architectural choices recur. One is the preference for cross-encoder-like joint encoding over bi-encoder retrieval when the benchmark already supplies candidate pairs. Another is the use of strong multilingual LLM backbones under an organizer-imposed parameter cap, with Gemma-3-12B, Qwen2/Qwen3 families, Qwen2.5-14B-Instruct, and multilingual encoder baselines all appearing in the literature (Nguyen-Ho et al., 29 Oct 2025). A third is task-aware serialization: hierarchical category markers for QC, language tags for multilingual robustness, and richer attribute serialization for QI (Lee et al., 21 Oct 2025).

The systems are therefore not architecturally exotic. The competition literature repeatedly favors robust and modular pair classifiers whose outputs can be calibrated, filtered, ensembled, or combined with lexical and structural signals.

4. Data-centric optimization, augmentation, and robustness

A striking regularity in the literature is the shift from model-centric novelty to data-centric engineering. DcuRAGONs explicitly describes its winning approach as data-centric, and the third-place data-centric case study makes the same point programmatically (Nguyen-Ho et al., 29 Oct 2025). In practice, this means that large gains often come from translation augmentation, label-noise mitigation, careful validation splits, and semantically informed negatives rather than from inventing a new encoder.

The Amazon winning report is especially explicit about stacking moderate improvements. Beyond multilingual pre-trained LMs, it uses text cleaning, translation-based augmentation into English, Spanish, and Japanese, English-specific modeling for the dominant language, label smoothing, self-distillation, pseudo-labeling, adversarial training with AWP and FGM, and ensemble reweighting (Zhang et al., 2022). Its self-distillation recipe combines hard and soft labels as

$$y_{\text{new}} = 0.7 \times \text{hard labels} + 0.3 \times \text{soft labels},$$

and its pseudo-labeling keeps only public-test predictions above 0.7 confidence (Zhang et al., 2022). The paper’s empirical lesson is not that one trick dominates, but that competitive multilingual ranking emerges from cumulative robustness improvements.

The 2025 reports extend this data-centric pattern in several directions. DcuRAGONs adds supervised task-adaptive pre-training across QC and QI, arguing that multilingual LLMs know many languages but not necessarily the relevance semantics of e-commerce search logs (Nguyen-Ho et al., 29 Oct 2025). Tredence_AICOE augments missing languages by translating queries with Gemini 2.5 Flash and experiments with hierarchy-based and embedding-based synthetic negatives, concluding that translation quality and semantic coherence matter more than aggressive balancing (R et al., 23 Oct 2025). DILAB adds a multi-stage refinement pipeline with language tagging, hierarchical category tagging, semantic item tagging, and generated descriptions, plus a filtering stage that removes suspect samples by combining model predictions with lexical similarity features (Lee et al., 21 Oct 2025).

The data-centric case study formalizes the same intuition more abstractly: translation-based augmentation for zero-train languages, semantic negative sampling using dense embeddings, and self-validation filtering of likely mislabeled instances (Yin et al., 24 Oct 2025). Its reported weighting for yes/no prediction is based on normalized yes/no token scores, and it calibrates QC and QI with distinct thresholds rather than assuming a universal decision boundary. EAR-MP pushes the same engineering tendency in a different direction by translating everything into English, cleaning aggressively, and then using task-specific additions such as hierarchical token injection for QC and lexical-semantic hybrid scoring for QI (Lim et al., 27 Oct 2025).

This body of work suggests a general principle: multilingual e-commerce competition systems often succeed by reducing distributional mismatch before modeling. Translation, normalization, metadata enrichment, and confidence-aware filtering are treated as first-order components of the search stack rather than peripheral preprocessing.

5. Evaluation, validation, and leaderboard dynamics

The metric structure of these competitions strongly shapes system design. In Amazon ESCI, the task-1 metric is NDCG with gains aligned to the ESCI categories, so ranking models naturally exploit the exact/substitute/complement distinctions at inference time (Zhang et al., 2022). In the Alibaba competition, the official score averages positive-class F1 across QC and QI, which encourages threshold tuning and validation strategies targeted at class-specific decision quality rather than rank calibration alone (Nguyen-Ho et al., 29 Oct 2025).

Validation methodology becomes a research contribution in its own right. DcuRAGONs argues that naive random splits leak repeated queries or semantically overlapping category paths across train and validation. It therefore groups QI folds by exact query string and groups QC folds by category-path prefixes, calling the result a category-aware split. This design is intended to estimate hidden-test generalization more faithfully and to avoid inflated offline performance (Nguyen-Ho et al., 29 Oct 2025). EAR-MP introduces per-leaf threshold tuning for QC by searching thresholds from 0.30 to 0.70 in steps of 0.02, while the data-centric third-place report finds that QI and QC require different operating points, with QI preferring a much lower threshold than QC (Lim et al., 27 Oct 2025).

Leaderboard behavior is also part of the technical picture. The Amazon winning report describes explicit public/private leaderboard divergence and uses ensemble weighting informed by both public scores and local cross-validation, while downweighting highly correlated models (Zhang et al., 2022). Its strongest single model achieved 0.9022 public and 0.9015 private, while the final ensemble reached 0.9057 public and 0.9043 private, winning task 1 (Zhang et al., 2022). Such results show that competition systems are often optimized not only for mean accuracy, but also for stability under leaderboard shift.

A broader implication is that multilingual e-commerce competitions reward validation design nearly as much as modeling design. Leakage-resistant folds, threshold calibration, public/private robustness, and task-aware score conversion are repeatedly reported as decisive.

6. Representative systems, broader context, and limitations

Published system reports provide a fairly clear leaderboard-centered record of the field. Team www won Amazon ESCI task 1 with a private NDCG of 0.9043 (Zhang et al., 2022). The DcuRAGONs team reports the highest score in the Alibaba International competition and is listed at 0.8931 overall in a comparative table from the third-place report (Nguyen-Ho et al., 29 Oct 2025, Yin et al., 24 Oct 2025). Team Tredence_AICOE reports 4th place with average private-test F1 0.8857 (R et al., 23 Oct 2025), and DILAB reports 5th place with overall score 0.8819 (Lee et al., 21 Oct 2025). The third-place data-centric report reaches 0.8865 overall, with 0.8896 on QC and 0.8833 on QI (Yin et al., 24 Oct 2025). These standings reinforce that the competition ecosystem is now documented through multiple independent technical reports rather than a single canonical system paper.

The competition literature also sits inside a larger research landscape. Amazon’s query reformulation work addresses multilingual tail-query robustness through behavior transfer and normalization rather than direct ranking (Zhang et al., 2024). PP-GLAM studies multilingual ESCI relevance as an interpretable ensemble of LLMs and behavior graphs over US, Spain, and Japan (Choudhary et al., 2024). CSRM-LLM addresses cold-start relevance matching with auxiliary translation tasks, retrieval-based query augmentation, and multi-round self-distillation (Wang et al., 1 Sep 2025). Production retrieval work on graph-based multilingual product retrieval (Lu et al., 2021) and multilingual semantic search at Uber Eats (Ling et al., 27 Jan 2026) tackles first-stage dense retrieval at a scale and latency regime mostly absent from the competition reports. Metadata-generation work, such as multilingual browse-page title generation, remains indirectly relevant because it improves searchable catalog text rather than pairwise relevance modeling itself (Mathur et al., 2018).

The major limitation of competition systems is therefore structural rather than purely algorithmic. Most published solutions address bounded-candidate ranking or pair classification, not open-corpus retrieval. Many rely heavily on leaderboard-guided ensembling, translation pivots, or competition-specific data quirks. This suggests that direct transfer into production search stacks is not automatic. At the same time, the competitions have been unusually informative: they have crystallized a now-stable recipe for multilingual commercial relevance modeling in which strong pretrained encoders, data-centric augmentation, robust validation, and calibrated ensemble design consistently dominate more speculative architectural departures.