Product Disambiguation Techniques

• Product disambiguation techniques are methods that determine if product records refer to the same SKU by using algorithmic, unsupervised, and human-in-the-loop strategies.
• They employ constrained clustering, combinatorial matching, and LLM-based reasoning to overcome challenges from missing identifiers and variable attribute quality.
• Practical implementations demonstrate high accuracy and scalability, enabling efficient deduplication and variant grouping in large e-commerce catalogs.

Product disambiguation techniques encompass algorithmic, semi-automated, and hybrid human-in-the-loop strategies for determining when two structured or unstructured product records refer to the same Stock Keeping Unit (SKU), product variant, or operationally-equivalent entity. The core aim is to correctly identify the equivalence (or distinctness) of product listings in the presence of missing identifiers, heterogeneous attribute coverage, and highly variable text descriptions. State-of-the-art research in this area draws upon constrained clustering, combinatorial unsupervised matching, semi-supervised deep clustering, question-driven LLM-based reasoning, and scalable multimodal embeddings.

1. Formal Problem Framing and Taxonomy

Product disambiguation is formalized as a binary classification, clustering, or mapping problem. Given two product representations $P_b$ (base) and $P_c$ (candidate), one must decide if they correspond to the same underlying item. This setting extends to three main operational regimes:

• Product Matching: Pairwise or n-way determination of product identity across listings, with or without explicit identifiers.
• Product Deduplication: Partitioning a large catalog into clusters, each containing only one SKU, collapsing duplicates across sources.
• Variant Grouping: Identifying families of products that share all invariant attributes except allowable variant axes (e.g., color, pack size).

Challenges include absence of unique global identifiers (SKU, GTIN, EAN), unstructured or inconsistent string-based attributes (titles, descriptions), missing or noisy structured fields (brand, model, specification), and the need for interpretability and business auditability.

2. Rule-Based and Constrained Clustering Approaches

Early and interpretable techniques focus on imposing business- and domain-driven hard constraints. For product variant grouping (West et al., 2021), a “non-parametric” constrained clustering is established over a product catalog $P$, leveraging both human-legible rules and crafted NLP features:

• Must-link Constraints: Enforce all items in a cluster to share brand, category, and family name, with model-number distance under a per-category threshold.
• Family Name Extraction: Apply domain-specific synonym normalization, punctuation/number/unit stripping, blacklist filtering, and removal of variant and categorical tokens to derive an invariant “core” name for clustering.
• Graph-Based Clustering: Build a product graph where edges correspond to must-link conditions, and output clusters as connected components, ensuring interpretability and direct traceability to rules.

Empirical results (West et al., 2021) show that adding family-name constraints boosts F1 score from 0.44 (classification baseline) to 0.74, with at least 51.3% of categories achieving ≥90% precision. The entire process is transparent and modifiable by domain experts.

3. Combinatorial and Unsupervised Matching via Title Analysis

The Unsupervised Product Matcher (UPM) (Akritidis et al., 2019) avoids both external data and pairwise O($N^2$) scaling by combinatorially generating k-sized unordered sets of tokens (“combinations”) per title and assigning each product to the highest-scoring combination cluster. Key steps include:

• Combination Generation: Morphological preprocessing yields title token sets $W_t$; only combinations up to $K = \lfloor \text{avg title len}/2 \rfloor$ are generated.
• Importance Scoring: Each combination is scored by frequency, information retrieval metrics (“hot” tokens via $idf$), field-weighting, and average token position; the final metric is $I(c) = \frac{Y_c^2}{\alpha + \bar{d}(c)\log f_c}$.
• One-pass Assignment: Each product joins the cluster specified by its maximal I(c) combination signature.
• Post-Verification: Enforce vendor uniqueness within clusters, realigning misassigned items based on high-dimensional cosine similarity to representative titles, with a threshold $\tau=0.4$.

On datasets such as PriceRunner and Skroutz, UPM attains aggregate F1 of 0.547–0.649 versus 0.329–0.391 for standard IDF-weighted baselines, while running 20–30x faster. All parameters are globally tuned and fixed, rendering the process parameter-free and fully unsupervised (Akritidis et al., 2019).

4. Semi-Supervised Clustering with Minimal Labels

Modern pipelines leverage semi-supervised clustering, as exemplified by the IDEC framework (Martinek et al., 2024), which combines autoencoder-based feature compression with deep clustering:

• Input Feature Extraction: Pairs of products are represented by a 5-dimensional feature vector incorporating fuzzy string ratios (Levenshtein-based), partial ratios, token set ratios, and Jaccard distances on both tokens and numeric-only tokens.
• Deep Embedded Clustering: The feature vector is compressed by an encoder $f_\theta$, clustered in latent space using Student-t kernels, and sharpened assignments via target distribution $P_c$0.
• Seed Constraints: Small sets of must-link and cannot-link constraints (a few percent of the data) are supplied from limited labeled examples. Must-link penalizes latent separation for matching points; cannot-link rewards separation for non-matching pairs.
• Optimization: The loss is a weighted combination of reconstruction, clustering, must-link, and cannot-link losses.

Empirical performance matches or exceeds supervised XGBoost models and fully unsupervised methods, with F1 reaching 0.893 (minimal constraints)–0.917 (higher constraint ratios) (Martinek et al., 2024). IDEC easily scales to millions of items due to its lightweight features and mini-batch training.

5. LLM and Multi-Agent Reasoning Frameworks

Recent advances push product disambiguation into interpretable multi-agent, LLM-based architectures, notably the Question-to-Knowledge (Q2K) pipeline (Seo et al., 1 Sep 2025):

• Reasoning Agent ($P_c$1): For a given product pair, decomposes the comparison into attribute-specific disambiguation questions (brand, core-name, variant, specification, quantity).
• Deduplication Agent ($P_c$2): Embeds the set of generated questions, searches a persistent trace database for similar question-answer pairs, and determines if existing knowledge suffices.
• Knowledge Agent ($P_c$3): If prior traces are insufficient, issues focused web searches and synthesizes authoritative answers, with each answer scored for document similarity $P_c$4.
• Human-in-the-Loop: Uncertain classifications (probability in [$P_c$5, $P_c$6]) are adjudicated by domain experts, with labeled traces fed back for continual learning.
• Efficiency and Interpretability: Q2K yields binary match decisions, full reasoning chains ($P_c$7), and an evolving database of reusable traces.

On a 72K-pair food & beverage dataset with only titles, Q2K achieves 95.62% accuracy, substantially outperforming zero/few-shot LLM (92–93%), rule-based methods (67%), and standard web-search LLM (93%), while reducing average web calls by 22%. Interpretable reasoning chains and trace reuse support both transparency and scalability (Seo et al., 1 Sep 2025).

6. Multimodal Embedding Pipelines for Large-Scale Deduplication

For high-volume e-commerce catalogs, scalable vector-based pipelines incorporate multimodal input and dense, compact representations:

• Text Model: In-domain BERTurk (12-layer, 768-dim) is pretrained via masked language modeling; features from multiple layers are aggregated via 1D convolution and compressed to 128-dim embeddings (Kulunk et al., 19 Sep 2025).
• Image Representation: DeiT-based Masked AutoEncoder encodes 9-patch images, including center, two random, and a full-image patch; patch features are concatenated and projected to 128 dims.
• Fusion Model: For a product pair $P_c$8, text and image vectors are concatenated ($P_c$9) and classified via a lightweight 1D conv + FC network.
• Efficient Indexing: All vectors are indexed with IVF_FLAT in Milvus, partitioning embeddings into 65,536 centroids and achieving ≤10 ms per query on catalogs of 200M+ items under 100GB RAM.
• Empirical Performance: The system achieves macro-F1 = 0.90 (vs. 0.83 for commercial baselines), with notable precision (Match: 0.91, Not-Match: 0.88); ablations confirm the effectiveness of PCA to 128 dim and fusion of intermediate features (Kulunk et al., 19 Sep 2025).

The primary failure mode is the misgrouping of visually near-identical but operationally distinct SKUs, suggesting that augmentation with attribute-aware logic would further improve boundary-case handling.

7. Trade-Offs, Interpretability, and Practical Considerations

The design space of product disambiguation techniques reflects several trade-offs:

• Accuracy vs. Cost: Sophisticated LLM-based and multimodal frameworks achieve highest accuracy at increased computational or search cost, but modular architectures such as Q2K reduce redundant retrieval through trace reuse (Seo et al., 1 Sep 2025).
• Interpretability: Rule-based constrained clustering and combinatorial unsupervised approaches maximize business transparency and enable direct human audit. Neural and deep clustering models offer varying degrees of black-box opacity.
• Scalability: Models leveraging vector compression and sublinear search enable real-time decision-making on catalogs spanning hundreds of millions of items.
• Domain Adaptation: Synonym dictionaries, attribute-exclusion lists, embedding models, and question-generation strategies are often domain-tunable; continual learning from human-in-the-loop corrections bridges evolving catalog drift.

A plausible implication is that future state-of-the-art systems will increasingly hybridize interpretable agent-based reasoning with multimodal, scalable matching—optimizing trade-offs in accuracy, transparency, and efficiency tailored to both operational and business requirements.