ShopGuru: Modular Shopping Assistant

Updated 4 July 2026

ShopGuru is a modular shopping-assistant system that integrates computer vision, speech processing, natural language understanding, and memory-augmented reasoning for in-store and online shopping.
It synthesizes benchmark tasks by converting storefront artifacts into executable tasks using domain-agnostic intent operators and multi-modal retrieval techniques.
The architecture supports robust multi-goal dialogues, accessibility features, and cross-catalog comparisons, offering actionable insights and scalable evaluation methods.

ShopGuru denotes a research-derived shopping-assistant architecture that combines in-store perception, conversational search, recommendation, policy lookup, accessibility support, and benchmarkable web interaction. In ShopGym, the name refers specifically to the task-synthesis layer that converts ShopArena storefront artifacts into executable benchmark tasks; in the broader shopping-assistant literature, the same name is used as a target system design integrating computer vision, speech processing, natural language processing, retrieval, and memory-augmented reasoning (Savadikar et al., 15 May 2026, Lai et al., 2020). This suggests that ShopGuru is best understood as a modular systems pattern rather than a single fixed algorithm.

1. Research lineage and conceptual boundaries

ShopGuru emerges at the intersection of several previously separate research lines: mobile in-store assistance, assistive perception for visually impaired shoppers, conversational faceted search, long-term preference-aware recommendation, visual shopping, multi-shop comparison, and reproducible web-agent evaluation. The resulting concept spans both deployment-oriented systems and evaluation infrastructure.

Research line	Contribution to ShopGuru	Source
Mobile in-store assistant	Camera/barcode capture, voice interaction, spec QA, purchase guidance	(Lai et al., 2020)
Assistive shopping accessibility	Offline object/OCR/barcode/TTS pipeline	(Deshmukh et al., 2022)
Conversational faceted search	CLU, intent operators, rules-based DST	(Manku et al., 2021)
Memory-augmented e-commerce agent	Joint preference retrieval and shopping assistance	(Yu et al., 16 Mar 2026)
Multi-shop web-agent benchmark	Cross-shop navigation, comparison, cart, checkout tasks	(Peeters et al., 18 Aug 2025)
Task synthesis over sandbox shops	Seven grounded benchmark skill categories	(Savadikar et al., 15 May 2026)

The conceptual scope is broader than a conventional slot-filling shopping bot. ShopTalk is designed for schemas with thousands of categories and cases in which a single category can have more than 8,900 attributes, motivating domain-agnostic intent operators rather than slot inventories (Manku et al., 2021). MG-ShopDial, by contrast, shows that realistic shopping dialogues mix recommendation, search, and question answering within the same session; its 64 dialogues and 2,196 utterances were explicitly collected to capture such multi-goal behavior (Bernard et al., 2023). Shopping Companion extends the same trajectory into long-term preference-aware assistance over a catalog of 1,298,797 products and 15–50-turn conversational histories (Yu et al., 16 Mar 2026). Taken together, these systems position ShopGuru as a shopping-specific orchestration layer rather than merely a product recommender.

2. Perception, product grounding, and physical-store interaction

A central ShopGuru function is product grounding in physical environments. ISA defines the canonical in-store interaction: a user takes a picture or scans the barcode of a product, the system identifies the product and retrieves database information, and the user then asks natural-language questions or requests recommendations and purchase guidance (Lai et al., 2020). ISA’s architecture separates a mobile client from back-end services over an HTTP REST API, with computer vision for product identification, third-party ASR/TTS for speech interaction, and NLP modules for intent recognition, domain processing, and output generation. Its four intent domains are Product Specification QA, Recommendation, Purchase, and Chit Chat. For intent recognition, ISA uses a Random Forest over bag-of-words features with 80 trees and test accuracy of 98.20% on a 1/3 held-out split from 500 labeled queries. For specification QA, it formulates answer selection over a product’s specification slots,

$i^{*} = \arg\max_{i \in \{1,\dots,M\}} s(Q,s_i),$

using IWAN, trained on 6,922 questions over 369 specification types and 148 products; held-out evaluation reports Top-1 accuracy 85.60%, Top-2 95.80%, and Top-3 97.60% (Lai et al., 2020).

Accessibility-oriented variants shift the same grounding problem toward offline assistance. SANIP implements three Python modules—custom object detection, text detection/OCR, and barcode/QR detection—then relays outputs by offline text-to-speech (Deshmukh et al., 2022). Its custom detector is YOLOv4-tiny trained in Darknet on a five-class dataset of 150 images total, with 30 images each for Parle-G, Lays, Tide, Shopping Cart, and EXIT sign. Text reading is based on pytesseract, barcode decoding on pyzbar, and speech output on pyttsx3. The system is reported to detect trained objects in real time, but precision, recall, mAP, latency, and user-study metrics are not reported (Deshmukh et al., 2022). The same accessibility lineage appears in earlier assisted-shopping recognition work, where fine-grained shelf recognition is built from OCR-derived word-to-class mappings, discriminative packaging patches, and active learning. On GroceryProducts, the full discriminative-patch plus spatial-pyramid plus SVM pipeline reaches 61.9% average classification accuracy, compared with 12.4% for a SURF BoW baseline; active learning raises accuracy from 60.5% to 64.4% at about 140 labeled images in one experiment (George et al., 2015).

ShopGuru also inherits visual robustness methods developed for noisy retail imagery. Product-90 introduces 142,466 images across 90 product categories with noisy labels derived from e-commerce review uploads, together with Guidance Learning, a teacher–student method combining a large noisy set with a small clean subset (Li et al., 2019). On Product-90, training on the mixed noisy and clean set with standard cross-entropy yields 66.78% accuracy, Guidance Learning raises this to 68.86%, and additional fine-tuning on the clean set yields 71.40% (Li et al., 2019). A plausible implication is that ShopGuru’s product-recognition layer is best served by a hybrid of clean in-domain supervision, noisy web-scale supervision, and fallback mechanisms such as barcode scan or OCR.

ARShopping adds a distinct physical-store presentation layer. It fuses AR.js marker tracking with EfficientDet-based object detection, retrieves product metadata from a server-side store built by crawling Target.com, and overlays radar-chart glyphs near detected products for multivariate comparison of factors such as nutrition, price, and ratings (Xu et al., 2022). Its rule-based fusion combines marker pose with object-detected centers and bounding boxes, and its virtual evaluation with 14 participants reports TAM means of 5.78 for perceived usefulness, 5.45 for ease of use, and 6.07 for intention of use (Xu et al., 2022). This establishes ShopGuru not only as a perception pipeline but also as a situated decision-support interface.

3. Conversational state, multi-goal dialogue, and long-term memory

ShopGuru’s dialogue substrate is shaped by two complementary lines: domain-agnostic conversational state tracking and explicit multi-goal shopping behavior. ShopTalk decouples dialog management from fulfillment. Its Contextual Language Understanding module is a sequence-to-sequence Transformer encoder-decoder with graph-structured inputs and a copy mechanism, while dialog-state updates are handled by a primarily rules-based DST over a minimal set of domain-agnostic intent operators such as SetValueOp, ClearValueOp, ClearFacetOp, NudgeFacetOp, and OrderByOp (Manku et al., 2021). The explicit rationale is web-scale schema complexity, with thousands of categories, ambiguous attribute reuse, negative preferences, ordered ranges, and ungrounded spans. ShopTalk was deployed in 2020 on the Google Assistant for Shopping searches, and post-launch measurements report a 9.5× increase in follow-on shopping queries, a 52% increase in average dialog length, and a 52% increase in engagement with displayed products (Manku et al., 2021).

MG-ShopDial provides the corresponding empirical evidence that shopping conversations are intrinsically mixed-goal. The dataset contains 64 human–human dialogues and 2,196 utterances across Sports and Outdoors, Books, Office Products, and Cell Phones and Accessories, with an average of 34.3 ± 14.9 utterances per dialogue and 7.5 ± 6.2 words per utterance (Bernard et al., 2023). Utterances are labeled by goal—search, recommendation, QA, or meta-communication—and by multi-label intent. The dominant observed pattern is an initial recommendation phase with preference elicitation, followed by information seeking through factual QA and exploratory search, and sometimes a second recommendation phase after constraints have been refined (Bernard et al., 2023). This gives ShopGuru a dialogue model in which goal switching is not exceptional but normal.

Shopping Companion extends the same logic into cross-session preference modeling. It uses a unified two-stage policy in which Stage 1 retrieves relevant memory turns, extracts implicit preferences, and asks for user confirmation, while Stage 2 performs shopping assistance through iterative product search and product-view verification (Yu et al., 16 Mar 2026). Memory retrieval uses all-MiniLM-L6-v2 turn embeddings and cosine similarity,

$s(q,m_i)=\frac{e(q)\cdot e(m_i)}{\|e(q)\|\|e(m_i)\|},$

over 15–50-turn histories. Product search is implemented with Pyserini BM25, and success at terminal state is defined by satisfying both instruction-driven needs and memory-derived preferences (Yu et al., 16 Mar 2026). On the paper’s benchmark, GPT-5 achieves Single Acc. 82%, Single Succ. 75%, Add-on Acc. 66%, Add-on Succ. 54%, and Average Succ. 64.5%, whereas the Shopping Companion configuration using Qwen3-4B with LoRA SFT, dual-reward RL, and tool-wise rewards reaches Single Acc. 90%, Single Succ. 84%, Add-on Acc. 55%, Add-on Succ. 43%, and Average Succ. 63.5% (Yu et al., 16 Mar 2026). The important architectural point is joint optimization: memory extraction and shopping execution are trained together rather than as separate modules.

4. Retrieval, recommendation, and comparison across catalogs and shops

Recommendation in ShopGuru spans product-level retrieval, substitute discovery, and explicit multi-shop comparison. ISA’s recommendation engine is intentionally simple—“search[ing] the database for related products”—but its placement in a multi-turn flow is consequential: specification questions can pivot into price inquiries, similar-item requests, and purchase orchestration within one interaction (Lai et al., 2020). A stronger retrieval lineage appears in Studio2Shop, which addresses exact-match fashion retrieval by learning a query encoder over studio images and matching it against static catalog embeddings. On a 20,000-query, 50,000-article internal Zalando test, the non-linear fDNA-based model reports top-1 0.238, top-5 0.496, top-10 0.613, top-20 0.722, top-50 0.834, top-1% 0.970, average rank 89, and median rank 5 (Lasserre et al., 2018). This shows that a ShopGuru visual-search subsystem can separate fast candidate generation from exact-match reranking.

Large-scale scene-based visual shopping introduces additional retrieval constraints. Pinterest’s Shop The Look combines Faster R-CNN-style object detection with visual embeddings and category-restricted retrieval over a corpus of hundreds of millions of products (Shiau et al., 2020). Moving from a global index to per-category indices reduced “Bad” top-5 results by 17.1% and increased “Similar” top-5 results by 9.9%, while adding gender restriction increased “Similar” by 5.9% and “Extremely Similar” by 12.7% (Shiau et al., 2020). Across successive system iterations, the paper reports cumulative relative gains of over 160% in end-to-end human relevance judgements and over 80% in engagement (Shiau et al., 2020). ShopGuru’s retrieval layer therefore combines nearest-neighbor search with hard semantic constraints rather than relying on visual similarity alone.

Multi-shop comparison requires another layer of structure. The comparison pipeline in “Are you sure?” uses prod2vec-based fast retrieval with top- $k=100$ neighbors, a Siamese substitute classifier, and a property-ranking stage based on normalized query frequency, PDP frequency, and property entropy (Chia et al., 2021). For display, it selects $W=3$ substitutes, uses 7 log-price bins while discarding the lowest and highest bins, and maximizes weighted Hamming diversity over one-hot property vectors. At fixed recall 0.8, the best tested configuration improves Shop A precision from 0.743 to 0.812 and Shop B precision from 0.573 to 0.743 relative to the image-cosine baseline (Chia et al., 2021). This suggests that ShopGuru comparison tables are not merely nearest-neighbor lists; they are structured outputs balancing substitutability, price diversification, and attribute informativeness.

WebMall recasts the same problem as web-agent execution over four simulated WooCommerce-based shops containing 4,421 offers and 91 tasks across 11 categories (Peeters et al., 18 Aug 2025). Tasks include finding specific products, comparing prices, identifying substitutes, finding compatible products, adding items to cart, and completing checkout. The best basic-task configuration—GPT-4.1 with AX-Tree + Memory—achieves Completion Rate 75.00%, Precision 91.60%, Recall 83.95%, and F1 87.61%, whereas the best advanced-task configuration—Claude Sonnet 4 with AX-Tree—achieves Completion Rate 53.49%, Precision 63.37%, Recall 63.41%, and F1 63.39% (Peeters et al., 18 Aug 2025). The study’s main result is that AX-Tree observations and persistent short-term memory materially reduce premature submission and cross-shop coverage failures, making them first-class ShopGuru components for online comparison-shopping.

5. Knowledge representation, multimodality, and benchmark synthesis

ShopGuru’s knowledge layer is informed by multi-modal e-commerce knowledge graphs. AliMe MKG centers product understanding on a cognitive chain linking Scenario, Problem, POI, Property_value, and Item, with images attached through has_image relations (Xu et al., 2021). Textual IE supplies item properties and POIs, images are processed through a two-stream ViT + StructBERT model, and cross-modal matching uses a dot product between pooled image and text embeddings. On the Beauty domain, the paper reports AUC 0.9861 for cross-modal matching and a 55.18× online inference speedup relative to Pixel-BERT; the deployed system serves hundreds of thousands of customers per day in Taobao (Xu et al., 2021). Reported graph scale includes 400+ scenarios, 1K+ user problems, 500K+ POIs, 300K+ items, and 28M+ images (Xu et al., 2021). For ShopGuru, this means that product assistance can be grounded not only in catalog fields but in scenario-oriented semantics and evidence-bearing media.

The most literal use of the name appears in ShopGym, where ShopGuru is the benchmark layer above ShopArena (Savadikar et al., 15 May 2026). It consumes products.json, collections.json, pages.json, stats.json, and capability flags, then emits validated tasks over seven skill categories: search-exact, search-substitute, browse, filter, shipping policy, returns/refund policy, and long-horizon shopping journeys (Savadikar et al., 15 May 2026). Tasks are formalized as $T=(E,s_0,p,V)$ , with binary success rate

$SR=\frac{1}{N}\sum_{i=1}^{N} V(\tau_i).$

The generation pipeline mixes deterministic short-horizon generators with LLM-authored long-horizon tasks, then applies a seven-rule validator covering unknown collections, unknown products, infeasible filters, intent-answer leakage, option mismatch, product-not-in-collection, and unknown pages (Savadikar et al., 15 May 2026). Across 224 generated tasks over six sandbox shops, BrowserGym long-horizon success rates are 47.9% for GPT-5-mini, 59.0% for Gemini 3 Flash, and 62.5% for GPT-5 (Savadikar et al., 15 May 2026). In this sense, ShopGuru is not only an assistant design but also a standardized task-construction methodology for evaluating shopping agents under stable conditions.

6. Evaluation regimes, limitations, and future directions

A recurrent property of ShopGuru-related research is asymmetry between module-level performance and end-to-end validation. ISA reports strong NLP results—98.20% intent accuracy and specification QA Top-1/Top-2/Top-3 accuracies of 85.60%, 95.80%, and 97.60%—but does not report WER, latency, computer-vision accuracy, or robustness to lighting, occlusion, barcode damage, or ambient noise; it also states that a user study had not yet been conducted (Lai et al., 2020). SANIP likewise reports qualitative real-time success but no precision/recall/mAP, no train/val/test split, and no user study (Deshmukh et al., 2022). ARShopping identifies connectivity, battery drain, glyph readability, and scanning precision as concerns, and its evaluation is virtual rather than in-store (Xu et al., 2022). These gaps indicate that ShopGuru evaluation must extend beyond isolated classifier accuracy to real-store acoustics, shelf clutter, end-to-end task success, and human factors.

Several lines also expose unresolved generalization problems. Shopping Companion shows that budget-constrained multi-item optimization remains difficult even for strong closed-source models and that tool-wise reward design is domain-specific (Yu et al., 16 Mar 2026). WebMall demonstrates that vision-only web agents are markedly weaker than AX-Tree-based agents and that advanced tasks involving vague requirements, substitution, or compatibility remain substantially below basic-task performance (Peeters et al., 18 Aug 2025). ShopGym explicitly preserves realism by grounding synthetic shops in live storefront structure, but it also deliberately removes operational noise such as inventory churn and third-party integration effects, so its benchmarks are not substitutes for live-web robustness tests (Savadikar et al., 15 May 2026). A plausible implication is that ShopGuru requires a dual evaluation stack: controlled sandbox benchmarking for reproducibility and live-environment testing for non-stationary failure modes.

The future-work agenda is correspondingly broad. ISA identifies broader QA coverage, support for other languages and informal text, and user-study evaluation as open directions, explicitly pointing to multilingual and social-style language handling (Lai et al., 2020). SANIP’s future scope includes app deployment via TensorFlow and larger datasets to improve accuracy (Deshmukh et al., 2022). ARShopping users requested recommendations based on goals, price comparisons across channels and over time, and more narrative explanations of specifications (Xu et al., 2022). ShopGym identifies checkout/payment-safe mock flows, multilingual evaluation, richer cart and total checks, and broader policy regimes as remaining extensions (Savadikar et al., 15 May 2026). Across these works, ShopGuru evolves toward a shopping-assistant stack that is simultaneously multimodal, memory-aware, accessibility-conscious, comparison-capable, and benchmarkable—yet still constrained by catalog heterogeneity, real-world robustness, and the difficulty of validating long-horizon shopping behavior under realistic conditions.