VimoRAG: Visual RAG for Retail FGC

• VimoRAG is a visual retrieval-augmented pipeline designed for few-shot fine-grained classification in retail, enabling dynamic class updates via a database of exemplars.
• It integrates image segmentation, OCR, and cosine-similarity based retrieval to extract structured product data including GTINs, prices, and discounts.
• Evaluated on a 367-way classification task, VimoRAG achieves up to 86.8% accuracy, outperforming conventional single-modal baselines in retail advertisement analysis.

VimoRAG is the “Visual RAG pipeline” introduced for few-shot Fine-Grained Classification (FGC) in retail advertisement leaflets, where it combines the Retrieval Augmented Generation (RAG) approach and Vision LLMs (VLMs) to extract product and promotion data and simultaneously predict fine-grained product ids along with price and discount information. Its key characteristic is that it allows the prediction of novel products without re-training, simply by adding a few class samples to the RAG database. In the reported comparison across GPT-4o, GPT-4o-mini, and Gemini 2.0 Flash back-ends, the approach achieves 86.8% accuracy on a diverse dataset (Lamm et al., 16 Apr 2025).

1. Problem setting and scope

Fine-Grained Classification still poses an open problem in many practically relevant applications. In the retail domain, the identification of fast changing and visually highly similar products and their properties is described as key to automated price-monitoring and product recommendation. VimoRAG is formulated for this setting: the input is a query advertisement image, and the system is designed to recover both categorical and promotional information rather than only a single class decision (Lamm et al., 16 Apr 2025).

The method targets advertisement leaflets from various retailers. Its output space is structured around product attributes such as brand, product_category, GTINs, price, discount, and related fields. This makes the system closer to a retrieval-augmented extraction-and-classification pipeline than to a conventional image-only classifier. A plausible implication is that the task couples visual discrimination, OCR-dependent context recovery, and schema-constrained generation within a single inference loop.

The core design decision is to externalize class knowledge into a retrievable database of examples. In the reported formulation, train-set examples $\{(x_i,t_i,y_i)\}$ are indexed, and inference proceeds by retrieving relevant exemplars and converting them into few-shot context for a VLM. This shifts adaptation from parameter updates to database maintenance.

2. Five-stage pipeline architecture

VimoRAG consists of five major stages: preprocessing; vector store creation; retrieval, classification, and relational query; prompt generation; and completion (Lamm et al., 16 Apr 2025).

In preprocessing, the input query advertisement image undergoes two parallel operations using LangSAM segment-anything. First, the product region is cropped out, yielding the image patch $x_{\rm prod}$. Second, the mask is inverted, meaning product pixels are erased to create a background image for OCR. GPT-4o is then used on the background image for OCR/text extraction to obtain a “visual” product description text $t_{\rm prod}$, with prices removed.

In vector store creation, the embedding models are “OpenCLIPEmbeddings” from xlm-roberta-large-ViT-H-14, which map image patches and product-description texts into a single shared $d$-dimensional space. All train-set examples are indexed in a Chroma vector database, storing image embeddings $v_i^{\rm img}$, text embeddings $v_i^{\rm txt}$, and metadata consisting of the ground-truth class label $y_i$.

The retrieval, classification, and relational query stage computes an embedding for the query image and performs cosine-similarity search in the vector store. The retrieved top-$K$ neighbors provide labels for majority-vote classification, after which the system filters the retrieved embeddings to those with the predicted label and fetches their full records $(x_{(j)}, t_{(j)}, y_{(j)})$ as few-shot context examples.

Prompt generation assembles a structured prompt comprising a system/task description, up to $M \leq 3$ few-shot examples, and the query image. If the token limit is exceeded, examples are dropped until the prompt fits. Completion then calls one of the supported VLM back-ends—GPT-4o, GPT-4o-mini, or Gemini 2.0-flash—and requests output in a schema via Pydantic so that each field is returned in structured form.

3. Retrieval-augmented classification mechanics

The retrieval step is written as embedding the query image into a shared space,

$x_{\rm prod}$0

and then computing cosine similarity against each database vector $x_{\rm prod}$1,

$x_{\rm prod}$2

The top-$x_{\rm prod}$3 indices are retrieved by descending similarity, and classification is performed by majority vote over their labels (Lamm et al., 16 Apr 2025).

The prediction rule is

$x_{\rm prod}$4

In the architecture description, ties are broken by the single nearest neighbor. This preserves a purely retrieval-based class decision before any generative reasoning occurs.

After classification, the system performs a relational query over the retrieved set. Only those retrieved examples whose label equals $x_{\rm prod}$5 are retained, and their full records become few-shot exemplars. The few-shot context therefore is not a generic nearest-neighbor set; it is a label-filtered subset aligned with the retrieved class hypothesis.

The prompt includes a system/task description of the form “Extract all features: brand, category, GTINs, price, …”, up to three few-shot examples, and the query image. The self-contained pseudocode also computes both $x_{\rm prod}$6 and $x_{\rm prod}$7 after preprocessing and OCR extraction, although the retrieval call itself is written as VectorStore.search(q_img, top=K). The output is parsed from a schema-constrained VLM completion rather than from free-form text.

4. Vector store design and open-set class addition

The few-shot database is implemented with Chroma, although the description states that it could equally use FAISS with cosine/HNSW. Each indexed example contributes image embeddings, text embeddings, and metadata containing the class label. This makes the database simultaneously a retrieval substrate and a class memory (Lamm et al., 16 Apr 2025).

The method’s open-set / dynamic classes mechanism is explicit. To add a new product class, one simply embeds its few sample images and texts and inserts them into the vector store. No model re-training is needed. At query time, retrieval will automatically surface nearest neighbors from the augmented index, and relational few-shot context will include the new class.

A common misunderstanding is to read this as ordinary classifier updating with prompt engineering layered on top. In the reported formulation, the decisive adaptation operation is database insertion rather than weight modification. The summary accordingly characterizes the system as a “re-train-free” few-shot FGC system.

This design also clarifies why the retrieval stage is central rather than auxiliary. Since class addition occurs through new vectors and metadata, retrieval quality, embedding quality, and vector-store hyperparameters directly govern the feasibility of recognizing novel products.

5. Experimental evaluation

The main GTIN classification benchmark is 367-way FGC with a test set of 1 101 images. The reported baselines are ResNet50 (image only) at 84.4%, BERT (text only) at 74.5%, and CLIP zero-shot at 45.9%. VimoRAG with GPT-4o-mini_2024-07-18 reaches 86.8%, outperforming those single-modal baselines on the reported dataset (Lamm et al., 16 Apr 2025).

The back-end comparison is reported per target field on the same test set. For GPT-4o-mini, the accuracies are brand 90.8%, product_category 88.1%, weight 80.0%, GTINs 81.2%, different_sorts 49.4%, price 87.9%, regular_price 36.5%, relative_discount 32.5%, and absolute_discount 89.4%. For GPT-4o, they are brand 86.2%, product_category 82.6%, weight 77.7%, GTINs 76.5%, different_sorts 48.9%, price 87.8%, regular_price 49.2%, relative_discount 38.6%, and absolute_discount 94.3%.

For Gemini 2.0 Flash, the reported accuracies are brand 90.7%, product_category 93.0%, weight 85.2%, GTINs 75.4%, different_sorts 49.0%, price 91.5%, regular_price 47.6%, relative_discount 31.8%, and absolute_discount 81.0%. For GPT-4o-mini+ with reduced context, they are brand 96.8%, product_category 93.0%, weight 84.7%, GTINs 86.0%, different_sorts 49.4%, price 91.8%, regular_price 32.0%, relative_discount 30.4%, and absolute_discount 88.2%.

The context-size ablation identifies a failure mode in which some invalid all-NULL VLM responses occur when too many shots are provided. The reported remedy is to re-invoke with only a single example if the VLM’s first response has all fields NULL. Under this change, GTIN accuracy rises from 81.2% to 86.0%. Under an alternative GTIN-matching metric that counts a prediction as correct if any single predicted GTIN matches the ground truth set, overall GTIN accuracy climbs to 92.3%.

Cost and latency are also reported for a full test-set run. GPT-4o-mini uses 92 978 average total tokens, 7.9 seconds average latency, and \$x_{\rm prod}$8241.91 total cost. Gemini 2.0-flash uses 104 436 average total tokens, 4.8 seconds average latency, and \$10.98 total cost, with the asterisked note that these are prompt + candidates tokens.

6. Limitations, potential extensions, and nomenclature

The reported limitations are concrete and largely pipeline-specific. VimoRAG depends on segmentation quality, and LangSAM sometimes misses or over-segments. The retrieval encoder choice and vector store hyperparameters were fixed, so better embeddings or re-ranking could help. Prompt-length and context management remain an issue because very large images or many shots may exceed token limits. The system relies on closed-source commercial VLMs, with the suggestion that future open models could be plugged in. The paper also notes context biases, such as the VLM reading printed numbers in the query image as GTIN, or biasing different_sorts from context (Lamm et al., 16 Apr 2025).

The potential extensions are likewise explicit. The same Visual RAG design could be applied to other fine-grained domains such as birds, cars, and seeds. Other proposed directions are to integrate a learned re-ranker on top of raw cosine retrieval, to fuse vision and text retrieval in a VisRAG-style multi-modal RAG setup, and to dynamically adapt context size and prompt templates via reinforcement or learned policies. These extensions indicate that the presented system is a specific instantiation of a broader retrieval-augmented multimodal classification pattern rather than an endpoint architecture.

The name “VimoRAG” is not unique across recent arXiv usage. “VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion LLMs” uses the same name for a framework in which a Gemini Motion Video Retriever supplies relevant 2D human motion signals from 425 K+ uncaptioned human videos to a motion LLM for 3D motion generation (Xu et al., 16 Aug 2025). Related nomenclature also includes “Visual RAG: Expanding MLLM visual knowledge without fine-tuning,” which uses retrieval-augmented in-context learning with Gemini 1.5 Pro, CLIP embeddings, and a FAISS IndexFlatL2 retriever for image classification across eight datasets (Bonomo et al., 18 Jan 2025). Complementary work includes the “Visual-RAG” benchmark for text-to-image retrieval augmented generation on visual knowledge intensive queries (Wu et al., 23 Feb 2025) and the multimodal design-space study “mRAG,” which examines retrieval, re-ranking, generation, and a unified self-reflection framework for LVLMs (Hu et al., 29 May 2025).

Within that broader landscape, the product-classification VimoRAG is distinguished by its retail leaflet setting, its explicit five-stage extraction-and-classification workflow, and its open-set / dynamic classes procedure based on inserting few samples into a vector store. This suggests that, in practice, “VimoRAG” is best interpreted by paper-specific scope rather than by name alone.