VL-CLIP: Vision-Language Retrieval Framework

• The paper introduces VL-CLIP, a framework that augments CLIP with visual grounding and LLM-based text refinement to overcome e-commerce domain challenges.
• VL-CLIP employs object-focused cropping and an iterative LLM process to produce concise, attribute-rich text for improved image–text alignment.
• Empirical results show significant gains in HITS@5, MRR, and Precision, along with reduced latency, demonstrating its scalability and practical impact.

VL-CLIP denotes two closely related concepts in the CLIP literature. In the broad sense, it refers to CLIP-style vision–language systems that use aligned image and text embeddings as the core representational interface for downstream tasks. In the narrow sense, it names the e-commerce recommendation framework introduced in "VL-CLIP: Enhancing Multimodal Recommendations via Visual Grounding and LLM-Augmented CLIP Embeddings," which augments CLIP with product-centric visual grounding and LLM-refined text to improve retrieval and recommendation quality at large scale (Giahi et al., 22 Jul 2025). The broader usage is explicit in work on multilingual continual learning, dense-caption fine-tuning, and zero-shot navigation, where "VL-CLIP" denotes a family of dual-encoder or CLIP-driven pipelines rather than a single fixed architecture (Yang et al., 2024, Doveh et al., 2023, Dorbala et al., 2022).

1. Terminology and conceptual scope

VL-CLIP is not uniformly used as the name of one model class. In CLL-CLIP, the term is stated generically: "VL-CLIP" generally refers to CLIP-style vision-language dual-encoder models (Yang et al., 2024). DAC uses the phrase "VL-CLIP-style" to denote the standard dual-encoder plus InfoNCE training loop, with the emphasis placed on improved caption alignment and caption density rather than on architectural novelty (Doveh et al., 2023). CLIP-Nav is even more explicit: it does not introduce or explicitly name a model called "VL-CLIP," but instead uses CLIP for language grounding and navigation in a zero-shot VLN pipeline (Dorbala et al., 2022).

The specific framework called VL-CLIP, however, is the 2025 recommendation system for e-commerce. Its stated motivation is that vanilla CLIP has weak object-level alignment, ambiguous textual representations, and domain mismatch in e-commerce settings. The framework therefore combines Visual Grounding to localize the primary product and an LLM-based agent to produce concise, visually grounded text, while retaining CLIP’s dual-encoder structure and symmetric InfoNCE fine-tuning (Giahi et al., 22 Jul 2025).

A common misconception is that VL-CLIP denotes only the 2025 recommendation architecture. The literature instead supports a split usage: one paper uses VL-CLIP as a proper model name, while several others use it as a category label for CLIP-centered multimodal systems (Giahi et al., 22 Jul 2025, Yang et al., 2024).

2. Core architecture of the named VL-CLIP framework

The named VL-CLIP framework is organized around a product-retrieval pipeline. Inputs consist of large-scale catalog items with images, titles, descriptions, and structured attributes such as product type and gender/age. Preprocessing includes image deduplication via perceptual hashing, CLIP image normalization, text sanitization that strips HTML and promotional noise, and category balancing via stratified sampling for training and evaluation splits (Giahi et al., 22 Jul 2025).

The model then applies Grounding DINO with product-type prompts such as "handbag" or "dress" to localize the primary product. The highest-confidence box is selected; if its confidence is below threshold, the full image is retained. The cropped region is the image sent to the CLIP vision encoder. The text side is handled by an LLM-based agent that converts noisy metadata into a short, visually focused query under an iterative Summarizer–Evaluator–Refiner loop. The final image and text are encoded by fine-tuned CLIP dual encoders into 512-dimensional normalized embeddings, and alignment is learned with symmetric InfoNCE using in-batch negatives (Giahi et al., 22 Jul 2025).

The image and text embeddings are formalized as

$$v=\frac{\phi_{\text{CLIP-image}}(I_{\text{crop}})}{\left\|\phi_{\text{CLIP-image}}(I_{\text{crop}})\right\|}, \qquad t=\frac{\phi_{\text{CLIP-text}}(q^{\text{final}})}{\left\|\phi_{\text{CLIP-text}}(q^{\text{final}})\right\|}.$$

The crop selection rule is

$$I_{\text{crop}}= \begin{cases} \mathrm{Crop}(I,b_{i^*}), & \text{if } s_{i^*}\ge \tau_{\text{thresh}} \ I, & \text{otherwise}, \end{cases}$$

where $i^*=\arg\max_i s_i$. Training uses the standard symmetric CLIP contrastive objective

$$\mathcal{L}_{IT}=-\frac{1}{N}\sum_{i=1}^{N} \left( \log\frac{\exp(\mathrm{sim}(v_i,t_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(v_i,t_j)/\tau)} + \log\frac{\exp(\mathrm{sim}(t_i,v_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(t_i,v_j)/\tau)} \right),$$

with cosine similarity and temperature $\tau$ (Giahi et al., 22 Jul 2025).

This design preserves the CLIP retrieval geometry while changing what is presented to each encoder: object-aware crops on the image side and disambiguated, attribute-aware text on the language side.

3. Visual grounding and LLM-based text enrichment

The visual grounding module uses Grounding DINO as an open-set detector guided by product-type text prompts. VL-CLIP does not combine global and local image features; the refined crop is the primary image input. The paper also states that it does not add region–attribute alignment losses, token–region matching, or hard-negative mining beyond in-batch negatives. The reported gains therefore come from better visual focus via grounding and better textual specificity via LLM refinement, not from additional fine-grained alignment losses (Giahi et al., 22 Jul 2025).

The text-enrichment stage addresses catalog ambiguity. The agent receives concatenated structured and unstructured signals,

$$t_{\text{concat}}=[t_p \,\|\, t_g \,\|\, t_{\text{raw}} \,\|\, t_{\text{in-context}}],$$

where $t_p$ is product type, $t_g$ is gender/age, $t_{\text{raw}}$ is titles and descriptions, and $t_{\text{in-context}}$ provides few-shot exemplars. The Summarizer generates an initial short query, the Evaluator scores visual faithfulness, conciseness, and attribute consistency and may return <STOP>, and the Refiner applies the feedback for at most $$I_{\text{crop}}= \begin{cases} \mathrm{Crop}(I,b_{i^*}), & \text{if } s_{i^*}\ge \tau_{\text{thresh}} \ I, & \text{otherwise}, \end{cases}$$0 iterations. The output is constrained to fewer than 20 words and emphasizes visible attributes such as color, texture, shape, pattern, and neckline, while removing brand language and non-visual claims (Giahi et al., 22 Jul 2025).

The paper presents this decomposition as a direct response to three failure modes of vanilla CLIP in e-commerce. Weak object-level alignment is addressed by cropping around the product. Ambiguous textual representations are addressed by converting noisy descriptions into concise attribute-rich queries. Domain mismatch is addressed by fine-tuning CLIP on 7 million products from Walmart fashion and home categories while keeping Grounding DINO and the LLM fixed as preprocessing modules (Giahi et al., 22 Jul 2025).

A second misconception is that VL-CLIP’s gains require end-to-end multimodal reasoning over regions and language tokens. The framework as described is simpler: grounding is offline or preprocessing-oriented, LLM text synthesis is fixed, and CLIP itself is optimized with a conventional contrastive loss (Giahi et al., 22 Jul 2025).

4. Data, optimization, and retrieval infrastructure

Fine-tuning uses 7 million products from Walmart fashion and home categories. Each item includes images plus structured attributes and unstructured metadata. Stratified sampling is used to produce balanced fashion and home evaluation sets of 10k items each, spanning 17 product types total. The method focuses on single-image embeddings per item for retrieval, even though grounding and cropping are applied per image (Giahi et al., 22 Jul 2025).

The CLIP backbone is ViT-B/32 for images together with the CLIP transformer for text, and both are fine-tuned on the e-commerce domain. Training uses symmetric InfoNCE with 512-dimensional normalized embeddings. Early stopping occurs at 6 epochs based on validation loss and Recall@10; the validation loss improved from approximately $$I_{\text{crop}}= \begin{cases} \mathrm{Crop}(I,b_{i^*}), & \text{if } s_{i^*}\ge \tau_{\text{thresh}} \ I, & \text{otherwise}, \end{cases}$$1 to approximately $$I_{\text{crop}}= \begin{cases} \mathrm{Crop}(I,b_{i^*}), & \text{if } s_{i^*}\ge \tau_{\text{thresh}} \ I, & \text{otherwise}, \end{cases}$$2, and Recall@10 peaked at approximately $$I_{\text{crop}}= \begin{cases} \mathrm{Crop}(I,b_{i^*}), & \text{if } s_{i^*}\ge \tau_{\text{thresh}} \ I, & \text{otherwise}, \end{cases}$$3. Specific values for the temperature parameter are not reported (Giahi et al., 22 Jul 2025).

Serving is organized around approximate nearest-neighbor search. HNSW indices are built and partitioned by product type for latency and relevance, and an image embedding repository is maintained to avoid recomputation. Product-type HNSW indexing improves Precision@1 by $$I_{\text{crop}}= \begin{cases} \mathrm{Crop}(I,b_{i^*}), & \text{if } s_{i^*}\ge \tau_{\text{thresh}} \ I, & \text{otherwise}, \end{cases}$$4 and reduces latency by $$I_{\text{crop}}= \begin{cases} \mathrm{Crop}(I,b_{i^*}), & \text{if } s_{i^*}\ge \tau_{\text{thresh}} \ I, & \text{otherwise}, \end{cases}$$5 versus IVF indexing, while pHash-based deduplication improves MRR by $$I_{\text{crop}}= \begin{cases} \mathrm{Crop}(I,b_{i^*}), & \text{if } s_{i^*}\ge \tau_{\text{thresh}} \ I, & \text{otherwise}, \end{cases}$$6. Distributed jobs across T4 GPUs are used for embedding generation, and the system is described as scalable to tens of millions of items (Giahi et al., 22 Jul 2025).

This deployment profile is central to the framework’s identity. VL-CLIP is not merely a representation-learning modification to CLIP; it is a retrieval system whose design couples grounding, LLM normalization of catalog text, domain fine-tuning, indexing, and product-type partitioning.

5. Empirical performance

Offline evaluation is reported on Walmart Fashion and Home for multimodal retrieval, on Google Shopping for cross-domain transfer, on additional zero-shot Art and Toys categories, and through VLM-as-judge protocols for both query-based retrieval and similar-item recommendation. The framework is compared against CLIP, FashionCLIP, and GCL, and it consistently reports higher HITS@5, MRR, and Precision@K than those baselines (Giahi et al., 22 Jul 2025).

Setting	CLIP	VL-CLIP
Fashion HITS@5 / MRR	0.3080 / 0.2387	0.6758 / 0.5252
Home HITS@5 / MRR	0.2355 / 0.1747	0.6692 / 0.5100
Google Shopping HITS@5 / MRR	0.2419 / 0.1714	0.6644 / 0.4936

The gains extend to attribute-sensitive zero-shot classification. On fashion attributes, neckline accuracy is reported as $$I_{\text{crop}}= \begin{cases} \mathrm{Crop}(I,b_{i^*}), & \text{if } s_{i^*}\ge \tau_{\text{thresh}} \ I, & \text{otherwise}, \end{cases}$$7 for CLIP, $$I_{\text{crop}}= \begin{cases} \mathrm{Crop}(I,b_{i^*}), & \text{if } s_{i^*}\ge \tau_{\text{thresh}} \ I, & \text{otherwise}, \end{cases}$$8 for FashionCLIP, $$I_{\text{crop}}= \begin{cases} \mathrm{Crop}(I,b_{i^*}), & \text{if } s_{i^*}\ge \tau_{\text{thresh}} \ I, & \text{otherwise}, \end{cases}$$9 for GCL, and $i^*=\arg\max_i s_i$0 for VL-CLIP. Pattern accuracy is $i^*=\arg\max_i s_i$1, $i^*=\arg\max_i s_i$2, $i^*=\arg\max_i s_i$3, and $i^*=\arg\max_i s_i$4, respectively. In VLM-as-judge evaluation on the Walmart dataset, query-based retrieval Precision@1 rises from $i^*=\arg\max_i s_i$5 for CLIP to $i^*=\arg\max_i s_i$6 for VL-CLIP, and similar-item recommendation Precision@1 rises from $i^*=\arg\max_i s_i$7 to $i^*=\arg\max_i s_i$8. Cross-domain evaluation on Google Shopping shows analogous behavior, with query-based Precision@1 rising from $i^*=\arg\max_i s_i$9 to $$\mathcal{L}_{IT}=-\frac{1}{N}\sum_{i=1}^{N} \left( \log\frac{\exp(\mathrm{sim}(v_i,t_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(v_i,t_j)/\tau)} + \log\frac{\exp(\mathrm{sim}(t_i,v_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(t_i,v_j)/\tau)} \right),$$0 and similar-item Precision@1 from $$\mathcal{L}_{IT}=-\frac{1}{N}\sum_{i=1}^{N} \left( \log\frac{\exp(\mathrm{sim}(v_i,t_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(v_i,t_j)/\tau)} + \log\frac{\exp(\mathrm{sim}(t_i,v_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(t_i,v_j)/\tau)} \right),$$1 to $$\mathcal{L}_{IT}=-\frac{1}{N}\sum_{i=1}^{N} \left( \log\frac{\exp(\mathrm{sim}(v_i,t_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(v_i,t_j)/\tau)} + \log\frac{\exp(\mathrm{sim}(t_i,v_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(t_i,v_j)/\tau)} \right),$$2 (Giahi et al., 22 Jul 2025).

The paper also reports online A/B testing over millions of interactions during four weeks. Relative to baseline, VL-CLIP increases CTR by $$\mathcal{L}_{IT}=-\frac{1}{N}\sum_{i=1}^{N} \left( \log\frac{\exp(\mathrm{sim}(v_i,t_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(v_i,t_j)/\tau)} + \log\frac{\exp(\mathrm{sim}(t_i,v_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(t_i,v_j)/\tau)} \right),$$3, ATC by $$\mathcal{L}_{IT}=-\frac{1}{N}\sum_{i=1}^{N} \left( \log\frac{\exp(\mathrm{sim}(v_i,t_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(v_i,t_j)/\tau)} + \log\frac{\exp(\mathrm{sim}(t_i,v_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(t_i,v_j)/\tau)} \right),$$4, and GMV by $$\mathcal{L}_{IT}=-\frac{1}{N}\sum_{i=1}^{N} \left( \log\frac{\exp(\mathrm{sim}(v_i,t_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(v_i,t_j)/\tau)} + \log\frac{\exp(\mathrm{sim}(t_i,v_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(t_i,v_j)/\tau)} \right),$$5. Confidence intervals and statistical significance are not reported. The ablation study attributes substantial gains to both main modules: removing Visual Grounding lowers Fashion performance from $$\mathcal{L}_{IT}=-\frac{1}{N}\sum_{i=1}^{N} \left( \log\frac{\exp(\mathrm{sim}(v_i,t_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(v_i,t_j)/\tau)} + \log\frac{\exp(\mathrm{sim}(t_i,v_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(t_i,v_j)/\tau)} \right),$$6 to $$\mathcal{L}_{IT}=-\frac{1}{N}\sum_{i=1}^{N} \left( \log\frac{\exp(\mathrm{sim}(v_i,t_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(v_i,t_j)/\tau)} + \log\frac{\exp(\mathrm{sim}(t_i,v_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(t_i,v_j)/\tau)} \right),$$7 in HITS@5/MRR, and removing both Visual Grounding and the LLM lowers it further to $$\mathcal{L}_{IT}=-\frac{1}{N}\sum_{i=1}^{N} \left( \log\frac{\exp(\mathrm{sim}(v_i,t_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(v_i,t_j)/\tau)} + \log\frac{\exp(\mathrm{sim}(t_i,v_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(t_i,v_j)/\tau)} \right),$$8. Similar drops are reported in Home, leading the paper to describe grounding as the larger contributor and LLM refinement as an additional gain (Giahi et al., 22 Jul 2025).

6. VL-CLIP as a broader research family

Outside the recommendation framework, the literature uses VL-CLIP to describe a broader class of CLIP-centered multimodal methods. In zero-shot navigation, CLIP-Nav and Seq CLIP-Nav use CLIP to ground navigation keyphrases and activity descriptions, select candidate views, and trigger stopping or backtracking in REVERIE without dataset-specific fine-tuning; this work explicitly notes that it does not define a model called VL-CLIP, but it is an instance of CLIP used for language grounding and sequential decision-making (Dorbala et al., 2022).

In data-centric training, DAC augments a standard VL-CLIP dual-encoder loop with aligned and dense caption bags plus a negatives-enhanced multiple instance learning objective,

$$\mathcal{L}_{IT}=-\frac{1}{N}\sum_{i=1}^{N} \left( \log\frac{\exp(\mathrm{sim}(v_i,t_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(v_i,t_j)/\tau)} + \log\frac{\exp(\mathrm{sim}(t_i,v_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(t_i,v_j)/\tau)} \right),$$9

and reports up to approximately $\tau$0 improvement over base CLIP on compositional reasoning benchmarks while largely preserving downstream transferability (Doveh et al., 2023). In multilingual continual learning, CLL-CLIP extends CLIP to 36 languages by freezing the image and text encoders and training only an expandable token embedding layer under cross-modal and cross-lingual objectives, with TEIR regularization reducing catastrophic forgetting and improving text-to-image average Recall@1 on XM3600 by $\tau$1 (Yang et al., 2024).

Domain-specific variants also fit the VL-CLIP pattern. SkinCLIP-VL freezes a CLIP ViT-L/14 visual encoder, projects patch embeddings into a quantized Qwen2.5-VL decoder, and optimizes a Consistency-aware Focal Alignment loss for multimodal skin cancer diagnosis; it reports B-ACC $\tau$2 on ISIC 2019 and $\tau$3 on ISIC 2024 OOD with $\tau$4 fewer parameters than a 13B baseline (Lu et al., 22 Mar 2026). At test time, CLIPTTA reframes adaptation for CLIP-like VLMs around a soft contrastive objective aligned with CLIP pretraining, rather than entropy minimization, and reports superior robustness across 75 datasets under distribution shift (Lafon et al., 18 Jul 2025).

Taken together, these works show that VL-CLIP is best understood as an architectural idiom: CLIP supplies a shared embedding geometry, and task-specific modules alter how images, texts, prompts, or adaptation signals are presented to that geometry.

7. Limitations and future directions

The named VL-CLIP framework inherits several limitations acknowledged in its paper. It depends on metadata-driven prompts for grounding, so incorrect metadata can degrade crops. The LLM agent can hallucinate non-visual details, although the Evaluator/Refiner loop, strict prompt schema, and few-shot exemplars are intended to mitigate this. The system uses a single crop or the global image if confidence is low, so multi-object or multi-attribute scenes may require multi-region fusion. Cross-domain tests are described as promising, but the paper notes that extreme shifts, such as industrial parts, may need additional adaptation. The implementation is not released, although prompts are provided and Grounding DINO and CLIP are publicly available (Giahi et al., 22 Jul 2025).

The future work listed for the framework is correspondingly concrete: end-to-end grounding plus alignment, attribute-level alignment with word–region matching or multi-instance contrastive losses, multilingual support and taxonomy normalization across locales, personalization through graph or sequence signals, and multi-view or multi-image fusion per listing (Giahi et al., 22 Jul 2025). Related work indicates adjacent directions for the broader VL-CLIP family: caption alignment and caption density as explicit data objectives (Doveh et al., 2023), continual language expansion without joint multilingual retraining (Yang et al., 2024), and contrastive test-time adaptation under distribution shift (Lafon et al., 18 Jul 2025).

A plausible implication is that future VL-CLIP systems will combine the object-aware grounding of the recommendation framework with the data-centric supervision of DAC, the language extensibility of CLL-CLIP, and the adaptation mechanisms of CLIPTTA. The existing literature already treats these elements as compatible extensions of the same CLIP-centered vision–language paradigm (Giahi et al., 22 Jul 2025, Doveh et al., 2023, Yang et al., 2024, Lafon et al., 18 Jul 2025).