PinCLIP: Large-scale Foundational Multimodal Representation at Pinterest

Abstract: While multi-modal Visual LLMs (VLMs) have demonstrated significant success across various domains, the integration of VLMs into recommendation and retrieval systems remains a challenge, due to issues like training objective discrepancies and serving efficiency bottlenecks. This paper introduces PinCLIP, a large-scale visual representation learning approach developed to enhance retrieval and ranking models at Pinterest by leveraging VLMs to learn image-text alignment. We propose a novel hybrid Vision Transformer architecture that utilizes a VLM backbone and a hybrid fusion mechanism to capture multi-modality content representation at varying granularities. Beyond standard image-to-text alignment objectives, we introduce a neighbor alignment objective to model the cross-fusion of multi-modal representations within the Pinterest Pin-Board graph. Offline evaluations show that PinCLIP outperforms state-of-the-art baselines, such as Qwen, by 20% in multi-modal retrieval tasks. Online A/B testing demonstrates significant business impact, including substantial engagement gains across all major surfaces in Pinterest. Notably, PinCLIP significantly addresses the "cold-start" problem, enhancing fresh content distribution with a 15% Repin increase in organic content and 8.7% higher click for new Ads.

• The paper introduces PinCLIP, a production-scale multimodal representation system trained on about 890 million images and 2.4 billion Pin-to-Pin neighbor pairs to align visual, textual, and collaborative signals.
• PinCLIP combines hybrid convolutional-Funnel ViT encoders, multilingual SigLIP text towers, cross-modal fusion, and neighbor contrastive learning, achieving over 20% offline retrieval gains versus strong public baselines.
• The system improves online engagement, including up to 1.84% more Related Pins Repins, 5.02% higher Ads CTR, and substantially stronger fresh-content performance that helps address recommendation cold start.

Overview

PinCLIP is a large-scale multimodal representation learning system developed at Pinterest to supply embeddings for retrieval and ranking models across the platform's major surfaces, including Homefeed, Related Pins, Search, and Ads. The system addresses two persistent obstacles in integrating vision-LLMs (VLMs) into production recommender systems: the mismatch between generative VLM training objectives and the discriminative objectives of recommendation models, and the serving-cost constraints that make billion-parameter models impractical for low-latency inference at scale. The authors train on approximately 890M unique Pinterest images with multiple text sources and roughly 2.4B Pin-to-Pin neighbor pairs derived from the Pin-Board graph, and report offline retrieval gains exceeding 20% over strong public baselines such as Qwen3-VL-Embedding-8B, alongside significant engagement improvements in online A/B tests.

Architecture

PinCLIP comprises three components: an image encoder $E_{\mathsf{img}}$, a text encoder $E_{\mathsf{txt}}$, and a fusion encoder $E_{\mathsf{fsn}}$. The image encoders are in-house pretrained "Hybrid ViT" models combining a convolutional stem (ResNet-9 or InceptionNeXt-Tiny) with a Funnel-ViT trunk; funneling operations reduce token counts progressively, cutting compute and memory while preserving retrieval quality. The text encoders are multilingual SigLIP text towers of two sizes (278M and 700M parameters). The fusion encoder concatenates unpooled visual and textual token sequences and applies a small Transformer aggregator ($L_f = 2$ layers) with vanilla self-attention, followed by a single multi-head attention pooler with a learned query to compress the sequence into one $\ell_2$-normalized embedding.

Two contrastive objectives are combined. The first is image-to-text alignment using SigLIP's sigmoid loss, applied independently to descriptive text (title, description, or synthetic caption from BLIP/LLaVA) and keyword text (search queries or extracted annotations). Notably, the authors coalesce text sources into these two groups rather than concatenating them, citing evidence that CLIP-style text encoders have short effective context lengths and that verbose descriptions can degrade contrastive learning. The second objective is novel: Pin-to-Pin neighbor alignment, which applies the same sigmoid loss to pairs of fused Pin embeddings sampled via random walks over the pruned Pin-Board bipartite graph. This injects collaborative curation signal into the content embedding space, distinguishing PinCLIP from purely content-aligned approaches such as VL-CLIP (Giahi et al., 22 Jul 2025), which uses separate encoders and object-level crops without graph-derived alignment.

Training recipe and efficiency

Training uses a global batch size of 32,768 for image-text alignment and 4,096 for neighbor alignment on 64 H100 GPUs over 150k steps (~6B images), with the Lion optimizer, cosine decay, and a 10x lower learning rate for the text tower. Several efficiency findings are practically important:

• Partial freezing of the vision tower: unlocking only the final 12 layers of the image encoder yields the best downstream performance while reducing GPU memory by 16% and increasing throughput by ~16%, consistent with but extending LiT-style locked-image tuning.
• FlashAttention-2 and activation checkpointing improve throughput by 15% and reduce memory by 32%.
• Matryoshka Representation Learning (MRL) trains nested prefixes at dimensions 64/128/256, allowing downstream consumers to use a 64d prefix for coarse candidate generation; the 64d prefix is reported as an acceptable quality/cost tradeoff.
• int8 global affine post-training quantization halves embedding footprint (4096 to 2048 bits per 256d vector) with negligible offline metric impact.

A plain ViT backbone would require 2–4x more GPU resources for equivalent performance, underscoring the value of the hybrid conv-plus-funnel design.

Offline results

Evaluation covers five Recall@K tasks against 1.5M distractors: PinText image-to-text, Related Pins image-to-image and multimodal, and Search text-to-image and multimodal retrieval. The headline results are substantial:

Task	Best baseline	PinCLIP
PinText image-to-text	MetaCLIP2-g: 63.8	76.7 (image-text) / 76.4 (fusion)
Search text-to-image	MetaCLIP2-g: 27.4	44.5 (fusion)
Search multimodal	OmniSearchSage: 34.9	47.1
Related Pins image-to-image	SigLIP2-g: 35.7	44.5 (fusion)
Related Pins multimodal	Qwen3-VL-Embedding-8B: 23.6	57.2

The fusion model outperforms the image-text-only variant by +34% on Search and +44% on Related Pins, attributing the gains jointly to cross-modal fusion and the neighbor alignment objective. A striking result is that Qwen3-VL-Embedding-8B underperforms even smaller baselines on several Pinterest-specific tasks (e.g., 32.0 on PinText versus 63.8 for MetaCLIP2-g), suggesting that general-purpose web-scale multimodal embedders transfer poorly to this domain — though the paper does not investigate why. Ablations confirm monotonic gains from scaling both towers (Hybrid-ViT-g + Transformer-L reaches 84.9 R@10 on PinText), benefits from larger curated datasets up to ~900M images (with diminishing returns beyond ~300M), and an optimum at partially frozen vision depth.

Online impact

A/B tests integrate PinCLIP as a ranking feature and as an HNSW-based candidate generator. As a ranking feature, it produces surface Repin gains of +0.91% (Homefeed), +1.84% (Related Pins), and +0.96% (Search), with fresh-content Repins (content younger than 28 days) up +5.02% to +15.35% depending on surface. In Ads ranking, CTR improves +5.02% and click volume on new Ads (<14 days old) rises +8.67%. As a candidate generator, it adds +0.36% sitewide Repins and +2.3% sitewide fresh Repins on Related Pins, plus +0.34% search fulfillment rate. The disproportionate gains on fresh-content metrics indicate that content-based multimodal representations substantially mitigate cold-start behavior, since new Pins lack the engagement history that engagement-based embeddings require.

Limitations and open questions

The paper concedes several constraints. Roughly 30% of Pins lack titles or descriptions or carry noisy text, motivating reliance on synthetic captions and search-query keywords; the dependence of final quality on captioner quality (BLIP/LLaVA) and on CLIPScore-based filtering thresholds is not ablated. The neighbor-pair sampling depends on the structure and pruning of the Pin-Board graph, and the paper leaves open whether session-based engagement pairs could substitute for graph random walks at equal quality. Offline evaluation uses sampled engaged pairs with random distractors rather than full-corpus retrieval, so absolute Recall@K values may not reflect production difficulty. Finally, the poor performance of Qwen3-VL-Embedding-8B on Pinterest tasks is reported but unexplained, and the interaction between MRL truncation and quantization at very small prefixes (below 64d) is not explored.

Conclusion

PinCLIP demonstrates that a domain-tuned hybrid ViT with cross-modal fusion, SigLIP-style sigmoid alignment, and a graph-derived neighbor alignment objective can outperform much larger general-purpose multimodal embedders on production retrieval tasks while remaining deployable through MRL truncation and int8 quantization. The consistent fresh-content gains across organic and Ads surfaces provide concrete evidence that learned multimodal content representations address cold-start recommendation at industrial scale.