VLM4Rec: Multimodal Semantic Recommendation

Updated 5 July 2026

The paper introduces the VLM4Rec framework that semantically aligns visual and textual inputs for improved recommendation ranking.
It leverages methods like shared-space alignment and language grounding to construct fine-grained, preference-relevant item representations.
Empirical results show significant gains on benchmark datasets while addressing challenges like prompt quality, domain mismatch, and scalability.

Multimodal Semantic Representation for Recommendation, often abbreviated as VLM4Rec, denotes a general framework for recommendation in which vision–LLMs or closely related multimodal large models are used to construct aligned, fine-grained item semantics for downstream ranking, retrieval, or sequential prediction. In this formulation, recommendation quality depends not only on combining visual and textual inputs, but on whether item content is organized in a semantic space matched to user preference formation. Across recent work, VLM4Rec appears in several concrete forms: CLIP-based multi-view alignment, LVLM-driven image grounding into natural language, MLLM-based user preference reasoning, lightweight adaptation of large vision-language backbones, and information-aware or graph-aware semantic refinement (Wu et al., 2024, Valencia et al., 13 Mar 2026, Dang et al., 21 Aug 2025).

1. Conceptual foundations

VLM4Rec emerges from a critique of conventional multimodal recommendation. Earlier systems often treated multimodal recommendation as a feature-fusion problem: visual and textual embeddings were extracted independently from upstream models and then concatenated, averaged, or propagated over user–item or item–item graphs. Several papers argue that this is insufficient because it leaves unresolved the cross-modal semantic gap, preserves nuisance factors from raw visual inputs, and often fails to represent the attributes that actually drive interaction decisions (Wu et al., 2024, Valencia et al., 13 Mar 2026).

Within this perspective, the central problem is top- $K$ recommendation under implicit feedback. A representative formulation uses a user set $\mathcal{U}$ , an item set $\mathcal{I}$ , and binary interactions $R \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|}$ , with the objective of learning user and item representations such that observed items rank above unobserved ones (Dang et al., 21 Aug 2025). What changes under VLM4Rec is not the ranking task itself, but the semantics of the representation space: raw modality vectors are replaced, aligned, or adapted so that proximity reflects preference-relevant factors such as style, material, function, occasion, or interaction motivation rather than low-level appearance similarity (Valencia et al., 13 Mar 2026).

This shift has antecedents in earlier multimodal recommenders that already used pre-trained vision-language encoders for aligned content understanding. “MM-Rec” employed ViLBERT with co-attentional Transformers over titles and image regions, then used crossmodal candidate-aware attention for user modeling in news recommendation (Wu et al., 2021). A plausible implication is that VLM4Rec does not introduce multimodal recommendation ex nihilo; rather, it systematizes a semantic-alignment view that recent large multimodal models make substantially more practical and explicit.

2. Semantic construction of item representations

A core question in VLM4Rec is how item semantics are built from images, text, and metadata. Recent work exhibits two dominant strategies: shared-space alignment and language grounding.

Shared-space alignment is exemplified by CLIPER. Each item’s metadata is segmented into multiple textual “views,” such as title, brand, categories, and description, and these views are aligned with the image in CLIP’s latent space. For item $i$ , the $j$ -th textual view and image are encoded as

$\mathbf{e}_{i, j}^{t} = E_{t}(T_{i}^{j}), \qquad \mathbf{e}_{i}^{v} = E_{v}(V_{i}),$

and view weights are computed by temperature-scaled cosine similarity. Self-Attention fusion then forms

$\mathbf{e}_{i}^{t} \leftarrow \sum\nolimits_{j=0}^{C-1} \mathbf{e}_{i, j}^{t} \cdot s_{i}^{j}.$

This treats multimodal semantic representation as alignment across multiple textual facets rather than simple concatenation, and empirically SA outperforms MLP, SUM, and Concat within CLIPER (Wu et al., 2024).

Language grounding instead converts the image into explicit text before encoding. In “VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-LLMs,” an LVLM first grounds the image $x_i$ into a semantic description $s_i$ ,

$\mathcal{U}$ 0

and a frozen text encoder then produces

$\mathcal{U}$ 1

Recommendation subsequently uses normalized item embeddings and a simple user profile built by mean pooling the last $\mathcal{U}$ 2 item embeddings (Valencia et al., 13 Mar 2026). MLLMRec follows a related route: Gemma3-27B generates a high-quality semantic description $\mathcal{U}$ 3 from each image, concatenates it with textual metadata $\mathcal{U}$ 4 to form

$\mathcal{U}$ 5

and encodes the result with a frozen Sentence-Transformer (Dang et al., 21 Aug 2025). VLIF applies the same “textification” principle with Qwen-VL 7B, but conditions the caption on the item title and then encodes both VLM-derived captions and raw text with Sentence-BERT in a shared latent space (Kieu et al., 3 Nov 2025).

The main item-side design patterns can be summarized as follows.

Framework	Item semantic strategy	Characteristic mechanism
CLIPER	Multi-view text–image alignment	CLIP similarity-weighted SA fusion
VLM4Rec	Image-to-language grounding	LLaVA-NeXT 7B + Sentence-BERT
MLLMRec	Semantic distillation from images	Gemma3-27B descriptions fused with metadata
VLIF	Title-guided visual enrichment	Qwen-VL 7B captions + PID-inspired fusion
SDA	Adapted LVLM item embeddings	CMSA + modality-disentangled adapters

Taken together, these variants indicate that VLM4Rec is not tied to a single encoder family. It includes CLIP-style latent alignment, explicit caption grounding, and lightweight LVLM adaptation, provided that the resulting item embeddings are semantically reorganized for recommendation rather than merely imported from generic pretraining (Wu et al., 2024, Rao et al., 7 Dec 2025).

3. User semantics and preference representation

VLM4Rec also changes how users are represented. A recurring claim in the literature is that users do not possess inherent multimodal features in the way items do, so naïve random user initialization or graph-only user encoding can create a semantic mismatch between user and item spaces (Dang et al., 21 Aug 2025, Xu et al., 16 Apr 2026).

The lightweight VLM4Rec formulation constructs a user profile from recent item semantics. After normalizing item vectors,

$\mathcal{U}$ 6

the profile is

$\mathcal{U}$ 7

and candidates are ranked by cosine similarity with $\mathcal{U}$ 8 (Valencia et al., 13 Mar 2026). This deliberately simple design isolates the effect of representation quality.

MLLMRec makes the user side substantially more explicit. For each user $\mathcal{U}$ 9, it builds a behavioral description list

$\mathcal{I}$ 0

feeds this list into an MLLM prompt that asks the model to reason about the user’s preferences, obtains a purified preference summary $\mathcal{I}$ 1, and encodes it as

$\mathcal{I}$ 2

A two-layer MLP adapter maps this vector into the recommendation space, and the resulting user embedding is scored against item embeddings by dot product (Dang et al., 21 Aug 2025). Reported ablations show that removing this reasoning module causes the largest degradation, and a variant that adds user–item GCN propagation degrades performance because of feature homogenization (Dang et al., 21 Aug 2025). This suggests that in some VLM4Rec systems, language-driven user semantics can substitute for part of the graph convolution typically used in multimodal recommendation.

Other work addresses the user side through initialization or structured cross-view learning. SG-URInit constructs each user’s initial representation from the modality embeddings of interacted items and the centroids of their clusters, with

$\mathcal{I}$ 3

thereby placing users in the same semantic space as items before training (Xu et al., 16 Apr 2026). BiVRec instead learns structured multi-interest user representations in an ID view and a multimodal view, then aligns them through coarse-grained overall semantic similarity and fine-grained interest allocation similarity (Hu et al., 2024). A plausible interpretation is that VLM4Rec increasingly treats user modeling as a semantic representation problem in its own right, not merely as a downstream consequence of item fusion.

4. Alignment, graph structure, and controlled fusion

A major branch of VLM4Rec research focuses on the fact that semantically improved item vectors are still not enough if graph topology, modality interactions, or fine-tuning dynamics remain misaligned.

One line of work refines item graphs. MLLMRec begins with cosine similarity over text-encoded item semantics,

$\mathcal{I}$ 4

forms a KNN graph, denoises it with a similarity threshold,

$\mathcal{I}$ 5

and then enhances topology using audience co-occurrence measured by Jaccard similarity,

$\mathcal{I}$ 6

The enhanced adjacency is

$\mathcal{I}$ 7

and LightGCN propagates over this refined item–item graph (Dang et al., 21 Aug 2025). SIGER pursues a related goal from the opposite direction: it extracts collaborative signals from the interaction graph and injects them into each modality-specific item graph through

$\mathcal{I}$ 8

then adds modulus-based perturbation and dual representation alignment to improve robustness to graph noise (Zhang et al., 8 Aug 2025).

A second line addresses misalignment inside large vision-language backbones. SDA identifies two problems: representation misalignment caused by domain gaps and gradient conflicts caused by shared adapters. Its Cross-Modal Structural Alignment aligns cross-modal similarity distributions to a structure-aware soft teacher,

$\mathcal{I}$ 9

while Modality-Disentangled Adaptation replaces shared LoRA with expertized, gated low-rank paths,

$R \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|}$ 0

The paper reports that gradient cosine similarity in Qwen2.5-VL’s last decoder layer changes from negative under LoRA ( $R \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|}$ 1 in $R \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|}$ 2, $R \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|}$ 3 in $R \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|}$ 4) to positive under MoDA ( $R \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|}$ 5 and $R \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|}$ 6), which is presented as evidence of reduced modality interference (Rao et al., 7 Dec 2025).

A third line concerns fusion control. VLIF argues from a Partial Information Decomposition perspective that multimodal recommendation should separate unique, redundant, and synergistic signals. It estimates synergy with cross-modal Transformers,

$R \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|}$ 7

derives a redundant component $R \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|}$ 8, removes redundancy from vision via orthogonal projection,

$R \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|}$ 9

and fuses item representations as

$i$ 0

MRdIB formulates a related objective via a multimodal information bottleneck, unique-information predictors, a MINE-based redundancy term, and a synergy objective on joint prediction (Kieu et al., 3 Nov 2025, Wang et al., 24 Sep 2025). Taken together, these papers suggest that VLM4Rec has moved beyond “use a larger encoder” toward explicit control of topology, alignment geometry, and cross-modal information flow.

5. Empirical evidence across representative systems

The empirical record for VLM4Rec is heterogeneous in architecture but relatively consistent in direction. On Amazon Baby, Sports, and Clothing, CLIPER reports average improvements over baselines of 8.40%/10.50% on Baby, 7.17%/8.32% on Sports, and 8.48%/7.99% on Clothing for Recall@K/NDCG@K, with the largest single improvement being FREEDOM-CLIPER on Clothing at +35.33% Recall@50 and +29.50% NDCG@50 (Wu et al., 2024).

MLLMRec reports larger gains on the same Amazon-style benchmark family. Against strong multimodal baselines, it achieves +65.51% Recall@20 and +75.52% NDCG@20 on Baby, +25.99% and +22.87% on Sports, and +25.05% and +18.85% on Clothing, with an average improvement of 38.53% over the best baselines (Dang et al., 21 Aug 2025). Its plug-and-play graph refinements also improve several existing item–item graph learners on Baby by +5.05% Recall@20 and +3.80% NDCG@20 on average (Dang et al., 21 Aug 2025).

The LVLM-grounded VLM4Rec paper emphasizes a different result: on its LLaVA-covered subset, text-only item representations derived from LLaVA-generated descriptions outperform all evaluated fusion variants. It reports Recall@10 = 0.354 for the LLaVA text-only model versus 0.228 for BERT text-only, a +54.9% improvement, while LLaVA-based attention, concatenation, naïve averaging, and SMORE fusion all remain below the LLaVA text-only variant (Valencia et al., 13 Mar 2026). This is the clearest empirical statement in the literature that, in at least one setting, representation quality can matter more than fusion complexity.

Evidence for adaptation and long-tail behavior appears in SDA. On Amazon Beauty, Sports, and Toys, SDA improves multimodal and sequential recommenders by an average of 6.15% in Hit@10 and 8.64% in NDCG@10, with gains up to 12.83% and 18.70% on long-tail items (Rao et al., 7 Dec 2025). VLIF, by contrast, reports smaller but consistent gains over strong baselines on Baby, Sports, and Clothing—for example, Baby $i$ 1 versus COHESION’s $i$ 2, and Baby $i$ 3 versus $i$ 4—which it attributes to title-guided visual enrichment and PID-inspired fusion (Kieu et al., 3 Nov 2025). A plausible synthesis is that stronger item semantics appear beneficial across multiple design choices, while the marginal gain from any specific fusion or alignment mechanism depends strongly on the quality of the initial semantic construction and the extent of structural noise in the graph or modality space.

6. Misconceptions, limitations, and research directions

A common misconception is that VLM4Rec is simply multimodal recommendation with larger encoders. The literature does not support that reduction. Several papers explicitly distinguish semantic alignment from direct feature fusion, and the VLM4Rec paper shows a case where raw-feature fusion loses to semantically grounded text-only retrieval (Valencia et al., 13 Mar 2026, Wu et al., 2024). Another misconception is that VLM4Rec necessarily requires end-to-end fine-tuning of very large models. CLIPER uses frozen CLIP/Long-CLIP encoders, MLLMRec uses Gemma3-27B in inference-only mode, and the lightweight VLM4Rec formulation keeps both LLaVA-NeXT 7B and Sentence-BERT frozen (Wu et al., 2024, Dang et al., 21 Aug 2025, Valencia et al., 13 Mar 2026).

The main limitations recur across papers. Prompt quality and caption quality are persistent bottlenecks in grounding-based systems; MLLMRec notes dependence on prompt quality and inference cost, VLIF notes sensitivity to noisy or ambiguous titles, and the restaurant POI work identifies caption hallucination and proxy-data mismatch as open issues (Dang et al., 21 Aug 2025, Kieu et al., 3 Nov 2025, Kanzawa et al., 2024). Domain mismatch remains a second concern: SDA frames it as representation misalignment between item data and general LVLM pretraining, while several graph-based systems treat noisy raw features as a source of structural distortion that must be denoised or aligned (Rao et al., 7 Dec 2025, Zhang et al., 8 Aug 2025). Scalability is a third recurring issue, especially when semantic generation, KNN graph construction, or distribution-level alignment must be applied to very large catalogs (Dang et al., 21 Aug 2025, Rao et al., 7 Dec 2025).

Future directions are correspondingly convergent. Reported extensions include online preference updates with streaming prompts, conversational user refinement, retrieval-augmented generation for catalog knowledge, task-aware LVLM pretraining, efficient inference through batching and caching, user-side multimodal modeling, and controlled end-to-end optimization that balances alignment with recommendation loss (Dang et al., 21 Aug 2025, Rao et al., 7 Dec 2025). More broadly, these directions suggest that VLM4Rec is evolving from a family of item-side semantic enhancement techniques into a broader representation-learning program in which item content, user semantics, graph structure, and modality interactions are jointly organized around preference-aligned meaning rather than raw modality similarity.