---
title: All-Modality Generative Recommendation
url: https://www.emergentmind.com/topics/all-modality-generative-recommendation
type: topic
---

# All-Modality Generative Recommendation

All-modality generative recommendation is an emerging paradigm in information access systems that unifies multimodal content understanding, user preference modeling, and personalized content generation within a single end-to-end generative framework. Rather than re-ranking existing items for user selection, all-modality generative recommenders synthesize new or predicted user-relevant items—spanning images, text, audio, and other modalities—directly conditioned on a user’s multimodal interaction history. This approach integrates advances in large multimodal models (LMMs), hierarchically discrete semantic tokenization, sequence modeling, and reinforcement learning-driven reward optimization, surpassing classic retrieval-based methods in both flexibility and personalization [2506.01704].

## 1. Paradigm Shift: From Retrieval to Generative, Multimodal Recommendation

Traditional recommender systems are fundamentally bounded by the limitation that user preferences are satisfied only by selecting from a fixed corpus of pre-existing items. Even with sophisticated multimodal fusion, these systems cannot invent novel content outside the training set [2506.01704, 2510.27157]. All-modality generative recommenders address this core limitation in two complementary ways:

- **Autoregressive generation**—Modeling the next user-relevant item as a sequence of discrete tokens (semantic IDs), potentially corresponding to quantized features across text, image, and other modalities, rather than as a label from a closed vocabulary [2511.15122, 2602.03713].
- **Modality-universal modeling**—Leveraging architectures capable of ingesting and emitting arbitrary combinations of textual, visual, and other content modalities; the same model can decode text, images, or even audio/video representations conditioned on a user's full multimodal interaction history [2506.01704, 2403.10667].

This results in a system that can, for example, produce a personalized image aligned with a user's demonstrated historical preferences—an operation not accessible to retrieval-based recommenders [2506.01704]. The approach is extensible to video and audio, provided paired data and tokenization mechanisms are available.

## 2. Model Architectures and Cross-Modal Tokenization

All-modality generative recommendation requires a modeling pipeline that (a) unifies multimodal user and item representations and (b) supports autoregressive or iterative content generation. This is achieved through:

- **Any-to-any Large Multimodal Models (LMMs)**—Unified transformers ingesting both text and image tokens (typically via wordpiece tokenizers for text and VQ-VAE or residual quantization VAE tokenizers for images) and outputting the same [2506.01704]. Autoregression is maintained over a joint vocabulary.
- **Hierarchical and cross-modal semantic quantization**—Item content in each modality is mapped to sequences of discrete tokens using schemes such as multi-level residual quantization, with codebooks trained either through reconstruction (VQ-VAE), self-supervised representation learning (DINO), or contrastive objectives [2511.15122, 2602.03713, 2506.16683]. Recent advances (e.g., SimCIT) foreground contrastive alignment over mere reconstruction to maximize code discrimination and multimodal integration [2506.16683].
- **Unified input format and fusion**—User histories are serialized as joint sequences of semantic IDs across modalities, sometimes with modality separator tokens or learned inter-modality alignments [2506.01704, 2510.27157, 2503.23333].
- **Personalization via prompt design**—Generation is conditioned on user context through template-based prompts, e.g., "Please generate a movie poster that would attract my interest. Here are the movies I have watched: …" [2506.01704]. All modalities in the user's history inform autoregressive self-attention.
  
Typical architectures are illustrated by the use of unified backbones with modality-agnostic decoding heads, facilitating the generation of either images, text, or multimodal outputs as required [2506.01704, 2403.10667]. For instance, UniMP uses a frozen vision-language foundation model with cross-attention at every alternate transformer block to blend visual and textual semantics [2403.10667]. 

## 3. Training Objectives: Supervised, Contrastive, and Reinforcement Learning

The training pipeline in all-modality generative recommendation proceeds through distinct but complementary stages:

- **Supervised Fine-Tuning (SFT)**—Models are trained to predict the next item (in its tokenized multimodal representation) given user history, minimizing cross-entropy over the true sequence of code tokens [2506.01704, 2511.15122, 2510.27157]. Typical loss formulations:

  $$
  \mathcal{L}_{\mathrm{SFT}} = - \sum_{u}\sum_{k=1}^{|\mathcal{H}_u|} \sum_{j=1}^{J} \log p_{\mathrm{LMM}}\bigl(\mathcal{I}_{u_k;j} \mid \mathcal{P}(\mathcal{H}_u[1:k-1]),\mathcal{I}_{u_k;<j}\bigr)
  $$

- **Cross-modal alignment and contrastive losses**—To ensure the semantic IDs learned from different modalities are sufficiently aligned, contrastive or InfoNCE-style objectives are used both at the codebook learning stage (contrastive quantization, cross-modal reconstruction alignment) and optionally for the entire model [2511.15122, 2506.16683, 2602.03713].

- **Reinforcement Learning Fine-Tuning**—Online RL mechanisms (notably Group Relative Policy Optimization, GRPO) are introduced to adjust generation policies based on downstream composite rewards, including future relevance, semantic similarity (CLIP/DINO), diversity, and content aesthetics (NIMA) [2506.01704]. The RL objective is:

  $$
  \mathcal{L}_{\mathrm{GRPO}} = \mathbb{E}_{\mathcal{H},\{\mathcal{I}_j\}}\left[\frac{1}{G}\sum_{j=1}^{G}(\min\{\omega_j A_j,\mathrm{clip}(\omega_j,1\pm\varepsilon)A_j\}-\beta\,\mathrm{KL}[\pi_\theta(\cdot|\mathcal{H})\|\pi_{\mathrm{ref}}(\cdot|\mathcal{H})])\right]
  $$

- **Constrained Decoding**—Inference leverages prefix-tree (Trie)-masked beam search over valid token paths, ensuring only known or permissible item code sequences are generated. This is essential for both reliability and computational efficiency at industrial scale [2602.03713, 2512.21543].

Leading models enhance tokenization robustness and codebook utilization through multi-aspect alignment (MACRec), hierarchical residual quantization (FusID), and explicit fusion of collaborative signals (CEMG, MSCGRec) [2511.15122, 2602.03713, 2512.21543, 2601.08764].

## 4. Modalities: Conditioning, Fusion, and Generalization

All-modality generative recommendation systems are designed to operate over arbitrary combinations of modalities:

- **Multimodal item representation**—Each item is represented with parallel tokenizations for different modalities (e.g., text, image, collaborative, spatial), either concatenated in late fusion or merged through explicit fusion layers (e.g., soft-gating, MLP, cross-attention) [2511.15122, 2602.03713, 2512.21543].
- **User-aware and context-aware conditioning**—User histories combine sequentially interleaved multimodal tokens, with personalized prompts guiding generation [2506.01704]. New modalities are incorporated by extending pretrained encoders and quantizers or adapting reward functions to modality-relevant metrics (e.g., LPIPS/SSIM for images, ASR+CLIP for speech) [2506.01704].
- **Generalization**—While most published models target text and images, frameworks are explicitly proposed as modality-agnostic, with potential for audio/video extension, given appropriate discrete tokenization and evaluation metrics [2506.01704, 2403.10667].
- **Collaborative and knowledge signals**—Recent models explicitly include collaborative features as an additional modality, encoded and quantized alongside content representations and fused prior to code generation (MSCGRec, CEMG) [2602.03713, 2512.21543].

A representative overview:

| Model         | Tokenization           | Fusion         | Modalities (paper)          |
|---------------|-----------------------|---------------|-----------------------------|
| Janus-Pro-1B  | VQ-VAE (images), text | Unified Transformer | Image, text             |
| MACRec        | RQ-VAE (text, image), cross-modal alignment | T5 + scoring ensemble | Text, image    |
| MSCGRec       | RQ (text, image, coll) | Token concat + custom rel. pos. | Text, image, collab |
| UniMP         | Flat sequence, VL cross-attn | Foundation LM + Cross-attn | Image, text, ID   |
| CEMG          | RQ-VAE (fusion)       | Soft-gating fusion (collab-guided) | Text, image, collab |

## 5. Empirical Evaluation and Comparative Performance

Evaluation frameworks for all-modality generative recommendation are grounded in both information retrieval and generation metrics:

- **IR-style Top-k metrics**: HitRate@K, Recall@K, NDCG@K—measuring ranking quality of predicted item sets; NDCG formulations per standard IR conventions [2511.15122, 2602.03713, 2510.27157].
- **Semantic and perceptual metrics**: CLIP similarity (historical relevance, potential future relevance), DINO (visual similarity), PCS (profile-conditioned similarity), LPIPS, SSIM, MS-SSIM, NIMA (aesthetics) for image generation [2506.01704].
- **Qualitative & Human Evaluation**: Pairwise win/lose/tie studies comparing generated samples; preference aggregation showing RL-augmented models preferred by human annotators [2506.01704].
- **Efficiency & Scalability**: Memory footprint, codebook utilization, inference speed (users/second), cold-start and long-tail item coverage, scaling laws [2602.03713, 2512.21543, 2506.16683].
  
Empirically, all-modality generative recommenders consistently outperform both single-modality and retrieval-based systems across domains and modalities. Notable empirical findings include:

- Janus+SFT+RL achieves the highest relevance and aesthetic scores on MovieLens and PixelRec; RL tuning lifts performance by up to 10 points on CLIP-based metrics [2506.01704].
- MACRec outperforms MQL4GRec by +5–6% NDCG@10 via deep cross-modal alignment [2511.15122].
- TriAlignGR’s multitask (SID, text, image) alignment surpasses baseline HR@5/NDCG@5 by +13–15% [2605.05249].
- MSCGRec, fusing semantic, collaborative, and constrained decoding, improves NDCG@10 by up to +36% versus prior generative methods and matches or exceeds top discriminative models at industrial scale [2602.03713].
- Large benchmark competitions (e.g., Tencent Advertising Algorithm Challenge 2025) identify key contributions from heavy multimodal quantization, explicit feedback/action-type conditioning, and conversion-weighted loss functions [2604.04976].

## 6. Challenges, Limitations, and Open Directions

Current all-modality generative recommendation approaches face several critical challenges and research frontiers:

- **Tokenization conflicts and codebook underutilization**—Without adequate contrastive or cross-modal alignment, different modalities can yield redundant or ambiguous semantic IDs, undermining discriminative power [2601.08764]. Strategies such as contrastive tokenization, modality-aware product quantization, and chain-of-thought interest mining mitigate these issues [2506.16683, 2601.08764, 2605.05249].
- **Content degradation and semantic opacity**—Sequential quantization can discard high-level semantic and latent user interest information (SID Content Degradation—SCD) and produce token sequences without explicit semantic mapping (SID Semantic Opacity—SSO). TriAlignGR addresses this via two-stage CMSA/MDIM before quantization and triangular multitask alignment [2605.05249].
- **Bias, fairness, and explanation**—Distributional and selection biases, particularly popularity bias, pose risks. LLM4Rec introduces causal debiasing (propensity scoring, do-calculus, adversarial losses) and enforces demographic fairness, alongside LLM-based explainable generation [2510.01622].
- **Scalability and deployment**—While compact code vocabulary and constrained decoding enable large-scale deployment, Trie-based decoding and long code sequences remain potential bottlenecks for massive catalogs [2512.21543]. Memory and inference costs must be continually monitored.
- **Generalization to novel modalities**—Audio, video, spatial, and multi-source context (e.g., sensor/IoT data) are only at the threshold of integration, requiring new tokenizers, reward functions, and training paradigms [2506.01704, 2403.10667]. Open benchmarks (e.g., TencentGR-10M) are starting to include such breadth at scale [2604.04976].
- **Robustness and hallucination**—Generated content must remain grounded in user history and actual item candidates; hallucination, spurious alignment, and reward hacking can undermine trust and effectiveness [2506.01704, 2510.27157].

## 7. Synthesis and Prospects

All-modality generative recommendation represents a foundational shift in recommender system design, dissolving modality boundaries and retrieval constraints in favor of end-to-end generative architectures capable of personalized content synthesis. This paradigm, exemplified by approaches leveraging hierarchical cross-modal tokenization, unified autoregressive modeling, RL-driven personalization, and rigorous empirical evaluation, demonstrates clear advantages in personalization, extensibility, and generative capacity [2506.01704, 2511.15122, 2602.03713]. 

The crucial architectural ingredients—discrete semantic token spaces, robust multimodal alignment, fusion with collaborative signals, and pragmatic deployment strategies—are rapidly converging toward scalability and effectiveness for real-world information systems. Ongoing research focuses on multimodal scalability, robustness, fairness, and support for new content types, with open industrial benchmarks accelerating progress.

Ultimately, all-modality generative recommendation provides both a practical framework and a conceptual blueprint for next-generation intelligent assistants and content platforms, capable of holistic, user-driven synthesis across text, vision, and beyond [2510.27157, 2403.10667].

Source: https://www.emergentmind.com/topics/all-modality-generative-recommendation