---
title: Multi-modal Embedding Extraction
url: https://www.emergentmind.com/topics/multi-modal-embedding-extraction
type: topic
---

# Multi-modal Embedding Extraction

Generative Recommendation Systems (GRSs) represent a paradigm shift in personalized information access by unifying candidate retrieval, ranking, and—even in some scenarios—content creation within autoregressive generative frameworks. Rooted in advances in large-scale language models, multimodal representation learning, and scalable Transformer architectures, GRSs frame recommendation as conditional sequence generation, treating user–item interaction modeling as a structured generative process over semantic or collaborative token spaces. This approach is distinguished from classical discriminative or multi-stage retrieval paradigms by its potential to integrate world knowledge, exploit scaling laws, and enable new capabilities in reasoning, multi-modality, and adaptive deployment at internet scale [2510.27157, 2507.06507, 2502.13783].

## 1. Core Concepts and Mathematical Foundations

A GRS learns a conditional distribution over recommended sequences $S = (s_1, \ldots, s_T)$ given user $u$ and context $c$—which may include user profile, session state, item features, or rich prompts:

$$
P(S \mid u, c;\theta) = \prod_{t=1}^T P(s_t \mid s_{<t}, u, c; \theta)
$$

This formulation subsumes classical next-item prediction, sequential slate generation, and multi-step recommendations. The end-to-end training objective is typically the sequence negative log-likelihood (NLL):

$$
\mathcal{L}_{\text{NLL}} = -\sum_{(u, c, S) \in D} \sum_{t=1}^{|S|} \log P(s_t \mid s_{<t}, u, c;\theta)
$$

GRSs may represent items as semantic IDs (discrete, quantized codewords reflecting content and collaborative attributes [2509.25522, 2405.07314]), text tokens (item titles or descriptions), or even rich multimodal feature tokens. In many industrial deployments, the tokenization is performed via learned, hierarchical codebooks designed to balance semantic expressiveness, collaborative alignment, and code assignment diversity [2405.07314, 2511.15122].

## 2. Architectural Paradigms

GRSs encompass multiple architectural strategies:

- **Decoder-only and encoder–decoder Transformers:** Sequence modeling frameworks where item (or action) tokens are generated autoregressively, conditioned on histories, user features, and possibly action or context tokens [2507.06507, 2604.14878].
- **LLM-based GRS:** Direct fine-tuning or prompting of large language models, with options to inject collaborative signals via prompt engineering, adapter layers, or explicit ID tokenization [2502.13783, 2507.06507, 2509.25522].
- **Multi-modal GRS:** Integration of text, image, or video modalities into the token space. Approaches include parallel quantization of multimodal embeddings, late-fusion with modality markers, contrastive cross-modal alignment during codebook learning, and explicit cross-modal generation losses [2511.15122, 2503.23333].
- **Retrieval-oriented generative frameworks:** Use of session-level embedding generation for high-cardinality retrieval, bypassing strict autoregressive positional constraints to enable efficient candidate generation [2508.11977].

A concise taxonomy is provided in the table below.

| Paradigm                | Key Features                                          | Example Methods         |
|-------------------------|------------------------------------------------------|-------------------------|
| SID-based GR            | Autoregressive on semantic ID tokens; codebook design for content/collab.| TIGER, LC-Rec, LETTER [2405.07314, 2509.25522] |
| LLM-as-RS               | Direct generative modeling over item text; no discrete codebook| Qwen3, OneRec          |
| Multi-modal GR          | Codebook/tokenization for text, image, audio; contrastive alignment | MACRec [2511.15122], MGR-LF++ [2503.23333] |
| End-to-end Retrieval    | Session-level, position-free embedding generation; ANN retrieval| TBGRecall [2508.11977], TencentGR [2604.04976] |

## 3. Tokenization, Multimodal Fusion, and Collaborative Alignment

Efficient and expressive item tokenization is central. SID-based architectures construct identifiers through residual-quantized VAEs or MoE-based codebooks, with regularization losses enforcing semantic reconstruction, collaborative similarity (via contrastive learning on CF embeddings), and codeword utilization diversity [2405.07314, 2511.15122]. Multimodal GRSs address the longstanding challenge of integrating text, vision, and other signals:

- **Parallel cross-modal quantization:** Jointly learns text/image codebooks, regularized by layerwise contrastive and alignment losses to minimize collision rates and maximize codebook utilization [2511.15122].
- **Late fusion with modality markers:** Concatenates modality-specific semantic code sequences, inserting special tokens to clarify modality transitions and applying cross-modal contrastive objectives for alignment [2503.23333].
- **Hybrid generation and ensemble scoring:** During decoding, constrained beam search is applied over per-modality vocabularies, with ensemble or explicit alignment losses optimizing the final selection [2511.15122].

Such mechanisms result in improved performance (up to 20% relative), enhanced code assignment distribution, and increased robustness to cold-start and multi-domain signals.

## 4. Training Objectives, Fine-Tuning, and Preference Optimization

Beyond classical sequence NLL, GRSs employ a suite of objectives to address exposure bias, preference ranking, and policy alignment:

- **GFlowNet-based fine-tuning:** Treats multi-step generation as trajectory sampling, optimizing the flow such that $\pi_\theta(\tau) \propto R(\tau)$ for trajectory $\tau$, with rewards incorporating observed positives, collaborative filtering signals, and token-level similarity to exemplars [2506.16114].
- **Listwise direct preference optimization:** Moves beyond independent token prediction by directly enforcing item-level partial-order preferences (e.g., purchase ≻ click ≻ exposure) through listwise softmax losses [2602.08575].
- **Page-wise and session-wise supervision:** Densifies gradients and corrects one-to-many ambiguities in paginated or sessionized requests, accelerating convergence and reducing hallucination rates [2604.14878, 2508.11977].
- **Reinforcement Learning (RL) and hybrid reward models:** Incorporates user preference models, group-relative policy optimization, and supervised NLL regularization to align generation with observed and inferred satisfaction targets [2604.14878, 2602.23639].
- **Reflection-correction mechanisms:** Applies structured error localization and correction after initial token generation, supervised and further optimized by RL under task/trajectory-level rewards [2602.23639].
- **Model editing for cold-start collapse:** Facilitates training-free injection of new item token sequences via targeted parameter updates and position-wise gating, yielding strong cold-item recall with an order-of-magnitude faster updates than retraining [2603.14259].

## 5. Industrial-Scale Datasets, Benchmarks, and System Considerations

With the deployment of GRSs in production, new public benchmarks and system frameworks have emerged:

- **All-modality advertising datasets:** TencentGR-1M/10M [2604.04976] offer large-scale, multimodal logs with multi-action signals (impression, click, conversion), supporting open evaluation of large generative recommenders. Feature schemas span hashed IDs, multi-modal embeddings, and temporally structured action sequences, with evaluation metrics weighted to reflect practical business objectives (e.g., conversion gain).
- **Scalability and system co-design:** TurboGR [2605.13433] demonstrates optimized, jagged-operator-aware training on NPUs, achieving >54% MFU and near-linear scalability via fused attention kernels, dynamic jagged load balancing, semi-asynchronous communication, and negative sampling offload. This enables training of >0.2B parameter GRSs with long (8k+) sequences on industrial hardware, a critical step toward production-scale deployment.
- **Best practices from challenge competitions:** Winning solutions in large advertising challenges highlight critical design levers including robust time encoding, semantic ID quantization with long-tail regularization, very large negative banks under contrastive losses, ANN-based scalable inference, and deep backbone architectures—though with sensitivity to model size, negative pool, and training stability.

| Dataset         | #Users     | #Items      | Interactions     | Modalities         | Key Features             |
|-----------------|------------|-------------|------------------|--------------------|--------------------------|
| TencentGR-1M    | 1M         | 4.7M        | 90M (avg. 91/user)| Text, Image        | Clicks, exposures        |
| TencentGR-10M   | 10M        | 17.5M       | 973M (avg. 97/user)| Text, Image        | Clicks, exposures, conversions |

## 6. Open Problems, Challenges, and Future Directions

GRSs introduce new capabilities but also surface domain-specific technical challenges:

- **Scaling and bottlenecks:** Discrete codebook tokenizations (SID-based) saturate in performance under size scaling, while LLM-as-RS models exhibit smooth scaling laws and up to 20% higher recall at fixed data scale [2509.25522, 2502.13783]. The limited capacity of learned codes to retain semantic information is a critical bottleneck, suggesting a need for hybrid or end-to-end tokenization schemes.
- **Benchmarking and robustness:** Static, single-turn datasets remain insufficient; the field lacks standardized, multi-turn, multi-modal interactive benchmarks [2510.27157]. Robustness to adversarial and cold-start scenarios is an ongoing concern, partially addressed by model editing [2603.14259] and synthetic agent-based simulations [2510.27157].
- **Preference and alignment:** Capturing nuanced, multi-objective, and long-horizon user preferences involves integrating reward models, RL fine-tuning, listwise and groupwise ranking objectives, and regularizer-guided token learning [2602.08575, 2604.14878].
- **Real-time efficiency:** GRSs must optimize Prompt/ANN/beam search, minimize latency, and manage resource-intense negative sampling. Industry solutions exploit unique system-level co-design, including hierarchical sparse parallelism, pipeline orchestration, and asynchronous updates [2605.13433].
- **Explainability, fairness, and ethical considerations:** Interpretable decision-making, fairness under streaming data and demographic shifts, and resilience to popularity and position biases receive increasing scrutiny [2510.27157].

Continued research is focusing on dynamic benchmarking with user agents, unified multi-task LLM backbones, hybrid continuous–discrete representations, and advanced training techniques for continual learning, prompt adaptation, and efficient inference. The confluence of data-centric engineering, scaling law–informed architecture, and optimized system design defines the present and near future of generative recommendation research at scale [2502.13783, 2605.13433, 2510.27157].

## 7. Summary Table: Key GRS Methodologies

| GRS Type                  | Tokenization & Input    | Objective(s)              | Major Algorithms/Models                  |
|---------------------------|------------------------|---------------------------|------------------------------------------|
| SID-based GenRec          | Quantized item codes   | NLL, InfoNCE, RL/DPO      | TIGER, LC-Rec, LETTER [2405.07314]      |
| Multi-Modal GR            | Codebooks per modality | Contrastive, reconstr.    | MACRec [2511.15122], MGR-LF++ [2503.23333] |
| LLM-as-RS                 | Item text, prompts     | Generative LM fine-tuning | Qwen3, OneRec                           |
| Retrieval-oriented GenRec | Embeddings, session IDs| Contrastive, InfoNCE      | TBGRecall [2508.11977], TencentGR [2604.04976] |
| RL/Preference Aligned     | SID/sequence tokens    | RL (GRPO, DPO), ranking   | GenRec [2604.14878], RankGR [2602.08575]|
| Error Correction/Editing  | Token-level sequences  | SFT, RL, parameter editing| GRC [2602.23639], GenRecEdit [2603.14259]|

Each methodology addresses fundamental requirements of large-scale, industrial recommendation: unification of retrieval and ranking, adaptation to rich multimodal content, alignment to true user preference hierarchies, and robustness to both system scaling and dynamic inputs.

---

Generative Recommendation Systems thus consolidate decades of research into a new, cohesive paradigm, leveraging LLM-induced scaling laws, end-to-end differentiability, and unified sequence modeling to create highly adaptable, knowledge-infused, and potentially fully conversational recommendation engines [2510.27157, 2502.13783, 2507.06507]. Key open directions include surmounting tokenization-induced bottlenecks, fully harnessing cross-modal context, and translating algorithmic advances into reliably scalable, fair, and interpretable production deployments.

Source: https://www.emergentmind.com/topics/multi-modal-embedding-extraction