MMRM: A Multiplex Multimodal Representation Model for Product Ranking in E-commerce Search

Abstract: Multimodal information is pivotal for e-commerce search ranking. Existing works leverage multimodal data typically by fine-tuning general Multimodal LLMs (MLLMs) via collaborative signals, subsequently integrating the derived representations into ranking models as item features. Despite their efficacy, these methods face two primary limitations: (1) they rely on a single collaborative signal for MLLM fine-tuning, failing to exploit the heterogeneous signals essential for multitask ranking; and (2) they treat multimodal representations as regular item features in ranking models, underutilizing their latent potential for user behavior modeling. To address these challenges, we propose the Multiplex Multimodal Representation Model (MMRM), a unified framework that aligns MLLMs with diverse collaborative signals. By employing a shared backbone with task-specific tokens and projection layers, MMRM simultaneously learns from multiple signals and generates comprehensive multiplex item representations in a single inference pass. Furthermore, we introduce a multiplex user representation strategy in ranking models, which derives task-specific user representations via search-based behavior sequence modeling leveraging multiplex item representations. Extensive experiments demonstrate MMRM's superior efficiency and effectiveness. Notably, MMRM has been successfully deployed in the JD e-commerce search engine, yielding significant performance gains for millions of daily users.

• The paper introduces MMRM, a multiplex multimodal representation model that disentangles heterogeneous collaborative signals to improve e-commerce product ranking.
• It employs task-specific tokens, projection layers, and a graph-based data curation pipeline to effectively align query-item clicks, carts, and orders.
• Empirical results demonstrate significant GAUC gains and online performance improvements, validating the practical impact of MMRM in large-scale production systems.

MMRM: A Multiplex Multimodal Representation Model for Product Ranking in E-commerce Search

Motivation and Problem Formulation

The effective integration of heterogeneous multimodal signals is critical for contemporary e-commerce product search and recommendation systems, where user interactions reflect diverse intents across multiple downstream objectives. Existing approaches predominantly adapt multimodal LLMs (MLLMs) using a single collaborative signal (e.g., query-to-item clicks), and then inject these representations as “flat” item features into downstream ranking models. This paradigm introduces two principal limitations: inadequate exploitation of task heterogeneity in collaborative signals (e.g., clicks, carts, orders) and the underutilization of latent multimodal representations for fine-grained user behavior modeling.

MMRM Architecture and Data Construction

MMRM (Multiplex Multimodal Representation Model) addresses these issues via a unified architecture that aligns to four distinct collaborative signals: search-to-item click (q2i), item-to-item click, item-to-item cart, and item-to-item order. Leveraging a shared MLLM backbone (initialized from Qwen3-VL-4B-Instruct), MMRM introduces task-specific tokens and projection layers to obtain disentangled item embeddings per behavioral signal, enabling one-pass inference segmentation for multitask outputs.

The data curation pipeline systematically constructs large-scale triplet datasets for each collaborative signal. Q2i datasets are derived from query-item clicks, capturing coarse relevance, while i2i datasets (click, cart, order) are built using a graph-based approach over user behavior sequences, extracting fine-grained relational structures and employing hard-negative mining within item categories. This denoises influence from high-frequency promoted items and ensures robust contrastive learning.

Multiplex User Representation in Ranking

Beyond item-level modeling, MMRM proposes a multiplex user representation strategy for ranking architectures. User embeddings are dynamically generated per task via attentive behavior sequence modeling—specifically, top-K multi-head target attention over observed multiplex item representations. These task-conditioned user representations are input to a modular multi-gate mixture-of-experts (MMoE) layer in the ranking model, followed by task-tower networks for final prediction heads (CTR, ACR, CVR).

Multitask objectives are optimized with separate, weighted losses for each behavioral target, facilitating balanced learning across non-identically distributed signals. The architecture is compute-optimized using GradCache and FlashAttention, enabling efficient large batch training at scale.

Empirical Results

Extensive offline experiments on held-out datasets (with rich test splits per collaborative signal) yield several key findings:

1. Disentanglement and Task Alignment: Single-task models excel solely on their target signal, while multitask representation sharing (vanilla multi-tasking) degrades performance due to embedding collapse. MMRM’s multiplex, token-conditioned architecture yields consistent improvements (e.g., F1@5 and NDCG@5 across all signals), highlighting robust cross-task representation disentanglement.
2. Ranking Gains: In multi-objective ranking, leveraging MMRM-generated multiplex user and item embeddings delivers substantial gains in GAUC across CTR, ACR, and CVR compared to all baselines, including prior state-of-the-art soft searching and embedding-table approaches.
3. Online Deployment Impact: Deployed on JD’s production search engine, MMRM provided relative lifts of 0.42% (UCTR), 0.37% (UACR), and 0.35% (UCVR) over a strong MLLM baseline. Given the scale of user traffic, the practical impact is salient.

Theoretical and Practical Implications

This work consolidates the evidence that multitask alignment to heterogeneous multimodal signals is necessary for high-fidelity behavioral modeling in e-commerce. The use of structured triplets per signal and hard-negative sampling enables MMRM to scale contrastive learning, while the joint modeling of user and item multiplex representations is essential for combating both embedding collapse and signal dilution observed in multitask learning with naive parameter sharing.

Practically, MMRM demonstrates deployment-readiness, with design accommodations for production constraints (single-pass inference, large-batch training), and is validated at scale in a top industrial search engine.

Future Directions

Possible extensions include increasing the granularity of collaborative signals (e.g., session-level or context-enriched signals), integrating more advanced user sequence modeling (e.g., transformer-based sequence encoders), and exploring expansion to other modalities (video, audio) or domains (social commerce, personalized news feeds). Furthermore, research into automated signal weighting and dynamic loss rebalancing could further improve multi-objective alignment without manual tuning.

Conclusion

MMRM advances the integration of multimodal signals in e-commerce by providing a scalable, extensible architecture for both item and user representation multiplexing, thereby improving multitask ranking performance and user intent modeling. Its practical utility and empirical results establish a new standard for MLLM-based e-commerce retrieval and ranking systems (2607.11030).