Multiplex Multimodal Representation Model (MMRM)

• The paper introduces MMRM, which generates four distinct task-specific item embeddings (SEARCH, CLICK, CART, ORDER) in a single pass, greatly enhancing efficiency.
• It employs a shared MLLM backbone with specialized tokens and MLP projections to integrate heterogeneous signals from clicks, cart, and order behaviors.
• MMRM advances ranking accuracy by modeling task-specific user behavior via search-based sequence attention, yielding improved CTR, ACR, and CVR.

The Multiplex Multimodal Representation Model (MMRM) is a unified framework for e-commerce search ranking that aligns Multimodal LLMs (MLLMs) with diverse collaborative signals, generates comprehensive multiplex item representations in a single inference pass, and introduces a multiplex user representation strategy based on search-based behavior sequence modeling (Chen et al., 13 Jul 2026). In the formulation reported for product ranking, MMRM addresses two limitations of prior multimodal ranking pipelines: reliance on a single collaborative signal for MLLM fine-tuning, and the treatment of multimodal representations as regular item features rather than as structures that can support user behavior modeling (Chen et al., 13 Jul 2026).

1. Problem setting and conceptual scope

MMRM is designed for e-commerce search, where ranking depends on both user intent and product semantics. The reported motivation is that conventional models, often ID-based, fail to utilize rich modalities such as text and visuals, and typically exploit only a single collaborative signal, for example query-to-item clicks. The framework therefore targets heterogeneous, multi-granular signals available in operational search systems, including query-to-item relevance and item-to-item behavioral relations derived from click, cart, and order sequences (Chen et al., 13 Jul 2026).

Within this setting, “multiplex” refers to the generation and downstream use of multiple task-specific representations rather than a single monolithic embedding. The model simultaneously learns from multiple signals and outputs four disentangled item embeddings associated with distinct behavioral objectives: SEARCH, CLICK, CART, and ORDER. The same multiplex principle is then applied to user modeling, where task-specific user representations are derived from behavior sequences using the corresponding multiplex item representations (Chen et al., 13 Jul 2026).

This design places MMRM at the intersection of multimodal representation learning and multitask ranking. A plausible implication is that the framework treats collaborative supervision not merely as a source of labels, but as a mechanism for structuring the embedding space into behavior-specific subspaces.

2. Item-side architecture and multimodal encoding

The reported MMRM architecture uses a shared multimodal backbone with task-specific tokens and projection layers. Inputs for an item include an image, a title, and auxiliary attributes such as shop name and price; queries are modeled as special-case items without image or attributes. During encoding, each item or query appends four special task tokens: [SEARCH], [CLICK], [CART], and [ORDER] (Chen et al., 13 Jul 2026).

The shared backbone is an MLLM, exemplified by Qwen3-VL-4B with a locked vision encoder. From the last hidden state corresponding to each task token, MMRM applies a task-specific MLP projection to obtain the task-conditioned item embedding:

$\mathbf{r}_i^t = \text{MLP}^t(\mathbf{h}_i^t)$

where $\mathbf{h}_i^t$ is the last hidden state of the backbone for token $t$, and $\text{MLP}^t$ is the task-specific projection for that token (Chen et al., 13 Jul 2026).

The architectural consequence is operationally significant: one forward pass produces all four task-specific embeddings. The paper describes these as multiplex item embeddings and reports that this makes MMRM 4x more efficient than running four single-task models (Chen et al., 13 Jul 2026). The framework summary further characterizes this as efficient multiplex embedding generation, accelerating both training and serving.

The backbone-and-token formulation distinguishes MMRM from systems that fine-tune a multimodal encoder for a single signal and then reuse its output as a static feature. Here, the representation interface itself is task-indexed.

3. Collaborative signals and multitask learning objective

MMRM is jointly trained on four collaborative signal datasets, each organized as triplets of anchor, positive, and negative samples. These datasets mix query-conditioned and behavior-conditioned supervision:

Signal	Description	Window
q2i_click	Query-to-item clicks	—
i2i_click	Sequential item-to-item clicks	10-minute session
i2i_cart	Consecutive cart additions	30-min window
i2i_order	Sequential orders	7-day window

For i2i graph-based sampling, the reported procedure selects positives among first-order and second-order neighbors weighted by transition probabilities, and hard negatives from the same sub-category. Anchor sampling is defined by

$P(v_a) = \sqrt{f(v_a)/t}$

where $f(v_a)$ is frequency and $t = 10^{-5}$ as in word2vec (Chen et al., 13 Jul 2026).

Training uses a multitask contrastive learning objective. The paper states that for each task $t$, MMRM employs a contrastive loss over a mixed batch of tasks, and that the total loss is a weighted sum over task-specific contrastive losses (Chen et al., 13 Jul 2026). Large-batch contrastive learning is reported to be enabled via GradCache and FlashAttention, which further boosts performance with manageable GPU memory.

This training regime ties the multiplex output heads to heterogeneous collaborative structure. The explicit inclusion of q2i and i2i signals means that broad relevance and finer-grained sequential similarity are learned within the same backbone (Chen et al., 13 Jul 2026). This suggests that MMRM is not simply multitask in the prediction sense; it is multiplex at the representation level because each signal family is associated with a distinct embedding channel.

4. Multiplex user representation and ranking

A central contribution of MMRM is that the multiplex item embeddings are not used only as item-side features. Instead, the reported ranking model derives task-specific user representations through search-based behavior sequence modeling. For a query $Q$, user behavior sequence $S$, and item $\mathbf{h}_i^t$0, the model first performs soft search to extract a top-$\mathbf{h}_i^t$1 relevant subsequence:

$\mathbf{h}_i^t$2

and then applies Multi-Head Target Attention (MHTA) to obtain the user representation for task $\mathbf{h}_i^t$3:

$\mathbf{h}_i^t$4

(Chen et al., 13 Jul 2026).

The reported intent is that user modeling should leverage task-specific, disentangled embeddings from historical interactions. This extends the multiplex principle from items to behavior sequences: SEARCH, CLICK, CART, and ORDER each induce their own user representation channel.

All task-specific user representations, together with additional features $\mathbf{h}_i^t$5, are then fed into a Multi-gate Mixture-of-Experts (MMoE) module:

$\mathbf{h}_i^t$6

After this stage, the pair $\mathbf{h}_i^t$7 is passed to a task-specific output tower for final prediction (Chen et al., 13 Jul 2026).

The framework summary identifies this component as a multiplex user representation strategy for multitask ranking. In that sense, MMRM is not only a representation learner but also a ranking architecture in which representation structure and behavior aggregation are coupled.

5. Empirical results, efficiency, and deployment

The reported training corpus mixes four datasets, each containing 0.3B triplets, for a total of more than 1B training triplets. Evaluation uses four 0.5M-sample test sets and an online A/B test on millions of daily users. Retrieval metrics are F1@5 and NDCG@5, while ranking is evaluated with GAUC for CTR, ACR, and CVR, corresponding to click, add-to-cart, and conversion rates (Chen et al., 13 Jul 2026).

For representation models, the summarized task-wise scores are reported as follows:

Model	q2i_click	i2i_click	i2i_cart	i2i_order
Single-task	0.1985	0.3254	0.3013	0.0904
Vanilla-Multi	0.1593	0.2987	0.2748	0.0676
MMRM	0.2055	0.3276	0.3037	0.0934

The paper states that MMRM outperforms all baselines on both retrieval and ranking across all tasks, and that multiplex item embeddings generated through task tokens and MLPs permit a single forward pass to produce all four task-specific representations (Chen et al., 13 Jul 2026). For ranking models, multiplex user representation using MMRM embeddings per task yields the highest GAUC scores across CTR, ACR, and CVR. The summary also states that soft search with multiplex embeddings, denoted SIMsoft(itemMMRM[[ALL](https://www.emergentmind.com/topics/cascading-annealed-language-learning-all)]), outperformed hard-category filtration and single-embedding baselines.

In online deployment, the reported A/B results are UCTR +0.42%, UACR +0.37%, and UCVR +0.35% over a strong production baseline. The system is described as fully deployed in JD.com’s e-commerce search engine and operational at massive scale (Chen et al., 13 Jul 2026).

6. Relation to adjacent multiplex and multimodal research

MMRM belongs to a broader research landscape in which “multiplex” and “multimodal” representation learning are instantiated through different mathematical and architectural choices. In brain network analysis, MGNet stacks multimodal graphs across subjects and modalities into a tensor $\mathbf{h}_i^t$8, learns a common latent space through multilinear tensor projection, constructs a consensus adjacency with K-nearest neighbors in the learned space, and applies multiplex GCNs followed by modality pooling for classification (Kong et al., 2021). In link prediction, MRGNN uses intra-layer node-level feature propagation, multi-layer projection into a shared latent space, and inter-layer representation aggregation with logistic or semantic attention, with a specific emphasis on weak ties in multiplex networks (Gu et al., 2024).

In latent-variable modeling, the MRF MVAE introduces Markov Random Fields into both prior and posterior distributions so that multimodal VAEs can explicitly model pairwise intermodal dependencies rather than relying on factorized aggregating schemes such as PoE, MoE, or MoPoE (Oubari et al., 2024). In shared-subspace learning, dMCCA uses parallel deep networks to transform multiple modalities into a common subspace by maximizing the ratio of between-modality to within-modality covariance, thereby targeting the common signal across more than two modalities (Somandepalli et al., 2019). In multilingual lexical modeling, a multimodal multiplex network is formalized as the quadruplet $\mathbf{h}_i^t$9, with a visual layer linked to lexical representations across languages (Huynh et al., 7 Nov 2025).

The acronym MMRM is also used in a different domain for the “Multimodal Multitask Restoring Model,” a system for restoring ancient ideographic texts by combining a RoBERTa context encoder, a ResNet50 image encoder, additive feature fusion, and task-specific text and image decoders (Duan et al., 2024). That usage is distinct from the product-ranking MMRM, although both employ multimodal inputs and multiple task heads.

Taken together, these works indicate that multiplex multimodal representation learning is not a single formalism. It includes tensorized multiplex GCN pipelines, attention-based multiplex graph learners, MRF-structured multimodal VAEs, covariance-ratio shared-subspace methods, multilayer lexical networks, and collaborative-signal-aligned multimodal rankers. Within that spectrum, the e-commerce MMRM is characterized by three specific features: a shared MLLM backbone with task-specific tokens and projections, joint learning from heterogeneous collaborative signals, and a ranking-stage multiplex user representation strategy (Chen et al., 13 Jul 2026).