DMAE: Distribution-Guided Multimodal Interest Auto-Encoder

• The paper introduces DMAE which dynamically fuses text and image signals at the behavior level to capture user interest more effectively.
• It employs a sophisticated encoding process combining discretization, sine-cosine encoding, and sliding-window self-attention for temporal context and modality interaction.
• Empirical results across multiple datasets show significant gains in AUC, Logloss, and user engagement metrics, demonstrating its practical efficacy in recommendation systems.

Searching arXiv for the DMAE paper and a few directly mentioned related methods for accurate citation.

arXiv search: DMAE

Distribution-Guided Multimodal-Interest Auto-Encoder (DMAE) is a multimodal recommendation model for click-through-rate-style prediction that is designed to dynamically fuse user multimodal interests at the behavior level rather than at the item level. It was introduced in "Distribution-Guided Auto-Encoder for User Multimodal Interest Cross Fusion" (Zhang et al., 20 Aug 2025). The model represents each historical click as a multimodal interest signal derived from the similarity between the target item and each clicked item under multiple modalities, specifically text and image, then encodes these signals into behavior-wise multimodal interest vectors, cross-fuses them within and across modalities using attention, and decodes them back into an interest distribution that supervises the encoder. In this formulation, DMAE is an auto-encoder for user multimodal interest whose latent representation is guided by a distribution reconstruction objective.

1. Motivation and problem formulation

DMAE is motivated by two limitations identified in prior recommendation pipelines (Zhang et al., 20 Aug 2025). First, traditional recommendation and CTR methods rely heavily on item ID embeddings to capture implicit collaborative filtering signals. The reported consequence is severe data sparsity, weak item representations, and an inability to exploit explicit semantic information carried by text and image features. Second, existing multimodal recommendation methods are described as predominantly early-fusion systems: they combine text and image features at the item level, but neglect the contextual influence of the user behavior sequence.

The method targets the gap created by this early-fusion bias. The reported claim is that prior multimodal methods typically use a single fused representation and therefore do not adapt multimodal interest according to the user behavior sequence. DMAE instead models behavior-level multimodal interest sequences, performs dynamic fusion, and preserves multimodal information through a distribution-guided decoder. This suggests that the model treats interest as a temporally structured, modality-conditioned signal rather than as a static multimodal descriptor.

The task setting is standard binary classification. Each instance is written as

$(\mathcal{X}, y),$

where $\mathcal{X}$ includes user profile $x_u$, user behavior sequence $\mathcal{S}$, and target item $x_i$, and $y \in \{0,1\}$ indicates click or no-click. The user behavior sequence is

$\mathcal{S} = \{x_1, \dots, x_j, \dots, x_T\},$

and the objective is to estimate

$P(y \mid \mathcal{X}).$

2. Architectural composition and input representations

DMAE consists of three modules: the Multimodal Interest Encoding Unit (MIEU), the Multimodal Interest Fusion Unit (MIFU), and the Interest-Distribution Decoding Unit (IDDU) (Zhang et al., 20 Aug 2025). The fused interest representation produced by these components is then combined with other features for click prediction.

The model consumes both ID and multimodal embeddings. For each example, the ID-side inputs are user embedding $\boldsymbol{v}_u$, target item embedding $\boldsymbol{v}_i$, and historical item embeddings $\mathcal{X}$0. The multimodal inputs are text embeddings $\mathcal{X}$1 and image embeddings $\mathcal{X}$2. These multimodal embeddings are frozen during recommendation training.

The pipeline can be summarized as follows:

Module	Function	Key mechanism
MIEU	Behavior-wise interest encoding	Similarity, discretization, sine-cosine encoding, position awareness
MIFU	Behavior-level multimodal cross fusion	Sliding-window self-attention and cross-modal attention
IDDU	Distribution-guided supervision	Masking, MLP decoding, KL divergence

A common misconception is to classify DMAE as another item-level multimodal early-fusion model. The method is explicitly described otherwise: its fusion occurs over interest vectors derived from user behaviors, not over raw item modalities. Another operational distinction is that the decoder is used only in training, not in online inference.

3. Multimodal Interest Encoding Unit

MIEU converts the similarity between the target item and each historical item into behavior-wise interest representations (Zhang et al., 20 Aug 2025). For each modality, the model constructs similarity score sequences

$\mathcal{X}$3

For modality $\mathcal{X}$4, similarity is defined as

$\mathcal{X}$5

This maps cosine similarity into the unit interval.

Because the similarity signal is continuous, MIEU applies both discretization and sine-cosine encoding. The discretization stage is written as

$\mathcal{X}$6

and

$\mathcal{X}$7

where $\mathcal{X}$8 and $\mathcal{X}$9 are learnable scalars, $x_u$0 is the number of buckets, $x_u$1 maps similarity into a bucket embedding, and $x_u$2 is the discretized vector.

The sine-cosine branch is introduced to preserve fine-grained differences between similarity values: $x_u$3

These encodings are combined and augmented with positional information: $x_u$4 where $x_u$5 denotes concatenation and $x_u$6 is the positional embedding for time index. The resulting modality-specific behavioral interest sequences are

$x_u$7

The reported rationale for this design is explicit: similarity alone is not enough; discretization captures coarse range information; sine-cosine encoding preserves continuous differences; and position encoding captures temporal dynamics of user interest. This suggests that MIEU is intended to bridge symbolic CTR-style sequence modeling and continuous metric structure in multimodal embedding spaces.

4. Behavior-level fusion and distribution-guided decoding

MIFU performs behavior-level multimodal cross fusion rather than early fusion of item modalities (Zhang et al., 20 Aug 2025). Its first stage is intra-modal interaction through sliding-window self-attention. For each modality,

$x_u$8

$x_u$9

The stated purpose is to capture intra-modal dependencies among behaviors while keeping complexity manageable.

Each modality is then summarized by average pooling: $\mathcal{S}$0 These summary vectors are used as contextual guidance to fuse the other modality. For modality $\mathcal{S}$1,

$\mathcal{S}$2

and for modality $\mathcal{S}$3,

$\mathcal{S}$4

The paper characterizes this as the key cross-fusion mechanism: one modality’s aggregated interest acts as the query or context, the other modality’s behavior sequence provides keys and values, and the result is a redefined modality-specific user interest vector. The reported outcome is both intra-modal crossover and inter-modal crossover.

IDDU reconstructs a true interest distribution from the latent interest representation. Its role is to reduce information loss during encoding and constrain latent vectors to preserve the original similarity structure. The ground-truth interest distribution is formed over time bins and similarity bins. The similarity partition is

$\mathcal{S}$5

For time bin $\mathcal{S}$6 and similarity bin $\mathcal{S}$7,

$\mathcal{S}$8

This yields modality-specific distributions

$\mathcal{S}$9

The decoder applies masking and MLP reconstruction: $x_i$0

$x_i$1

This is performed separately for $x_i$2 and $x_i$3, producing $x_i$4 and $x_i$5. The decoding loss is

$x_i$6

The source notes that the expanded form of the loss appears to contain minor notation typos in the second term, while the intended objective is clearly $x_i$7. Within the reported interpretation, IDDU acts as a self-supervised regularizer that preserves multimodal information and improves the quality of the encoder’s latent representations.

5. Prediction layer, optimization, and implementation setting

For final prediction, DMAE uses DIN as the backbone sequence encoder (Zhang et al., 20 Aug 2025). The sequence representation is

$x_i$8

Prediction is then defined as

$x_i$9

The prediction loss is binary cross entropy: $y \in \{0,1\}$0 The full DMAE objective is written as

$y \in \{0,1\}$1

where $y \in \{0,1\}$2 controls the strength of decoder regularization.

The experimental implementation uses TensorFlow with Adam, initial learning rate $y \in \{0,1\}$3, and exponential decay rate $y \in \{0,1\}$4. On public datasets, the MLP layers are $y \in \{0,1\}$5; on the industrial dataset they are $y \in \{0,1\}$6. The ID embedding dimension is grid searched over $y \in \{0,1\}$7, while the industrial ID embedding dimension is $y \in \{0,1\}$8. The MIFU window size is matched to the time-bin parameter $y \in \{0,1\}$9.

The datasets are Amazon-Books, Amazon-Electro, MovieLens-20M, and an industrial dataset. Amazon-Books contains 75,053 users, 358,367 items, and 1,583 categories. Amazon-Electro contains 192,403 users, 63,001 items, 801 categories, and 1,689,188 samples. MovieLens-20M contains 138,493 users, 27,278 movies, 21 categories, and 20,000,263 samples, with a binary label defined by rating $\mathcal{S} = \{x_1, \dots, x_j, \dots, x_T\},$0. The industrial dataset is described as a large-scale e-commerce display advertising dataset with train equal to the last 20 days and test equal to the next day, and it includes text and images. The multimodal features are extracted using GPT-3 for text embeddings and EVA-02 for image embeddings.

6. Empirical results, ablations, and reported significance

DMAE is reported to achieve the best performance on all evaluated datasets (Zhang et al., 20 Aug 2025). The evaluation metrics are AUC and Logloss, with GAUC$\mathcal{S} = \{x_1, \dots, x_j, \dots, x_T\},$1 additionally reported on the industrial dataset. The baseline set includes ID-based methods—Wide & Deep, FRNet, Average Pooling, GRU4Rec, DIN, and BST—and multimodal methods—UniSRec, LRD, DIMO, and SimTier.

Dataset	Reported performance	Notes
Amazon-Books	AUC 0.7645, Logloss 0.5873	Best reported result
Amazon-Electro	AUC 0.9105, Logloss 0.3334	Best reported result
MovieLens-20M	AUC 0.7570, Logloss 0.5793	Best reported result
Industrial dataset	AUC 0.7319, GAUC$\mathcal{S} = \{x_1, \dots, x_j, \dots, x_T\},$2 0.6249, Logloss 0.1028	Best reported result

Compared to DIN, the paper reports average gains of $\mathcal{S} = \{x_1, \dots, x_j, \dots, x_T\},$3 AUC and $\mathcal{S} = \{x_1, \dots, x_j, \dots, x_T\},$4 Logloss. The ablation study removes four components: similarity-score encoding in MIEU, time information in MIEU, the MIFU module, and the IDDU module. The main reported conclusions are that removing MIFU causes the largest drop, removing IDDU also hurts performance substantially, and removing either similarity encoding or time encoding degrades performance. The stated interpretation is that behavior-aware multimodal fusion is essential, distribution-guided decoding helps preserve information, and both similarity encoding and temporal context matter.

The hyperparameter study examines $\mathcal{S} = \{x_1, \dots, x_j, \dots, x_T\},$5, $\mathcal{S} = \{x_1, \dots, x_j, \dots, x_T\},$6 for time bins, and $\mathcal{S} = \{x_1, \dots, x_j, \dots, x_T\},$7 for similarity bins. The reported findings are that the best $\mathcal{S} = \{x_1, \dots, x_j, \dots, x_T\},$8 differs by dataset, but moderate-to-strong decoder weight works best; too small a value makes the decoder too weak, while too large a value distracts the prediction task. On industrial data, the best performance occurs at $\mathcal{S} = \{x_1, \dots, x_j, \dots, x_T\},$9 and $P(y \mid \mathcal{X}).$0. The accompanying interpretation is that bins that are too coarse lose temporal or similarity structure, while bins that are too fine make the task too hard and hurt convergence.

The paper further attributes DMAE’s performance to four factors: modeling multimodal interest at behavior granularity, dynamically adapting modality fusion, preserving multimodal information via distribution guidance, and retaining practical efficiency through sliding-window attention, training-only decoding, DIN-like online inference, and reduced storage by quantization in deployment. It also reports online A/B testing improvements for new-item exposure and overall business metrics: $P(y \mid \mathcal{X}).$1 $P(y \mid \mathcal{X}).$2, $P(y \mid \mathcal{X}).$3 $P(y \mid \mathcal{X}).$4, overall CTR $P(y \mid \mathcal{X}).$5, and revenue $P(y \mid \mathcal{X}).$6. A plausible implication is that behavior-level multimodal interest modeling is particularly beneficial when ID-only signals are sparse, such as in cold-start and new-item exposure scenarios.