Deep Multimodal Group Interest Network (DMGIN)

• The paper introduces DMGIN, which integrates multimodal representation learning with group-based sequence compression to enhance CTR prediction.
• It employs a dual-tower network, clustering, intra-group transformers, and temporal evolution transformers to capture long-term user behavior dynamics.
• The model achieves efficient industrial-scale performance by reducing computational overhead while improving key metrics like AUC, GAUC, CTR, and RPM.

The Deep Multimodal Group Interest Network (DMGIN) is an end-to-end architecture for recommendation systems that addresses the dual challenges of leveraging multimodal user interaction data and modeling long post-click behavior sequences. By combining multimodal representation learning, behavioral grouping strategies informed by Multimodal LLMs (MLLMs), intra- and inter-group temporal modeling, and candidate-aware attention mechanisms, DMGIN achieves efficient, accurate, and scalable user interest modeling suitable for industrial-scale CTR prediction and recommendation applications (Wei et al., 29 Aug 2025).

1. Architecture and Core Modules

DMGIN comprises several sequential modules, each targeting a distinct aspect of multimodal and temporal user behavior representation:

• Cross-Modal Representation Learning Module (CMRLM): Trains a CLIP-style dual-tower model to align multiple modalities (e.g., shop name, image, food description) in a shared latent space, producing robust joint embeddings for shopping entities.
• Interest-Driven Entity Clustering Module (IDECM): Applies K-means clustering to the multimodal embeddings, transforming long user behavior sequences (potentially $O(10^4)$ events) into much shorter sequences of interest groups (typically $O(10^2)$), thus dramatically reducing sequence length and computational complexity.
• Intra-Group Interest Enhancement Module (IGIEM): Enhances group-level signals using two strategies:
  • Interest Statistics: Aggregates counts of behavior types, timestamp statistics, and monetary features per group, providing a static profile.
  • Intra-Group Transformers: Models dynamic evolution within each group using multi-head self-attention (MHSA) over behavior-level concatenated embeddings (e.g., timestamps, locations, behavior types), with output representations refined by mean pooling.
• Temporal Group Evolution Transformer (TGET): Utilizes hierarchical sequential transduction units (HSTU) to model temporal dependencies and evolution across ordered interest groups, maintaining the integrity of long-range sequential patterns essential for lifelong behavior modeling.
• Candidate-Aware Group Attention Module (CAGAM): Computes attention scores between the candidate item and each group embedding using similarity in the shared latent space, enabling the model to selectively focus on the most relevant interest groups for CTR prediction.

2. Grouping Strategy and Multimodal Representation

A central feature of DMGIN is its group-based compression of behavior sequences, which mitigates redundancy caused by repeated interactions on the same or related entities and maintains the critical multimodal semantic context.

• Grouping via MLLMs: Multimodal embeddings from CMRLM are clustered so that post-click records on visually and semantically similar entities are assigned to the same group. Human-in-the-loop procedures, such as t-SNE visualizations and cluster balancing, validate and maintain group cohesion.
• Statistical and Dynamic Features: Within each group, summary statistics (stat_counts, stat_max_time, stat_avg_price) provide static interest measures, while intra-group transformers encode dynamic, fine-grained behavioral signals. For group $s$, the aggregated feature is:

$$\text{stat}_s = [\text{stat\_counts}, \text{stat\_max\_time}, ..., \text{stat\_avg\_price}]$$

Dynamic evolution is captured by:

$\text{dyna}_s = \text{meanpool}(\text{MHSA}(e_b))$

where $e_b$ is the sequence of concatenated behavior attribute embeddings.

This grouping approach enables substantial computational savings when processing lifelong sequences, with almost no additional overhead compared to directly inputting multimodal embeddings (Wei et al., 29 Aug 2025). Grouping is not limited to identity attributes; other semantic keys are feasible if supported by downstream tasks.

3. Temporal Modeling across Groups

Temporal inter-group modeling is conducted via the TGET module, which processes temporally sorted group representations to capture the evolution of higher-level interests.

• HSTU Operations: Each transformer layer splits the group representation into multiple projections (queries, keys, values, gating signals), applies multi-head attention to model interdependencies, and fuses information with gating and normalization:

$$U(G), V(G), Q(G), K(G) = \text{Split}(\phi_1(f_1(G)))$$

$$\mathcal{A}(G)V(G) = \phi_2(Q(G)K(G) + \text{bias}^p) V(G)$$

$$Y(G) = f_2(\text{Norm}(\mathcal{A}(G)V(G)) \odot U(G))$$

This bidimensional sequence processing preserves global sequential context and supports effective long-term interest modeling.

4. Candidate-Aware Attention and CTR Prediction

Recommendation output is produced by computing attention between the candidate (target item) and each evolved group embedding:

$$r_{\text{long}} = \frac{\exp(\text{sim}(W_t e, W_g g'_s))}{\sum_{s'} \exp(\text{sim}(W_t e, W_g g'_{s'}))}$$

where $e$ is the candidate item's embedding, $g'_s$ is the temporal-evolved representation of group $s$, and $W_t$, $W_g$ are learned projections. This mechanism allows the network to emphasize groups with the most predictive value specific to the candidate context.

5. Computational and Implementation Considerations

• Scalability: DMGIN is designed to process industrial-scale user histories, leveraging group compression (reducing sequence length from $O(10^4)$ to $O(10^2)$), which enables efficient data-parallel computation and substantially lowers memory and computational requirements.
• System Overhead: Grouping incurs minimal computational cost, as CMRLM embeddings are computed once per entity, not per behavior. Inference latency (7–8 ms) and model size (4.0 GB) were observed to be modest considering the scale (billions of interactions, two years of user histories).
• Training and Deployment: Both industrial datasets and public benchmarks (e.g., Amazon Grocery) were employed for model evaluation. Offline metrics included AUC and Group AUC (GAUC), while online metrics used in production include CTR and Revenue per Mille (RPM). An A/B test in an LBS advertising system demonstrated $4.7\%$ CTR and $2.3\%$ RPM improvements against business baselines (Wei et al., 29 Aug 2025).

Component	Function	Technical Mechanism
CMRLM	Cross-modal embedding	Dual-tower CLIP-like network
IDECM	Interest group clustering	K-means over multimodal vectors
IGIEM	Group-level interest encoding	Statistics + intra-group MHSA
TGET	Temporal evolution of group interests	HSTU (transformer) blocks
CAGAM	Candidate-focused group weighting for prediction	Softmax attention

6. Comparative Performance and Empirical Validation

DMGIN outperforms previous state-of-the-art baselines such as DIN, DIEN, TWIN, and DSIN Full on both public and industrial datasets, achieving higher AUC/GAUC and real-world business metric improvements:

• Offline: Grouping and dynamic intra-group modeling both yield measurable performance gains in ablation studies, with the dynamic branch cited as particularly crucial to overall accuracy.
• Online: In production testing, increased inference latency and model size were outweighed by substantial commercial benefits in CTR and revenue.
• Interpretability: The explicit grouping, statistical summarization, and temporal modeling yield interpretable intermediate representations, aiding model introspection and diagnosis.

7. Significance and Implications

DMGIN introduces a scalable approach for lifelong, multimodal interest modeling suitable for large-scale recommendation scenarios. Its architecture demonstrates how multimodal LLMs and self-attention mechanisms can be coordinated to compress, refine, and leverage heterogeneous behavioral signals without sacrificing predictive power or efficiency (Wei et al., 29 Aug 2025).

This approach addresses key limitations of prior models that either ignored long-range dependencies, processed multimodal features at excessive computational cost, or failed to model group dynamics. The combination of multimodal clustering and hierarchical transformer-based sequence modeling sets a technical precedent for advancing recommender systems capable of handling vast, heterogeneous interaction logs with high fidelity. The modularity of the framework also suggests extensibility to additional modalities, context features, or adaptation for group-level recommendation and interest detection in both social and industrial contexts.