Dual-Stream MLP is All You Need for CTR Prediction

Abstract: Click-through rate (CTR) prediction holds a pivotal role in online advertising and recommendation systems, where even small improvements can significantly boost revenue. Existing research primarily focuses on designing dual-stream architectures to capture effective complex feature interactions from both explicit and implicit perspectives. However, these approaches are faced with two major challenges: 1) the high complexity of feature interaction learning, which increases computational demands and the overfitting risk, and 2) the imbalance between explicit and implicit modules, where one module's output may dominate the final prediction. To address these issues, in this paper, we propose Dual-Stream MLP (DS-MLP), a novel feature interaction framework for the CTR prediction task. Specially, it leverages knowledge distillation to consolidate the capacity of learning explicit feature interaction into a main MLP network, while a parallel MLP simultaneously captures implicit feature interactions as a complement. To effectively optimize the dual-stream MLP architecture, we further design a specific learning approach with two alignment strategies for enhancing the compatibility of the two MLP components. Experiments demonstrate that DS-MLP, though merely a vanilla MLP structure (the final model), can achieve state-of-the-art performance across three widely used benchmarks, offering a scalable and efficient solution for large-scale recommendation systems. Our code is available at https://github.com/RUCAIBox/DS-MLP.

• The paper introduces a DS-MLP model that leverages knowledge distillation and dual-stream alignment to integrate explicit and implicit feature interactions for CTR prediction.
• Empirical results on benchmarks like Criteo, Avazu, and Movielens demonstrate that DS-MLP achieves superior AUC and LogLoss performance with low inference latency.
• Theoretical analysis validates that the distilled student MLP internalizes the teacher’s explicit interaction patterns, ensuring robustness, scalability, and effective correction of teacher bias.

Dual-Stream MLP for CTR Prediction via Knowledge Distillation and Alignment

Introduction

Click-through rate (CTR) prediction models have evolved from simple explicit interaction mechanisms such as factorization machines, to complex hybrid architectures featuring high-order feature crossing and deep neural embedding. However, contemporary two-stream architectures that coalesce explicit and implicit interaction modules suffer from compounded architectural complexity and an imbalance where the explicit module tends to dominate the implicit one in the final predictive aggregation. The paper "Dual-Stream MLP is All You Need for CTR Prediction" (2606.04944) proposes a dual-stream MLP (DS-MLP) architecture that overcomes these challenges by employing knowledge distillation (KD) and dual-MLP alignment, yielding a lightweight MLP model that matches or surpasses the performance of state-of-the-art (SOTA) sophisticated CTR architectures.

Dual-Stream MLP Design and Optimization Procedure

The central design of DS-MLP is outlined in (Figure 1). The entire pipeline is partitioned into two principal stages: knowledge distillation from high-capacity teacher models and fine-tuned dual-MLP alignment.

Figure 1: The dual-stream MLP architecture of DS-MLP, with explicit (main) and implicit (parallel) branches trained via knowledge distillation and fine-tuning/aligning.

In the first stage, a parameter-efficient student MLP (main MLP; three layers) absorbs explicit interaction knowledge from a powerful dual-stream teacher (e.g., GDCN or DCNv2) that combines feature crossing and deep implicit fusion. KD minimizes the cross-entropy between logits (teacher and student) and leverages temperature-scaling for regularization.

Due to the magnitude dominance of the teacher’s explicit stream, the distilled student regresses primarily to explicit interactions, insufficiently modeling implicit dependencies. Consequently, a parallel MLP is introduced as an auxiliary implicit feature learner. The two streams are additively aggregated after alignment. To mitigate output scale mismatch and encourage orthogonal learning, DS-MLP adopts:

• Hidden state alignment with batch normalization,
• Prediction alignment via independent binary cross-entropy (BCE) supervision for each branch.

These strategies enable robust balancing and synergy between explicit and implicit feature interaction modeling.

Theoretical Analysis

Theoretical justification is provided for the distillation of explicit interaction capacity into a plain MLP. By leveraging the universal approximation property of ReLU-based MLPs and advances in Sobolev-space–based generalization bounds, the authors establish that the distilled student not only approximates the teacher's predictive mapping in the $L^2$ sense but also reconstructs its higher-order interaction structure reflected in Hessian (second derivative) alignment. The alignment in interaction sensitivity metrics:

$$I_{ij}(f) = \mathbb{E}_\mathcal{D}\left[ \left| \frac{\partial^2 f}{\partial x_i \partial x_j} \right| \right]$$

implies that the student MLP genuinely internalizes explicit crossing patterns originally captured via specialized modules in the teacher network. Thus, the efficacy of distilling dual-stream cross-network models into lightweight MLPs is not merely empirical but admits rigorous analytical support.

Empirical Results

Overall Performance

DS-MLP is exhaustively evaluated on three canonical CTR benchmarks: Criteo, Avazu, and Movielens. Across all metrics (AUC, LogLoss, RelaImpr), DS-MLP exhibits consistent, statistically significant performance improvements over prior SOTA methods, including GDCN, DCNv2, FinalMLP, and ensemble KD approaches. For example, on Criteo, DS-MLP achieves 0.8152 AUC and 0.4366 LogLoss—superior to all explicit, implicit, and hybrid baselines. Importantly, DS-MLP achieves these results with low inference latency, confirming its practical utility for large-scale industrial deployments.

Scalability Analysis

Figure 2: The scaling effect for explicit-interaction teacher models versus dual-stream student MLP under progressive capacity scaling.

DS-MLP maintains monotonic improvements with increased MLP capacity, whereas cross network models saturate and ultimately degrade, indicating enhanced robustness and reduced overfitting risk for MLPs with explicit-implicit alignment compared to specialized interaction modules.

Ablation and Interpretability Studies

Ablation demonstrates that both streams and their alignment mechanisms are indispensable: eliminating the alignment loss, the parallel MLP, KD, or switching the dual-MLP for CrossNet each results in marked accuracy drops. Fidelity analysis (via MSE on synthetic polynomials) confirms the student MLP not only mimics output but closely matches the teacher’s explicit interaction logic, whereas the parallel MLP specializes in complementary implicit patterns.

Figure 3: MSE comparison per stream/component in capturing explicit interactions, demonstrating the main MLP’s faithful distillation and the parallel MLP’s divergence (implicit focus).

Correction of Teacher Bias

A case study on difficult instances highlights that errors inherited from the teacher during KD are subsequently corrected by the parallel MLP after alignment, reducing the risk of overfitting to teacher bias and improving robustness on out-of-distribution or ambiguous test cases.

Figure 4: Case study visualizing prediction correction for outlier samples by the parallel MLP in the dual-stream DS-MLP.

Hyperparameter Robustness

DS-MLP is shown to be robust to wide ranges of loss weights (KD/align coefficients). However, there are identifiable "sweet spots" for these weights that optimize the balance between teacher mimicry and stream complementarity.

Figure 5: Sensitivity of DS-MLP performance to knowledge distillation (λ) and alignment (α) loss coefficients.

Practical and Theoretical Implications

The proposed approach demonstrates that highly parameterized explicit-interaction modules are not necessary in the final deployed model. With proper teacher selection (any SOTA dual-stream teacher can be utilized), DS-MLP can inherit arbitrary feature combination capacity into a plain MLP, yielding a teacher-agnostic, hardware-optimized, and highly scalable architecture. This paradigm is theoretically grounded, outperforming complex architectures even under scaling.

Practically, this unlocks straightforward model simplification without sacrificing accuracy—a compelling asset for real-world recommender platforms where latency, reproducibility, and maintainability are paramount. From a methodological perspective, this challenges the necessity of increasingly intricate cross architectures, potentially redirecting future research into MLP-centric distillation and alignment frameworks and extension to multi-stream (e.g., mixture-of-experts) paradigms.

Conclusion

"Dual-Stream MLP is All You Need for CTR Prediction" establishes a robust blueprint for unifying explicit and implicit feature interaction learning within a dual-stream, knowledge-distilled MLP framework. Through careful architectural and loss alignment design, DS-MLP achieves both theoretical fidelity to complex cross interaction patterns and practical benefits in accuracy, efficiency, and scalability. The work provides a strong argument for relegating specialized explicit interaction modules to the teacher phase, allowing the deployment of universal MLPs that retain SOTA predictive performance and offer seamless extensibility for future advances in scalable recommendation systems.