Dual-Stream MLP for CTR

• Dual-Stream MLP (DS-MLP) is a CTR prediction framework that processes a shared embedding via two parallel MLP streams—one guided by a teacher to learn explicit feature interactions and one learning implicit interactions from scratch.
• The model fuses outputs from both streams through simple averaging after applying alignment techniques like BatchNorm to ensure scale consistency and harmonized representations.
• Empirical results on datasets such as Criteo, Avazu, and MovieLens highlight DS-MLP’s strong AUC performance with reduced computational complexity and lower risk of overfitting.

Searching arXiv for the target and closely related papers to ground the article in current literature. Dual-Stream MLP (DS-MLP) is a feature interaction framework for click-through rate (CTR) prediction in which a shared embedding vector is processed by two parallel multilayer perceptron streams: a main MLP that, through knowledge distillation, absorbs explicit high-order crossing behavior from a teacher model, and a parallel MLP trained from scratch to capture complementary implicit nonlinear interactions. The two branch logits are averaged and passed through a sigmoid to produce the CTR estimate, so the final deployed predictor remains a vanilla MLP-based dual-stream architecture rather than a hybrid cross-network or attention-based system (Ou et al., 3 Jun 2026).

1. Position within two-stream CTR modeling

CTR prediction is a fundamental task in online advertising and recommendation, and the immediate research backdrop for DS-MLP is the line of two-stream architectures that attempt to combine explicit and implicit feature interaction learning. In the formulation summarized here, prior work is characterized as designing dual-stream architectures to capture complex feature interactions from both explicit and implicit perspectives, but facing two recurrent difficulties: the high complexity of feature interaction learning, which increases computational demands and overfitting risk, and the imbalance between explicit and implicit modules, where one module’s output may dominate the final prediction (Ou et al., 3 Jun 2026).

An important precursor is FinalMLP, which reported that a well-tuned two-stream MLP that simply combines two MLPs can achieve surprisingly good performance, despite the prevailing emphasis on coupling an MLP with a dedicated explicit-interaction module such as those used in DeepFM or DCN-style systems. In FinalMLP, this observation led to an enhanced two-stream MLP design with feature gating and interaction aggregation layers (Mao et al., 2023). DS-MLP inherits the central premise that two MLP streams can constitute a strong CTR model, but reformulates the division of labor: one MLP is explicitly shaped by distillation from a teacher, while the other remains free to learn complementary implicit interactions.

A common misconception is that an “all-MLP” final model must therefore abandon explicit feature crossing. DS-MLP is explicitly designed to reject that premise. Its main MLP is not merely another generic dense network; it is trained so that the capacity for explicit feature interaction learning is consolidated into that branch through teacher-guided distillation (Ou et al., 3 Jun 2026).

2. Core architecture and prediction rule

Each raw instance is represented as

$x = \{x_1,\ldots,x_m\},$

a set of $m$ categorical or numerical fields. All fields are embedded through a single shared embedding table $E$, producing

$v = [e(x_1);\ldots;e(x_m)] \in \mathbb{R}^d.$

This same dense feature vector $v$ is fed in parallel to two MLP streams (Ou et al., 3 Jun 2026).

The first stream is the main MLP, also described as the student, whose role is to absorb explicit, high-order crossing behavior from a strong teacher. The second stream is a parallel MLP that is constructed identically to the main MLP but initialized and trained from scratch. The two streams output un-sigmoided logits $y_M$ and $y_P$, and the final click-through logit is fused by simple averaging:

$$\hat y = \sigma\bigl(\tfrac12 \cdot (y_M + y_P)\bigr).$$

This fusion rule is deliberately simple. Unlike architectures that place the explicit interaction mechanism in the deployed model, DS-MLP places the burden of explicit-cross induction largely in training, via distillation, rather than in the inference-time structure (Ou et al., 3 Jun 2026).

The main MLP is specified as a simple 3-layer feed-forward network. For layer $k$,

$$h^{(k)} = \sigma(W^{(k)} h^{(k-1)} + b^{(k)}), \qquad h^{(0)} = v.$$

The parallel branch follows the same structural template:

$m$0

A ReLU, or another nonlinear $m$1, is applied at each hidden layer. Because the parallel branch is not forced to match any explicit-cross teacher, it is described as freely learning rich implicit, nonlinear interactions (Ou et al., 3 Jun 2026).

3. Distilling explicit feature interactions into the main MLP

The defining mechanism of DS-MLP is knowledge distillation from a dual-stream teacher. The teacher logit is denoted $m$2, and one example given is a GDCN-like teacher:

$m$3

where $m$4 models explicit feature crosses via gated CrossNet layers and $m$5 is a standard deep network for implicit entry (Ou et al., 3 Jun 2026).

Distillation is formulated through a soft-target cross-entropy between teacher and student outputs. With temperature $m$6, the teacher and student logits are scaled as

$m$7

and the distillation loss is

$m$8

where

$m$9

Stage I training combines this distillation term with the ordinary CTR cross-entropy:

$E$0

with

$E$1

Here $E$2 controls the distillation weight (Ou et al., 3 Jun 2026).

The intended effect is not merely teacher imitation in a generic sense. The paper’s formulation is that DS-MLP leverages knowledge distillation to consolidate the capacity of learning explicit feature interaction into a main MLP network, while a parallel MLP simultaneously captures implicit feature interactions as a complement. This suggests a transfer of representational burden from structurally explicit modules into a compact dense network, with the teacher used only during training rather than deployment (Ou et al., 3 Jun 2026).

4. Alignment strategies and two-stage optimization

After Stage I, only the main MLP has been directly guided by the teacher. The architecture therefore introduces two alignment strategies during Stage II to improve compatibility between the two MLP components (Ou et al., 3 Jun 2026).

The first is hidden-state, or feature-level, alignment. A separate BatchNorm is applied to each MLP’s last hidden layer, $E$3 and $E$4. The stated purpose is to ensure that the two feature vectors have similar scales before they are linearly projected into final logits. In this design, BatchNorm is not merely a generic stabilization device; it is part of an explicit inter-branch harmonization mechanism.

The second is prediction-level alignment. Each branch is required to be a competent predictor on its own by adding branch-wise CTR losses:

$E$5

$E$6

Stage II then optimizes

$E$7

where $E$8 and $E$9 trade off the enforcement of each stream’s independent predictive power against preservation of complementarity (Ou et al., 3 Jun 2026).

Regularization comprises BatchNorm, standard weight decay, and early stopping on both MLPs, while the teacher’s parameters remain frozen throughout. The stated rationale is that the auxiliary alignment losses prevent one stream from collapsing into an over-specialized predictor, and that BatchNorm constrains numeric ranges. Within the paper’s interpretation, these mechanisms jointly address the imbalance problem typical of heterogeneous two-stream systems (Ou et al., 3 Jun 2026).

5. Computational profile, scalability, and representation

DS-MLP is presented as a simplification of the computational pathway for CTR interaction modeling. An MLP layer of width $v = [e(x_1);\ldots;e(x_m)] \in \mathbb{R}^d.$0 is described as costing only $v = [e(x_1);\ldots;e(x_m)] \in \mathbb{R}^d.$1 per example, with highly optimized dense kernels on modern hardware. The model uses only two 3-layer MLPs and thereby avoids the $v = [e(x_1);\ldots;e(x_m)] \in \mathbb{R}^d.$2 pairwise products or $v = [e(x_1);\ldots;e(x_m)] \in \mathbb{R}^d.$3 cross layers found in typical dual-stream models (Ou et al., 3 Jun 2026).

The paper associates this simplification with lower overfitting risk. By distilling explicit crossing into a smaller 3-layer MLP, DS-MLP keeps its parameter count low; BatchNorm and the auxiliary parallel stream are described as serving as regularizers. Empirically, the system is reported to yield strong generalization even on large, sparse CTR data (Ou et al., 3 Jun 2026).

The experimental characterization further emphasizes scalability. Figure 1 is described as plotting AUC versus parameter count for DS-MLP and GDCN, with DS-MLP performance rising smoothly as width grows whereas GDCN saturates or overfits. Figure 2 is described as confirming that the distilled main MLP approximates the teacher’s explicit CrossNet with very low MSE, while the parallel MLP remains orthogonal (Ou et al., 3 Jun 2026). A plausible implication is that DS-MLP is not simply compressing a teacher into a single branch; it is using the second branch to preserve residual modeling capacity that is not collapsed into the distilled representation.

6. Empirical results and relation to FinalMLP

On the reported benchmark suite, DS-MLP is evaluated on Criteo, Avazu, and MovieLens. The paper reports the following results (Ou et al., 3 Jun 2026):

Dataset	AUC	LogLoss
Criteo	0.8152	0.4366
Avazu	0.7670	0.3657
MovieLens	0.9752	0.1971

The same table also reports RelaImpr values of 3.21% on Criteo, 2.73% on Avazu, and 6.02% on MovieLens. The summary states that DS-MLP outperforms or matches every prior explicit, implicit or two-stream model, at comparable inference latency of approximately 60 ms (Ou et al., 3 Jun 2026).

The ablation study attributes measurable performance to each major design choice. Removing alignment losses $v = [e(x_1);\ldots;e(x_m)] \in \mathbb{R}^d.$4 causes an approximately 0.002–0.003 drop in AUC; removing the parallel MLP loses approximately 0.003 in AUC; removing knowledge distillation altogether drops approximately 0.001–0.002 in AUC; and replacing the dual-MLP backbone with a CrossNet without MLP underperforms. Table 11 further reports that DS-MLP can digest many different teachers—AutoInt, DCNv2, xDeepFM, GDCN, and FinalMLP—and always improves over the bare teacher, which is presented as evidence of a teacher-agnostic design (Ou et al., 3 Jun 2026).

The relation to FinalMLP is especially instructive. FinalMLP starts from a base DualMLP in which the same embedding vector $v = [e(x_1);\ldots;e(x_m)] \in \mathbb{R}^d.$5 is fed to two independent MLP streams, and then adds two plug-in enhancements: a feature gating layer, which produces stream-specific soft weights $v = [e(x_1);\ldots;e(x_m)] \in \mathbb{R}^d.$6 to re-weight $v = [e(x_1);\ldots;e(x_m)] \in \mathbb{R}^d.$7 and differentiate the two inputs, and an interaction aggregation layer, which replaces simple summation or concatenation with full bilinear fusion or multi-head bilinear fusion (Mao et al., 2023). FinalMLP reports that this design achieves better performance than many sophisticated two-stream CTR models on four open benchmark datasets and in an online A/B test, with specific AUC gains over DualMLP and models such as DCN V2, xDeepFM, and EDCN (Mao et al., 2023).

The comparison can be summarized as follows:

Aspect	FinalMLP	DS-MLP
How the two streams differ	Feature gating produces stream-specific re-weighted inputs	Main MLP is distilled from a teacher; parallel MLP is trained from scratch
How stream outputs are fused	Full bilinear fusion or multi-head bilinear fusion	Average of branch logits followed by sigmoid
Optimization emphasis	Binary cross-entropy with $v = [e(x_1);\ldots;e(x_m)] \in \mathbb{R}^d.$8 regularization and dropout	Stage I distillation, Stage II alignment, BN, weight decay, early stopping

Taken together, these two papers support a broader reinterpretation of dual-stream CTR modeling. FinalMLP shows that two simple MLPs, when differentiated at the input and fused through stream-level interactions, form a strong CTR model; DS-MLP shows that two simple MLPs can also be made competitive by moving explicit interaction learning into the training procedure through knowledge distillation and then aligning the branches during fine-tuning (Mao et al., 2023). In both cases, the conventional assumption that strong two-stream CTR models require heterogeneous inference-time modules is directly challenged. DS-MLP’s code is reported as available at https://github.com/RUCAIBox/DS-MLP, while FinalMLP’s source code is reported as available at MindSpore/models (Ou et al., 3 Jun 2026).