Dual-MLP Models: Complementary MLP Designs

• Dual-MLP models are a family of architectures that pair complementary MLPs to divide representational labor across different axes such as tokens, channels, and modalities.
• They are applied in diverse areas—from video recognition to sequential recommendation and CTR prediction—by using tailored mixers and gating techniques to manage distinct data interactions.
• These models leverage specialized training strategies and efficient optimization methods to balance performance gains with reduced computational costs.

Searching arXiv for recent and foundational papers relevant to Dual-MLP models. Dual-MLP models are a family of architectures in which two complementary multilayer perceptrons are paired to divide representational labor across axes, temporal scales, modalities, or branches. In the cited literature, this pairing appears in several distinct forms: token-mixing MLP plus channel MLP for video and vision, global plus local causal token mixers for sequence modeling, dual-stream MLPs for click-through rate prediction, dual permutation-equivariant MLPs aggregated by dot-product for set encoding, and dual-branch MLPs for multimodal hyperspectral reconstruction (Qiu et al., 2022, Jiang et al., 2023, Mao et al., 2023, Fei et al., 2022, Ou et al., 3 Jun 2026). The term therefore does not denote a single canonical block, but a recurring architectural pattern in which two MLP components are deliberately specialized and then composed.

1. Conceptual definition and recurring design pattern

A recurrent property of Dual-MLP models is complementarity rather than duplication. In MLP-3D, each block contains one MLP applied across tokens and one MLP applied independently to each token; in TriMLP, the pair is a causal global mixer followed by a causal local mixer; in FinalMLP and DS-MLP, the pair is realized as two CTR streams with different functional roles; and in DuMLP-Pin, two permutation-equivariant MLPs are combined by a permutation-invariant dot-product aggregation (Qiu et al., 2022, Jiang et al., 2023, Mao et al., 2023, Fei et al., 2022, Ou et al., 3 Jun 2026).

The block-level archetype is explicit in MLP-3D. With pre-normalization and residual connections, the block computes

$$Y = X + \mathrm{TokenMLP}(\mathrm{LN}(X)), \qquad Z = Y + \mathrm{ChannelMLP}(\mathrm{LN}(Y)),$$

and the token mixer itself is decomposed as

$\hat Y = \mathrm{FC}(X_H + X_W + X_T),$

where $X_H$ and $X_W$ are spatial mixers and $X_T$ is a temporal mixer implemented by Grouped Time Mixing (GTM) (Qiu et al., 2022). This is a strict two-MLP decomposition: one MLP mixes across positions, the other across channels.

In sequential recommendation, TriMLP replaces the token/channel split with a long-range/short-range split. Its Triangular Mixer alternates a global masked MLP and a local masked MLP in series, enforcing strict chronological causality by upper-triangular masking. The recurrence is

$X_2 = \mathrm{MixL}(\mathrm{MixG}(X_1)),$

with no residual connection between the two mixers in the default implementation (Jiang et al., 2023).

In set modeling, DuMLP-Pin uses a different duality. Two permutation-equivariant MLPs $\Phi$ and $\Psi$ are applied row-wise to set elements, producing $U=\Phi(X)$ and $V=\Psi(X)$, after which the invariant representation is

$\hat Y = \mathrm{FC}(X_H + X_W + X_T),$0

followed by a readout $\hat Y = \mathrm{FC}(X_H + X_W + X_T),$1 (Fei et al., 2022). Here the pair is not stream fusion in the CTR sense, but dual equivariant feature maps whose interaction yields a global invariant descriptor.

This suggests that “Dual-MLP” is best understood as an architectural schema defined by functional complementarity: two MLP components are retained as distinct computational objects because they are intended to learn different classes of interactions.

2. Token–channel Dual-MLP models in video and vision

MLP-3D is a canonical video instantiation of the token/channel Dual-MLP pattern. A video clip of size $\hat Y = \mathrm{FC}(X_H + X_W + X_T),$2 is tokenized into tubelets with window $\hat Y = \mathrm{FC}(X_H + X_W + X_T),$3 and stride $\hat Y = \mathrm{FC}(X_H + X_W + X_T),$4, producing

$\hat Y = \mathrm{FC}(X_H + X_W + X_T),$5

or equivalently $\hat Y = \mathrm{FC}(X_H + X_W + X_T),$6 with $\hat Y = \mathrm{FC}(X_H + X_W + X_T),$7 and $\hat Y = \mathrm{FC}(X_H + X_W + X_T),$8 (Qiu et al., 2022). The network uses a four-stage hierarchical design in which spatial resolution is reduced and channel width increased, while temporal resolution stays $\hat Y = \mathrm{FC}(X_H + X_W + X_T),$9 until global average pooling.

Its principal technical contribution is GTM, which equips the token-mixing MLP with temporal modeling while avoiding the prohibitive complexity of full temporal mixing. Full time mixing has time complexity $X_H$0 and parameter count $X_H$1, whereas GTM with group size $X_H$2 reduces this to $X_H$3 with parameters $X_H$4, or $X_H$5 with relative-offset sharing; shift-token GTM uses $X_H$6 parameters (Qiu et al., 2022). Several GTM variants are defined: short-range GTM using contiguous temporal groups, long-range GTM using strided grouping, shift-window GTM using overlapping windows, and shift-token GTM using circular temporal shifts concatenated along channels.

The model composes these GTM variants by a two-step greedy architecture search. A supernet is first pre-trained by randomly assigning GTM type and group size per block with shared parameters; a blockwise greedy selection then maximizes

$X_H$7

repeating the pass three times for stability (Qiu et al., 2022). Under ImageNet-1K pretraining, MLP-3D-L achieves 68.5% top-1 on Something-Something V2 and 81.4% top-1 on Kinetics-400; on SS-V1 it reaches up to 56.5% top-1. Representative inference costs are 60/108/183/336 GFLOPs on SS-V2 single-view inference and 57/102/170/308 GFLOPs on Kinetics-400 multi-view inference for XS/S/M/L variants (Qiu et al., 2022).

DynaMixer occupies a related position in vision. It follows the same token-mixing/channel-mixing decomposition but replaces static token mixing with content-adaptive dynamic mixing matrices generated from the tokens themselves. The block produces row mixing, column mixing, and channel mixing, aggregated as

$X_H$8

and uses dimensionality reduction together with multi-segment fusion to keep dynamic mixing efficient (Wang et al., 2022). On ImageNet-1K, DynaMixer-S/M/L achieve 82.7%, 83.7%, and 84.3% top-1 accuracy with 26M, 57M, and 97M parameters, respectively (Wang et al., 2022).

The significance of these models is architectural: they show that the token/channel Dual-MLP factorization can be extended from image classification to spatiotemporal recognition and can accommodate either static or dynamic token mixing. In MLP-3D this extension is temporal and grouped; in DynaMixer it is content-adaptive and axis-wise.

3. Causal Dual-MLP formulations for sequential modeling

TriMLP adapts the Dual-MLP idea to auto-regressive sequence modeling by pairing two masked token mixers rather than a token mixer and a channel mixer. Its core observation is that unrestricted MLP token mixing leaks future information. The remedy is a triangular mask that drops the lower-triangular neurons of the mixing kernel, so each position aggregates only from itself and its past (Jiang et al., 2023).

For an input sequence representation $X_H$9, the global mixer uses an upper-triangular masked kernel $X_W$0 and normalized scores $X_W$1 to compute

$X_W$2

The local mixer applies another masked MLP, but with an additional block-diagonal session mask $X_W$3, yielding

$X_W$4

where only tokens in the same session and satisfying $X_W$5 may contribute (Jiang et al., 2023). The final mixer output is $X_W$6.

The two mixers are serial, not parallel. The default macro-architecture is Embedding $X_W$7 Triangular Mixer $X_W$8 Classifier, with two MLP mixing layers total (Jiang et al., 2023). Softmax over tokens is used by default; positional embeddings and FFN are not used by default; a variant with FFN, pre-layer normalization, and residuals brings limited gains in most datasets and increases parameter scale (Jiang et al., 2023).

The model is evaluated on 12 datasets from Amazon, MovieLens, Tenrec, and LBSN, covering 50K–10M interactions. Its asymptotic complexity is $X_W$9, the same order as self-attention in sequence length and dimension, but without query/key/value projections and without FFN in the default setting (Jiang et al., 2023). On ML-10M with $X_T$0, $X_T$1, and batch size 512, TriMLP uses 2.15G MACs, 0.03M parameters, 1391MB GPU memory, and 0.8946s inference time, compared with 13.04G/0.19M/1810MB/1.1730s for GRU4Rec and 25.94G/0.40M/2503MB/2.5037s for SASRec (Jiang et al., 2023).

Empirically, the abstracted average improvement is up to 14.88% with 8.65% less inference cost. Ablations isolate the necessity of the dual masked design: unmasked SqrMLP degrades performance by −77.99% on average across 12 datasets, while TriMLP_G and TriMLP_L each improve over the identity baseline but the full MixG $X_T$2 MixL composition is consistently best (Jiang et al., 2023). A common misconception is that a single global causal mixer should suffice; the reported ablations instead show that separating long-range and short-range causal mixing is materially beneficial.

4. Dual-stream MLPs in CTR prediction

In CTR prediction, Dual-MLP models are implemented as two-stream architectures. FinalMLP begins from the empirical observation that a well-tuned two-stream model composed only of two MLPs can match or surpass many two-stream designs that pair an MLP with an explicit interaction module (Mao et al., 2023). Its base DualMLP computes

$X_T$3

and then augments this baseline with stream-specific feature gating and stream-level bilinear interaction aggregation.

The feature gating mechanism differentiates the two streams at the input:

$X_T$4

where $X_T$5 and $X_T$6 are produced by MLP-based gates (Mao et al., 2023). Fusion then uses bilinear interaction aggregation,

$X_T$7

with a multi-head factorization reducing the parameter and FLOP cost from $X_T$8 to $X_T$9 (Mao et al., 2023). On public benchmarks, FinalMLP reaches AUC 81.49 on Criteo, 76.66 on Avazu, 97.20 on MovieLens, and 98.61 on Frappe; in an online news recommendation system it records 72.93 AUC with 47 ms latency in the 8-head variant and a +1.6% average CTR lift over 5 days (Mao et al., 2023).

DS-MLP takes a different dual-stream route. It targets two stated problems in dual-stream CTR models: high-complexity explicit interaction learning and imbalance between explicit and implicit modules. Its solution is a main MLP $X_2 = \mathrm{MixL}(\mathrm{MixG}(X_1)),$0 distilled from a teacher and a parallel MLP $X_2 = \mathrm{MixL}(\mathrm{MixG}(X_1)),$1 that captures complementary implicit interactions (Ou et al., 3 Jun 2026). The two logits are fused by a fixed weighted sum,

$X_2 = \mathrm{MixL}(\mathrm{MixG}(X_1)),$2

with $X_2 = \mathrm{MixL}(\mathrm{MixG}(X_1)),$3 by default (Ou et al., 3 Jun 2026).

Training is two-stage. In stage I, the main stream is optimized by CTR loss plus knowledge-distillation loss from a teacher such as GDCN:

$X_2 = \mathrm{MixL}(\mathrm{MixG}(X_1)),$4

In stage II, both streams are jointly fine-tuned with hidden-state alignment via batch normalization and branch-wise BCE alignment,

$X_2 = \mathrm{MixL}(\mathrm{MixG}(X_1)),$5

The deployed model drops the teacher and retains only the two MLP streams with BN (Ou et al., 3 Jun 2026).

Across Criteo, Avazu, and MovieLens, DS-MLP reports 0.8152 / 0.4366, 0.7670 / 0.3657, and 0.9752 / 0.1971 in AUC / LogLoss, respectively, with inference latencies of 59.43 ms, 25.89 ms, and 2.65 ms (Ou et al., 3 Jun 2026). The ablations are structurally informative: removing alignment loss, removing the student MLP, removing the parallel MLP, or removing KD all degrades performance; replacing the dual MLP with CrossNet also underperforms (Ou et al., 3 Jun 2026). In this line of work, Dual-MLP means not merely two MLPs in parallel, but two MLPs with explicitly separated responsibilities and a training scheme designed to keep their outputs compatible.

5. Set encoding, multimodal reconstruction, and other domain-specific realizations

DuMLP-Pin shows that Dual-MLP design is not confined to token mixing or dual-stream fusion. For a set $X_2 = \mathrm{MixL}(\mathrm{MixG}(X_1)),$6, it applies two permutation-equivariant MLPs $X_2 = \mathrm{MixL}(\mathrm{MixG}(X_1)),$7 and $X_2 = \mathrm{MixL}(\mathrm{MixG}(X_1)),$8 elementwise and aggregates by

$X_2 = \mathrm{MixL}(\mathrm{MixG}(X_1)),$9

after which a readout head maps $\Phi$0 to the target (Fei et al., 2022). The paper proves that any permutation-invariant function implemented in this way can be decomposed into two or more permutation-equivariant ones in a dot-product way as the cardinality of the input set exceeds a threshold; specifically, for $\Phi$1, such a decomposition exists if and only if $\Phi$2 (Fei et al., 2022). Under certain conditions, the architecture can also be viewed as Deep Sets with strong low-rank constraints.

The empirical profile is unusually strong for a global aggregation model. DuMLP-Pin achieves 0.48% $\Phi$3 0.034 error on MNIST pixel sets in the large variant, 66.8% $\Phi$4 0.43 OA on CelebA attribute set anomaly detection, 92.3% $\Phi$5 0.12 OA on ModelNet40 with 0.29M parameters, and 84.8% $\Phi$6 0.04 mIoU on ShapeNetPart with 0.98M parameters (Fei et al., 2022). On ModelNet40 its MACs per sample are 19M, lower than PointNet at 440M, Set Transformer at 1284M, and Point Transformer at approximately 18,314M (Fei et al., 2022). The principal limitation remains the absence of explicit local-neighborhood modeling.

MLP-AMDC provides a multimodal, physics-informed Dual-MLP realization for adaptive-mask dual-camera snapshot hyperspectral imaging. A 50/50 beamsplitter yields an RGB path and a CASSI path; the RGB measurement

$\Phi$7

is fed to an AdaptiveMaskNet to generate a scene-adaptive mask $\Phi$8, while the coded and dispersed CASSI path yields the 2D measurement

$\Phi$9

(Cai et al., 2023). Reconstruction uses a dual-stream MLP architecture with a CASSI residual branch and an RGB branch, updated by

$\Psi$0

The network uses a SpectralSpatial MLP block for channel mixing over the spectral dimension and a Swin-Spatial-MLP block for windowed spatial mixing, without self-attention (Cai et al., 2023).

On KAIST, AMDC-9stg reaches 48.61 dB PSNR and 99.6% SSIM; on ARAD_1K, AMDC-9stg (MLP) reports MRAE 0.0420, RMSE 0.0044, PSNR 47.99 dB, and SSIM 99.35% (Cai et al., 2023). The MLP variants are also faster than transformer counterparts at matched stage counts; for example, AMDC-3stg records 70.4 FPS versus 45.8 FPS for the transformer version, and AMDC-9stg records 14.12 FPS versus 11.1 FPS (Cai et al., 2023). This domain makes the duality modality-specific rather than axis-specific.

Several adjacent architectures explicitly position themselves relative to the token/channel Dual-MLP decomposition while extending it beyond strict two-way form. X-MLP is patch-embedding-free and replaces a single token axis with width-cross, height-cross, and channel-cross MLPs, thereby turning token/channel duality into multi-axis mixing (Wang et al., 2023). SplitMixer likewise preserves the separation of spatial mixing and channel mixing, but instantiates both paths with efficient convolutions rather than dense MLPs; it is explicitly described as a convolutional instantiation of the dual-MLP idea (Borji et al., 2022). These works indicate that the dual-MLP principle can be generalized, factorized, or convolutionalized without abandoning the underlying division of labor.

6. Efficiency, optimization strategies, and limitations

A central motivation for Dual-MLP models is efficiency. MLP-3D replaces full temporal mixing with GTM to reduce temporal complexity from $\Psi$1 to $\Psi$2 and to keep parameters bounded by $\Psi$3 or $\Psi$4 with offset sharing (Qiu et al., 2022). TriMLP removes query–key–value projections and FFN from the default path while keeping parallel masked matrix multiplications; across 12 datasets it reduces inference time by an average 8.64–8.65% against the fastest baseline, with reductions growing from 2.08% on Tiny datasets to 20.72% on Large ones (Jiang et al., 2023). DS-MLP pushes explicit interaction modeling into an offline teacher and serves only MLP streams, which is why the final deployed model is described as “merely a vanilla MLP structure” (Ou et al., 3 Jun 2026).

Optimization schemes are correspondingly diverse. MLP-3D uses ImageNet-1K CycleMLP initialization, AdamW with base learning rate $\Psi$5, weight decay 0.05, cosine decay over 32 epochs, RandAugment, Random Erasing, DropPath, and GTM-type regularization, while architecture composition is selected by a greedy search over GTM variants and group sizes (Qiu et al., 2022). TriMLP uses Adam with learning rate $\Psi$6, dropout 0.5, early stopping after 10 epochs of non-improvement, and teacher forcing for auto-regressive training (Jiang et al., 2023). FinalMLP is tuned through FuxiCTR with batch size 4096, embedding dimension 10, learning rate $\Psi$7 or $\Psi$8, and approximately 30 runs per model on average (Mao et al., 2023). DS-MLP uses a two-stage Adam-based training procedure with KD and branch-alignment hyperparameters $\Psi$9, $U=\Phi(X)$0, $U=\Phi(X)$1, and temperature $U=\Phi(X)$2 (Ou et al., 3 Jun 2026).

The systems-level extreme of this efficiency theme is ParallelMLPs, which is a training procedure rather than a representational architecture. It enables simultaneous training of many independent MLPs by replacing a standard output-layer matrix multiplication with Modified Matrix Multiplication (M3), composed of a broadcasted element-wise product and a scatter-add. On synthetic datasets with 10,000 different models, reported speedups range from 1 to 4 orders of magnitude relative to sequential training; on GPU, a representative setting with 10,000 models, 10,000 samples, 100 features, and batch size 32 reports approximately 1854.881 s sequential versus 4.438 s parallel, while batch size 256 reports 566.248 s versus 2.75 s (Farias et al., 2022). For Dual-MLP training, this work shows that independent MLP towers can be fused computationally without mixing their gradients.

The limitations are equally domain-specific. MLP-3D notes that temporal interactions are linear within groups and that extremely long-range dependencies may require larger $U=\Phi(X)$3 or complementary long-range GTM, increasing cost (Qiu et al., 2022). TriMLP’s strict upper-triangular masking precludes bidirectional conditioning and leaves the method with $U=\Phi(X)$4 complexity in sequence length (Jiang et al., 2023). DS-MLP depends on teacher quality and is sensitive to $U=\Phi(X)$5, $U=\Phi(X)$6, $U=\Phi(X)$7, and $U=\Phi(X)$8; suboptimal settings may reintroduce stream dominance or underfit (Ou et al., 3 Jun 2026). FinalMLP’s single-head bilinear fusion can be memory-heavy, as shown by out-of-memory behavior on Criteo for $U=\Phi(X)$9, and the model loses gains when the two streams are insufficiently diversified (Mao et al., 2023). DuMLP-Pin does not explicitly model local neighborhoods and becomes rank-limited when $V=\Psi(X)$0 (Fei et al., 2022). MLP-AMDC may be sensitive to misregistration or calibration errors between RGB and CASSI paths (Cai et al., 2023).

Taken together, these works establish Dual-MLP models as a broad architectural family rather than a single formula. Their common claim is that two specialized MLP components can be more effective than a single undifferentiated MLP when the decomposition matches the structure of the task: tokens versus channels in video, long-range versus short-range dependencies in sequence recommendation, explicit versus implicit interaction learning in CTR prediction, equivariant branches in set encoding, or modality-specific streams in hyperspectral imaging.