RankMixer Architecture for Scalable Recommenders

• RankMixer is a hardware-aware ranking model that replaces memory-bound feature-cross modules and quadratic self-attention with an efficient token mixing mechanism.
• It employs per-token feed-forward networks and a Sparse-MoE extension to scale parameters to over a billion while maintaining fixed inference latency.
• Empirical results show significant AUC improvements and measurable business impact in industrial recommender and advertising systems.

The RankMixer architecture is a hardware-aware, large-scale ranking model designed specifically to address the unique demands of industrial recommender systems. It systematically replaces memory-bound and parameter-inefficient CPU-era feature-cross modules and the quadratic compute of transformer self-attention, employing a lightweight, highly parallelizable token-mixing mechanism. RankMixer achieves superior model FLOPs utilization (MFU), scales to over a billion parameters without breaching strict inference latency/QPS budgets, and preserves the modeling of both distinct feature subspaces and deep cross-feature interactions. The architecture serves as a foundation for subsequent evolutions such as TokenMixer-Large, demonstrating persistent gains under aggressive scaling regimes and deployment across major real-world recommendation and advertising systems (Zhu et al., 21 Jul 2025, Jiang et al., 6 Feb 2026).

1. Architectural Motivation and Context

In industrial recommendation, ranking backbones traditionally rely on a mix of feature-crossing modules—including FM, DCN, and handcrafted cross layers—whose memory-bound CPU-optimized kernels yield very low GPU arithmetic throughput. These modules resulted in model MFU values in the 3–6% range and limited the effective scaling of model parameters, as GPU utilization is constrained by memory traffic rather than compute (Zhu et al., 21 Jul 2025). In contrast, self-attention mechanisms in transformers, while successful in natural language processing, exhibit two major drawbacks for recommendation: quadratic cost with respect to token count ($O(T^2 D)$), and a shared similarity metric across tokens that typically encode highly heterogeneous feature fields, leading to poor inductive bias and hardware inefficiency.

RankMixer is motivated by the need to (i) increase parallelism and MFU, (ii) decouple parameter expansion from inference cost, and (iii) preserve deep, expressive interaction modeling for large, semantically diverse feature sets.

2. Core Building Blocks

Multi-Head Token Mixing

RankMixer replaces quadratic self-attention with a parameter-free Multi-Head Token Mixing layer. Given $X_0 \in \mathbb{R}^{T \times D}$ as token embeddings:

• Each D-dimensional token is split into $H$ equal-size "heads."
• For each head $h$, the $h$-th subspace of every token is concatenated into a new "mixed token," effectively transposing token and feature axes.
• For $H=T$, the mixing operation produces $T$ mixed tokens of dimension $D$, achieving $O(T D)$ cost.
• This operation is entirely parameter-free and consists of fusing large tensor rearrangements, allowing for maximal GPU kernels and bandwidth-efficient execution (Zhu et al., 21 Jul 2025).

In comparison, transformer self-attention incurs $O(T^2 D)$ operations and $X_0 \in \mathbb{R}^{T \times D}$0 memory for the attention matrix, which is intractable under production latency constraints.

Per-Token Feed-Forward Networks (PFFNs)

Immediately after token mixing, a two-layer MLP (feed-forward network) is applied independently to each token, with parameters unshared between tokens:

• For token $X_0 \in \mathbb{R}^{T \times D}$1, the transforms are $X_0 \in \mathbb{R}^{T \times D}$2 and $X_0 \in \mathbb{R}^{T \times D}$3.
• Output: $X_0 \in \mathbb{R}^{T \times D}$4, where $X_0 \in \mathbb{R}^{T \times D}$5 is the post-mixing embedding.
• This "isolation" prevents dominant tokens (e.g., frequent IDs) from overwhelming signals from infrequent tokens and maintains cross-token information through the residual path.

By allocating separate parameters per token, the PFFN multiplicatively increases model capacity, but the compute cost remains $X_0 \in \mathbb{R}^{T \times D}$6, identical to a shared-parameter MLP (Zhu et al., 21 Jul 2025, Jiang et al., 6 Feb 2026).

Sparse Mixture-of-Experts (Sparse-MoE) Extension

To scale beyond dense PFFNs, RankMixer replaces each token’s MLP with a lightweight MoE module:

• Each PFFN is expanded into $X_0 \in \mathbb{R}^{T \times D}$7 experts per token.
• ReLU Routing provides non-negative gate activations $X_0 \in \mathbb{R}^{T \times D}$8.
• Dense-Training/Sparse-Inference (DTSI) maintains two router versions: full expert exposure during training (avoiding "dead" experts), sparse activation at inference, and applies an $X_0 \in \mathbb{R}^{T \times D}$9 penalty on gate activations to enforce activation budgets.
• High sparsity (as low as $H$0 active experts) retains $H$1 of dense model AUC with up to $H$2 throughput improvement and balanced expert load (Zhu et al., 21 Jul 2025).

3. Computational Complexity and Parallelism

RankMixer is engineered for large-scale matrix operations aligned with GPU strengths. With $H$3 layers, $H$4 tokens, hidden size $H$5, and expansion $H$6:

• Dense configuration: $H$7Parameters $H$8, FLOPs $H$9 per sample.
• Empirical MFU increases to $h$0 (vs. $h$1 for self-attention or legacy hybrid modules).
• Parameter growth (e.g., scaling from 16M to 1B parameters) is decoupled from inference latency, which remains nearly fixed due to reduced FLOPs/parameter, increased MFU, and low-precision compute (e.g., fp16).

For instance, RankMixer-1B (fp16) achieves $h$2 parameter count over a baseline DCN+MLP block yet reduces latency ($h$3 ms vs. $h$4 ms) (Zhu et al., 21 Jul 2025).

Model	Params	MFU	Latency	AUC gain
DCN+MLP	16M	~4.5%	14.5ms	Baseline
RankMixer-100M	107M	44%		+0.64%–1.33%
RankMixer-1B	1.1B	45%	14.3ms	+0.95%–1.82%

4. Empirical Results and Deployment

RankMixer has been validated on trillion-scale production logs (e.g., Douyin platform):

• RankMixer-100M outperforms dense DLRM/DCN/HiFormer/Wukong models at identical parameter scale, with $h$5 to $h$6 AUC/UAUC improvements at lower or equivalent FLOPs.
• RankMixer-1B Dense achieves $h$7 Finish-AUC, $h$8 UAUC, $h$9 Skip-AUC, $h$0 Skip-UAUC.
• Sparse-MoE variant (1/8 experts active) sustains $h$1 dense AUC while halving inference cost.

Online A/B tests report measurable business impact:

• Full-traffic serving: $h$2 in active user days, $h$3 app duration, $h$4 likes.
• Larger gains for low-activity users.
• In advertisement ranking: $h$5 AUC, $h$6 ADVV (Zhu et al., 21 Jul 2025).

5. Evolution: TokenMixer-Large and Beyond

TokenMixer-Large (Jiang et al., 6 Feb 2026) builds upon RankMixer, resolving limitations encountered in deep model regimes:

• Introduces explicit mixing-and-reverting within each block, ensuring $h$7 input-output alignment and preventing vanishing gradients.
• Adds inter-layer (interval) residuals (skip connections every $h$8 layers) and auxiliary losses at regular depths, ensuring stable optimization when depth $h$9.
• Adopts a "sparse-train, sparse-infer" fully differentiable Sparse Per-token MoE, reducing training and inference cost proportionally to the active expert count $H=T$0.
• Achieves up to 15B parameter scale in live ByteDance recommender deployments, $H=T$1 GMV improvement in e-commerce, and $H=T$2 ADSS for advertising.
• MFU reached up to $H=T$3 using grouped-GEMM kernels and token-parallel all-to-all schemes across multiple GPUs.

6. Significance and Impact

The RankMixer architecture constitutes a paradigm shift for hardware-scaled industrial recommenders:

• Replaces a heterogeneous assembly of legacy feature-cross and deep models with a unified, highly parallel "mix then isolate" pattern.
• Enables dramatic increases in parameter count and representational capacity without incurring inference latency or resource footprint penalties.
• Generalizes across recommendation and advertisement ranking, with empirical scaling laws monotonic in parameter count, as demonstrated in online and offline production.
• Facilitates subsequent innovations, including TokenMixer-Large, that maintain both high MFU and stable training at multi-billion parameter scales.

The practical significance is substantiated by production adoption and persistent business metric improvements, making RankMixer and its derivatives foundational layers for large-scale, high-throughput modern recommender systems (Zhu et al., 21 Jul 2025, Jiang et al., 6 Feb 2026).