---
title: 'SparseCTR: Scalable CTR Prediction'
url: https://www.emergentmind.com/topics/sparsectr
type: topic
---

# SparseCTR: Scalable CTR Prediction

SparseCTR is a CTR prediction framework designed for efficient and effective modeling of long-term user behaviors in large-scale industrial recommender systems. It addresses the prohibitive computational complexity associated with conventional full self-attention architectures when applied to long user behavior sequences, while capturing domain-specific personalization and temporal dynamics. SparseCTR introduces a personalized time-aware chunking algorithm combined with a three-branch sparse self-attention module and head-wise composite relative temporal encoding, achieving improved accuracy and computational efficiency across several industry datasets [2601.17836].

## 1. Model Architecture

The architecture of SparseCTR consists of an input stage, multiple stacked SparseBlock layers, and a prediction layer. 

**Input Representation**:
- For each user, profile features $U = \{u_1, …, u_{|U|}\}$, historical behaviors $B = \{b_1, …, b_{|B|}\}$ (with item embeddings and timestamps), and candidate items $C = \{c_1, …, c_{|C|}\}$ are collected.
- All are concatenated into a mixed sequence $S = [b_1, …, b_{|B|}, c_1, …, c_{|C|}] \in \mathbb{R}^{n \times d}$, $n = |B| + |C|$, and profile features are embedded into $e_U \in \mathbb{R}^{|U| d}$.
- A composite mask $M$ ensures causality (position $i$ only attends to $j < i$) and candidate isolation (candidates do not attend to each other).

**SparseBlock Layer** (repeated $l$ times):
- Pre-normalization is performed with RMSNorm.
- EvoAttention with composite relative temporal bias is applied.
- SwiGLU-Feedforward network: 
  $$
  E_S^{\mathrm{FFN}} = (\phi(E_S^{\mathrm{Evo}} W_1) \odot E_S^{\mathrm{Evo}} W_2) W_3
  $$
  where $\phi = \mathrm{Swish}$, $W_1,W_2 \in \mathbb{R}^{d \times 3d}$, $W_3 \in \mathbb{R}^{3d \times d}$.

**Prediction Layer**:
- Candidate embeddings $E_C^{(l)}$ are fused with $e_U$ and passed through an MLP for CTR prediction:
  $$
  \hat{Y} = \mathrm{MLP}( \mathrm{ReLU}(E_C^{(l)} W_4) \odot \mathrm{sigmoid}(e_U W_5) )
  $$
- The model is optimized using binary cross-entropy loss on all candidates.

## 2. Personalized Time-Aware Chunking

A key innovation in SparseCTR is the personalized time-aware chunking (“TimeChunking”) algorithm, which segments long behavior sequences based on the largest time-gaps to preserve the continuity of user actions and facilitate parallelism.

**Procedure**:
- Compute time differences $\Delta t_k = t_k - t_{k-1}$, $k=1…|B|$.
- Select the top-$|P|$ largest time-gaps as chunk boundaries.
- Define variable-length chunks $\{p_1, …, p_P\}$, each containing consecutive behaviors between boundaries.

| Step               | Operation                                              | Output                     |
|--------------------|--------------------------------------------------------|----------------------------|
| 1                  | $\Delta t$ computation on $|B|$ behaviors              | Length-$|B|$ array         |
| 2                  | Select largest $|P|$ indices in $\Delta t$             | Chunk boundaries indices   |
| 3                  | Sort, pad with 0 and $|B|$, split accordingly          | $P$ chunks                 |

This chunking ensures that sequences with abrupt temporal gaps are split appropriately and continuous interaction patterns are maintained.

## 3. Three-Branch EvoAttention Mechanism

SparseCTR's EvoAttention module sparsifies self-attention by jointly modeling three aspects of user behavior:

**Global Attention** (long-term interests):
- For each chunk $p_j$, aggregate its behaviors via an MLP:
  $$
  k_{p_j} = \mathrm{MLP}(\{k_b: b \in p_j\}), \quad v_{p_j} = \mathrm{MLP}(\{v_b: b \in p_j\})
  $$
  Then attend globally:
  $$
  A_{\mathrm{glob}}(Q, K_P, V_P) = \mathrm{softmax}((Q K_P^\top)/\sqrt{d} + M) V_P
  $$

**Transition Attention** (interest shifts):
- For each chunk, use the last $m$ behaviors ($B'$), aggregate their key-value pairs, and compute attention:
  $$
  A_{\mathrm{trans}}(Q, K_{B'}, V_{B'}) = \mathrm{softmax}((Q K_{B'}^\top)/\sqrt{d} + M) V_{B'}
  $$

**Local Attention** (short-term interests and user profile):
- For each position $i$, local window $B''_i = \{u_c\} \cup \{b_{i-w+1}, ..., b_i\}$.
- Apply local self-attention:
  $$
  A_{\mathrm{loc}}(Q, K_{B''}, V_{B''}) = \mathrm{softmax}((Q K_{B''}^\top)/{\sqrt{d}} + M) V_{B''}
  $$

**Gated Fusion**:
- For each head, fuse outputs:
  $$
  [\alpha_1, \alpha_2, \alpha_3] = \mathrm{softmax}([A_{\mathrm{glob}},A_{\mathrm{trans}},A_{\mathrm{loc}}] \cdot W_\mathrm{gate})
  $$
  $$
  A = \alpha_1 A_{\mathrm{glob}} + \alpha_2 A_{\mathrm{trans}} + \alpha_3 A_{\mathrm{loc}}
  $$
  Multi-head outputs are concatenated and projected:
  $$
  E_S^{\mathrm{Evo}} = \mathrm{concat}(A^{(1)},…,A^{(H)}) W_O
  $$

## 4. Composite Relative Temporal Encoding

The composite relative temporal encoding (“RelTemporal”) enhances time awareness in each attention branch via learnable, head-specific biases, capturing the following three temporal relations:

1. **Relative Time (Bucketing)**:
   - $\Delta t_{ij} = |t_i - t_j|$, $b = \lfloor \log_2 \Delta t_{ij} \rfloor$
   - $ \mathrm{bias1}_{ij}^{(h)} = -b s_1^{(h)} $
   - $s_1^{(h)}$ is learnable, initialized as a geometric sequence.

2. **Relative Hour (Periodicity)**:
   - $H_{ij} = ($hour difference$) \bmod 24$
   - $\mathrm{bias2}_{ij}^{(h)} = -\sin(\pi H_{ij}/24) s_2^{(h)}$

3. **Relative Weekend (Weekday vs. Weekend)**:
   - $W(t) = 1$ if weekend; $0$ else
   - $\mathrm{bias3}_{ij}^{(h)} = 0$ if $W(t_i)=W(t_j)$, $-1 \cdot s_3^{(h)}$ otherwise

**Combined Bias**:
- $\mathrm{bias}_{ij}^{(h)} = \mathrm{bias1} + \mathrm{bias2} + \mathrm{bias3}$

The composite bias is incorporated into each attention computation:
$$
\mathrm{softmax}((QK^\top)/\sqrt{d} + \mathrm{bias}) V
$$
For global attention, $t_i, t_j$ denote average chunk timestamps.

## 5. Computational Complexity

SparseCTR achieves significant computational savings relative to full self-attention:

- **SparseCTR per block:** 
  $O(B \cdot l \cdot [n P d + n m P d + n w d])$
- **Full self-attention:** 
  $O(B \cdot l \cdot n^2 d)$

Here, $B$ is batch size, $l$ the number of SparseBlocks, $n$ the sequence length, $d$ the model dimension, $P$ chunk count, $m$ transition window size, and $w$ local window; with $P, m, w \ll n$.

**Parameter Count** (excluding embeddings):
- EvoAttention: $O(4l d^2 + 3l d)$
- SwiGLU-FFN: $O(9l d^2)$

This sparsification enables handling industrial-scale sequences (e.g., length 1024) with constrained compute and memory budgets.

## 6. Empirical Evaluation

SparseCTR has been extensively validated on large-scale industrial datasets:
- **Industry (Meituan):** 86.7M users, 20.2M items, 1.456B samples (2 years)
- **Alibaba Display Ads:** 1.14M users, 461K items, 700M samples (22 days)
- **Ele.me click logs:** 14.4M users, 7.45M items, 128M samples (30 days)

**Baselines**:
- DIN, CAN (Group I); SoftSIM, HardSIM, ETA, TWIN-V2 (long-seq, Group II); BST, HSTU, LONGER, SUAN (attention-based, Group III).

**Metrics**: AUC and relative improvement (RelImpr over DIN).

| Model          | AUC (Industry)    | RelImpr (%) |
|----------------|-------------------|-------------|
| SparseCTR      | 0.7083 ±0.00004   | 8.49        |
| SUAN (best baseline) | 0.7040    | 6.25        |

**Ablation Studies**:
- Removing EvoAttention for full self-attention reduces AUC (0.7083 → 0.7051).
- Removing the global branch yields 0.7023.
- Removing all temporal encodings yields 0.7015.
- Each component contributes to state-of-the-art performance.

**Comparative Analysis**:
- EvoAttention vs Dilated/NSAttention/KernelAttention: best AUC-speed trade-off.
- TimeChunking outperforms SimChunking and Fixed-length chunking.
- RelTemporal outperforms T5Bias, HSTUBias, Alibi, and RoPE.

**Efficiency and Scaling Laws**:
- Varying $(P, m, w)$ delivers up to $8.9\times$ speedup over full self-attention, reducing GPU memory without sacrificing AUC.
- AUC fits a power-law: $AUC(X) = E - A/X^\alpha$, $R^2 \approx 1$, demonstrating that higher FLOPs consistently yield better AUC across three orders of magnitude in FLOPs.

## 7. Deployment and Practical Considerations

**Deployment Settings**:
- Embedding size $d = 32$; MLP: $[32 \rightarrow 1]$ with ReLU
- $l = 2$ SparseBlock layers, $H = 8$ heads
- Sequence lengths: $1024$ (Industry, Alibaba), $50$ (Ele.me)
- Training: 1 epoch, Adam optimizer, NVIDIA A100-80G GPUs

**Data Processing**:
- Offline data: listwise samples $(U, B, C, Y)$ per exposure, enabling simultaneous evaluation of all $|C|$ candidates per forward pass.

**Inference and Online Performance**:
- Causal encoder and mask yield single-pass inference for all candidates; average latency $\approx$ 40ms.
- Online A/B testing (1% traffic, 7 days): CTR increased by 1.72%, CPM by 1.41%, with no regression in latency or memory profile.

*This suggests that the architecture is suitable for deployment in latency-sensitive industrial recommendation settings, providing both accuracy and resource efficiency* [2601.17836].

Source: https://www.emergentmind.com/topics/sparsectr