---
title: Stacked Target-to-History Cross Attention
url: https://www.emergentmind.com/topics/stacked-target-to-history-cross-attention-stca
type: topic
---

# Stacked Target-to-History Cross Attention

Stacked Target-to-History Cross Attention (STCA) is an attention mechanism introduced for industrial-scale long-sequence modeling, specifically designed to process extremely long user histories—on the order of 10,000 tokens—in real-time recommender systems operating under stringent latency and compute budgets. STCA substitutes conventional full self-attention over targets and history with a stacked strategy of single-query, multi-head cross-attention from target to history, achieving linear complexity with sequence length. This architectural innovation enables end-to-end modeling at a scale previously impractical for production settings, as demonstrated in deployment at full traffic on Douyin [2511.06077].

## 1. Motivation and Conceptual Foundations

Traditional recommender systems leveraging self-attention encounter prohibitive resource consumption when ingesting sequences of length \( L \sim 10,000 \). The per-layer FLOPs and memory usage for standard Transformers is \( O(L^2 d) \), making them infeasible for low-latency, high-throughput applications. Empirically, the majority of utility for ranking a candidate target \( t \) arises from its direct interactions with history items, rather than the inter-history dependencies modeled by self-attention. STCA addresses this by restricting intra-history modeling and focusing attention solely on target-to-history (T→H) interactions, yielding an attention block whose costs scale as \( O(L d h) \), where \( d \) is the embedding dimension and \( h \) the number of heads. This tradeoff substantially reduces compute requirements while retaining the most relevant signal for target ranking.

## 2. Formal Definition and Layer Structure

A single STCA layer operates as follows:

- **Notation**:
  - History embeddings \( \mathcal H = \{x_1, \ldots, x_L\},\; x_j \in \mathbb R^d \)
  - Target embedding \( x_t \in \mathbb R^d \)
  - Number of heads \( h \), head dimension \( d_h = d/h \)
  - Parameters: \( W_Q^{(i,r)}, W_K^{(i,r)}, W_V^{(i,r)} \in \mathbb R^{d \times d_h} \); projection \( W_O^{(i)} \in \mathbb R^{d\times d} \)

- **Step 1: Input Encoding (Eq. 1)**
  $$
  \widetilde X^{(i)} = \mathrm{LN}\bigl(\mathrm{SwiGLUFFN}^{(i)}(X)\bigr) \in \mathbb{R}^{L \times d}
  $$
  $$
  q^{(1)} = \mathrm{LN}\bigl(\mathrm{SwiGLUFFN}^{(1)}(x_t)\bigr) \in \mathbb{R}^d
  $$
  with SwiGLU-FFN defined as
  $$
  \mathrm{SwiGLUFFN}(z) = \big((zW_u)\odot\mathrm{SiLU}(zW_v)\big)W_o
  $$

- **Step 2: Multi-Head Target→History Cross Attention (per layer \( i \), head \( r \))**
  - Raw scores:
    $$
    s^{(i,r)} = \frac{(q^{(i)}W_Q^{(i,r)})\,(\widetilde X^{(i)} W_K^{(i,r)})^\top}{\sqrt{d_h}} \in \mathbb{R}^{1 \times L}
    $$
  - Weights:
    $$
    \alpha^{(i,r)} = \mathrm{softmax}(s^{(i,r)}) \in \mathbb{R}^{1 \times L}
    $$
  - Weighted sum:
    $$
    o^{(i,r)} = \alpha^{(i,r)} \; (\widetilde X^{(i)} W_V^{(i,r)}) \in \mathbb{R}^{1 \times d_h}
    $$
  - Output projection:
    $$
    o^{(i)} = [o^{(i,1)} \;\|\; \cdots \;\|\; o^{(i,h)}] W_O^{(i)} \in \mathbb{R}^{d}
    $$

- **Step 3: Layer Stacking and Target-Conditioned Query Fusion**
  For layer \( i+1 \), the query is constructed by fusing previous outputs and the raw target:
  $$
  q^{(i+1)} = \mathrm{SwiGLUFFN}^{(i+1)}\big([o^{(1)} \;\|\; \ldots \;\|\; o^{(i)} \;\|\; x_t] W_C^{(i+1)}\big)
  $$
  After \( M \) layers, the history summary is
  $$
  Z_\mathcal{H} = \begin{bmatrix} o^{(1)} \\ \vdots \\ o^{(M)} \end{bmatrix} \in \mathbb{R}^{M \times d}
  $$
  The final target-conditioned representation for prediction is
  $$
  z = \mathrm{SwiGLUFFN}([o^{(1)} \;\|\; \cdots \;\|\; o^{(M)} \;\|\; x_t] W_Z)
  $$

## 3. Computational Complexity and Scalability

STCA reduces per-layer complexity from quadratic \( O(L^2 d) \) in standard self-attention to linear \( O(L d h) \). With the additional "reordering trick" for efficient computation (Eq. attn-optimized), the cost per head becomes
$$
O(d\,d_h) + O(Ld) + O(d\,d_h) = O(Ld + h d^2) \approx O(L d h)
$$
when \( L \gg d \). Empirical scaling (Fig. 5) demonstrates that, as history length increases from 500 to 10,000, STCA’s FLOPs grow by approximately \( 20\times \), compared to \( 114\times \) for Transformer self-attention. In production, a 4-layer STCA model at \( L=10,000 \) (21 GFLOPs) achieves parity or superiority (in NLL) over a 4-layer Transformer at \( L=8,000 \) (156 GFLOPs), representing roughly \( 7.4\times \) lower compute for matched quality.

## 4. Implementation Details and System Integration

Typical deployment hyperparameters for STCA include embedding dimension \( d=256 \), SwiGLU multilayer ratio \( r=4 \), number of heads \( h=8 \), and stack depth \( M=4 \) layers. In production, Request-Level Batching (RLB) further amplifies the efficiency gains by sharing user-history encoding across \( m=8 \) targets, leading to approximately \( 77\%-84\% \) host–device bandwidth reduction and \( 2.2\times-5.1\times \) end-to-end throughput uplift, and extending the feasible history length by \( 8\times \) at a fixed GPU RAM footprint.

Training follows a “train sparsely / infer densely” paradigm: stochastic training sequence lengths \( L_{\mathrm{train}} \in [L_{\min}, L_{\max}] \), with average \( L_{\mathrm{avg}} \approx 2,000 \); inference at \( L_{\mathrm{infer}} = 10,000 \). Sampling uses a U-shaped beta distribution with \( \alpha \approx 0.02 \), and the most recent tokens are always prioritized (suffix truncation). Sequence sparsity (SS) at \( 20\% \) achieves \( \sim 80\% \) of full-length AUC gain for one-third of the compute.

## 5. Pseudocode and Workflow

The STCA forward pass is succinctly captured by the following pseudocode:

```python
# Inputs: X[L×d]: history embeddings, x_t[d]: target, M: stack depth, h: num_heads
q = LayerNorm(SwiGLUFFN^{(1)}(x_t))  # initial query
for i in 1..M:
    X_i = LayerNorm(SwiGLUFFN^{(i)}(X))  # encode history
    for r in 1..h:
        qQ = q · W_Q^{(i,r)}             # [1×d_h]
        K  = X_i · W_K^{(i,r)}           # [L×d_h]
        V  = X_i · W_V^{(i,r)}           # [L×d_h]
        scores = qQ · K^T / sqrt(d_h)    # [1×L]
        α = softmax(scores)              # [1×L]
        o_r = α · V                      # [1×d_h]
    o^{(i)} = concat(o_1,…,o_h) · W_O^{(i)}  # [d]
    if i < M:
        concat_q = [o^{(1)}…o^{(i)}, x_t]    # [(i+1)d]
        q = SwiGLUFFN^{(i+1)}(concat_q · W_C^{(i+1)})
# Final output
z = SwiGLUFFN([o^{(1)}…o^{(M)}, x_t] · W_Z)
return z
```

This process yields a target-conditional history summary for subsequent prediction layers in the recommendation pipeline.

## 6. Empirical Results and Comparative Analysis

STCA demonstrates both efficiency and effectiveness in large-scale recommender deployments. In matched-compute offline experiments without retrieval features, STCA+RLB+Extrapolation achieves a +0.49% finish AUC lift and −1.16% NLL, compared to +0.31%/−0.86% for HSTU and +0.25%/−0.46% for standard Transformer architectures. Ablation studies indicate additional incremental benefits from deeper stacks (+0.18% AUC, 2→4 layers), SwiGLU activation (+0.11%), enlarged sparse ID embeddings (+0.08%), time-delta side information (+0.08%), increased head count (+0.05%), and enhanced query fusion (+0.06%). These findings elucidate the modular and extensible nature of STCA within end-to-end recommendation systems at billion-scale traffic.

## 7. Limitations and Theoretical Trade-Offs

While STCA prioritizes target–history interactions critical for ranking, the omission of history–history dependencies (modeled in full self-attention) constitutes an explicit trade-off. This is theoretically justified by the empirical observation that such dependencies contribute marginally once history length becomes large and only a limited portion of history is most relevant to the target. STCA’s linear scaling unlocks practical modeling of sequences at lengths previously considered infeasible for real-time systems, while experimental ablations confirm that the accuracy-cost tradeoff is favorable compared to existing alternatives.

In summary, STCA redefines the design landscape for long-sequence attention modules in recommender systems by enabling linear compute scaling, maintaining end-to-end differentiability, and matching Transformer accuracy at a small fraction of the cost. Its integration with Request Level Batching and length-extrapolative training further extends its capability for industrial-scale deployments with strict latency and resource constraints [2511.06077].

Source: https://www.emergentmind.com/topics/stacked-target-to-history-cross-attention-stca