---
title: 'FuXi-γ: Efficient Sequential Recommendation Transformer'
url: https://www.emergentmind.com/topics/fuxi
type: topic
---

# FuXi-γ: Efficient Sequential Recommendation Transformer

FuXi-$γ$ is an autoregressive, decoder-only Transformer framework for efficient and effective sequential recommendation. It introduces a pair of principled innovations: an exponential-power temporal encoder for fine-grained modeling of temporal intervals in sequential data, and a diagonal-sparse positional mechanism for structured pruning of relative positional attention. The architecture is motivated by the computational demands of standard Transformer-based recommenders and is designed to accelerate both training and inference for very long sequences, without sacrificing accuracy. Empirical evaluation on large-scale benchmarks demonstrates that FuXi-$γ$ achieves state-of-the-art recommendation performance while reducing attention-related FLOPs by over 70% in key scenarios [2512.12740].

## 1. Architectural Overview: Decoder-Only Transformer

FuXi-$γ$ employs a stack of $L$ identical decoder-only Transformer blocks, each strictly autoregressive: every position $i$ attends only to $\{1, \dots, i\}$, ensuring causal masking. The architecture dispenses with separate encoders or bidirectional self-attention typical of designs such as BERT4Rec. Instead, it utilizes a single-headed dual-channel attention in each block, which decomposes the attention mechanism into separate temporal and positional channels rather than standard Q-K-V multi-head structure.

Within each block, the canonical attention is replaced by:

- Dual channel formulation: temporal weights (from the exponential-power temporal encoder) and positional weights (from the diagonal-sparse positional mechanism) are combined.
- SwiGLU-FFN follows each attention layer for non-linear mixing.

Critically, FuXi-$γ$ replaces $\mathrm{Softmax}(QK^T/\sqrt{d})$ with two interpretable learned weight matrices—one temporal, one positional—mixed with $U,V$ projections and combined via Hadamard interaction.

## 2. Exponential-Power Temporal Encoder

FuXi-$γ$ models a sequence $S=\{(v_1, t_1), (v_2, t_2), \ldots, (v_n, t_n)\}$, with $v_i$ item indices and $t_i$ timestamps.

The model precomputes the pairwise relative-time matrix:
$$
T \in \mathbb{R}^{n \times n}, \qquad T_{ij} = |t_i - t_j|,
$$
always cast to float32 for computational efficiency. The attention on temporal intervals is given by:
$$
A_{\text{ts}} = \alpha\,\gamma^{T^{\beta}}
$$
or equivalently
$$
(A_{\text{ts}})_{ij} = \alpha\,\gamma^{|t_i - t_j|^\beta},
$$
with:

- $\alpha$ (learnable base-intensity scalar),
- $\beta$ (learnable power, shaping short- versus long-term emphasis),
- $\gamma \in (0,1)$ (hyperparameter, exponential decay rate).

Interpretation: smaller $\gamma$ sharpens attention toward recent events; $\gamma\rightarrow1$ extends memory for long-term dependencies. Practical implementations offset $T$ by a small $\varepsilon$ if necessary for stability ($\beta < 1$ risks non-Lipschitz regions at $0$).

Matrix-vectorized implementation (per block with $n$ tokens, $d$ channels):

1. Precompute $T \to$ float32.
2. Calculate $P = T^{\beta}$ (elementwise).
3. $E = \exp(P \log \gamma)$.
4. $A_{\text{ts}} = \alpha E \in \mathbb{R}^{n\times n}$.
5. Compute temporal-weighted values: $Z_{\text{ts}} = A_{\text{ts}} V$, $V \in \mathbb{R}^{n\times d}$.

All steps are batchable tensor operations, yielding $O(n^2 d)$ time per block, amenable to parallel GPU/TPU computation.

## 3. Diagonal-Sparse Positional Mechanism

For positional encoding, FuXi-$γ$ learns a relative-position matrix $W_{\text{pos}} \in \mathbb{R}^{n\times n}$ under a Toeplitz constraint:
$$
W_{\text{pos}}^{i,j} = W_{\text{pos}}^{i+1,j+1}
$$
(enforcing diagonal stationarity). The model achieves FLOPs reduction by block-pruning attention weights using a diagonally sliding mask:

- **Block-partition**: Divide $W_{\text{pos}}$ into $\frac{n}{s}\times\frac{n}{s}$ blocks of size $s\times s$. Pad as needed.
- **Scoring**: Each block in the *leftmost* column scores the sum of absolute values. For block $b$:
  $$
  \mathrm{score}_b = \sum_{u=1}^s\sum_{v=1}^s |W_{\text{pos}}[bs+u, v]|
  $$
- **Pruning**: Given a prune ratio $\tau$, the $\lfloor (\frac{n}{s})\tau \rfloor$ lowest-scoring diagonal blocks in the leftmost column are pruned, representing full diagonals. Block indices spaced by $(\frac{n}{s}+1)$ are masked.
- **Sparse value computation**:
  $$
  Z_{\text{pos}} = (M \odot W_{\text{pos}}) V
  $$
  with $\odot$ as elementwise block-masking.

Empirical results show $\tau=50\%\text{--}60\%$ cuts 70–75% positional FLOPs at negligible performance cost.

## 4. Block Structure, Training, and Inference

Each FuXi-$γ$ block proceeds as:

1. **RMSNorm**: $X \in \mathbb{R}^{n\times d}$.
2. **Linear \& SiLU split**: $[U, V]=\mathrm{Split}(\mathrm{SiLU}(XW_{uv}))$, $U\in\mathbb{R}^{n\times 2d}$, $V\in\mathbb{R}^{n\times d}$.
3. **Channel computation**:
   - $Z_{\text{ts}}=A_{\text{ts}}V$
   - $Z_{\text{pos}}=(M\odot W_{\text{pos}})V$
4. **Hadamard-mixed integration**:
   $$
   I = \mathrm{RMSNorm}([\![Z_{\text{ts}}, Z_{\text{pos}}]\!]) \odot U
   $$
   with $[\![\cdot]\!]$ denoting concatenation.
5. **Linear projection + residual**: $O=I W_o + X$
6. **SwiGLU-FFN + residual**: 
   $$
   O' = \mathrm{RMSNorm}(O), \quad H = (\mathrm{SiLU}(O'W_1) \odot (O'W_2))W_3 + O.
   $$
Final output projects to the item vocabulary using sampled softmax.

Training is conducted on MovieLens-1M, MovieLens-20M, KuaiRand (short video), and a large industrial music dataset (28M users, 1.3B interactions), with metrics HR@10/50, NDCG@10/50, and MRR. Models use 2 and 8 layers ($d=256$), dropout 0.2, AdamW (lr$=10^{-3}$), batch size 128, and 128 negatives. Typical values: $\gamma=0.8$ (movie/video), $\gamma=0.9$ (music).

## 5. Empirical Performance and Ablations

FuXi-$γ$ delivers:

- 3–5% relative improvement in HR@10/NDCG@10 versus previous generative models.
- Up to 4.74$\times$ training and 6.18$\times$ inference speedup versus LLaMa on long sequences.
- Ablations:
  - Removing temporal encoder: $-9\%$ HR@10.
  - Removing positional channel: $-4\%$ HR@10.
  - Removing SwiGLU-FFN: $-2\%$ HR@10.

Sensitivity analysis indicates that $\gamma \in [0.7,0.85]$ (movies/videos) and $[0.85,0.95]$ (music) yield stable results, while prune ratio $\tau \leq 60\%$ causes $<2\%$ NDCG degradation with $70\%+$ FLOPs reduction.

| Component           | Effect on HR@10 (removal) | Recommended Setting     |
|---------------------|--------------------------|------------------------|
| Temporal encoder    | −9%                      | $\gamma$ set per domain |
| Positional channel  | −4%                      | $\tau \leq 60\%$        |
| SwiGLU-FFN          | −2%                      | present                |

## 6. Implementation and Extensions

Key recommendations:

- Always cast $T\rightarrow$float32 prior to training.
- Calculate sparse positional mask $M$ offline and store for inference.
- Leverage fused kernels for layer norm and SiLU where available.
- For $n>2000$, consider further block-sparsification or low-rank factorization.

Potential future modifications:

- The $O(n^2d)$ per-layer complexity could be reduced via joint time-position compression.
- Mixture-of-exponentials may provide multi-scale memory.
- In sparse or cold-start regimes, incorporating user profile or auxiliary context via cross-attention may benefit performance.

## 7. Context and Significance

FuXi-$γ$ synthesizes cognitively-inspired temporal attention (exponential decay reflecting human memory, cf. Ebbinghaus forgetting curve) with principled matrix-structural sparsity (Toeplitz-diagonal pruning), instantiated within a strictly causal, decoder-only Transformer. This yields a model capable of state-of-the-art recommendation at low computational cost for industrial-scale, long-sequence data [2512.12740]. All steps are fully parallelizable and compatible with modern GPU/TPU deployment, further strengthening FuXi-$γ$'s position as a practical architecture for both academic research and large-scale recommender systems.

Code for reproducing results is available at [https://github.com/Yeedzhi/FuXi-gamma](https://github.com/Yeedzhi/FuXi-gamma).

Source: https://www.emergentmind.com/topics/fuxi