FuXi-γ: Efficient Sequential Recommendation Transformer

• FuXi-γ is an autoregressive, decoder-only Transformer that integrates an exponential-power temporal encoder and a diagonal-sparse positional mechanism for efficient sequential recommendation.
• It achieves over 70% reduction in attention FLOPs, significantly accelerating training and inference on very long sequences without sacrificing accuracy.
• Empirical evaluations demonstrate 3–5% improvement in HR@10 and up to 6.18× inference speedup, validating its performance on industrial-scale benchmarks.

FuXi-$γ$ is an autoregressive, decoder-only Transformer framework for efficient and effective sequential recommendation. It introduces a pair of principled innovations: an exponential-power temporal encoder for fine-grained modeling of temporal intervals in sequential data, and a diagonal-sparse positional mechanism for structured pruning of relative positional attention. The architecture is motivated by the computational demands of standard Transformer-based recommenders and is designed to accelerate both training and inference for very long sequences, without sacrificing accuracy. Empirical evaluation on large-scale benchmarks demonstrates that FuXi-$γ$ achieves state-of-the-art recommendation performance while reducing attention-related FLOPs by over 70% in key scenarios (Yi et al., 14 Dec 2025).

1. Architectural Overview: Decoder-Only Transformer

FuXi-$γ$ employs a stack of $L$ identical decoder-only Transformer blocks, each strictly autoregressive: every position $i$ attends only to $\{1, \dots, i\}$, ensuring causal masking. The architecture dispenses with separate encoders or bidirectional self-attention typical of designs such as BERT4Rec. Instead, it utilizes a single-headed dual-channel attention in each block, which decomposes the attention mechanism into separate temporal and positional channels rather than standard Q-K-V multi-head structure.

Within each block, the canonical attention is replaced by:

• Dual channel formulation: temporal weights (from the exponential-power temporal encoder) and positional weights (from the diagonal-sparse positional mechanism) are combined.
• SwiGLU-FFN follows each attention layer for non-linear mixing.

Critically, FuXi-$γ$ replaces $\mathrm{Softmax}(QK^T/\sqrt{d})$ with two interpretable learned weight matrices—one temporal, one positional—mixed with $U,V$ projections and combined via Hadamard interaction.

2. Exponential-Power Temporal Encoder

FuXi-$γ$ models a sequence $γ$0, with $γ$1 item indices and $γ$2 timestamps.

The model precomputes the pairwise relative-time matrix:

$γ$3

always cast to float32 for computational efficiency. The attention on temporal intervals is given by:

$γ$4

or equivalently

$γ$5

with:

• $γ$6 (learnable base-intensity scalar),
• $γ$7 (learnable power, shaping short- versus long-term emphasis),
• $γ$8 (hyperparameter, exponential decay rate).

Interpretation: smaller $γ$9 sharpens attention toward recent events; $γ$0 extends memory for long-term dependencies. Practical implementations offset $γ$1 by a small $γ$2 if necessary for stability ($γ$3 risks non-Lipschitz regions at $γ$4).

Matrix-vectorized implementation (per block with $γ$5 tokens, $γ$6 channels):

1. Precompute $γ$7 float32.
2. Calculate $γ$8 (elementwise).
3. $γ$9.
4. $L$0.
5. Compute temporal-weighted values: $L$1, $L$2.

All steps are batchable tensor operations, yielding $L$3 time per block, amenable to parallel GPU/TPU computation.

3. Diagonal-Sparse Positional Mechanism

For positional encoding, FuXi-$L$4 learns a relative-position matrix $L$5 under a Toeplitz constraint:

$L$6

(enforcing diagonal stationarity). The model achieves FLOPs reduction by block-pruning attention weights using a diagonally sliding mask:

• Block-partition: Divide $L$7 into $L$8 blocks of size $L$9. Pad as needed.
• Scoring: Each block in the leftmost column scores the sum of absolute values. For block $i$0:

$i$1

• Pruning: Given a prune ratio $i$2, the $i$3 lowest-scoring diagonal blocks in the leftmost column are pruned, representing full diagonals. Block indices spaced by $i$4 are masked.
• Sparse value computation:

$i$5

with $i$6 as elementwise block-masking.

Empirical results show $i$7 cuts 70–75% positional FLOPs at negligible performance cost.

4. Block Structure, Training, and Inference

Each FuXi-$i$8 block proceeds as:

1. RMSNorm: $i$9.
2. Linear & SiLU split: $\{1, \dots, i\}$0, $\{1, \dots, i\}$1, $\{1, \dots, i\}$2.
3. Channel computation:
   • $\{1, \dots, i\}$3
   • $\{1, \dots, i\}$4
4. Hadamard-mixed integration:

$\{1, \dots, i\}$5

with $\{1, \dots, i\}$6 denoting concatenation.

1. Linear projection + residual: $\{1, \dots, i\}$7
2. SwiGLU-FFN + residual:

$\{1, \dots, i\}$8

Final output projects to the item vocabulary using sampled softmax.

Training is conducted on MovieLens-1M, MovieLens-20M, KuaiRand (short video), and a large industrial music dataset (28M users, 1.3B interactions), with metrics HR@10/50, NDCG@10/50, and MRR. Models use 2 and 8 layers ($\{1, \dots, i\}$9), dropout 0.2, AdamW (lr$γ$0), batch size 128, and 128 negatives. Typical values: $γ$1 (movie/video), $γ$2 (music).

5. Empirical Performance and Ablations

FuXi-$γ$3 delivers:

• 3–5% relative improvement in HR@10/NDCG@10 versus previous generative models.
• Up to 4.74$γ$4 training and 6.18$γ$5 inference speedup versus LLaMa on long sequences.
• Ablations:
  • Removing temporal encoder: $γ$6 HR@10.
  • Removing positional channel: $γ$7 HR@10.
  • Removing SwiGLU-FFN: $γ$8 HR@10.

Sensitivity analysis indicates that $γ$9 (movies/videos) and $\mathrm{Softmax}(QK^T/\sqrt{d})$0 (music) yield stable results, while prune ratio $\mathrm{Softmax}(QK^T/\sqrt{d})$1 causes $\mathrm{Softmax}(QK^T/\sqrt{d})$2 NDCG degradation with $\mathrm{Softmax}(QK^T/\sqrt{d})$3 FLOPs reduction.

Component	Effect on HR@10 (removal)	Recommended Setting
Temporal encoder	−9%	$\mathrm{Softmax}(QK^T/\sqrt{d})$4 set per domain
Positional channel	−4%	$\mathrm{Softmax}(QK^T/\sqrt{d})$5
SwiGLU-FFN	−2%	present

6. Implementation and Extensions

Key recommendations:

• Always cast $\mathrm{Softmax}(QK^T/\sqrt{d})$6float32 prior to training.
• Calculate sparse positional mask $\mathrm{Softmax}(QK^T/\sqrt{d})$7 offline and store for inference.
• Leverage fused kernels for layer norm and SiLU where available.
• For $\mathrm{Softmax}(QK^T/\sqrt{d})$8, consider further block-sparsification or low-rank factorization.

Potential future modifications:

• The $\mathrm{Softmax}(QK^T/\sqrt{d})$9 per-layer complexity could be reduced via joint time-position compression.
• Mixture-of-exponentials may provide multi-scale memory.
• In sparse or cold-start regimes, incorporating user profile or auxiliary context via cross-attention may benefit performance.

7. Context and Significance

FuXi-$U,V$0 synthesizes cognitively-inspired temporal attention (exponential decay reflecting human memory, cf. Ebbinghaus forgetting curve) with principled matrix-structural sparsity (Toeplitz-diagonal pruning), instantiated within a strictly causal, decoder-only Transformer. This yields a model capable of state-of-the-art recommendation at low computational cost for industrial-scale, long-sequence data (Yi et al., 14 Dec 2025). All steps are fully parallelizable and compatible with modern GPU/TPU deployment, further strengthening FuXi-$U,V$1's position as a practical architecture for both academic research and large-scale recommender systems.

Code for reproducing results is available at https://github.com/Yeedzhi/FuXi-gamma.