FuXi-Linear: Efficient Long-term Sequential Recommendation

• FuXi-Linear is a linear-complexity architecture for long-term, time-aware sequential recommendation that addresses the efficiency–effectiveness bottleneck of quadratic attention.
• It employs a three-channel pipeline—semantic retention, linear positional channel, and temporal retention—to decouple temporal semantics and accurately model periodicity.
• The design supports chunkwise recurrent training with cached state updates, enabling scalable online recommendations and superior performance on long-sequence datasets.

Searching arXiv for the cited FuXi-Linear paper and related FuXi papers to ground the article. FuXi-Linear is a linear-complexity architecture for long-term, time-aware sequential recommendation that is designed to address the efficiency–effectiveness limitations of quadratic attention on long user histories. The model targets three stated challenges in applying linear attention to recommendation: temporal–semantic crosstalk, insufficient positional information in linear frameworks, and the concentration of prior linear recommendation studies on short sequences and shallow architectures. Its core design combines a semantic retention mechanism with two additional channels—the Temporal Retention Channel and the Linear Positional Channel—so that temporal periodicity and fine-grained positional structure can be modeled while preserving linear complexity in sequence length (Ye et al., 27 Feb 2026).

1. Nomenclature and scope

FuXi-Linear is introduced in the paper "FuXi-Linear: Unleashing the Power of Linear Attention in Long-term Time-aware Sequential Recommendation" (Ye et al., 27 Feb 2026). It is a recommendation model rather than a weather forecasting model. This distinction matters because the terms "FuXi" and "FuXi Weather" also denote earlier machine-learning systems for global weather forecasting, and neither of those papers introduces a component named "FuXi-Linear" (Sun et al., 2024).

A concise disambiguation is useful because the shared "FuXi" prefix can obscure the fact that these are different research lines.

Paper	Domain	Relevance to "FuXi-Linear"
"FuXi-Linear: Unleashing the Power of Linear Attention in Long-term Time-aware Sequential Recommendation" (Ye et al., 27 Feb 2026)	Sequential recommendation	Introduces FuXi-Linear
"FuXi Weather: A data-to-forecast machine learning system for global weather" (Sun et al., 2024)	Weather forecasting and data assimilation	Does not introduce or use the term "FuXi-Linear"
"FuXi: A cascade machine learning forecasting system for 15-day global weather forecast" (Chen et al., 2023)	Weather forecasting	Does not define a "FuXi-Linear" variant

The recommendation model is therefore best understood as an architecture for long-sequence ranking under chronological interaction histories, not as a submodule of the weather systems. A plausible implication is that the name reuses the broader "FuXi" branding while addressing a separate technical problem: scalable sequence modeling for recommender systems.

2. Problem setting and design motivation

FuXi-Linear is positioned against SASRec-like baselines whose quadratic attention incurs $O(n^2)$ compute and memory, restricting usable sequence length and slowing inference (Ye et al., 27 Feb 2026). The motivating application is industrial recommendation with sequences that may exceed $10^4$ interactions, where quadratic mechanisms become a bottleneck.

The paper identifies three concrete obstacles that limit prior linear-attention recommendation models. First, temporal signals are often introduced by scaling semantic attention weights with time-aware factors, which entangles temporal and semantic information and produces temporal–semantic crosstalk. Second, decay-based linear mechanisms encode order only coarsely and cannot sharply distinguish adjacent positions, while stronger relative-position formulations typically sacrifice recurrence and revert to quadratic cost. Third, prior work is described as focusing mainly on sequence lengths of at most $100$ and on one- or two-layer architectures, leaving behavior at thousand-length scale insufficiently verified (Ye et al., 27 Feb 2026).

FuXi-Linear addresses these issues through what the paper describes as a three-channel pipeline: semantic retention, linear positional kernel, and decoupled periodic temporal retention, followed by lightweight fusion and a feed-forward module optimized for linear complexity. The central claim is that long-sequence recommendation requires not merely replacing full attention with a linear surrogate, but rethinking how temporal and positional signals enter the model so that recurrence, fine positional resolution, and periodic structure can coexist.

3. Architectural organization

The model takes a user sequence truncated or padded to a fixed length $n$. Items are embedded through $E \in \mathbb{R}^{|I|\times d}$ and fused with learnable absolute positional embeddings (Ye et al., 27 Feb 2026). Each block then applies RMSNorm, a Linear Multi-Channel Attention module, and a Multi-stage Feed-forward Network, with residual fusion around the block.

Within each FuXi-Linear block, three channels are computed in parallel:

1. Retention channel: a RetNet-style linear attention mechanism providing semantic or content context.
2. Linear Positional Channel (LPC): a learnable kernelized channel that injects relative positional information while preserving recurrence.
3. Temporal Retention Channel (TRC): a timestamp-only channel that models multi-scale periodicity without mixing temporal features into semantic attention.

The channel outputs are normalized, concatenated, and gated elementwise by a projection of the original input. The paper defines this fusion as

$$\mathrm{LMCA}(X) = \mathrm{Concat}(\mathrm{Norm}(Y_{\mathrm{ret}}), \mathrm{Norm}(Y_p), \mathrm{Norm}(Y_t)) \odot (XW_u),$$

where $W_u$ is a learned projection and $\odot$ denotes elementwise product (Ye et al., 27 Feb 2026).

The subsequent MFFN merges channel information with residual structure:

$Y_{\mathrm{stage1}} = XW_0 + X_0,$

$$\widetilde{Y}_{\mathrm{stage1}} = \mathrm{Norm}(Y_{\mathrm{stage1}}),$$

$10^4$0

with $10^4$1 set to SiLU and Norm given by RMSNorm (Ye et al., 27 Feb 2026).

This architecture is explicitly both recurrent and parallelizable. That duality is central: it allows chunkwise parallel training while preserving constant-in-$10^4$2 incremental decoding once states are cached.

4. Core mechanisms

Semantic retention

For the semantic channel, the paper defines

$10^4$3

with $10^4$4 as SiLU. A lower-triangular decay mask $10^4$5 is used per head:

$10^4$6

where $10^4$7 is learnable per head (Ye et al., 27 Feb 2026).

The parallel-form head is

$10^4$8

and the recurrent update is

$10^4$9

This mechanism yields linear-time sequence processing and $100$0 step cost during decoding when the recurrent states are cached (Ye et al., 27 Feb 2026).

Linear Positional Channel

The LPC is motivated by the claim that strong relative positional encodings are usually quadratic, whereas decay-only linear encodings are too coarse. The method approximates a relative-position function $100$1 through a low-rank kernel:

$100$2

with $100$3 (Ye et al., 27 Feb 2026).

Using projected values $100$4, the recurrent state is

$100$5

and the output is

$100$6

for learnable $100$7 (Ye et al., 27 Feb 2026).

In parallel form,

$100$8

where $100$9 is a learnable positional embedding matrix. The stated purpose is to approach the expressiveness of relative positional bias while maintaining linear complexity through low-rank kernelization and recurrence.

Temporal Retention Channel

The TRC is the main temporal innovation. Its objective is to capture multi-scale periodicity from timestamps while avoiding temporal–semantic crosstalk by deriving queries and keys from timestamps alone (Ye et al., 27 Feb 2026). The paper introduces a complex-domain time-evolution view:

$n$0

which leads to damped sinusoidal behavior across time intervals.

For each period parameter pair $n$1, two heads encode cosine and sine components. With timestamps $n$2, the temporal heads in parallel form are

$n$3

$n$4

These are fused with current-state projections through learnable balancing coefficients $n$5 (Ye et al., 27 Feb 2026).

The recurrent form defines timestamp-only queries and keys. For example,

$n$6

$n$7

and the recurrent state update is

$n$8

The period grid is given by

$n$9

with modular reduction used to avoid float32 precision loss, and $E \in \mathbb{R}^{|I|\times d}$0 initialized as

$E \in \mathbb{R}^{|I|\times d}$1

ensuring $E \in \mathbb{R}^{|I|\times d}$2 (Ye et al., 27 Feb 2026).

The stated significance of TRC is twofold: it isolates temporal signals from semantic attention and explicitly models periodicity rather than treating time only as an auxiliary bias.

5. Training procedure, recurrence, and computational properties

A unifying feature of FuXi-Linear is that all three channels admit the same algebraic form,

$E \in \mathbb{R}^{|I|\times d}$3

together with recurrent state evolution

$E \in \mathbb{R}^{|I|\times d}$4

This shared structure permits chunkwise recurrent training: within a chunk of size $E \in \mathbb{R}^{|I|\times d}$5, vectorized computation is used, while chunk boundary states propagate recurrence across the sequence (Ye et al., 27 Feb 2026).

For chunk $E \in \mathbb{R}^{|I|\times d}$6, the paper writes

$E \in \mathbb{R}^{|I|\times d}$7

and

$E \in \mathbb{R}^{|I|\times d}$8

This enables parallel training while retaining linear complexity in sequence length (Ye et al., 27 Feb 2026).

The reported complexity claims are explicit. For one layer with sequence length $E \in \mathbb{R}^{|I|\times d}$9, embedding dimension $$\mathrm{LMCA}(X) = \mathrm{Concat}(\mathrm{Norm}(Y_{\mathrm{ret}}), \mathrm{Norm}(Y_p), \mathrm{Norm}(Y_t)) \odot (XW_u),$$0, and chunk size $$\mathrm{LMCA}(X) = \mathrm{Concat}(\mathrm{Norm}(Y_{\mathrm{ret}}), \mathrm{Norm}(Y_p), \mathrm{Norm}(Y_t)) \odot (XW_u),$$1:

• Training space complexity: $$\mathrm{LMCA}(X) = \mathrm{Concat}(\mathrm{Norm}(Y_{\mathrm{ret}}), \mathrm{Norm}(Y_p), \mathrm{Norm}(Y_t)) \odot (XW_u),$$2.
• Training time complexity: $$\mathrm{LMCA}(X) = \mathrm{Concat}(\mathrm{Norm}(Y_{\mathrm{ret}}), \mathrm{Norm}(Y_p), \mathrm{Norm}(Y_t)) \odot (XW_u),$$3.
• Inference complexity: $$\mathrm{LMCA}(X) = \mathrm{Concat}(\mathrm{Norm}(Y_{\mathrm{ret}}), \mathrm{Norm}(Y_p), \mathrm{Norm}(Y_t)) \odot (XW_u),$$4 per step in $$\mathrm{LMCA}(X) = \mathrm{Concat}(\mathrm{Norm}(Y_{\mathrm{ret}}), \mathrm{Norm}(Y_p), \mathrm{Norm}(Y_t)) \odot (XW_u),$$5 due to cached recurrent states, with decoding cache of $$\mathrm{LMCA}(X) = \mathrm{Concat}(\mathrm{Norm}(Y_{\mathrm{ret}}), \mathrm{Norm}(Y_p), \mathrm{Norm}(Y_t)) \odot (XW_u),$$6 per head (Ye et al., 27 Feb 2026).

The paper further distinguishes prefill, which computes caches for a full history and is linear in $$\mathrm{LMCA}(X) = \mathrm{Concat}(\mathrm{Norm}(Y_{\mathrm{ret}}), \mathrm{Norm}(Y_p), \mathrm{Norm}(Y_t)) \odot (XW_u),$$7, from decode, which updates the next-step state incrementally and is constant in $$\mathrm{LMCA}(X) = \mathrm{Concat}(\mathrm{Norm}(Y_{\mathrm{ret}}), \mathrm{Norm}(Y_p), \mathrm{Norm}(Y_t)) \odot (XW_u),$$8. This division is important for online recommendation serving, where full-history processing and per-event updates have different latency requirements.

Training uses PyTorch and multi-GPU execution via Accelerate, with RMSNorm and SiLU throughout (Ye et al., 27 Feb 2026). The loss is sampled softmax with $$\mathrm{LMCA}(X) = \mathrm{Concat}(\mathrm{Norm}(Y_{\mathrm{ret}}), \mathrm{Norm}(Y_p), \mathrm{Norm}(Y_t)) \odot (XW_u),$$9 negatives. Given last-step representation $W_u$0, the next-item distribution is

$W_u$1

Training is autoregressive over targets $W_u$2 (Ye et al., 27 Feb 2026).

The reported default training regime uses 100 epochs, with dataset-specific settings. Kuairand-27K and KuaiRec use batch size 128 and learning rate $W_u$3; MovieLens-20M uses batch size 1024 and learning rate $W_u$4. For unstable baselines such as SASRec and Mamba, the paper uses learning rate $W_u$5 and 200 epochs (Ye et al., 27 Feb 2026). Model sizes are 4 layers with $W_u$6 on Kuairand-27K and KuaiRec, and 8 layers with $W_u$7 on MovieLens-20M. The positional dimension is set to $W_u$8, and the temporal channel uses $W_u$9 (Ye et al., 27 Feb 2026).

6. Empirical evaluation, ablations, and scaling behavior

The model is evaluated on MovieLens-20M, Kuairand-27K, and KuaiRec, all of which preserve timestamps for time-aware modeling (Ye et al., 27 Feb 2026). The datasets differ markedly in sequence length:

• MovieLens-20M: 138,493 users, 26,744 items, 20,000,263 interactions, average length 144.41.
• Kuairand-27K: 27,284 users, 131,090 items, 97,010,279 interactions, average length 3555.57.
• KuaiRec: 7,176 users, 9,958 items, 12,529,113 interactions, average length 1745.97. (Ye et al., 27 Feb 2026)

Evaluation uses HR@K, NDCG@K, and MRR with full-item ranking and $\odot$0 (Ye et al., 27 Feb 2026). On the three benchmarks, the paper reports the following top-line results for FuXi-Linear:

• Kuairand-27K: NG@10 0.0609, NG@50 0.0906, HR@10 0.1124, HR@50 0.2488, MRR 0.0540.
• MovieLens-20M: NG@10 0.2131, NG@50 0.2700, HR@10 0.3592, HR@50 0.6161, MRR 0.1830.
• KuaiRec: NG@10 0.1368, NG@50 0.1851, HR@10 0.2242, HR@50 0.4486, MRR 0.1235. (Ye et al., 27 Feb 2026)

For the long-sequence datasets Kuairand-27K and KuaiRec, the reported relative improvements over state-of-the-art baselines average 9.26%, 7.24%, 9.01%, 5.11%, and 8.33% for NG@10, NG@50, HR@10, HR@50, and MRR respectively (Ye et al., 27 Feb 2026).

The efficiency experiments are conducted up to sequence length 8k with batch size 64 for prefill and 1024 for decode, using $\odot$1, $\odot$2, $\odot$3, chunk size $\odot$4, and FlashAttention-2 across models (Ye et al., 27 Feb 2026). At length 8k, the reported speedups are:

• Prefill: 10$\odot$5 versus FuXi-$\odot$6, 3.1$\odot$7 versus FuXi-$\odot$8, 7.8$\odot$9 versus HSTU.
• Decode: 21$Y_{\mathrm{stage1}} = XW_0 + X_0,$0 versus FuXi-$Y_{\mathrm{stage1}} = XW_0 + X_0,$1, 4.2$Y_{\mathrm{stage1}} = XW_0 + X_0,$2 versus FuXi-$Y_{\mathrm{stage1}} = XW_0 + X_0,$3, 18$Y_{\mathrm{stage1}} = XW_0 + X_0,$4 versus HSTU. (Ye et al., 27 Feb 2026)

The ablations directly support the architectural decomposition. On Kuairand-27K, removing TRC reduces NG@10 from 0.0609 to 0.0471, HR@50 from 0.2488 to 0.2122, and MRR from 0.0540 to 0.0424, which is described as the largest drop among channel removals (Ye et al., 27 Feb 2026). Removing the timestamp-only queries and keys degrades NG@10 to 0.0566 and HR@50 to 0.2388, indicating that explicit periodic queries and keys matter. Removing LPC lowers NG@10 to 0.0568 and HR@50 to 0.2385, while removing the semantic retention channel still degrades the model, though less sharply, to NG@10 0.0588 and HR@50 0.2409 (Ye et al., 27 Feb 2026).

The paper also reports direct comparisons for temporal and positional methods on Kuairand-27K. For temporal modeling, FuXi-Linear obtains NG@10 0.0609 at $Y_{\mathrm{stage1}} = XW_0 + X_0,$5, compared with HSTU bias at 0.0538 with $Y_{\mathrm{stage1}} = XW_0 + X_0,$6, absolute sinusoid at 0.0504 with $Y_{\mathrm{stage1}} = XW_0 + X_0,$7, and TiSSD at 0.0468 with $Y_{\mathrm{stage1}} = XW_0 + X_0,$8 (Ye et al., 27 Feb 2026). For positional modeling, FuXi-Linear obtains NG@10 0.0609 at $Y_{\mathrm{stage1}} = XW_0 + X_0,$9, compared with FuXi-$$\widetilde{Y}_{\mathrm{stage1}} = \mathrm{Norm}(Y_{\mathrm{stage1}}),$$0 RPE at 0.0607 with $$\widetilde{Y}_{\mathrm{stage1}} = \mathrm{Norm}(Y_{\mathrm{stage1}}),$$1, RoPE at 0.0575 with $$\widetilde{Y}_{\mathrm{stage1}} = \mathrm{Norm}(Y_{\mathrm{stage1}}),$$2, T5 bias at 0.0580 with $$\widetilde{Y}_{\mathrm{stage1}} = \mathrm{Norm}(Y_{\mathrm{stage1}}),$$3, and Alibi at 0.0574 with $$\widetilde{Y}_{\mathrm{stage1}} = \mathrm{Norm}(Y_{\mathrm{stage1}}),$$4 (Ye et al., 27 Feb 2026). This suggests that the LPC captures much of the utility associated with stronger relative-position schemes without incurring their quadratic cost.

A further result emphasized by the paper is a power-law scaling trend at sequence length $$\widetilde{Y}_{\mathrm{stage1}} = \mathrm{Norm}(Y_{\mathrm{stage1}}),$$5. As non-embedding parameter count $$\widetilde{Y}_{\mathrm{stage1}} = \mathrm{Norm}(Y_{\mathrm{stage1}}),$$6 grows from 188K to 20M, NG@10 increases from 0.0472 to 0.0710 and HR@10 from 0.0881 to 0.1288 (Ye et al., 27 Feb 2026). The paper describes this as a robust power-law scaling property at thousand-length scale, which it argues has been largely unexplored in prior linear recommendation studies.

7. Limitations, deployment implications, and related work

The paper states several limitations. Temporal periodicity choices, including the grid parameters $$\widetilde{Y}_{\mathrm{stage1}} = \mathrm{Norm}(Y_{\mathrm{stage1}}),$$7 and $$\widetilde{Y}_{\mathrm{stage1}} = \mathrm{Norm}(Y_{\mathrm{stage1}}),$$8 and the learned decays $$\widetilde{Y}_{\mathrm{stage1}} = \mathrm{Norm}(Y_{\mathrm{stage1}}),$$9, may struggle with extreme periodicity variance or irregular timestamp noise; adaptivity under domain shift is not tested (Ye et al., 27 Feb 2026). Cold-start users and multi-behavior sequences are not directly addressed, and future work is proposed toward more diverse multi-behavior settings and further efficiency optimization. The LPC is also framed as an approximation to strong relative positional encoding rather than an exact full-resolution RPE, whose exact form would still incur quadratic cost (Ye et al., 27 Feb 2026).

In implementation terms, the model is designed for incremental serving. The paper specifies that each block can cache semantic retention states $10^4$00, positional state $10^4$01, and temporal states $10^4$02, so that each new event $10^4$03 updates the states and computes outputs in $10^4$04 per layer (Ye et al., 27 Feb 2026). It further notes that production deployment can store per-user channel states in a KV store keyed by user ID. This suggests that FuXi-Linear is intended not only as a theoretical complexity improvement but also as an architecture aligned with real-time recommendation systems.

Within the cited related-work landscape, the relevant full-attention baselines include SASRec and HSTU, while the quadratic FuXi-$10^4$05 and FuXi-$10^4$06 variants integrate time and position through quadratic maps (Ye et al., 27 Feb 2026). Linear-attention or recurrent recommendation alternatives discussed in the paper include LinRec, LRURec/RecBLR, Mamba4Rec, SSD4Rec, TTT4Rec, EchoMamba4Rec, and SIGMA. FuXi-Linear’s stated novelty is therefore threefold: timestamp-only periodic retention that eliminates temporal–semantic crosstalk, a linear positional kernel that approximates RPE-like behavior with recurrence, and explicit verification on thousand-length sequences and deeper architectures with substantial speedups and state-of-the-art recommendation quality (Ye et al., 27 Feb 2026).

Taken together, FuXi-Linear occupies a specific position in sequential recommendation research: it is not merely a faster attention substitute, but a structured linear model in which semantic, positional, and temporal dependencies are decomposed into recurrent channels with distinct inductive biases. The empirical evidence in the paper indicates that this decomposition is particularly consequential for long, timestamped interaction histories, where both periodicity and fine positional resolution remain important even under aggressive complexity constraints (Ye et al., 27 Feb 2026).