Diagonal-Sparse Positional Mechanism

• Diagonal-sparse positional mechanism is a structured sparsification strategy that prunes low-importance diagonals from Toeplitz-structured relative attention matrices.
• It employs a diagonal-sliding block pruning algorithm that partitions the attention matrix into blocks and scores them to identify and remove low-contribution diagonals.
• Empirical results show up to 75% FLOPs reduction with minimal accuracy loss, making it highly effective for long-sequence modeling in Transformer-based systems.

The diagonal-sparse positional mechanism is a structured sparsification strategy for relative positional attention matrices in decoder-only Transformer architectures, introduced in the FuXi-$\gamma$ sequential recommendation framework. Its core function is to prune low-importance diagonals in the learned Toeplitz-structured positional attention, delivering significant reductions in computational complexity and memory overhead while maintaining ranking accuracy. The mechanism systematically exploits the persymmetric (Toeplitz) property of the positional weight matrix, enabling principled and hardware-friendly block pruning for long-sequence modeling (Yi et al., 14 Dec 2025).

1. Mathematical Foundations

Central to the mechanism is the parametrization of the relative positional weight matrix $W_{\rm pos}\in\mathbb{R}^{n\times n}$, for sequence length $n$. The Toeplitz constraint is strictly enforced during training, ensuring

$$W_{\rm pos}^{\,i,j} = W_{\rm pos}^{\,i+m,\,j+m}\ \quad\forall\ i,j,m$$

so that every descending diagonal is constant.

The matrix is partitioned into $B\times B$ non-overlapping blocks of size $s\times s$ (stride size $s$), with $B=n/s$ after appropriate padding and decoder mask application. The pruning process evaluates blockwise importance using the leftmost column: for block $k$ on the leftmost column (row indices $[(k-1)s+1,ks]$, column indices $1$ to $s$), the score is the sum of the absolute values

$$\mathrm{score}_k = \sum_{(i,j)\in\mathcal{B}_k}\bigl|W_{\rm pos}^{\,i,j}\bigr|.$$

A pruning ratio $\tau\in[0,1]$ determines the selection of the lowest-scoring $\lfloor B\tau\rfloor$ diagonals, which are then pruned in all diagonal-aligned blocks, utilizing the uniqueness of each diagonal under the Toeplitz property.

2. Diagonal-Sliding Block Pruning Algorithm

The mechanism operationalizes diagonal sparsification through a diagonal-sliding strategy, which is central to its computational efficiency. The core algorithm is as follows:

1. Block Partitioning: $W_{\rm pos}$ is reshaped so that leftmost column blocks are indexed independently.
2. Scoring: For each block $k$ ($k=0,\ldots,B-1$), compute

$$\mathrm{score}[k] = \sum_{i=1}^s\sum_{j=1}^s \left|\text{leftmost\_blocks}[k,i,j]\right|.$$

1. Pruning Block Selection: Select indices $\mathcal{K}$ of the $q=\lfloor B\tau\rfloor$ least-important leftmost blocks.
2. Diagonal Extension: For each $k\in\mathcal{K}$, generate block indices along the corresponding diagonal by iterating with stride $B+1$ until the end of the $B\times B$ grid, and collect these in the mask set $M$.
3. Masking: At inference or training, all masked block positions have their $W_{\rm pos}$ entries zeroed:

$$W_{\rm pos}^{(\text{sparse})}[b] = \begin{cases} 0,& \text{if block-idx }b\in M\ W_{\rm pos}[b],&\text{otherwise.} \end{cases}$$

This operation results in structured removal of low-contribution diagonals, directly reducing both memory access and compute complexity.

3. Computational Complexity and Efficiency

Dense positional attention requires $O(n^2)$ operations. With diagonal-sparse pruning at pruning ratio $\tau$, only $(1-\tau)B^2$ blocks (each of size $s^2$; $B=n/s$) are retained, yielding attention cost

$O((1-\tau)n^2).$

Empirical results confirm that, for $\tau$ in $[0.4,0.6]$, attention FLOPs are reduced by 70–75\% (e.g., 74.56\% on ML-20M), with corresponding reductions in memory bandwidth and intermediate storage. This enables end-to-end inference speedups up to $6.18\times$ on long-sequence recommendation tasks (Yi et al., 14 Dec 2025).

4. Empirical Evaluation and Ablation Studies

Extensive evaluations across public and industrial benchmarks support the practical robustness and utility of the mechanism:

• Ablation: Removing the entire positional channel (not just pruning) yields noticeable drops in HR@10 and NDCG@10 (e.g., –1.41\% and –1.10\% respectively on ML-1M).
• Pruning ratio $\tau$: At $\tau=0.6$ on ML-20M, NDCG@10 retains 98.92\% of the dense baseline, while FLOPs decrease by 74.56\%. On KuaiRand, mild diagonal sparsity slightly increases NDCG@10 (+0.2\%), with 70.7\% lower FLOPs. On an industrial-scale dataset, $\tau=0.6$ induces $<0.1\%$ accuracy degradation.
• Stride size $s$: On ML-20M, $s=8$ yields an optimal trade-off: HR@10=0.3576, NDCG@10=0.2129 at 33.3\% density. Very small $s$ may reduce hardware efficiency, while large $s$ blunts pruning selectivity.
• Sequence length: As $n$ increases, the method becomes more forgiving due to increasing redundancy among diagonals, permitting greater sparsity without performance penalty.

Table: Key Experimental Outcomes

Setting	Metric Change	FLOPs Reduction
Ablation (ML-1M)	HR@10 –1.41%, NDCG@10 –1.10%	N/A
$\tau = 0.6$ (ML-20M)	NDCG@10 retains 98.92%	74.56%
$s=8$ (ML-20M)	HR@10=0.3576, NDCG@10=0.2129	66.7% sparsity
Industrial dataset	$<0.1\%$ drop at $\tau=0.6$	$>70\%$

5. Practical Guidelines and Operational Limitations

Hyperparameter Selection

• Stride size $s$: Recommended values are $s\in\{4,\,8,\,16\}$, with $s=8$ providing a favorable balance between parallelism and pruning precision.
• Pruning ratio $\tau$: Values in $[0.4,\,0.6]$ preserve $>98\%$ of dense accuracy at $70$-$75\%$ block sparsity. Smaller $\tau$ is advised for short sequences ($n<200$); larger $\tau$ for long sequences ($n>500$) due to increased redundancy.

Operational Guidance

• Pruning is post-hoc; $\tau$ may be tuned via A/B validation after model training.
• Always re-score $W_{\rm pos}$ after training before pruning. The correct Toeplitz structure must be strictly enforced during this process.

Limitations

• Block Granularity: Semi-structured, block-level pruning may underutilize hardware for very small $s$ and may miss fine-grained dependencies within blocks.
• Toeplitz Violation: If $W_{\rm pos}$ deviates from the Toeplitz constraint, the leftmost-column scoring misidentifies important diagonals. Enforcement of the constraint is therefore essential.
• Task Structure: Highly non-stationary tasks with off-diagonal relative position patterns may not be well captured, as the method assumes diagonal significance.

6. Context and Significance in Transformer-Based Systems

The diagonal-sparse positional mechanism represents a systematic approach to sparsifying structured attention without heavy reliance on heuristic pruning or blockwise regularization. It is particularly suitable in scenarios where sequence lengths are large, and hardware throughput is critical. By leveraging the learned Toeplitz structure, the mechanism aligns algorithmic efficiency with matrix-theoretic properties, enabling scalable deployment of Transformer architectures in recommendation and similar long-sequence domains with minimal loss of modeling efficacy (Yi et al., 14 Dec 2025).