XicorAttention: Rank Correlation in Transformers

• The paper introduces XicorAttention, which leverages a differentiable version of Chatterjee’s rank correlation coefficient to replace dot-product attention in Transformers.
• XicorAttention employs continuous relaxations via SoftSort and FastSoftRank, achieving up to a 9.1% MAE reduction on time series forecasting benchmarks.
• Integrating XicorAttention into Transformer models provides robust performance improvements for capturing complex nonlinear dependencies, despite moderate computational overhead.

Nonlinear Rank Correlation-Based Attention (XicorAttention) is an attention mechanism constructed on Chatterjee’s rank correlation coefficient $\xi$, designed to replace the standard dot-product attention in Transformers with a differentiable measure of nonlinear dependency between query and key pairs. XicorAttention leverages rank correlation to capture complex relationships in data, particularly for time series forecasting, where nonlinear structure and regime shifts are prevalent and often unaddressed by conventional attention measures. Its key innovation is the replacement of the standard inner-product kernel with a soft (differentiable) version of Chatterjee's coefficient, computed via relaxations of sorting and ranking to ensure gradient flow. Empirically, XicorAttention achieves significant performance improvements on standard forecasting benchmarks, with up to 9.1% MAE reduction over baselines, at a moderate computational overhead (Kimura et al., 3 Jun 2025).

1. Chatterjee’s Rank Correlation Coefficient and Theoretical Foundations

Chatterjee’s rank correlation coefficient $\xi$ is a rank-based statistic for measuring general (including nonlinear) dependence between two variables. For a pair of random variables $(X, Y)$ with joint CDF $F(x, y)$ and marginals $F_X(x)$, $F_Y(y)$, the population correlation is defined as

$$\xi(X, Y) = \frac{\int \operatorname{Var}_X[ p(y \mid X) ] \, dF_Y(y)}{\int \operatorname{Var}[ I\{Y \geq y\} ] \, dF_Y(y)},$$

where $p(y \mid X) := \mathbb{P}(Y \geq y \mid X)$.

Given data $\{(X_i, Y_i): i = 1, \dots, n\}$, the finite-sample estimator under no ties is

$$\xi_n(X, Y) = 1 - \frac{3}{n^2 - 1} \sum_{i=1}^{n-1} \left| r_{i+1} - r_i \right|,$$

where $r_i$ is the rank of $Y_{(i)}$ after sorting $X_{(1)} < \cdots < X_{(n)}$ (Shi et al., 2020, Kimura et al., 3 Jun 2025).

Chatterjee’s $\xi$ equals zero if and only if $X$ and $Y$ are independent. It is a consistent estimator for general dependence and is distribution-free under the null. However, its local power is rate-suboptimal: under weak dependencies ($\Theta(1/\sqrt{n})$), $\xi_n$ does not exhibit power exceeding the nominal level, contrasting with alternatives such as Hoeffding’s $D$, Blum-Kiefer-Rosenblatt’s $R$, and Bergsma-Dassios-Yanagimoto’s $\tau^*$, which are rate-optimal for such alternatives (Shi et al., 2020).

2. Differentiable Relaxations: SoftSort and FastSoftRank

The original computation of $\xi_n$ uses non-differentiable operations (sorting and ranking), which are not compatible with standard neural network backpropagation. XicorAttention resolves this via continuous relaxations:

• SoftSort: Given a vector $\mathbf{q}\in\mathbb{R}^n$, SoftSort computes a doubly-stochastic soft permutation matrix via a row-wise softmax:

$$\widehat{P} = \mathrm{SoftSort}_\tau(\mathbf{q}) = \mathrm{softmax}_{\text{row}}\left( -\frac{d(\operatorname{sort}(\mathbf{q})\,\mathbf{1}^\top,\, \mathbf{1}\,\mathbf{q}^\top)}{\tau} \right),$$

where $d(\cdot,\cdot)$ is a smooth distance (typically $\ell_1$), and $\tau > 0$ is a temperature.

• FastSoftRank: The ranking operator is approximated via strongly convex isotonic regression on the permutahedron. For input $\mathbf{k}$, the soft rank is given by

$$r^\varepsilon_\Phi(\mathbf{k}) = \arg\min_{y \in \text{Perm}(\rho)} \frac12 \| y + \frac{\mathbf{k}}{\varepsilon} \|^2,$$

where $\rho = (n, n-1, \ldots, 1)$, and $\varepsilon$ controls regularization.

Both SoftSort and FastSoftRank are $O(n \log n)$ per sample, ensuring practical applicability for Transformer head dimensions ($d$).

3. XicorAttention Mechanism

XicorAttention reconceptualizes attention weights as nonlinear rank correlations between projected queries and keys:

1. For an input $\mathbf{X} \in \mathbb{R}^{T \times C}$, obtain queries, keys, and values via linear maps, then split into $h$ heads of dimension $d$.
2. For each head, and for every $(i,j)$ pair of query/key positions, consider $\bm{q} = \mathbf{Q}^m_{i,:}$, $\bm{k} = \mathbf{K}^m_{j,:}$ as vector samples of size $n = d$.
3. Apply SoftSort to $\bm{q}$, permute both $\bm{q}$ and $\bm{k}$, and obtain differentiable soft ranks of $\bm{k}$.
4. Compute the relaxed Chatterjee coefficient:

$$\widehat\xi_d(\bm q, \bm k) = 1 - \frac{3}{d^2-1} \sum_{\ell=1}^{d-1} |\widehat r_{\ell+1} - \widehat r_\ell|,$$

yielding an attention score in $[0,1]$.

1. Compile these into an attention matrix $\mathcal{A}^m \in [0,1]^{T \times T}$ and output:

$$\mathrm{XicorAttn}^m(\mathbf{Q}^m, \mathbf{K}^m, \mathbf{V}^m) = \mathcal{A}^m \mathbf{V}^m.$$

Unlike standard attention, no additional softmax is applied.

Pseudocode (single head):

INPUT: Q ∈ R^{T×d}, K ∈ R^{T×d}, V ∈ R^{T×d}, temperature τ, regularizer ε OUTPUT: Y ∈ R^{T×d} for i in 1..T: for j in 1..T: q = Q[i,:] k = K[j,:] # 1) soft-sort q, permute k Psoft = SoftSort_τ(q) # (d×d) q_sorted = Psoft @ q k_sorted = Psoft @ k # 2) soft-rank k_sorted r_soft = FastSoftRank_ε(k_sorted) # vector in R^d # 3) compute soft-xi ξ̂ = 1 - 3/(d^2-1) * sum(|r_soft[ℓ+1] - r_soft[ℓ]| for ℓ in 1..d-1) A[i,j] = ξ̂ Y = A @ V

(Kimura et al., 3 Jun 2025)

4. Transformer Integration and Empirical Evaluation

XicorAttention is integrated into standard time series Transformer architectures by replacing the multi-head self-attention dot-product block. Embedding, positional encoding, feed-forward networks, and output projections remain unaltered. The method is implemented in several state-of-the-art forecasting backbones, including PatchTST, iTransformer, TimeXer, Informer, Autoformer, FEDformer, and vanilla Transformer (Kimura et al., 3 Jun 2025).

Empirical evaluation spans six real-world multivariate time-series datasets (ETTh1/2, ETTm1/2, Exchange, Weather, Electricity, Traffic), using mean absolute error (MAE) and mean squared error (MSE) as metrics. XicorAttention consistently improves forecasting performance, achieving up to 9.1% MAE reduction on the Exchange dataset, with 3–5% average gains across all datasets and lookaheads. It achieves best absolute MAE/MSE on 5 out of 6 benchmarks.

Ablation studies on head dimension $d$ indicate that performance increases until $d \approx 128$, beyond which gains plateau; thus, a per-head dimension of at least 128 is recommended. Main computational overhead arises from SoftSort ($O(d^2)$); empirically, XicorAttention is about 1.5$\times$ slower than the standard attention mechanism for moderate $T$ (Kimura et al., 3 Jun 2025).

5. Computational Complexity, Differentiability, and Comparative Analysis

The standard attention mechanism incurs $O(T^2 d)$ complexity per layer. In contrast, XicorAttention incurs $O(T^2 d^2)$ per head, driven by SoftSort ($O(d^2)$) and FastSoftRank ($O(d \log d)$) operations. End-to-end differentiability is preserved via analytic gradients through both soft permutation and soft ranking steps.

Comparison to other independence measures yields the following:

Statistic	Null Distribution	Local Power	Complexity
$\xi_n$	$N(0, 2/5n)$	Suboptimal	$O(n \log n)$
$D_n$	Non-degenerate	Optimal	$O(n \log n)$
$R_n$	Non-degenerate	Optimal	$O(n \log n)$
$\tau^*_n$	Non-degenerate	Optimal	$O(n \log n)$
$\xi_n^*$	Degenerate (null)	Consistent	$O(n^{5/3})$

Chatterjee's $\xi_n$ is distribution-free under the null but demonstrates low sensitivity to dependencies of order $O(1/\sqrt{n})$. For strong nonlinear signals, as often encountered in time series forecasting, this limitation is less impactful, justifying the use of XicorAttention for moderate to strong dependencies. The earlier Dette–Siburg–Stoimenov kernel estimator $\xi_n^*$ is harder to tune and computationally less favorable (Shi et al., 2020).

6. Practical Recommendations and Limitations

• XicorAttention is suited for scenarios where detection of nonlinear dependence is crucial, and moderate-to-strong signals are expected.
• The primary performance gain derives from the use of Chatterjee’s $\xi$ as a principled, permutation-invariant, and nonlinear similarity, distinguishing it from inner-product attention.
• In regimes where only very weak dependencies ($\sim 1/\sqrt{n}$) are present, alternative U-statistic-based attention mechanisms may be preferred for local power optimality (Shi et al., 2020).
• Recommended head dimensions are $d \ge 128$. Model parameters such as SoftSort temperature ($\tau = 1.0$) and FastSoftRank regularization ($\varepsilon = 0.01$) are used in the original implementation.
• Future optimization may include more efficient differentiable sorting techniques to mitigate the quadratic overhead of SoftSort.

7. Conceptual Impact and Relation to Broader Research

XicorAttention contributes a new hybrid between classical nonparametric dependence measures and modern self-attention architectures. It demonstrates that differentiable rank-based statistics can serve as effective plug-ins for neural attention, enabling detection of a broader spectrum of statistical dependencies. The dual requirements of computational tractability and differentiability are addressed via continuous relaxations, connecting algorithmic developments in statistical dependence testing (Shi et al., 2020) with deep learning advances (Kimura et al., 3 Jun 2025). This approach highlights the possibility of adopting a wide variety of statistical kernels in neural models, subject to constraints on local power and computational efficiency. A plausible implication is the potential for further exploration of rank- or kernel-based attention mechanisms in settings beyond time series forecasting.