Ask Self, Ask Others: Relation Is All You Need

Abstract: Attention directly derives normalized information flow from pairwise scores. We introduce Relation, an alternative token-mixing primitive that first organizes pairwise evidence into explicit Self and Exchange relations and derives information flow afterward. This relational organization gives rise to Full Relation, FlashRelation, Linear Relation, Hybrid Relation, and a KV-style Relation Cache. Across matched decoder-only models at approximately 10M, 30M, and 100M parameters, Full Relation achieves lower final validation NLL than MHA at all three scales. In a fixed-context reference benchmark, FlashRelation is 3.60-4.41x faster than the materialized Full Relation implementation. Across scale-matched production workloads, it reaches 76.4-84.9% of PyTorch FlashAttention throughput while executing the Full Relation operator. Hybrid Relation uses 75% Linear Relation layers and achieves strong language-modeling quality. These results support a relation-first view of token mixing: ask Self, ask Others, then let Flow follow Relation.

• The paper introduces Relation, a token-mixing method that separates pairwise evidence into Self and Exchange roles before normalizing information flow, yielding a two-stage process that decides whether to use history and where to retrieve it.
• The paper reports lower validation NLL than matched multi-head attention at 10M, 30M, and 100M parameters, with improvements of 0.0412, 0.0151, and 0.0310 respectively, although results show higher small-model variance and mixed downstream-task performance.
• The paper derives FlashRelation, Linear Relation, Hybrid Relation, and Relation Cache implementations, including exact tiled execution with 3.60–4.41× speedups over materialized Relation and recurrent variants that reduce history storage to an O(d²/H) decode state.

Overview

This paper proposes Relation, a token-mixing primitive that reorganizes how information flow is computed in decoder-only Transformers. Rather than normalizing raw pairwise compatibility scores directly into attention weights, Relation first organizes pairwise evidence into two explicit structural roles — Self (a token's relation to itself) and Exchange (its relation to other tokens) — and derives normalized flow only afterward. The paper develops this idea into a family of operators: Full Relation, FlashRelation (a tiled, FlashAttention-style execution), Linear Relation (a recurrent-state variant), Hybrid Relation (an interleaved composition), and a KV-style Relation Cache for autoregressive decoding. The central claim is that relation formation should be a distinct stage preceding flow allocation — "Flow follows Relation" — in contrast to canonical scaled dot-product attention, where a single score $U_{ij}$ simultaneously encodes relational structure and determines flow (2608.20172).

The Self–Exchange Relation operator

The basic Self–Exchange Relation (SER) projects the input into two relation spaces $P_1, P_2$ and an information space $I$, computes pairwise evidence $U_{ij} = p_{1,i}^\top p_{2,j}/\sqrt{d_h}$ (with RoPE on $P_1$ and $P_2$ but not $I$), and then maps evidence into role-specific entries: a bounded sigmoid Self entry $S_i = \sigma(U_{ii}/\tau_S)$ on the diagonal and SiLU Exchange entries $E_{ij}$ over the causal history, with a learnable layer-wise count correction $-\lambda_\ell \log i$. The causal Relation matrix $P_1, P_2$0 assembled from these entries is normalized into flow $P_1, P_2$1, which transports the (Givens-mixed, unrotated) information states. Multi-Head Relation (MHR) extends SER across heads, mixing information states across adjacent head pairs with learnable Givens rotations whose pairing pattern alternates between layers.

The paper derives an exact factorization of the resulting flow. The Self–Exchange mass split is governed by the gap $P_1, P_2$2, where $P_1, P_2$3: the Self mass is $P_1, P_2$4 and the aggregate Exchange mass is $P_1, P_2$5, summing exactly to one. Conditional on Exchange, the count correction cancels in pairwise differences, so allocation within history depends entirely on relative Exchange relations. This yields a two-level semantics — whether to consult history, then where in history to look — and, critically, an algebraic identity $P_1, P_2$6 that separates historical aggregation from final allocation.

Practical realizations

The factorization directly motivates FlashRelation, an exact tiled execution in the style of FlashAttention (2608.20172). A causal tiled scan maintains running maximum, normalizer, and information accumulator over historical Exchange entries only; the Self–Exchange allocation is completed after the scan. FlashRelation computes exactly the Full Relation output without materializing $P_1, P_2$7 matrices.

Linear Relation compresses historical Others into a recurrent matrix state $P_1, P_2$8 with input-dependent, KDA-style channel-wise retention. The current Self relation $P_1, P_2$9 modulates both the read from history and the write into the state, with the read strictly preceding the write to preserve causal support. The paper summarizes this as "Ask Self → Ask Others → Answer → Become Others." Linear Relation achieves $I$0 token-mixing cost and an $I$1 decode state, versus $I$2 history for Full Relation. Hybrid Relation interleaves nine Linear and three Full Relation layers in an $I$3 layout, mirroring the hybrid design of Kimi Linear. Finally, the paper derives an exact Relation Cache: during decoding, only $I$4 states and information states need be cached (the current token's $I$5 is never reused), giving an $I$6 projected cache analogous to KV caching.

Empirical results

Language-modeling comparisons use matched decoder-only models at approximately 10M, 30M, and 100M parameters with paired seeds 42–44, TinyStories (10M, 30M) and SmolLM-corpus (100M) data, and final-checkpoint validation NLL as the primary metric. Full Relation achieves lower mean final NLL than MHA at all three scales:

Scale	MHA NLL	Relation NLL	Δ
10M	1.6853 ± 0.0042	1.6441 ± 0.0124	−0.0412
30M	1.3001 ± 0.0044	1.2850 ± 0.0136	−0.0151
100M	2.9373 ± 0.0093	2.9063 ± 0.0061	−0.0310

Per-seed results show eight wins out of nine paired comparisons, with the single loss at 30M seed 42 (+0.0055). Notably, the Relation models exhibit higher seed variance at 10M and 30M, and the 30M margin is the smallest — the consistency claim rests more on the 10M and 100M results. Secondary evaluations are mixed: Relation wins BLiMP at 10M and 30M but loses at 100M (−0.0081), and results on ARC, OpenBookQA, HellaSwag, PIQA, and LAMBADA are largely within noise, with some regressions (e.g., 30M LAMBADA test perplexity worsens by ~70 points).

Structural ablations at 10M show that removing count calibration ($I$7, +0.0506) and collapsing to a single head (+0.0507) are the most damaging; Exchange-only transport (+0.0320) and Raw-$I$8 communication (+0.0366) also hurt, while removing Givens mixing costs only +0.0032, indicating the rotations contribute marginally at this scale.

On systems benchmarks (RTX 5090, BF16), FlashRelation achieves 3.60–4.41× speedup over a materialized Full Relation reference at $I$9, and reaches 76.4–84.9% of PyTorch FlashAttention throughput on scale-matched production workloads while executing the exact Full Relation operator. Hybrid Relation with 75% Linear Relation layers attains 1.2780 ± 0.0050 NLL under a 30M-class setting — better than the 30M MHA baseline, though on a non-identical geometry, which the paper explicitly flags.

Limitations and open questions

The paper is candid about scope. All experiments cover decoder-only LLMs up to ~100M parameters and at most 1.071B training tokens; behavior at substantially larger scales, in multimodal settings, and under post-training is untested. Linear and Hybrid Relation are evaluated only in selected configurations, and the Relation Cache is established mathematically rather than validated in a serving system. FlashRelation's throughput gap to FlashAttention (76–85%) leaves open whether the overhead is intrinsic to the Self–Exchange construction or an artifact of kernel maturity. The higher seed variance of Relation models at smaller scales, and whether the NLL advantage persists or grows with scale, are empirical questions the paper does not resolve. The fixed Self temperature $U_{ij} = p_{1,i}^\top p_{2,j}/\sqrt{d_h}$0 is acknowledged as not theoretically unique.

Conclusion

The paper presents Relation as a coherent reorganization of token mixing in which pairwise evidence is explicitly structured into Self and Exchange roles before normalization, with information flow derived afterward. The exact Self–Exchange factorization is the load-bearing result: it yields a principled tiled implementation, a recurrent-state variant, and a KV-style cache within a single framework. Full Relation's consistent NLL advantage over matched MHA at three scales, together with FlashRelation's practical throughput and Hybrid Relation's quality with 75% linear layers, supports Relation as a viable alternative primitive — with the caveat that all evidence comes from small-scale, pre-training-only experiments, and the central open question is whether the relational organization survives at production scale.