Distance Attention Projection

• Distance attention projection is a conceptual umbrella that integrates explicit distance measures with projection operations to compute similarities and guide attention mechanisms.
• It includes both projection-free methods, such as Gaussian kernel attention, and projection-based techniques that use distance masks or biases to modify standard Transformer projections.
• Empirical evaluations show that these methods can balance parameter efficiency with performance improvements in tasks like image classification, language modeling, and 360° depth estimation.

Distance attention projection denotes a family of mechanisms in which an attention or distance operator is organized by an explicit notion of distance together with an explicit notion of projection. The expression is not a single standardized method name in the cited literature; instead, closely related constructions appear in several forms: projection-free Gaussian kernel attention over token-feature distances in Transformers, distance-masked self-attention over projected features in human-object interaction detection, relative-distance masking in sentence encoding, semantic- and distance-aware cross-projection fusion for 360° depth estimation, and attention-like weighting over projection directions in sliced-Wasserstein methods (Kundu et al., 4 May 2026, Wang et al., 2022, Im et al., 2017, Ai et al., 2024, Zhang et al., 2024).

1. Terminological scope and core design axes

In the cited work, the same three words—distance, attention, and projection—refer to different technical objects. In some models, distance is Euclidean distance in feature space; in others it is image-space center distance, token-index distance, 3D Cartesian coordinate distance on the sphere, or Wasserstein discrepancy after projection. Likewise, projection may mean Transformer query/key/value projections, geometric map projections, ERP-to-ICOSAP cross-projection fusion, or projection directions on a sphere or Stiefel manifold. This suggests that “distance attention projection” is best understood as a conceptual umbrella rather than a single canonical architecture (Kundu et al., 4 May 2026, Wang et al., 2022, Ai et al., 2024, Zhang et al., 2024).

Research line	Distance notion	Projection notion
Gaussian Kernel Attention	Euclidean distance between per-head token features	Removes $W_Q,W_K,W_V$; keeps $W_O$
Far-Near Distance Attention / DSAN	Bounding-box center distance; token-index distance	Standard attention projections remain
Elite360D B2F	3D coordinate distance and feature-space distance	Fusion between ERP and ICOSAP projections
DSSW / amortized v-DSW	Projected Wasserstein discrepancy	Projection directions or projection distributions

A central distinction runs through the literature. One family uses distance to replace projected dot-product similarity; another uses distance to constrain or bias an otherwise standard projected attention mechanism. A third family uses projection not inside attention blocks but as the object over which a distance itself is defined and adaptively weighted. That distinction is essential for avoiding category errors when comparing methods.

2. Projection-free distance attention in Transformers

The clearest instance of distance attention replacing learned attention projections is "Projection-Free Transformers via Gaussian Kernel Attention" (Kundu et al., 4 May 2026). The method starts from the standard self-attention baseline

$$\mathrm{Attn}(X)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,\qquad Q=XW_Q,\;K=XW_K,\;V=XW_V,$$

and asks whether the learned input projections $W_Q,W_K,W_V$ are necessary for effective token mixing. Gaussian Kernel Attention (GKA) answers this by computing affinities directly from per-head token features $\tilde{x}^{(h)}_i$ using a Gaussian radial basis function:

$$K^{(h)}_{ij}=\exp\!\left(-\frac{\|\tilde{x}^{(h)}_i-\tilde{x}^{(h)}_j\|_2^2}{2\sigma_h^2}\right),$$

with row normalization and head output

$Y_i^{(h)}=\sum_j W_{ij}^{(h)}\,\tilde{x}_j^{(h)}.$

The final multi-head output is

$$\mathrm{GKA}(\tilde{X})=\mathrm{Concat}_{h=1}^H(Y^{(h)})\,W_O.$$

The defining architectural claim is explicit: no $W_Q,W_K,W_V$ are learned; only $\{\ell_h\}_{h=1}^H$ and $W_O$0 remain, with $W_O$1. “Projection-free” therefore means free of the canonical input-side attention projections, not free of linear maps altogether, because the output projection $W_O$2 is retained for compatibility with the standard Transformer interface. In vision, the per-head features are obtained by reshaping normalized tokens, such as pre-attention LayerNorm output, into $W_O$3 heads of width $W_O$4. In language modeling, the implementation applies RoPE and per-head normalization consistent with nanochat’s QK normalization, then computes the Gaussian kernel on those features.

The method is interpretable as normalized kernel regression over tokens, Gaussian kernel smoothing, and one step of feature-space diffusion on a fully connected similarity graph. The row-stochastic normalized kernel matrix gives each head an explicit locality scale: smaller $W_O$5 yields sharper, more localized attention, whereas larger $W_O$6 produces broader, more uniform attention. A practical consequence noted in the supplementary material is that self-attention on the diagonal can dominate when $W_O$7 is small, making some heads nearly identity-like in early layers.

Masking is applied directly to kernel affinities before normalization. For autoregressive language modeling, $W_O$8 enforces causal masking $W_O$9; an optional sliding-window constraint additionally requires $$\mathrm{Attn}(X)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,\qquad Q=XW_Q,\;K=XW_K,\;V=XW_V,$$0. After masking, the weights are renormalized over only the allowed positions. This preserves a proper normalized attention distribution over the valid context set.

Empirically, the paper frames GKA as an efficiency–accuracy trade-off rather than a strict replacement for projected attention. On ImageNet-1K with DeiT-style ViTs, GKA reduces total parameters by roughly 23–25% and FLOPs by 21–24% across Tiny, Small, and Base variants, while attention-module parameters drop by about 75%. Accuracy drops modestly: GKA-Ti obtains 71.54% versus 72.2% for DeiT-Ti, GKA-S obtains 78.23% versus 79.8%, and GKA-B obtains 80.3% versus 81.8%. The ViT-Ti projection ablation is especially diagnostic: standard DeiT-Ti with $$\mathrm{Attn}(X)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,\qquad Q=XW_Q,\;K=XW_K,\;V=XW_V,$$1 has 5.72M parameters, 2.51G FLOPs, and 72.2% top-1; VLT-Ti with $$\mathrm{Attn}(X)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,\qquad Q=XW_Q,\;K=XW_K,\;V=XW_V,$$2 only has 5.28M parameters, 2.33G FLOPs, and 73.5% top-1; GKA-Ti with $$\mathrm{Attn}(X)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,\qquad Q=XW_Q,\;K=XW_K,\;V=XW_V,$$3 only has 4.38M parameters, 1.98G FLOPs, and 71.54% top-1. In a depth-20 LLM at sequence length 2048, the baseline has 896M parameters, $$\mathrm{Attn}(X)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,\qquad Q=XW_Q,\;K=XW_K,\;V=XW_V,$$4 FLOPs/token, and $$\mathrm{Attn}(X)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,\qquad Q=XW_Q,\;K=XW_K,\;V=XW_V,$$5 total train FLOPs, whereas GKA has 378M parameters, $$\mathrm{Attn}(X)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,\qquad Q=XW_Q,\;K=XW_K,\;V=XW_V,$$6 FLOPs/token, and $$\mathrm{Attn}(X)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,\qquad Q=XW_Q,\;K=XW_K,\;V=XW_V,$$7 total train FLOPs; the corresponding BPB values are 0.7800 train and 0.7884 val for the baseline, versus 0.818256 train and 0.819080 val for GKA. The measured near-zero train-validation gap suggests reduced overfitting and/or under-training at that scale, but the paper is clear that language-model quality remains below the projected baseline at the reported compute point (Kundu et al., 4 May 2026).

3. Distance-conditioned attention with learned projections

A second family retains standard attention projections but uses distance to constrain or bias which projected compatibilities matter. In "Distance Matters in Human-Object Interaction Detection" (Wang et al., 2022), the relevant component is Far-Near Distance Attention (FNDA) inside the Spatially Differentiated Transformer. The model first obtains DETR-based human and object tokens, enriches them by Intra-Class Diversification and spatial fusion, and then applies self-attention masked by distance-based far/near partitions. The distance is the $$\mathrm{Attn}(X)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,\qquad Q=XW_Q,\;K=XW_K,\;V=XW_V,$$8 distance between bounding-box centers; the split is adaptive per token using the median distance of that token to all others. For token $$\mathrm{Attn}(X)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,\qquad Q=XW_Q,\;K=XW_K,\;V=XW_V,$$9, tokens farther than $W_Q,W_K,W_V$0 are “far,” the remainder are “near,” and the diagonal is always unmasked so that each token can attend to itself.

FNDA does not replace attention projections. Queries, keys, and values are still formed by learnable matrices $W_Q,W_K,W_V$1, but the attention map is masked by either the far mask or the near mask. The paper states that FNDA allows one token to propagate information with two glances: the first glance involves only far away tokens while the second glance focuses solely on near tokens. These masks are alternately applied across token encoder blocks, creating a form of distance-conditioned message passing over detected instance tokens. The design aim is to avoid competition between local clutter and genuinely informative long-range partners in crowded HOI scenes.

The optimization side is complemented by a Distance-Aware loss, in which the per-pair weight is $W_Q,W_K,W_V$2. This increases the relative influence of distant interactions during learning and addresses the long-tail distribution over distance. The paper explicitly combines DA loss with focal loss in full-model training.

The empirical gains are reported as state-of-the-art results on HICO-DET and V-COCO. On HICO-DET with ResNet-50, SDT reaches 32.45 mAP under the default setting and 35.95 mAP under the known-object setting, compared with UPT’s 31.66 and 35.05; with ResNet-101, the results are 32.97 and 36.32. On V-COCO, SDT with ResNet-50 obtains 60.3/65.7 under scenarios 1/2, and with ResNet-101 reaches 61.8/67.6. The FNDA ablation is more specific to the distance-attention mechanism: plain MHSA yields 30.66 full / 26.06 rare / 32.03 non-rare on HICO-DET, while FNDA yields 31.70 / 26.26 / 33.32; with spatial fusion and ICD, MHSA + spatial fusion + ICD gives 30.90 / 26.74 / 32.14, whereas FNDA + spatial fusion + ICD gives 32.45 / 28.09 / 33.75.

An older but conceptually related instance appears in "Distance-based Self-Attention Network for Natural Language Inference" (Im et al., 2017). DSAN uses standard Transformer-style multi-head projections $W_Q,W_K,W_V$3, but modifies the logits by adding two masks before softmax: a directional mask $W_Q,W_K,W_V$4 and a distance mask $W_Q,W_K,W_V$5 whose $W_Q,W_K,W_V$6-entry is $W_Q,W_K,W_V$7. The directional mask gives a hard forward/backward constraint; the distance mask gives a soft locality bias, scaled by $W_Q,W_K,W_V$8, with $W_Q,W_K,W_V$9 heads. Distance therefore affects the normalized attention distribution without altering the Q/K/V projections themselves. On SNLI, the model without the distance mask reaches 86.0 test accuracy and the Distance-based Self-Attention Network reaches 86.3. The paper stresses that the benefit is clearer on longer sentences: using SNLI-best hyperparameters on MultiNLI, DiSAN obtains 71.0 / 71.4 (matched / mismatched), whereas Distance-based SAN obtains 74.1 / 72.9. The mechanism remains quadratic in sequence length and uses a fixed hand-designed relative bias, but it demonstrates that distance can act as a strong inductive prior even when standard projections are preserved.

Taken together, FNDA and DSAN show a consistent pattern: distance may operate as a hard mask, a soft logit penalty, or an objective reweighting term, while projected semantic compatibility remains intact. This is distinct from projection-free kernels such as GKA, where distance defines similarity geometry directly.

4. Cross-projection distance fusion in 360° vision

"Elite360D: Towards Efficient 360 Depth Estimation via Semantic- and Distance-Aware Bi-Projection Fusion" (Ai et al., 2024) uses the term in yet another sense: distance-aware attention across projection domains. The framework inputs an ERP panorama and an ICOSAP point set. ERP provides the prediction space and compatibility with perspective-pretrained backbones; ICOSAP is described as undistorted and spatially continuous, and is used to provide global context. The architecture has three components—ERP image encoder, ICOSAP point encoder, and Bi-projection Bi-attention Fusion (B2F)—followed by a decoder with upsampling and skip connections.

The ICOSAP representation is a discrete point set. At subdivision level $\tilde{x}^{(h)}_i$0, the paper states that ICOSAP has $\tilde{x}^{(h)}_i$1 faces and $\tilde{x}^{(h)}_i$2 vertices. The center point of each face is used as a representative point, and each point stores $\tilde{x}^{(h)}_i$3. The default settings are $\tilde{x}^{(h)}_i$4 and final point feature number $\tilde{x}^{(h)}_i$5. The ICOSAP encoder is a Point Transformer encoder with three down-sample blocks. The ERP encoder can be a standard 2D backbone such as ResNet, EfficientNet, Swin Transformer, or DilateFormer.

The B2F module lets each ERP pixel feature attend to the entire ICOSAP feature set through two branches. The semantic-aware branch is standard cross-attention, producing $\tilde{x}^{(h)}_i$6 from ERP queries and ICOSAP keys/values. The distance-aware branch is more distinctive. For ERP pixel $\tilde{x}^{(h)}_i$7, it computes a spatial distance embedding

$\tilde{x}^{(h)}_i$8

where the coordinate-wise distances are taken between the ERP pixel’s 3D spherical location and each ICOSAP point coordinate. It also computes a semantic distance embedding

$\tilde{x}^{(h)}_i$9

These are summed, reduced over channels, normalized with softmax, and used to aggregate ICOSAP values into $$K^{(h)}_{ij}=\exp\!\left(-\frac{\|\tilde{x}^{(h)}_i-\tilde{x}^{(h)}_j\|_2^2}{2\sigma_h^2}\right),$$0. Gated fusion then combines the two outputs:

$$K^{(h)}_{ij}=\exp\!\left(-\frac{\|\tilde{x}^{(h)}_i-\tilde{x}^{(h)}_j\|_2^2}{2\sigma_h^2}\right),$$1

The result is a local-with-global ERP-format feature map.

The empirical claim is that explicit distance-aware cross-projection fusion is stronger than semantic-only fusion and substantially lighter than prior ERP+CP/TP systems. On Matterport3D with ResNet-34, the ERP baseline has 24.50M parameters and 59.27 GFLOPs, with Abs Rel 0.1255, Sq Rel 0.1048, RMSE 0.5173, and $$K^{(h)}_{ij}=\exp\!\left(-\frac{\|\tilde{x}^{(h)}_i-\tilde{x}^{(h)}_j\|_2^2}{2\sigma_h^2}\right),$$2; Elite360D has 25.54M parameters and 65.29 GFLOPs, with Abs Rel 0.1115, Sq Rel 0.0914, RMSE 0.4875, and $$K^{(h)}_{ij}=\exp\!\left(-\frac{\|\tilde{x}^{(h)}_i-\tilde{x}^{(h)}_j\|_2^2}{2\sigma_h^2}\right),$$3. The B2F ablation is especially relevant: Only SA gives Abs Rel 0.1204, RMSE 0.5121, and $$K^{(h)}_{ij}=\exp\!\left(-\frac{\|\tilde{x}^{(h)}_i-\tilde{x}^{(h)}_j\|_2^2}{2\sigma_h^2}\right),$$4; Only DA gives 0.1184, 0.4944, and 87.06; full B2F gives 0.1115, 0.4875, and 88.15. A projection-choice ablation on Matterport3D with ResNet-18 reports ERP-CP at 25.66M parameters, 54.15G FLOPs, Abs Rel 0.1369, RMSE 0.5401; ERP-TP (N=18) at 25.66M, 50.58G, 0.1328, 0.5385; and ERP-ICOSAP at 15.43M, 45.91G, 0.1272, 0.5270. Here “projection” is neither Q/K/V projection nor projection-free removal of them; it is the interaction of two geometric representations of the same spherical scene.

5. Projection attention in Wasserstein-type distances

In optimal-transport settings, distance attention projection often refers not to token mixing but to the adaptive selection or weighting of projection directions. "Self-Attention Amortized Distributional Projection Optimization for Sliced Wasserstein Point-Cloud Reconstruction" (Nguyen et al., 2023) uses self-attention as a symmetric, permutation-invariant encoder of two point clouds. The encoder does not model pairwise transport plans directly and does not use cross-attention over all support pairs. Instead, it separately encodes each set, pools the resulting descriptors, sums them symmetrically, and normalizes the result to produce either a projection direction $$K^{(h)}_{ij}=\exp\!\left(-\frac{\|\tilde{x}^{(h)}_i-\tilde{x}^{(h)}_j\|_2^2}{2\sigma_h^2}\right),$$5 for Max-SW-style amortization or, in the proposed method, the location parameter $$K^{(h)}_{ij}=\exp\!\left(-\frac{\|\tilde{x}^{(h)}_i-\tilde{x}^{(h)}_j\|_2^2}{2\sigma_h^2}\right),$$6 of a von Mises–Fisher projection distribution for v-DSW. The paper’s main theoretical point is that a non-optimal single projected Wasserstein distance is only a pseudo metric, whereas non-optimal v-DSW with any $$K^{(h)}_{ij}=\exp\!\left(-\frac{\|\tilde{x}^{(h)}_i-\tilde{x}^{(h)}_j\|_2^2}{2\sigma_h^2}\right),$$7 and $$K^{(h)}_{ij}=\exp\!\left(-\frac{\|\tilde{x}^{(h)}_i-\tilde{x}^{(h)}_j\|_2^2}{2\sigma_h^2}\right),$$8 remains a valid metric. Efficient attention variants make the amortizer linear in the number of supports. Empirically, on ModelNet40 reconstruction, non-amortized v-DSW at $$K^{(h)}_{ij}=\exp\!\left(-\frac{\|\tilde{x}^{(h)}_i-\tilde{x}^{(h)}_j\|_2^2}{2\sigma_h^2}\right),$$9 costs 633 time units, whereas the best attention-amortized v-DSW variants are around 123–125 and achieve lower EMD; the $Y_i^{(h)}=\sum_j W_{ij}^{(h)}\,\tilde{x}_j^{(h)}.$0 ablation reports that $Y_i^{(h)}=\sum_j W_{ij}^{(h)}\,\tilde{x}_j^{(h)}.$1 works best among $Y_i^{(h)}=\sum_j W_{ij}^{(h)}\,\tilde{x}_j^{(h)}.$2.

"Towards Better Spherical Sliced-Wasserstein Distance Learning with Data-Adaptive Discriminative Projection Direction" (Zhang et al., 2024) makes the attention interpretation explicit at the level of projection directions. Standard SSW averages circular projected Wasserstein discrepancies uniformly over $Y_i^{(h)}=\sum_j W_{ij}^{(h)}\,\tilde{x}_j^{(h)}.$3. DSSW replaces that uniform average with a normalized weighted average:

$Y_i^{(h)}=\sum_j W_{ij}^{(h)}\,\tilde{x}_j^{(h)}.$4

In the non-parametric version, $Y_i^{(h)}=\sum_j W_{ij}^{(h)}\,\tilde{x}_j^{(h)}.$5 is the projected Wasserstein discrepancy itself and $Y_i^{(h)}=\sum_j W_{ij}^{(h)}\,\tilde{x}_j^{(h)}.$6 is chosen as $Y_i^{(h)}=\sum_j W_{ij}^{(h)}\,\tilde{x}_j^{(h)}.$7, $Y_i^{(h)}=\sum_j W_{ij}^{(h)}\,\tilde{x}_j^{(h)}.$8, or $Y_i^{(h)}=\sum_j W_{ij}^{(h)}\,\tilde{x}_j^{(h)}.$9. In the parametric version, $$\mathrm{GKA}(\tilde{X})=\mathrm{Concat}_{h=1}^H(Y^{(h)})\,W_O.$$0 is a learned neural network over projected samples, and one variant uses an internal attention mechanism. The normalized weights satisfy $$\mathrm{GKA}(\tilde{X})=\mathrm{Concat}_{h=1}^H(Y^{(h)})\,W_O.$$1 and $$\mathrm{GKA}(\tilde{X})=\mathrm{Concat}_{h=1}^H(Y^{(h)})\,W_O.$$2. The paper proves positivity and symmetry but does not establish identity of indiscernibles. This suggests a soft attention distribution over projection directions rather than a metric projection search. Empirically, DSSW variants improve on SSW and S3W across several tasks; for gradient flows, ARI-S3W (30) reports NLL $$\mathrm{GKA}(\tilde{X})=\mathrm{Concat}_{h=1}^H(Y^{(h)})\,W_O.$$3 and $$\mathrm{GKA}(\tilde{X})=\mathrm{Concat}_{h=1}^H(Y^{(h)})\,W_O.$$4, whereas DSSW (attention) reports NLL $$\mathrm{GKA}(\tilde{X})=\mathrm{Concat}_{h=1}^H(Y^{(h)})\,W_O.$$5 and $$\mathrm{GKA}(\tilde{X})=\mathrm{Concat}_{h=1}^H(Y^{(h)})\,W_O.$$6. On the Earthquake density-estimation task, ARI-S3W gives 0.78 test NLL, DSSW (exp) gives 0.70, DSSW (linear) gives 0.69, and DSSW (attention) gives 0.70.

These works use “projection attention” in a mathematically different way from Transformer attention. The attended objects are sampled directions or projection distributions rather than tokens, and the output is a weighted discrepancy estimate rather than a contextualized sequence. The shared principle is pair-dependent emphasis: some projections receive more mass because they are more discriminative for the distributions being compared.

6. Distinctions, misconceptions, and adjacent meanings

A common misconception is that any distance-aware attention is projection-free. The literature shows the opposite. GKA removes $$\mathrm{GKA}(\tilde{X})=\mathrm{Concat}_{h=1}^H(Y^{(h)})\,W_O.$$7 and keeps only $$\mathrm{GKA}(\tilde{X})=\mathrm{Concat}_{h=1}^H(Y^{(h)})\,W_O.$$8, so distance defines similarity directly (Kundu et al., 4 May 2026). FNDA and DSAN keep standard projected attention and use distance as a mask or bias (Wang et al., 2022, Im et al., 2017). Elite360D uses distance-aware cross-attention across two geometric projections of the same 360° scene (Ai et al., 2024). DSSW and amortized v-DSW attend over projection directions rather than over tokens (Zhang et al., 2024, Nguyen et al., 2023).

A second misconception is that “projection” always means a learned linear map. In adjacent areas it instead denotes a geometric or variational transformation. "Two-point Equidistant Projection and Degree-of-interest Filtering for Smooth Exploration of Geo-referenced Networks" uses a distance-preserving map projection defined with respect to the start and end points of a transition, together with Degree-of-Interest filtering that selects which vertices deserve screen space. The preliminary study analyzed 72 participants across four distance intervals from $$\mathrm{GKA}(\tilde{X})=\mathrm{Concat}_{h=1}^H(Y^{(h)})\,W_O.$$9–$W_Q,W_K,W_V$0 up to $W_Q,W_K,W_V$1–$W_Q,W_K,W_V$2, reporting few conclusive results but trends suggesting that TPEQD and AZEQD are promising for longer-distance zoom-and-pan transitions (Franke et al., 2024). Here “attention” is an interface-level selection mechanism, not a neural operator.

A third adjacent usage appears in differentiable geometry and optimization. "A Differentiable Distance Metric for Robotics Through Generalized Alternating Projection" defines a differentiable set-to-set metric

$W_Q,W_K,W_V$3

where witness points are found by generalized alternating projection. The construction guarantees $W_Q,W_K,W_V$4 under overlap and $W_Q,W_K,W_V$5-times differentiability for disjoint sets under the stated assumptions (Gonçalves et al., 1 Jul 2025). Likewise, projection robust Wasserstein learns a Stiefel projection $W_Q,W_K,W_V$6 maximizing transport discrepancy, and the ReALM+iRBBS framework returns an $W_Q,W_K,W_V$7-stationary point of the PRW problem within $W_Q,W_K,W_V$8 iterations (Jiang et al., 2022). These are not attention models, but they show that “distance projection” also has a well-developed optimization meaning outside neural sequence architectures.

The broad historical significance of distance attention projection is therefore not the emergence of a single dominant operator, but a convergence of design ideas. Distance may define similarity geometry directly, restrict projected compatibility, mediate information transfer across spatial representations, or allocate probability mass over projection directions. Projection may mean removing Q/K/V, preserving them, crossing between ERP and ICOSAP, or searching over geodesic and transport subspaces. A plausible implication is that future work will continue to separate these axes more explicitly: what is being projected, what distance is being measured, and whether distance replaces, biases, or reweights attention.