Spatially-Enhanced Attention (SE-Attention)

• Spatially-Enhanced Attention is a family of methods that embed spatial structure directly into the attention operation through techniques like group-equivariance, deformable sampling, and spatial-channel gating.
• The approach modifies standard self-attention by integrating spatial cues into descriptor computation, support evaluation, and transformation laws, ensuring invariance and precise spatial relationships.
• These mechanisms have demonstrated practical benefits in diverse applications such as 3D geometry processing, image segmentation, and speech enhancement by balancing performance, efficiency, and computational cost.

Spatially-Enhanced Attention (SE-Attention) denotes a family of attention mechanisms in which spatial structure is built directly into the attention operation rather than appended as a weak positional cue or discarded by global pooling. In the works surveyed here, the term spans several distinct but related constructions: group-equivariant attention for 2D and 3D geometry, channel-attention modules whose descriptors encode spatial relationships, top-down spatial-channel gating, deformable or sparse attention that learns where to sample, and graph-based or multichannel mechanisms that propagate information according to spatial adjacency or sensor geometry (Fuchs et al., 2020, Nikzad et al., 2024, Xia et al., 2023).

1. Conceptual scope and basic formulation

Standard self-attention computes

$$\text{Attn}( \mathbf{q}_i, \{\mathbf{k}_j\}, \{\mathbf{v}_j\} ) = \sum_{j=1}^n \alpha_{ij}\,\mathbf{v}_j,\qquad \alpha_{ij} = \frac{\exp(\mathbf{q}_i^\top \mathbf{k}_j)}{\sum_{j'}\exp(\mathbf{q}_i^\top \mathbf{k}_{j'})}.$$

In vanilla Transformers, the queries, keys, and values are arbitrary learned embeddings, and “they do not respect any particular symmetry of the space” (Fuchs et al., 2020). In the literature considered here, SE-Attention modifies at least one of three ingredients: the descriptor from which attention weights are produced, the support over which attention is evaluated, or the transformation law imposed on features and messages under spatial transformations (Fuchs et al., 2020, Nikzad et al., 2024).

A recurring distinction is that “spatial attention” does not name a single mechanism. Some works use the term for exact geometric equivariance under $\mathrm{SE}(3)$ or $\mathrm{SE}(2)$, some for channel weighting driven by spatial autocorrelation rather than pure global statistics, some for multiplicative masks $G(x,y,c)$ over feature maps, and some for dynamic restriction of attention to sparse or deformable spatial supports (Pronovost et al., 24 Jul 2025, Hu et al., 5 Jun 2025, Calian et al., 2019).

Family	Spatial mechanism	Representative works
Group-equivariant attention	Relative poses, irreducible representations, invariant scores, equivariant values	(Fuchs et al., 2020, Pronovost et al., 24 Jul 2025, Diaconu et al., 2019)
Spatially informed channel attention	Spatial autocorrelation, local tiles, graph-based channel/spatial interactions	(Nikzad et al., 2024, Vosco et al., 2021, Sabharwal et al., 2024)
Dynamic spatial support	Deformable sampling, PatchMatch sparsity, selective recomputation	(Xia et al., 2023, Calian et al., 2019, Borji, 2024)
Context- or sensor-conditioned gating	Top-down masks, cross-channel TF masks, spatially separable local/global attention	(Hu et al., 5 Jun 2025, Opochinsky et al., 19 Jan 2026, Mourya et al., 2022)

This diversity makes SE-Attention better understood as a design orientation than as a single module type. Across these works, the common requirement is that attention should reflect how relations in space are organized, transformed, or selected.

2. Geometric and symmetry-constrained formulations

The most formalized version appears in the SE(3)-Transformer, which “hard-wires 3D geometry” into self-attention so that the layer is equivariant to 3D rotations and translations (Fuchs et al., 2020). Point clouds or graphs are treated as tensor fields

$$\mathbf{f}(\mathbf{x}) = \sum_{j=1}^N \mathbf{f}_j\,\delta(\mathbf{x} - \mathbf{x}_j),$$

with each node feature decomposed into type-$\ell$ components $\mathbf{f}_j^\ell \in \mathbb{R}^{2\ell+1}$. Queries are node-local equivariant embeddings, while keys and values depend on relative positions $\mathbf{x}_j-\mathbf{x}_i$ through Tensor Field Network kernels built from spherical harmonics, Clebsch–Gordan matrices, and learnable radial functions. The defining constraint is that attention weights are invariant to global pose, whereas value updates transform equivariantly; this is the mechanism by which attention becomes a geometric operator rather than a purely content-based set function (Fuchs et al., 2020).

In this formulation, the attention score remains invariant because the query and key vectors transform under the same orthogonal representation, so their inner product is unchanged under rotation. The value map $\mathbf{W}_V^{\ell k}(\mathbf{x}_j-\mathbf{x}_i)$ obeys the kernel constraint

$$\mathbf{W}^{\ell k}(R^{-1}\mathbf{x}) = \mathbf{D}_\ell(R)\,\mathbf{W}^{\ell k}(\mathbf{x})\,\mathbf{D}_k(R)^{-1},$$

which guarantees that every message rotates predictably with the input (Fuchs et al., 2020). The paper reports lower MSE than a non-equivariant Set Transformer and an equivariant but non-attention TFN on N-body prediction, and states that the equivariance error $\mathrm{SE}(3)$0 is near machine precision ($\mathrm{SE}(3)$1) (Fuchs et al., 2020). On ScanObjectNN, it also introduces an “SE(3)+z” variant that deliberately breaks full SE(3) by treating the $\mathrm{SE}(3)$2-coordinate as a scalar and $\mathrm{SE}(3)$3 as a vector, showing that partial symmetry can be imposed selectively rather than absolutely.

A 2D counterpart appears in “Linear Memory SE(2) Invariant Attention,” where every token has a pose $\mathrm{SE}(3)$4, and attention depends only on relative pose $\mathrm{SE}(3)$5 (Pronovost et al., 24 Jul 2025). The central factorization

$\mathrm{SE}(3)$6

allows geometry-aware pre-processing of $\mathrm{SE}(3)$7, followed by standard scaled dot-product attention and pose-aware post-processing, yielding linear memory in the number of objects (Pronovost et al., 24 Jul 2025). On a 33M-scenario autonomous-driving dataset, the SE(2) Fourier variant matches 2D RoPE on NLL and stationary trajectories, is slightly worse on straight trajectories, and improves turning minADE from $\mathrm{SE}(3)$8 to $\mathrm{SE}(3)$9 (Pronovost et al., 24 Jul 2025). The paper also makes explicit that extending the same linear-memory construction to $\mathrm{SE}(2)$0 remains an open problem.

A related precursor is “Affine Self Convolution,” which reframes local self-attention in vision as a data-dependent convolution with an affine spatial map

$\mathrm{SE}(2)$1

and proves translation equivariance, later extending the construction to the discrete roto-translation group $\mathrm{SE}(2)$2 (Diaconu et al., 2019). The same work generalizes squeeze-and-excitation to group domains by averaging over the full group and then applying channel-wise reweighting, so that the squeeze operator is invariant to the underlying spatial symmetry (Diaconu et al., 2019). This line of work makes a strong claim: spatial enhancement can be imposed as an exact transformation law, not merely as a learned preference.

3. Spatially informed channel and joint channel-spatial recalibration

A separate lineage treats SE-Attention as a refinement of squeeze-and-excitation, where the main limitation is that global average pooling collapses all spatial structure into one scalar per channel. CSA-Net addresses this by keeping global statistics as a base descriptor and adding a second-order descriptor derived from local Moran’s $\mathrm{SE}(2)$3 over channels (Nikzad et al., 2024). Starting from

$\mathrm{SE}(2)$4

CSA computes pairwise distances $\mathrm{SE}(2)$5, converts them into a normalized spatial weight matrix $\mathrm{SE}(2)$6, then defines

$\mathrm{SE}(2)$7

followed by normalization to a spatially-enhanced descriptor $\mathrm{SE}(2)$8 and an SE-style one-hidden-layer MLP to obtain channel weights (Nikzad et al., 2024). The point is not to create a spatial mask over $\mathrm{SE}(2)$9, but to let channel attention depend on how channels’ spatial patterns relate to each other. On ImageNet with ResNet-50, the paper reports top-1 error $G(x,y,c)$0 and top-5 error $G(x,y,c)$1; on COCO detection with Faster R-CNN and ResNet-50, it reports $G(x,y,c)$2 mAP@[.5,.95], best among the compared methods (Nikzad et al., 2024).

“Tiled Squeeze-and-Excite” reaches a related conclusion from a hardware perspective: full global context is not always necessary for channel attention (Vosco et al., 2021). TSE replaces a single global descriptor with several local descriptors per channel, obtained by non-overlapping tiled average pooling and a shared SE excitation network. On RegNetY-800MF, the paper reports Top-1 $G(x,y,c)$3 for baseline SE, $G(x,y,c)$4 for $G(x,y,c)$5, $G(x,y,c)$6 for $G(x,y,c)$7, and $G(x,y,c)$8 for $G(x,y,c)$9, concluding that “limited local context on the order of seven rows or columns of the original image is sufficient to match the performance of global context” (Vosco et al., 2021). It further reports that in EfficientDetD2, TSE reduces activation pipeline buffering by $$\mathbf{f}(\mathbf{x}) = \sum_{j=1}^N \mathbf{f}_j\,\delta(\mathbf{x} - \mathbf{x}_j),$$0, “from 50M to 4.77M,” without loss of accuracy (Vosco et al., 2021). This suggests that one practical interpretation of SE-Attention is not stronger globality, but better placement of spatial aggregation.

STEAM extends the same family toward explicit joint channel-spatial modeling through graph attention (Sabharwal et al., 2024). Its Channel Interaction Attention treats channels as nodes on a cyclic graph, while Spatial Interaction Attention operates on an Output Guided Pooling map arranged as a grid graph. Both branches use multi-head graph-transformer-style attention on pooled scalar descriptors, keeping parameter count independent of $$\mathbf{f}(\mathbf{x}) = \sum_{j=1}^N \mathbf{f}_j\,\delta(\mathbf{x} - \mathbf{x}_j),$$1 and $$\mathbf{f}(\mathbf{x}) = \sum_{j=1}^N \mathbf{f}_j\,\delta(\mathbf{x} - \mathbf{x}_j),$$2 (Sabharwal et al., 2024). On ResNet-50 classification, the reported result is Top-1 $$\mathbf{f}(\mathbf{x}) = \sum_{j=1}^N \mathbf{f}_j\,\delta(\mathbf{x} - \mathbf{x}_j),$$3, Params $$\mathbf{f}(\mathbf{x}) = \sum_{j=1}^N \mathbf{f}_j\,\delta(\mathbf{x} - \mathbf{x}_j),$$4, and GFLOPs $$\mathbf{f}(\mathbf{x}) = \sum_{j=1}^N \mathbf{f}_j\,\delta(\mathbf{x} - \mathbf{x}_j),$$5, outperforming SE, CBAM, ECA, GCT, and MCA in that table (Sabharwal et al., 2024). The abstract states that STEAM “achieves a 2% increase in accuracy over the standard ResNet-50 model with only a meager increase in GFLOPs,” and that it achieves “a three-fold reduction in GFLOPs” relative to ECA and GCT while outperforming them in accuracy (Sabharwal et al., 2024).

4. Dynamic spatial support: deformable, sparse, graph-based, and selective computation

Another major interpretation of SE-Attention concerns not what weights are assigned, but where attention is allowed to look. DAT and DAT++ exemplify this through deformable multi-head attention, where reference points $$\mathbf{f}(\mathbf{x}) = \sum_{j=1}^N \mathbf{f}_j\,\delta(\mathbf{x} - \mathbf{x}_j),$$6 on a 2D feature map are shifted by learned offsets $$\mathbf{f}(\mathbf{x}) = \sum_{j=1}^N \mathbf{f}_j\,\delta(\mathbf{x} - \mathbf{x}_j),$$7 predicted from the query features (Xia et al., 2023). Keys and values are then sampled at the continuous locations $$\mathbf{f}(\mathbf{x}) = \sum_{j=1}^N \mathbf{f}_j\,\delta(\mathbf{x} - \mathbf{x}_j),$$8 via bilinear interpolation,

$$\mathbf{f}(\mathbf{x}) = \sum_{j=1}^N \mathbf{f}_j\,\delta(\mathbf{x} - \mathbf{x}_j),$$9

and attention uses these sparsely sampled, image-dependent positions together with deformable relative position bias (Xia et al., 2023). The result is a receptive field that remains global in scope but sparse in cardinality. DAT++ reports $\ell$0 ImageNet accuracy, $\ell$1 box AP and $\ell$2 mask AP on MS-COCO instance segmentation, and $\ell$3 mIoU on ADE20K semantic segmentation (Xia et al., 2023). In the paper’s interpretation, deformable attention resolves the tension between dense global attention, which is expensive and often distracted by irrelevant regions, and fixed windowing, which is efficient but data-agnostic.

SCRAM addresses a similar problem from the approximation side rather than by changing the attention rule itself (Calian et al., 2019). It assumes that attention maps in natural images exhibit “spatial coherence” and “sparse structure,” uses PatchMatch to find high-compatibility keys for each query, and then computes softmax exactly on a sparse, query-specific support (Calian et al., 2019). The reported complexity is $\ell$4 rather than $\ell$5, and the sparsity structure is data-driven rather than fixed (Calian et al., 2019). This suggests a second form of spatial enhancement: exploiting regularities of image geometry to make non-local attention computationally viable.

LGA replaces pairwise global attention with graph propagation on a locally connected latent graph, where each spatial location is a node, edges follow 8-connectivity plus a self-loop, and propagation proceeds by

$\ell$6

with $\ell$7 a degree-normalized adjacency matrix (Singh et al., 2023). Stacking layers expands context radius while retaining linear complexity in the number of nodes. The paper reports that on transparent object segmentation, SqueezeNet improves from mIoU $\ell$8 to $\ell$9 with LGA; on dehazing, reduced BPPNet improves from SSIM $\mathbf{f}_j^\ell \in \mathbb{R}^{2\ell+1}$0, PSNR $\mathbf{f}_j^\ell \in \mathbb{R}^{2\ell+1}$1 to SSIM $\mathbf{f}_j^\ell \in \mathbb{R}^{2\ell+1}$2, PSNR $\mathbf{f}_j^\ell \in \mathbb{R}^{2\ell+1}$3; and on optical flow, ARFlow improves EPE-all on Sintel Final from $\mathbf{f}_j^\ell \in \mathbb{R}^{2\ell+1}$4 to $\mathbf{f}_j^\ell \in \mathbb{R}^{2\ell+1}$5 (Singh et al., 2023). Because distant interactions emerge only through chained local paths, the mechanism encodes intermediate spatial consistency rather than only endpoint similarity.

A more radical use of the phrase appears in “Addressing a fundamental limitation in deep vision models: lack of spatial attention,” where spatial attention is defined as selective computation rather than weighted aggregation (Borji, 2024). The proposed change-map mechanism recomputes convolution or pooling only at locations whose receptive fields have changed, while unchanged regions reuse cached outputs (Borji, 2024). On repeated MNIST-like frames, the selective implementation preserves $\mathbf{f}_j^\ell \in \mathbb{R}^{2\ell+1}$6 accuracy while reducing runtime from $\mathbf{f}_j^\ell \in \mathbb{R}^{2\ell+1}$7 s to $\mathbf{f}_j^\ell \in \mathbb{R}^{2\ell+1}$8 s relative to a naive sequential baseline; on shifted images, runtime becomes $\mathbf{f}_j^\ell \in \mathbb{R}^{2\ell+1}$9 s instead of $\mathbf{x}_j-\mathbf{x}_i$0 s (Borji, 2024). In this line of work, SE-Attention names compute allocation over space rather than a similarity kernel.

5. Context-conditioned, cross-sensor, and domain-specific variants

SE-Attention also appears as explicit spatial-channel gating conditioned on external context. A neural network model of spatial and feature-based attention uses a frozen “function network” pretrained to approximately $\mathbf{x}_j-\mathbf{x}_i$1 accuracy on single-digit MNIST, and a trainable “context network” that produces gate values $\mathbf{x}_j-\mathbf{x}_i$2 (Hu et al., 5 Jun 2025). Attention is implemented as multiplicative modulation of an intermediate feature map,

$\mathbf{x}_j-\mathbf{x}_i$3

For spatial attention, the context network receives only a symbolic cue such as left/right or left/middle/right; for feature-based attention, it receives both the image and a cue indicating Group 1 versus Group 2 digits (Hu et al., 5 Jun 2025). Quantitatively, the two-digit spatial task improves from $\mathbf{x}_j-\mathbf{x}_i$4 to $\mathbf{x}_j-\mathbf{x}_i$5, the three-digit spatial task from $\mathbf{x}_j-\mathbf{x}_i$6 to $\mathbf{x}_j-\mathbf{x}_i$7, and the two-digit feature-based task from $\mathbf{x}_j-\mathbf{x}_i$8 to $\mathbf{x}_j-\mathbf{x}_i$9 (Hu et al., 5 Jun 2025). The paper stresses that this is not Transformer self-attention but a top-down spatial-and-channel-wise multiplicative mask.

In multichannel audio, AMDM-SE realizes spatial enhancement through a cross-channel time-frequency attention block inside a score-based diffusion model for speech enhancement (Opochinsky et al., 19 Jan 2026). A reference microphone spectrogram $\mathbf{W}_V^{\ell k}(\mathbf{x}_j-\mathbf{x}_i)$0 and the remaining channels $\mathbf{W}_V^{\ell k}(\mathbf{x}_j-\mathbf{x}_i)$1 are processed so that each non-reference channel produces a time-frequency mask through average pooling, $\mathbf{W}_V^{\ell k}(\mathbf{x}_j-\mathbf{x}_i)$2 convolutions, ReLU, and sigmoid; each mask is then applied element-wise to the reference spectrogram, and the resulting $\mathbf{W}_V^{\ell k}(\mathbf{x}_j-\mathbf{x}_i)$3 masked reference channels are concatenated with the original $\mathbf{W}_V^{\ell k}(\mathbf{x}_j-\mathbf{x}_i)$4 channels, yielding $\mathbf{W}_V^{\ell k}(\mathbf{x}_j-\mathbf{x}_i)$5 output channels (Opochinsky et al., 19 Jan 2026). On simulated data, AMDM-SE improves over MC-SGMSE from PESQ $\mathbf{W}_V^{\ell k}(\mathbf{x}_j-\mathbf{x}_i)$6 to $\mathbf{W}_V^{\ell k}(\mathbf{x}_j-\mathbf{x}_i)$7, SI-SDR $\mathbf{W}_V^{\ell k}(\mathbf{x}_j-\mathbf{x}_i)$8 to $\mathbf{W}_V^{\ell k}(\mathbf{x}_j-\mathbf{x}_i)$9, and eSTOI $$\mathbf{W}^{\ell k}(R^{-1}\mathbf{x}) = \mathbf{D}_\ell(R)\,\mathbf{W}^{\ell k}(\mathbf{x})\,\mathbf{D}_k(R)^{-1},$$0 to $$\mathbf{W}^{\ell k}(R^{-1}\mathbf{x}) = \mathbf{D}_\ell(R)\,\mathbf{W}^{\ell k}(\mathbf{x})\,\mathbf{D}_k(R)^{-1},$$1; on CHiME-3, it reports PESQ $$\mathbf{W}^{\ell k}(R^{-1}\mathbf{x}) = \mathbf{D}_\ell(R)\,\mathbf{W}^{\ell k}(\mathbf{x})\,\mathbf{D}_k(R)^{-1},$$2, eSTOI $$\mathbf{W}^{\ell k}(R^{-1}\mathbf{x}) = \mathbf{D}_\ell(R)\,\mathbf{W}^{\ell k}(\mathbf{x})\,\mathbf{D}_k(R)^{-1},$$3, ovrl $$\mathbf{W}^{\ell k}(R^{-1}\mathbf{x}) = \mathbf{D}_\ell(R)\,\mathbf{W}^{\ell k}(\mathbf{x})\,\mathbf{D}_k(R)^{-1},$$4, and sig $$\mathbf{W}^{\ell k}(R^{-1}\mathbf{x}) = \mathbf{D}_\ell(R)\,\mathbf{W}^{\ell k}(\mathbf{x})\,\mathbf{D}_k(R)^{-1},$$5, exceeding single-channel SGMSE on all of those metrics except bak (Opochinsky et al., 19 Jan 2026). Here “spatial” refers to inter-microphone directional and coherence cues embedded in the multichannel time-frequency representation.

In massive MIMO CSI feedback, STNet uses spatially separable attention composed of Locally Grouped Self-Attention and Global Sub-Sampled Attention (Mourya et al., 2022). CSI is represented as a 2D tensor in angular-delay space, and the attention cost is reduced from $$\mathbf{W}^{\ell k}(R^{-1}\mathbf{x}) = \mathbf{D}_\ell(R)\,\mathbf{W}^{\ell k}(\mathbf{x})\,\mathbf{D}_k(R)^{-1},$$6 for full global attention to

$$\mathbf{W}^{\ell k}(R^{-1}\mathbf{x}) = \mathbf{D}_\ell(R)\,\mathbf{W}^{\ell k}(\mathbf{x})\,\mathbf{D}_k(R)^{-1},$$7

through local windows plus sub-sampled global tokens (Mourya et al., 2022). The paper states that STNet outperformed state-of-the-art models in some scenarios with approximately $$\mathbf{W}^{\ell k}(R^{-1}\mathbf{x}) = \mathbf{D}_\ell(R)\,\mathbf{W}^{\ell k}(\mathbf{x})\,\mathbf{D}_k(R)^{-1},$$8 of the resources. For indoor $$\mathbf{W}^{\ell k}(R^{-1}\mathbf{x}) = \mathbf{D}_\ell(R)\,\mathbf{W}^{\ell k}(\mathbf{x})\,\mathbf{D}_k(R)^{-1},$$9, it reports NMSE $\mathrm{SE}(3)$00 dB for STNet versus $\mathrm{SE}(3)$01 dB for TransNet, with $\mathrm{SE}(3)$02M versus $\mathrm{SE}(3)$03M FLOPs; for indoor $\mathrm{SE}(3)$04, STNet reports $\mathrm{SE}(3)$05 dB versus $\mathrm{SE}(3)$06 dB for TransNet (Mourya et al., 2022). This is a domain-specific example in which spatial enhancement means explicit exploitation of the antenna–subcarrier grid.

6. Recurring principles, misconceptions, and open problems

Across these works, at least five design principles recur. First, spatial enhancement may be exact and group-theoretic, as in invariant attention weights and equivariant value updates under $\mathrm{SE}(3)$07 or $\mathrm{SE}(3)$08 (Fuchs et al., 2020, Pronovost et al., 24 Jul 2025). Second, it may operate at the descriptor level, replacing pure first-order statistics with second-order or local spatial context, as in CSA-Net and TSE (Nikzad et al., 2024, Vosco et al., 2021). Third, it may be realized through explicit spatial-channel gates conditioned on task or context, as in the dual-network MNIST model (Hu et al., 5 Jun 2025). Fourth, it may alter the support of attention through deformable sampling, randomized sparse search, or graph-limited propagation, as in DAT++, SCRAM, and LGA (Xia et al., 2023, Calian et al., 2019, Singh et al., 2023). Fifth, it may mean selective execution of computation over changed regions rather than full-frame processing, as argued in the change-map framework (Borji, 2024).

A common misconception is that all such mechanisms are merely variants of Transformer self-attention. The surveyed literature contradicts that view. Some modules are SE-style channel recalibrators, some are graph message-passing operators, some are multiplicative masks generated by a separate context network, and some are exact equivariant operators with no need for ad hoc positional encoding (Sabharwal et al., 2024, Hu et al., 5 Jun 2025, Fuchs et al., 2020). Another misconception is that “spatial attention” always means a 2D mask over $\mathrm{SE}(3)$09. CSA-Net, for example, leaves attention channel-wise but spatially enriches the descriptor through local Moran’s $\mathrm{SE}(3)$10, while TSE keeps attention channel-wise but makes it tile-dependent rather than global (Nikzad et al., 2024, Vosco et al., 2021).

The limitations are equally heterogeneous. CSA relies on an $\mathrm{SE}(3)$11 distance between feature maps and a $\mathrm{SE}(3)$12 spatial weight matrix, which the paper itself describes as a heuristic and a source of overhead for very large $\mathrm{SE}(3)$13 (Nikzad et al., 2024). LGA improves up to four layers and then degrades at eight layers, which the paper attributes to over-smoothing (Singh et al., 2023). The selective-computation view of spatial attention depends on temporal coherence; when changes are dense, little computation can be skipped (Borji, 2024). AMDM-SE does not encode explicit array geometry, and identifies Relative Transfer Functions as a promising extension (Opochinsky et al., 19 Jan 2026). Linear-memory SE(2) attention requires a Fourier approximation whose error must be controlled by basis size, and the paper explicitly leaves the $\mathrm{SE}(3)$14 extension as future work (Pronovost et al., 24 Jul 2025).

A broader conceptual challenge concerns representation itself. “On learning spatial sequences with the movement of attention” argues that spatial patterns should be represented on “different levels of abstraction,” that this introduces redundancy but is necessary for recognition and generalization, and that learning may require “selectionism learning” and “new data structures, not neural network architectures” (Osaulenko, 2023). A plausible implication is that future SE-Attention research may move beyond tensor-local operators toward mechanisms that traverse multi-scale spatial abstractions, store reusable instruction-like patterns, and integrate attention with explicit spatial memory.

Taken together, these works do not define a single canonical SE-Attention layer. They define a research program: attention should encode spatial relations, respect relevant symmetries, allocate support and compute according to geometry, and expose controllable trade-offs between expressiveness, invariance, and efficiency.