Sampling-Equivariant Self-Attention

• Sampling-equivariant self-attention is a mechanism that ensures output transformations precisely follow changes in the sampling process.
• It employs strategies like group-indexed token lifting, mask-based local sampling, and invariant inner products to embed symmetry in attention operations.
• Empirical studies in aerial detection, 3D geometry, and point clouds highlight its benefits in accuracy, robustness, and computational efficiency.

Sampling-equivariant self-attention denotes a class of attention mechanisms in which changes to the sampling of the underlying signal—such as token permutations, rotations or reflections of a sampling grid, patch extraction, down-sampling, up-sampling, or learned token subsampling—either leave the attention operator invariant or induce a prescribed transformation of its outputs. In the literature, the term is explicit in aerial-image detection, where local self-attention is recast as convolutional sampling with masks rather than locations (Yang et al., 2021), and it is also realized by group-equivariant transformers in vision, 3D geometry, Lie-group domains, and agent modeling, where queries, keys, values, positional structure, and resampling operators are constrained to commute with a symmetry action (Fu et al., 8 Feb 2026, Fuchs et al., 2020, Hutchinson et al., 2020, Xu et al., 1 Apr 2026).

1. Formal notion and scope

At the most general level, equivariance is the requirement that transforming the input by a group element $g$ transforms the output by the corresponding output action. One standard statement is

$S_g[\phi(v)] = \phi(T_g[v]),$

and, in an image setting with feature-space action $\pi_{\tilde g}^F$, an equivariant mapping $\Psi$ satisfies

$$\Psi\big[\pi_{\tilde g}^I\big](I) = \pi_{\tilde g}^F\big[\Psi\big](I), \quad \forall \tilde g.$$

These formulations underlie SE(3)-equivariant attention, discrete planar-equivariant ViTs, and Lie-group attention (Fuchs et al., 2020, Fu et al., 8 Feb 2026, Hutchinson et al., 2020).

The qualifier “sampling-equivariant” emphasizes that the transformation does not merely change feature values; it changes the way the data are sampled. In aerial detection, this is formulated by comparing the sampling graph produced on a transformed image with the ideally transformed sampling graph, and the discrepancy is measured by Earth Mover’s Distance and aggregated as AEMD; lower AEMD means better sampling equivariance (Yang et al., 2021). In group-equivariant ViTs, the same idea appears when rotations or reflections alter patch locations, group channels, and down/up-sampling lattices, but the network output still co-transforms exactly (Fu et al., 8 Feb 2026).

A recurring distinction is between exact equivariance and equivariance in expectation. Exact equivariance is established for several constructions on discrete planar groups, SE(3), SE(2), and general Lie groups (Fu et al., 8 Feb 2026, Fuchs et al., 2020, Xu et al., 1 Apr 2026, Hutchinson et al., 2020). By contrast, stochastic sampling schemes for efficient attention on sets or sequences typically preserve the target symmetry in distribution or in expectation over the sampling randomness (Li et al., 2023, Zeng et al., 2021).

2. Structural mechanisms that make attention sampling-equivariant

A common mechanism is to ensure that attention logits are invariant scalars while value messages are equivariant objects. In the SE(3)-Transformer, queries $q_i$ and keys $k_{ij}$ are equivariant embeddings of node and edge features, and the inner product $q_i^\top k_{ij}$ is invariant because both transform under the same orthogonal representation. Attention weights are therefore invariant scalars, and the weighted sum of equivariant value messages remains SE(3)-equivariant (Fuchs et al., 2020). LieTransformer expresses the same principle on a general Lie group by making the location term depend only on the relative group element $g^{-1}g'$; under global left multiplication, $g^{-1}g'$ is unchanged, so the attention kernel reindexes compatibly with the group action (Hutchinson et al., 2020).

A second mechanism is to lift tokens into a representation space that explicitly carries group structure. In the discrete planar ViT construction, the token tensor is

$S_g[\phi(v)] = \phi(T_g[v]),$0

where $S_g[\phi(v)] = \phi(T_g[v]),$1 indexes group elements. Queries, keys, and values are not formed by unconstrained dense maps; instead, each slice is computed with group-tied weights, for example

$S_g[\phi(v)] = \phi(T_g[v]),$2

The resulting block-structured $S_g[\phi(v)] = \phi(T_g[v]),$3 commute with the induced permutation of the group index, so the attention layer becomes exactly equivariant once positional encodings and resampling operators are made compatible (Fu et al., 8 Feb 2026).

A third mechanism is to replace location-based sampling by mask-based sampling. In SESAN, local self-attention is interpreted as convolutional sampling with masks rather than offsets. The mask is computed as

$S_g[\phi(v)] = \phi(T_g[v]),$4

positional encoding is removed from the mask-regression path, and a transformation embedding module injects the mask $S_g[\phi(v)] = \phi(T_g[v]),$5 back into the output feature $S_g[\phi(v)] = \phi(T_g[v]),$6. This design is meant to make the local sampling mask transform with the object under rotation, reflection, skew, or scaling (Yang et al., 2021).

A fourth mechanism is to encode geometry in an invariant algebraic representation rather than in explicit pairwise relative positional embeddings. DriveGATr represents scene elements as multivectors in $S_g[\phi(v)] = \phi(T_g[v]),$7, computes logits from an SE(2)-invariant inner product and distance-aware invariant terms, and then applies standard scaled dot-product attention over the resulting embeddings. Because the logits are invariant and the values are equivariant multivectors, the attention output is SE(2)-equivariant without explicit pairwise relative positional encodings (Xu et al., 1 Apr 2026).

Mechanism	Sampling object	Equivariance device
Group-indexed token lifting	Patch/grid tokens with group channel	Group-tied $S_g[\phi(v)] = \phi(T_g[v]),$8, invariant or equivariant PE
Relative-group attention	Points or lifted group elements	Kernels on $S_g[\phi(v)] = \phi(T_g[v]),$9
Mask-based local attention	Local image patch	Content-derived mask, no absolute PE in mask path
Algebraic geometric attention	Agents, map nodes	Invariant inner products over equivariant multivectors

These constructions differ in representation and domain, but they share one invariant pattern: the sampling geometry is not external bookkeeping. It is part of the representation on which attention is defined.

3. Sampling operations beyond the attention kernel

Sampling-equivariant attention is typically not achieved by modifying logits alone. In the discrete planar ViT framework, patch embedding is replaced by a group-equivariant convolution with stride,

$\pi_{\tilde g}^F$0

so that rotating or reflecting the input induces a spatial permutation plus a cyclic shift in the group dimension. Down-sampling is defined slice-wise by

$\pi_{\tilde g}^F$1

and equivariant Pixel-Shuffle uses the analogous

$\pi_{\tilde g}^F$2

These operators are intended to make patch extraction, multiscale reduction, and reconstruction commute with the group action rather than silently break it (Fu et al., 8 Feb 2026).

In point-cloud and graph attention, the relevant sampling issue is variable cardinality and density rather than a square lattice. The SE(3)-Transformer is a set/graph network: attention is defined per node and per neighbourhood, no assumption is made about the number of nodes, and permutation equivariance ensures that changing the order of points or the particular sampling order does not change the functional mapping beyond the same permutation (Fuchs et al., 2020). LieTransformer addresses an even broader setting by lifting samples on a homogeneous space $\pi_{\tilde g}^F$3 to functions on $\pi_{\tilde g}^F$4, then applying attention over the lifted support; a rigid transformation of the sampled input left-multiplies the lifted support, so the attention layer co-transforms with the sampling geometry (Hutchinson et al., 2020).

A different use of the term appears in self-learning Monte Carlo. There, the Transformer produces an equivariant effective spin field $\pi_{\tilde g}^F$5, then an invariant effective Hamiltonian $\pi_{\tilde g}^F$6, and the proposal transition kernel satisfies

$\pi_{\tilde g}^F$7

The paper explicitly interprets this as sampling-equivariance of the proposal chain: transforming the whole trajectory by a symmetry yields another valid trajectory with the same statistical properties (Nagai et al., 2023).

A related, non-attentional development replaces a fixed group discretization by an input-dependent sampling matrix $\pi_{\tilde g}^F$8 satisfying

$\pi_{\tilde g}^F$9

This yields exact equivariance for any number of group samples $\Psi$0 in a steerable-network nonlinearity (Inal et al., 2024). A plausible implication is that future group-attentional layers can decouple equivariance from sampling density by generating their group samples adaptively rather than fixing them a priori.

4. Stochastic and subquadratic variants

A separate branch of the literature studies sampling as an efficiency device. The sampled transformer for point sets randomly splits the set into subsets, applies Hamiltonian self-attention within each subset, and interprets the global sparse pattern as a random Hamiltonian cycle in the complete attention graph. A fixed realization is not strictly permutation-equivariant, but the distribution over sampled attention graphs is invariant under permutations, and permutation equivariance holds in expectation over the sampling (Li et al., 2023). The same work proves that sampled transformers remain universal approximators of continuous set-to-set functions.

YOSO rewrites self-attention as a sum of value vectors weighted by Bernoulli random variables derived from LSH collisions. For a single head,

$\Psi$1

with

$\Psi$2

The resulting operator is linear in $\Psi$3, its expectation is a deterministic collision-based attention map, and the implementation has linear complexity in sequence length for fixed hash parameters (Zeng et al., 2021). This is not exact group-equivariant attention, but it is an explicit instance of attention defined through stochastic sampling while preserving the attention form in expectation.

Sampling Foundational Transformer inserts a learned sampling-without-replacement step before attention on keys and values, while retaining all queries. It uses a pairwise Maxout score,

$\Psi$4

and a leaky ReLU-based probability map with a learned leak term. The paper emphasizes context aware sampling-without-replacement, linear asymptotic computational complexity, and a pseudoconvex formulation of the transformer layer (Nguyen et al., 2024). In point-cloud experiments, the same framework is used under rotational-invariant constraints by feeding only relative geometric information through $\Psi$5, not raw coordinates.

These stochastic or sampled-efficient constructions differ from exact symmetry-enforced designs. Their primary guarantee is usually linear complexity and equivariance in distribution, expectation, or with respect to a chosen invariant relative encoding, rather than an exact intertwining relation layer by layer.

5. Empirical behavior across domains

Representative reported outcomes show that sampling-equivariant attention is not confined to one modality (Yang et al., 2021, Fu et al., 8 Feb 2026, Fuchs et al., 2020, Xu et al., 1 Apr 2026).

Domain	Reported outcome	Interpretation
Aerial detection and classification	Rotation AEMD drops from $\Psi$6 to $\Psi$7 for SAN19 vs. SES-Net19; DOTA mAP rises from $\Psi$8 to $\Psi$9 for Gliding vs. Gliding+SES	Better alignment of local sampling masks with object transformations
Discrete planar-equivariant ViTs	COCO Box mAP $$\Psi\big[\pi_{\tilde g}^I\big](I) = \pi_{\tilde g}^F\big[\Psi\big](I), \quad \forall \tilde g.$$0, Mask mAP $$\Psi\big[\pi_{\tilde g}^I\big](I) = \pi_{\tilde g}^F\big[\Psi\big](I), \quad \forall \tilde g.$$1, ADE20K mIoU $$\Psi\big[\pi_{\tilde g}^I\big](I) = \pi_{\tilde g}^F\big[\Psi\big](I), \quad \forall \tilde g.$$2; in SR, SwinIR-EQ uses $$\Psi\big[\pi_{\tilde g}^I\big](I) = \pi_{\tilde g}^F\big[\Psi\big](I), \quad \forall \tilde g.$$3M vs $$\Psi\big[\pi_{\tilde g}^I\big](I) = \pi_{\tilde g}^F\big[\Psi\big](I), \quad \forall \tilde g.$$4M parameters	Exact equivariance can be maintained across patch embedding, attention, and multiscale resampling
SE(3)-equivariant attention	N-body position MSE $$\Psi\big[\pi_{\tilde g}^I\big](I) = \pi_{\tilde g}^F\big[\Psi\big](I), \quad \forall \tilde g.$$5; position equivariance error $$\Psi\big[\pi_{\tilde g}^I\big](I) = \pi_{\tilde g}^F\big[\Psi\big](I), \quad \forall \tilde g.$$6 for Set Transformer vs. SE(3)-Transformer	Invariant logits plus equivariant value messages improve both accuracy and symmetry fidelity
SE(2)-equivariant agent modeling	RMM $$\Psi\big[\pi_{\tilde g}^I\big](I) = \pi_{\tilde g}^F\big[\Psi\big](I), \quad \forall \tilde g.$$7 for plain Transformer vs. $$\Psi\big[\pi_{\tilde g}^I\big](I) = \pi_{\tilde g}^F\big[\Psi\big](I), \quad \forall \tilde g.$$8 for DriveGATr-3M; Transformer + RPE uses $$\Psi\big[\pi_{\tilde g}^I\big](I) = \pi_{\tilde g}^F\big[\Psi\big](I), \quad \forall \tilde g.$$9 GB peak memory vs. $q_i$0 GB for DriveGATr-3M	Geometry encoded in the representation can avoid explicit pairwise RPE overhead

The efficient-sampling literature reports a different empirical pattern. On ModelNet40, MAE-sampled attains $q_i$1 while MAE-dense attains $q_i$2, indicating that random Hamiltonian-cycle attention can remain competitive with dense attention on point sets (Li et al., 2023). YOSO reports results consistent with softmax self-attention on Long Range Arena while providing sizable speed-ups and memory savings (Zeng et al., 2021). SFT reports competitive results on point cloud, graph, and sequence benchmarks, while faster inference is tied to the learned key/value sampling ratio (Nguyen et al., 2024).

Across these results, two empirical regimes recur. Exact group-equivariant methods mainly improve robustness, sample efficiency, and geometric consistency under prescribed symmetries. Sampling-based efficient methods mainly trade dense all-pairs interactions for learned or random subsets while trying to preserve task performance and permutation structure.

6. Misconceptions, limitations, and open directions

A common misconception is that standard self-attention is already geometry-equivariant because it is permutation-equivariant. Several papers make the opposite point: vanilla attention is permutation-equivariant in the token index, but patch embedding, positional encodings, and ordinary resampling break rotation or reflection equivariance unless they are redesigned explicitly (Fu et al., 8 Feb 2026). In 3D, permutation equivariance over point labels does not by itself yield SE(3)-equivariance; invariant attention weights must be paired with equivariant feature representations and relative-coordinate kernels (Fuchs et al., 2020).

A second misconception is that all “sampling-aware” attention is exact. Exact layerwise equivariance is proved for discrete planar ViTs, SE(3)-Transformer attention, LieSelfAttention, and DriveGATr under their respective symmetry assumptions (Fu et al., 8 Feb 2026, Fuchs et al., 2020, Hutchinson et al., 2020, Xu et al., 1 Apr 2026). By contrast, sampled Hamiltonian attention on sets is permutation-equivariant in expectation, and Bernoulli-sampled YOSO preserves a deterministic collision-attention only in expectation over hashes (Li et al., 2023, Zeng et al., 2021). For continuous-group steerable networks, fixed discretizations improve equivariance only as the number of group samples increases, although adaptive equivariant sampling can remove that dependency in the non-attentional setting (Inal et al., 2024).

The main technical limitations are consistent across papers. Exact discrete-planar constructions are restricted to rotations by multiples of $q_i$3 and reflections or other finite subgroups, and complexity grows with group size $q_i$4 (Fu et al., 8 Feb 2026). SE(3)-equivariant attention relies on spherical harmonics, Clebsch–Gordan structure, and neighbourhood-based message passing, with spherical harmonics as a bottleneck and higher representation degrees increasing memory and compute (Fuchs et al., 2020). Lie-group and geometric-algebra formulations require explicit representation theory or algebraic primitives, and extensions to irregular domains or larger groups are non-trivial (Hutchinson et al., 2020, Xu et al., 1 Apr 2026).

Current directions point toward two converging themes. One is to make more of the transformer stack—not only logits, but tokenization, positional structure, and multiscale resampling—commute exactly with symmetry actions. The other is to make the sampling pattern itself learned, adaptive, and still symmetry-compatible. The first theme is already explicit in holistic equivariant ViTs (Fu et al., 8 Feb 2026); the second is explicit in efficient sampled transformers and in adaptive group sampling for steerable networks (Nguyen et al., 2024, Inal et al., 2024). A plausible implication is that future sampling-equivariant attention will combine exact symmetry constraints with adaptive discretization, so that changing the sampling pattern alters neither the represented geometry nor the functional action of attention except through a known group transformation.