Temporal-Spatial-Sample Attention

• Temporal-Spatial-Sample Attention is a design space that defines explicit attention mechanisms over time, space, and a third axis often interpreted as sample or token selection.
• It distinguishes true temporal and spatial attention from sample-adjacent methods such as client weighting, region sampling, and sample generation.
• Empirical evidence suggests that structured combinations of these attentions yield performance benefits in tasks like video analysis and neuromorphic processing.

Temporal-Spatial-Sample Attention is not a single standardized architecture in the cited literature; rather, it is a comparative label for models that combine attention or attention-like weighting across temporal structure, spatial structure, and a third axis that may correspond to samples, instances, regions selected as effective samples, or analogous units. Across the literature, the “temporal” and “spatial” components are often explicit, whereas the “sample” component is frequently absent, only loosely analogous, or replaced by token/region selection, client-level relation weighting, or sample generation. The most technically faithful reading therefore requires distinguishing true temporal attention, true spatial attention, and mechanisms that are merely sample-adjacent. Some papers implement temporal-spatial attention but not sample attention at all, such as STAtten (Lee et al., 2024), STSAN (Lin et al., 2020), ST-SAN (Lin et al., 2019), and the geometry-guided attention model for monocular depth estimation (Ruhkamp et al., 2021). Others realize a closer approximation to a temporal-spatial-sample formulation by adding region/token sampling, as in ST-SampleNet (Sao et al., 2024), or by combining temporal-spatial attention with sample generation rather than sample attention, as in TSA-SAN (Li et al., 16 May 2025).

1. Conceptual scope and taxonomy

Temporal-Spatial-Sample Attention can be decomposed into three axes. The first is temporal attention, meaning an explicit mechanism that measures relations across time steps, temporal segments, future horizons, or stage-wise “moments.” The second is spatial attention, meaning explicit weighting across spatial positions, graph nodes, regions, grids, or patches. The third is sample attention, which in a strict sense would mean attention over samples, instances, batch elements, or analogous units distinct from time and space.

The literature in the supplied corpus shows that these three axes are unevenly represented. STAtten explicitly defines spatial-only, temporal-only, and spatial-temporal attention scopes over spike tensors $$\mathbf{Q},\mathbf{K},\mathbf{V}\in\mathbb{R}^{T\times N\times D}$$, but also states that there is no attention over the batch or sample dimension (Lee et al., 2024). STSAN and ST-SAN similarly implement temporal-spatial attention within a single sample, not attention across samples (Lin et al., 2020, Lin et al., 2019). FedASTA uses explicit spatial attention over clients and temporal modeling via GRU and decomposition, but not temporal attention in the narrow self-attention sense, nor a distinct sample-attention operator (Li et al., 2024). TSA-SAN adds sample generation and interpolation, not sample-level attention (Li et al., 16 May 2025).

A plausible implication is that “Temporal-Spatial-Sample Attention” functions more as an analytical taxonomy than as a settled model class. In the cited papers, the third axis often appears as one of four substitutes: region/token selection (Sao et al., 2024), client-level relational weighting (Li et al., 2024), channel attention mistaken for a third axis (Nie et al., 2023), or sample generation (Li et al., 16 May 2025).

2. Explicit temporal-spatial attention formulations

Several papers in the corpus present explicit temporal and spatial attention, though with different factorizations.

STAtten defines three scopes for spike attention on tensors of shape $\mathbb{R}^{T\times N\times D}$: $$\mathrm{S\_Attn}[t,:,:] = \text{LIF}(\mathbf{Q}[t,:,:]\mathbf{K^\top}[t,:,:]\mathbf{V}[t,:,:]\cdot \alpha),$$

$$\mathrm{T\_Attn}[:,n,:] = \text{LIF}(\mathbf{Q}[:,n,:]\mathbf{K^\top}[:,n,:]\mathbf{V}[:,n,:]\cdot \alpha),$$

$$\mathrm{ST\_Attn}[:,:,d] = \text{LIF}(\mathbf{Q}[:,:,d]\mathbf{K}^\top[:,:,d]\mathbf{V}[:,:,d]\cdot \alpha).$$

These equations make explicit that spatial attention correlates token positions $n$ at fixed $t$, temporal attention correlates timesteps $t$ at fixed token $n$, and spatial-temporal attention jointly correlates time and token dimensions for each feature channel $d$ (Lee et al., 2024). The model then realizes spatial-temporal attention blockwise: $\mathbb{R}^{T\times N\times D}$0 with temporal blocks $\mathbb{R}^{T\times N\times D}$1, which preserves the asymptotic complexity $\mathbb{R}^{T\times N\times D}$2 claimed for spatial-only spiking attention (Lee et al., 2024).

STSAN preserves an explicit spatial-temporal tensor

$\mathbb{R}^{T\times N\times D}$3

and computes scaled dot-product attention for each spatial position: $\mathbb{R}^{T\times N\times D}$4 followed by attention weights $\mathbb{R}^{T\times N\times D}$5 and a multi-aspect aggregation over space and feature heads (Lin et al., 2020). The design keeps spatial indexing explicit and performs attention over temporal slices for each spatial location.

ST-SAN adopts a related but distinct tensorized design. Its scaled dot-product kernel is

$\mathbb{R}^{T\times N\times D}$6

with $\mathbb{R}^{T\times N\times D}$7 written as spatial-temporal tensors rather than flattened token lists (Lin et al., 2019). The paper argues that this allows spatial and temporal dependencies to be measured in one attention framework, though its printed equations indicate that the actual softmax is mainly over temporal positions for each spatial site (Lin et al., 2019).

AttentionNAS defines a search space over temporal, spatial, and spatiotemporal jointly attended dimensions for video features of shape $\mathbb{R}^{T\times N\times D}$8 (Wang et al., 2020). It distinguishes map-based attention,

$\mathbb{R}^{T\times N\times D}$9

from dot-product attention,

$$\mathrm{S\_Attn}[t,:,:] = \text{LIF}(\mathbf{Q}[t,:,:]\mathbf{K^\top}[t,:,:]\mathbf{V}[t,:,:]\cdot \alpha),$$0

and applies the resulting weights through

$$\mathrm{S\_Attn}[t,:,:] = \text{LIF}(\mathbf{Q}[t,:,:]\mathbf{K^\top}[t,:,:]\mathbf{V}[t,:,:]\cdot \alpha),$$1

This is an explicit treatment of temporal-only, spatial-only, and joint spatiotemporal attention, but again not sample attention (Wang et al., 2020).

3. Models that resemble the third axis but do not implement true sample attention

The strongest caution in the literature is that many methods are only partially relevant to a temporal-spatial-sample formulation.

GSABT is expressly described as not implementing full Temporal-Spatial-Sample Attention. Its attention is spatial only: a graph-constrained local attention

$$\mathrm{S\_Attn}[t,:,:] = \text{LIF}(\mathbf{Q}[t,:,:]\mathbf{K^\top}[t,:,:]\mathbf{V}[t,:,:]\cdot \alpha),$$2

and Top-$$\mathrm{S\_Attn}[t,:,:] = \text{LIF}(\mathbf{Q}[t,:,:]\mathbf{K^\top}[t,:,:]\mathbf{V}[t,:,:]\cdot \alpha),$$3 sparse global self-attention

$$\mathrm{S\_Attn}[t,:,:] = \text{LIF}(\mathbf{Q}[t,:,:]\mathbf{K^\top}[t,:,:]\mathbf{V}[t,:,:]\cdot \alpha),$$4

with

$$\mathrm{S\_Attn}[t,:,:] = \text{LIF}(\mathbf{Q}[t,:,:]\mathbf{K^\top}[t,:,:]\mathbf{V}[t,:,:]\cdot \alpha),$$5

Temporal dependency is handled by bidirectional temporal convolution,

$$\mathrm{S\_Attn}[t,:,:] = \text{LIF}(\mathbf{Q}[t,:,:]\mathbf{K^\top}[t,:,:]\mathbf{V}[t,:,:]\cdot \alpha),$$6

not temporal attention, and the paper explicitly states that it has no explicit sample-level attention (Zhang et al., 2024).

FedASTA is similarly asymmetric. It contains masked spatial attention over client hidden states,

$$\mathrm{S\_Attn}[t,:,:] = \text{LIF}(\mathbf{Q}[t,:,:]\mathbf{K^\top}[t,:,:]\mathbf{V}[t,:,:]\cdot \alpha),$$7

with masked scaled dot-product attention over clients, but temporal dependency is modeled by decomposition, GRU encoding of the seasonal component, and Fourier trend extraction rather than temporal attention (Li et al., 2024). Its adaptive graph and client-to-client weighting are the closest analog to sample attention, but the paper does not present a dedicated sample-attention module (Li et al., 2024).

TSA-SAN adds an explicit temporal-spatial attention mechanism over GRU features, but the sample-related contribution is sample generation rather than sample attention. Its temporal and spatial attention branches are formed from average and standard deviation pooled descriptors: $$\mathrm{S\_Attn}[t,:,:] = \text{LIF}(\mathbf{Q}[t,:,:]\mathbf{K^\top}[t,:,:]\mathbf{V}[t,:,:]\cdot \alpha),$$8 which are fused into

$$\mathrm{S\_Attn}[t,:,:] = \text{LIF}(\mathbf{Q}[t,:,:]\mathbf{K^\top}[t,:,:]\mathbf{V}[t,:,:]\cdot \alpha),$$9

then combined by matrix product into a temporal-spatial attention map $$\mathrm{T\_Attn}[:,n,:] = \text{LIF}(\mathbf{Q}[:,n,:]\mathbf{K^\top}[:,n,:]\mathbf{V}[:,n,:]\cdot \alpha),$$0 and applied through

$$\mathrm{T\_Attn}[:,n,:] = \text{LIF}(\mathbf{Q}[:,n,:]\mathbf{K^\top}[:,n,:]\mathbf{V}[:,n,:]\cdot \alpha),$$1

But the “sample” axis is supplied by Distribution Alignment Sample Generation and Interpolation-based Sample Synthesis, not by learned attention over samples (Li et al., 16 May 2025).

A strict misconception therefore arises when any third mechanism beyond time and space is called “sample attention.” The cited papers repeatedly distinguish between actual sample attention and substitutes such as channel attention (Nie et al., 2023), region selection (Sao et al., 2024), frame weighting (Meng et al., 2018), or sample synthesis (Li et al., 16 May 2025).

4. Sample-adjacent mechanisms: frame weighting, region sampling, and grouped areas

Although explicit sample attention is uncommon, several papers provide mechanisms that are adjacent to it.

The action-recognition model with interpretable spatial-temporal attention computes temporal frame weights

$$\mathrm{T\_Attn}[:,n,:] = \text{LIF}(\mathbf{Q}[:,n,:]\mathbf{K^\top}[:,n,:]\mathbf{V}[:,n,:]\cdot \alpha),$$2

and aggregates frame features as

$$\mathrm{T\_Attn}[:,n,:] = \text{LIF}(\mathbf{Q}[:,n,:]\mathbf{K^\top}[:,n,:]\mathbf{V}[:,n,:]\cdot \alpha),$$3

The paper explicitly notes that this can be interpreted as frame-level instance weighting inside a video, but not as sample attention over dataset examples or batch elements (Meng et al., 2018). A plausible implication is that frame-attention sometimes functions as an intra-video proxy for sample attention.

Temporal-Spatial Mapping for action recognition constructs a VideoMap

$$\mathrm{T\_Attn}[:,n,:] = \text{LIF}(\mathbf{Q}[:,n,:]\mathbf{K^\top}[:,n,:]\mathbf{V}[:,n,:]\cdot \alpha),$$4

then applies a temporal attention vector $$\mathrm{T\_Attn}[:,n,:] = \text{LIF}(\mathbf{Q}[:,n,:]\mathbf{K^\top}[:,n,:]\mathbf{V}[:,n,:]\cdot \alpha),$$5 broadcast across channels,

$$\mathrm{T\_Attn}[:,n,:] = \text{LIF}(\mathbf{Q}[:,n,:]\mathbf{K^\top}[:,n,:]\mathbf{V}[:,n,:]\cdot \alpha),$$6

to reweight feature maps,

$$\mathrm{T\_Attn}[:,n,:] = \text{LIF}(\mathbf{Q}[:,n,:]\mathbf{K^\top}[:,n,:]\mathbf{V}[:,n,:]\cdot \alpha),$$7

This is closer to frame/sample weighting than to spatial attention, but the paper remains a temporal-attention model rather than a full temporal-spatial-sample one (Song et al., 2018).

Area Attention generalizes attention from single items to adjacent groups. For an area $$\mathrm{T\_Attn}[:,n,:] = \text{LIF}(\mathbf{Q}[:,n,:]\mathbf{K^\top}[:,n,:]\mathbf{V}[:,n,:]\cdot \alpha),$$8, the key and value are

$$\mathrm{T\_Attn}[:,n,:] = \text{LIF}(\mathbf{Q}[:,n,:]\mathbf{K^\top}[:,n,:]\mathbf{V}[:,n,:]\cdot \alpha),$$9

with an optional parameterized area key

$$\mathrm{ST\_Attn}[:,:,d] = \text{LIF}(\mathbf{Q}[:,:,d]\mathbf{K}^\top[:,:,d]\mathbf{V}[:,:,d]\cdot \alpha).$$0

The paper explicitly covers 1D temporal adjacency and 2D spatial adjacency, and notes that the machinery would in principle generalize to additional structured axes, though it does not itself define a sample axis (Li et al., 2018). This suggests a route toward sample-level grouping only if a meaningful adjacency relation among samples exists.

ST-SampleNet presents the clearest third-axis mechanism in the corpus. It combines local CNN spatial encoding, transformer-based global spatial attention, transformer-based temporal attention, and a learned region sampling module. The sampler computes

$$\mathrm{ST\_Attn}[:,:,d] = \text{LIF}(\mathbf{Q}[:,:,d]\mathbf{K}^\top[:,:,d]\mathbf{V}[:,:,d]\cdot \alpha).$$1

predicts keep/drop probabilities

$$\mathrm{ST\_Attn}[:,:,d] = \text{LIF}(\mathbf{Q}[:,:,d]\mathbf{K}^\top[:,:,d]\mathbf{V}[:,:,d]\cdot \alpha).$$2

and retains only top-$$\mathrm{ST\_Attn}[:,:,d] = \text{LIF}(\mathbf{Q}[:,:,d]\mathbf{K}^\top[:,:,d]\mathbf{V}[:,:,d]\cdot \alpha).$$3 regions before spatial attention, reducing spatial-attention complexity from $$\mathrm{ST\_Attn}[:,:,d] = \text{LIF}(\mathbf{Q}[:,:,d]\mathbf{K}^\top[:,:,d]\mathbf{V}[:,:,d]\cdot \alpha).$$4 to $$\mathrm{ST\_Attn}[:,:,d] = \text{LIF}(\mathbf{Q}[:,:,d]\mathbf{K}^\top[:,:,d]\mathbf{V}[:,:,d]\cdot \alpha).$$5 (Sao et al., 2024). The paper explicitly frames this as a sample or token selection mechanism over spatial regions, not sample attention over dataset instances (Sao et al., 2024).

5. Non-attention temporal modeling paired with spatial attention

A recurrent pattern in the literature is that the temporal axis is often modeled without explicit temporal attention, even in papers whose titles suggest spatial-temporal attention.

GSABT uses bidirectional temporal convolution rather than temporal attention. Its temporal branch is built from stacked TCN layers with dilation coefficients

$$\mathrm{ST\_Attn}[:,:,d] = \text{LIF}(\mathbf{Q}[:,:,d]\mathbf{K}^\top[:,:,d]\mathbf{V}[:,:,d]\cdot \alpha).$$6

and forward-backward fusion

$$\mathrm{ST\_Attn}[:,:,d] = \text{LIF}(\mathbf{Q}[:,:,d]\mathbf{K}^\top[:,:,d]\mathbf{V}[:,:,d]\cdot \alpha).$$7

The paper states explicitly that there is no temporal attention mechanism (Zhang et al., 2024).

FedASTA similarly uses time decomposition, GRU encoder-decoder, and filtered Fourier transform. Its adaptive graph is constructed from trend representations,

$$\mathrm{ST\_Attn}[:,:,d] = \text{LIF}(\mathbf{Q}[:,:,d]\mathbf{K}^\top[:,:,d]\mathbf{V}[:,:,d]\cdot \alpha).$$8

followed by Fourier Sparse Distance

$$\mathrm{ST\_Attn}[:,:,d] = \text{LIF}(\mathbf{Q}[:,:,d]\mathbf{K}^\top[:,:,d]\mathbf{V}[:,:,d]\cdot \alpha).$$9

Yet the explicit attention remains spatial and client-wise, not temporal (Li et al., 2024).

TAU is another example. Its Temporal Attention Unit is not transformer-style self-attention but a factorized gating mechanism: $n$0

$n$1

$n$2

The temporal component is encoded by channel/frame gating after folding time into channels, while spatial structure is handled by convolutional attention-like maps (Tan et al., 2022). The paper explicitly states that it has no sample attention (Tan et al., 2022).

These cases matter because they delimit the term. A model that has spatial attention plus temporal convolution, recurrent modeling, or decomposition is not necessarily a temporal-attention model, and still less a temporal-spatial-sample attention model.

6. Temporal-spatial attention under alternative third axes: channel, geometry, and hierarchy

Some papers replace a putative sample axis with a different third axis.

Triplet Attention Transformer for spatiotemporal predictive learning alternates attention over time, space, and channel: $n$3 Its temporal attention uses a causal mask,

$n$4

its spatial attention uses a grid-unshuffle formulation with relative positional bias $n$5, and its channel attention operates over grouped channel tokens (Nie et al., 2023). The paper explicitly notes that channel attention is the closest analogue to a third axis but is not sample attention (Nie et al., 2023).

The geometry-guided spatial-temporal attention model for self-supervised monocular depth estimation replaces a third axis with geometry. Spatial attention is defined by 3D point distances: $n$6

$n$7

while temporal attention is cross-frame feature attention,

$n$8

The paper contains no sample-attention mechanism; its nearest analogue is the use of photometric and cycle masks to gate reliable frame or pixel contributions (Ruhkamp et al., 2021).

EEG emotion recognition via spatial-temporal transformers compares four variants: spatial, temporal, sequential spatial-temporal, and simultaneous spatial-temporal attention. Its joint model allows

$n$9

for all $t$0 within a sample, but not across samples $t$1 (Liu et al., 2021). This paper is explicit that the model operates inside one EEG sample and does not define a separate sample-attention axis (Liu et al., 2021).

7. Empirical patterns and evidence across application domains

The corpus supports several recurrent empirical conclusions.

First, explicit temporal-spatial attention tends to outperform single-axis or purely convolutional alternatives in domains where long-range dependency matters. STAtten improves spiking transformer backbones on sequential CIFAR, ImageNet, and neuromorphic datasets while preserving the claimed complexity $t$2 (Lee et al., 2024). In EEG, simultaneous spatial-temporal attention outperforms spatial-only, temporal-only, and sequential spatial-temporal variants on SEED and SEED-IV (Liu et al., 2021). In video, searched spatiotemporal attention cells exceed non-local blocks while using less compute (Wang et al., 2020).

Second, structured or factorized attention often outperforms naïve full flattening. DSAN’s Multi-Space Attention and switch-attention are explicitly motivated as remedies for over-averaging in large flattened spatial-temporal token spaces; replacing them with flattened multi-head attention causes a severe degradation on Taxi-NYC (Lin et al., 2020). STAtten uses local temporal blocks rather than global full space-time attention because distant spike correlations produce dead/silent neurons and memory overhead (Lee et al., 2024). ST-SAN similarly keeps spatial structure explicit and outperforms a plain Transformer baseline that treats spatial-temporal information as a sequence (Lin et al., 2019).

Third, many reported “third axes” are not sample axes in the strict sense. TSA-SAN’s strongest ablation effect comes from DASG sample generation rather than its temporal-spatial attention module (Li et al., 16 May 2025). GSABT’s Top-$t$3 sparse attention is over nodes, not samples (Zhang et al., 2024). STAN’s temporal component is built from stage-wise spatial features over “multiple moments,” not from attention across examples (Sun et al., 2022).

A plausible implication is that genuine sample attention remains underdeveloped relative to temporal and spatial attention. The literature more commonly addresses computational economy through token pruning (Sao et al., 2024), interpretability through frame weighting (Meng et al., 2018), or robustness through sample synthesis (Li et al., 16 May 2025) than through explicit cross-sample attention.

8. Encyclopedia assessment

The literature represented here does not support a single canonical definition of Temporal-Spatial-Sample Attention. Instead, it supports a layered distinction.

In the strict sense, a method deserves the label only if it contains all three of the following: explicit temporal attention, explicit spatial attention, and explicit attention over a third sample or instance axis. Within the supplied papers, none provides an uncontested, fully general realization of that strict definition. STAtten is a clean temporal-spatial attention mechanism without sample attention (Lee et al., 2024). STSAN, ST-SAN, AttentionNAS, and the EEG simultaneous spatial-temporal transformer are also temporal-spatial rather than temporal-spatial-sample models (Lin et al., 2020, Lin et al., 2019, Wang et al., 2020, Liu et al., 2021). GSABT, FedASTA, TAU, and DSAN pair spatial attention with non-attention temporal modeling or with only sample-adjacent mechanisms (Zhang et al., 2024, Li et al., 2024, Tan et al., 2022, Lin et al., 2020). TSA-SAN adds sample generation rather than sample attention (Li et al., 16 May 2025).

In a broader, engineering-oriented sense, ST-SampleNet comes closest to a practical temporal-spatial-sample formulation by combining temporal transformer attention, spatial transformer attention, and learned region sampling as a third-axis selection mechanism (Sao et al., 2024). But even there, the third axis is sampled region tokens rather than dataset samples.

This suggests that Temporal-Spatial-Sample Attention is presently better understood as a methodological design space than as a settled architecture family. The most stable components are temporal attention and spatial attention; the least stable is the sample axis, whose meaning varies from frame instance weighting (Meng et al., 2018), to grouped area construction (Li et al., 2018), to client-level relational weighting (Li et al., 2024), to sample generation (Li et al., 16 May 2025), to region pruning (Sao et al., 2024). Any rigorous use of the term therefore requires explicit specification of what “sample” denotes and whether it is truly attended over, merely selected, or only generated.