---
title: Temporal-Spatial-Sample Attention
url: https://www.emergentmind.com/topics/temporal-spatial-sample-attention
type: topic
---

# Temporal-Spatial-Sample Attention

Temporal-Spatial-Sample Attention is not a single standardized architecture in the cited literature; rather, it is a comparative label for models that combine attention or attention-like weighting across temporal structure, spatial structure, and a third axis that may correspond to samples, instances, regions selected as effective samples, or analogous units. Across the literature, the “temporal” and “spatial” components are often explicit, whereas the “sample” component is frequently absent, only loosely analogous, or replaced by token/region selection, client-level relation weighting, or sample generation. The most technically faithful reading therefore requires distinguishing true temporal attention, true spatial attention, and mechanisms that are merely sample-adjacent. Some papers implement temporal-spatial attention but not sample attention at all, such as STAtten [2409.19764], STSAN [2002.09693], ST-SAN [1912.07663], and the geometry-guided attention model for monocular depth estimation [2110.08192]. Others realize a closer approximation to a temporal-spatial-sample formulation by adding region/token sampling, as in ST-SampleNet [2411.06836], or by combining temporal-spatial attention with sample generation rather than sample attention, as in TSA-SAN [2505.11083].

## 1. Conceptual scope and taxonomy

Temporal-Spatial-Sample Attention can be decomposed into three axes. The first is **temporal attention**, meaning an explicit mechanism that measures relations across time steps, temporal segments, future horizons, or stage-wise “moments.” The second is **spatial attention**, meaning explicit weighting across spatial positions, graph nodes, regions, grids, or patches. The third is **sample attention**, which in a strict sense would mean attention over samples, instances, batch elements, or analogous units distinct from time and space.

The literature in the supplied corpus shows that these three axes are unevenly represented. STAtten explicitly defines spatial-only, temporal-only, and spatial-temporal attention scopes over spike tensors $\mathbf{Q},\mathbf{K},\mathbf{V}\in\mathbb{R}^{T\times N\times D}$, but also states that there is no attention over the batch or sample dimension [2409.19764]. STSAN and ST-SAN similarly implement temporal-spatial attention within a single sample, not attention across samples [2002.09693] [1912.07663]. FedASTA uses explicit spatial attention over clients and temporal modeling via GRU and decomposition, but not temporal attention in the narrow self-attention sense, nor a distinct sample-attention operator [2405.13090]. TSA-SAN adds sample generation and interpolation, not sample-level attention [2505.11083].

A plausible implication is that “Temporal-Spatial-Sample Attention” functions more as an analytical taxonomy than as a settled model class. In the cited papers, the third axis often appears as one of four substitutes: region/token selection [2411.06836], client-level relational weighting [2405.13090], channel attention mistaken for a third axis [2310.18698], or sample generation [2505.11083].

## 2. Explicit temporal-spatial attention formulations

Several papers in the corpus present explicit temporal and spatial attention, though with different factorizations.

STAtten defines three scopes for spike attention on tensors of shape $\mathbb{R}^{T\times N\times D}$:
\[
\mathrm{S\_Attn}[t,:,:] = \text{LIF}(\mathbf{Q}[t,:,:]\mathbf{K^\top}[t,:,:]\mathbf{V}[t,:,:]\cdot \alpha),
\]
\[
\mathrm{T\_Attn}[:,n,:] = \text{LIF}(\mathbf{Q}[:,n,:]\mathbf{K^\top}[:,n,:]\mathbf{V}[:,n,:]\cdot \alpha),
\]
\[
\mathrm{ST\_Attn}[:,:,d] = \text{LIF}(\mathbf{Q}[:,:,d]\mathbf{K}^\top[:,:,d]\mathbf{V}[:,:,d]\cdot \alpha).
\]
These equations make explicit that spatial attention correlates token positions $n$ at fixed $t$, temporal attention correlates timesteps $t$ at fixed token $n$, and spatial-temporal attention jointly correlates time and token dimensions for each feature channel $d$ [2409.19764]. The model then realizes spatial-temporal attention blockwise:
\[
\text{STAtten}(\mathbf{X}[b]) = \text{LIF}(\mathbf{Q}[b] \mathbf{K}^\top[b] \mathbf{V}[b] \cdot \alpha),
\]
with temporal blocks $[b]=[iB:(i+1)B,:,d]$, which preserves the asymptotic complexity $\mathcal{O}(TND^2)$ claimed for spatial-only spiking attention [2409.19764].

STSAN preserves an explicit spatial-temporal tensor
\[
Q,K,V \in \mathbf{R}^{I \times J \times |T| \times d},
\]
and computes scaled dot-product attention for each spatial position:
\[
s_{i,j}^h = \frac{Q_{i,j}^h \cdot (K^h_{i,j})^T}{\sqrt{d_h}},
\]
followed by attention weights $\alpha_{i,j,t}^h$ and a multi-aspect aggregation over space and feature heads [2002.09693]. The design keeps spatial indexing explicit and performs attention over temporal slices for each spatial location.

ST-SAN adopts a related but distinct tensorized design. Its scaled dot-product kernel is
\[
\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,
\]
with $Q,K,V$ written as spatial-temporal tensors rather than flattened token lists [1912.07663]. The paper argues that this allows spatial and temporal dependencies to be measured in one attention framework, though its printed equations indicate that the actual softmax is mainly over temporal positions for each spatial site [1912.07663].

AttentionNAS defines a search space over **temporal**, **spatial**, and **spatiotemporal jointly** attended dimensions for video features of shape $(T,H,W,C)$ [2007.12034]. It distinguishes map-based attention,
\[
W_\text{map} = Diag(\phi(G_2(AvgPool(G_1(f'_\text{in}))))),
\]
from dot-product attention,
\[
W_\text{dot-prod} = \phi(G_1(f'_\text{in})G_2(f'_\text{in})^{T}),
\]
and applies the resulting weights through
\[
f_\text{out}=ReshapeTo2D^{-1}(W\,ReshapeTo2D(G_3(f_\text{in}))).
\]
This is an explicit treatment of temporal-only, spatial-only, and joint spatiotemporal attention, but again not sample attention [2007.12034].

## 3. Models that resemble the third axis but do not implement true sample attention

The strongest caution in the literature is that many methods are only partially relevant to a temporal-spatial-sample formulation.

GSABT is expressly described as **not** implementing full Temporal-Spatial-Sample Attention. Its attention is spatial only: a graph-constrained local attention
\[
G_A = \mathrm{Softmax}(G_M A_M)
\]
and Top-$U$ sparse global self-attention
\[
S_M(ij)=
\begin{cases}
A_M(ij), & \text{if } A_M(ij)\ge u_i\\
-\infty, & \text{if } A_M(ij)<u_i
\end{cases}
\]
with
\[
O_{SA}(X)=\mathrm{Softmax}(S_M)V.
\]
Temporal dependency is handled by bidirectional temporal convolution,
\[
O_{BT}(X)=H^F(X)+H^B(X),
\]
not temporal attention, and the paper explicitly states that it has no explicit sample-level attention [2412.19842].

FedASTA is similarly asymmetric. It contains masked spatial attention over client hidden states,
\[
Q_i^S = H W_{Q_i^S},\qquad K_i^S = H W_{K_i^S},\qquad V_i^S = H W_{V_i^S},
\]
with masked scaled dot-product attention over clients, but temporal dependency is modeled by decomposition, GRU encoding of the seasonal component, and Fourier trend extraction rather than temporal attention [2405.13090]. Its adaptive graph and client-to-client weighting are the closest analog to sample attention, but the paper does not present a dedicated sample-attention module [2405.13090].

TSA-SAN adds an explicit temporal-spatial attention mechanism over GRU features, but the sample-related contribution is sample generation rather than sample attention. Its temporal and spatial attention branches are formed from average and standard deviation pooled descriptors:
\[
\bm{a}_{\text{TAP}},\ \bm{a}_{\text{TSD}},\ \bm{a}_{\text{SAP}},\ \bm{a}_{\text{SSD}},
\]
which are fused into
\[
\bm{a}_{\text{T}}=\text{Conv}_\text{T}(\bm{a}_{\text{TAP}\oplus \bm{a}_{\text{TSD}}),\qquad
\bm{a}_{\text{S}}=\text{Conv}_\text{S}(\bm{a}_{\text{SAP}\oplus \bm{a}_{\text{SSD}}),
\]
then combined by matrix product into a temporal-spatial attention map $\bm{a}_{\text{TS}}$ and applied through
\[
\bm{f}_{\text{TS}}=\bm{a}_{\text{TS}}\odot \bm{f}_{\text{GRU}}.
\]
But the “sample” axis is supplied by Distribution Alignment Sample Generation and Interpolation-based Sample Synthesis, not by learned attention over samples [2505.11083].

A strict misconception therefore arises when any third mechanism beyond time and space is called “sample attention.” The cited papers repeatedly distinguish between actual sample attention and substitutes such as channel attention [2310.18698], region selection [2411.06836], frame weighting [1810.04511], or sample synthesis [2505.11083].

## 4. Sample-adjacent mechanisms: frame weighting, region sampling, and grouped areas

Although explicit sample attention is uncommon, several papers provide mechanisms that are adjacent to it.

The action-recognition model with interpretable spatial-temporal attention computes temporal frame weights
\[
e_{ti} = \Phi(H_{t-1}, \tilde{X}_i), \qquad
w_{ti}=\frac{\exp(e_{ti})}{\sum_{i=1}^{n}\exp(e_{ti})},
\]
and aggregates frame features as
\[
Y_t = \frac{1}{n}\sum_{i=1}^{n} w_{ti}\tilde{X}_i.
\]
The paper explicitly notes that this can be interpreted as frame-level instance weighting inside a video, but not as sample attention over dataset examples or batch elements [1810.04511]. A plausible implication is that frame-attention sometimes functions as an intra-video proxy for sample attention.

Temporal-Spatial Mapping for action recognition constructs a VideoMap
\[
\mathcal{M} = [f_1^\top; f_2^\top; \cdots; f_T^\top] \in \mathbb{R}^{T \times L},
\]
then applies a temporal attention vector $a_i(\mathcal{M})$ broadcast across channels,
\[
A_i(\mathcal{M}) = a_i(\mathcal{M}) \otimes \mathbf{1},
\]
to reweight feature maps,
\[
\tilde{F_i} = A_i(\mathcal{M}) \circ F_i(\mathcal{M}).
\]
This is closer to frame/sample weighting than to spatial attention, but the paper remains a temporal-attention model rather than a full temporal-spatial-sample one [1809.03669].

Area Attention generalizes attention from single items to adjacent groups. For an area $r_i$, the key and value are
\[
\mu_{i}=\frac{1}{|r_i|}\sum_{j=1}^{|r_i|}k_{i,j},
\qquad
v_i^{r_i}=\sum_{j=1}^{|r_i|}v_{i,j},
\]
with an optional parameterized area key
\[
k_i^r=\phi(\mu_{i}W_{\mu} + \sigma_{i}W_{\sigma} + e_{i}W_{e})W_{d}.
\]
The paper explicitly covers 1D temporal adjacency and 2D spatial adjacency, and notes that the machinery would in principle generalize to additional structured axes, though it does not itself define a sample axis [1810.10126]. This suggests a route toward sample-level grouping only if a meaningful adjacency relation among samples exists.

ST-SampleNet presents the clearest third-axis mechanism in the corpus. It combines local CNN spatial encoding, transformer-based global spatial attention, transformer-based temporal attention, and a learned region sampling module. The sampler computes
\[
\mathcal{Z}^{sampler} = MLP(\mathcal{Z}^{poi} + \mathcal{Z}_{t-i}^{time}) \in \mathbb{R}^{N \times d},
\]
predicts keep/drop probabilities
\[
\rho = Softmax(\mathcal{Z'}) \in \mathbb{R}^{N \times 2},
\]
and retains only top-$k$ regions before spatial attention, reducing spatial-attention complexity from $\mathcal{O}(N^2)$ to $\mathcal{O}(k^2)$ [2411.06836]. The paper explicitly frames this as a sample or token selection mechanism over spatial regions, not sample attention over dataset instances [2411.06836].

## 5. Non-attention temporal modeling paired with spatial attention

A recurrent pattern in the literature is that the temporal axis is often modeled without explicit temporal attention, even in papers whose titles suggest spatial-temporal attention.

GSABT uses bidirectional temporal convolution rather than temporal attention. Its temporal branch is built from stacked TCN layers with dilation coefficients
\[
1,\;2,\;4,\;4
\]
and forward-backward fusion
\[
O_{BT}(X)=H^F(X)+H^B(X).
\]
The paper states explicitly that there is no temporal attention mechanism [2412.19842].

FedASTA similarly uses time decomposition, GRU encoder-decoder, and filtered Fourier transform. Its adaptive graph is constructed from trend representations,
\[
\Lambda_i = \mathcal{F}\{X_i\},
\qquad
\Lambda_{i,\mu}(k) = \Lambda_i(k)\cdot \mathbbm{1}(|\Lambda_i(k)|>\mu),
\]
followed by Fourier Sparse Distance
\[
\text{FSD}(v_i,v_j)= \sqrt{ \frac{1}{T_u} \sum_{k\in \mathbf{H}_u} \left| \Lambda_{i,\mu}(k)-\Lambda_{j,\mu}(k) \right|^2 }.
\]
Yet the explicit attention remains spatial and client-wise, not temporal [2405.13090].

TAU is another example. Its Temporal Attention Unit is not transformer-style self-attention but a factorized gating mechanism:
\[
\text{SA} = \mathrm{Conv}_{1\times 1}\big(\mathrm{DW\text{-}D\ Conv}(\mathrm{DW\ Conv}(H))\big),
\]
\[
\text{DA} = \mathrm{FC}(\mathrm{AvgPool}(H)),
\]
\[
H' = (\text{SA} \otimes \text{DA}) \odot H.
\]
The temporal component is encoded by channel/frame gating after folding time into channels, while spatial structure is handled by convolutional attention-like maps [2206.12126]. The paper explicitly states that it has no sample attention [2206.12126].

These cases matter because they delimit the term. A model that has spatial attention plus temporal convolution, recurrent modeling, or decomposition is not necessarily a temporal-attention model, and still less a temporal-spatial-sample attention model.

## 6. Temporal-spatial attention under alternative third axes: channel, geometry, and hierarchy

Some papers replace a putative sample axis with a different third axis.

Triplet Attention Transformer for spatiotemporal predictive learning alternates attention over time, space, and channel:
\[
\mathcal A_{\text{temporal}},\qquad
\mathcal A_{\text{spatial}},\qquad
\mathcal A_{\text{channel}}.
\]
Its temporal attention uses a causal mask,
\[
\mathcal A(\mathbf Q_i,\mathbf K_i,\mathbf V_i) =
\operatorname{Softmax}\!\left( \frac{\mathcal M(\mathbf Q_i^{\top}\mathbf K_i)}{\sqrt{C_k}} \right)\mathbf V_i^{\top},
\]
its spatial attention uses a grid-unshuffle formulation with relative positional bias $\mathbf B$, and its channel attention operates over grouped channel tokens [2310.18698]. The paper explicitly notes that channel attention is the closest analogue to a third axis but is not sample attention [2310.18698].

The geometry-guided spatial-temporal attention model for self-supervised monocular depth estimation replaces a third axis with geometry. Spatial attention is defined by 3D point distances:
\[
\mathbf{P}_i = \mathbf{K}^{-1}( d_i \cdot \mathbf{C}_i),\quad
\mathbf{P}_j = \mathbf{K}^{-1}( d_j \cdot \mathbf{C}_j),
\]
\[
\mathcal{A}_{i,j}^{spatial} = \exp \bigg( -\frac  { \| \mathbf{P}_i - \mathbf{P}_j  \|_2 }  {\sigma}\bigg),
\]
while temporal attention is cross-frame feature attention,
\[
\mathcal{A}_{i,j}^{temporal} = \text{Softmax}_j\big((\mathbf{F}_i^q)^\top \mathbf{F}_j^k\big).
\]
The paper contains no sample-attention mechanism; its nearest analogue is the use of photometric and cycle masks to gate reliable frame or pixel contributions [2110.08192].

EEG emotion recognition via spatial-temporal transformers compares four variants: spatial, temporal, sequential spatial-temporal, and simultaneous spatial-temporal attention. Its joint model allows
\[
z_{n,t,p} \leftrightarrow z_{n,t',p'}
\]
for all $t',p'$ within a sample, but not across samples $n$ [2110.06553]. This paper is explicit that the model operates inside one EEG sample and does not define a separate sample-attention axis [2110.06553].

## 7. Empirical patterns and evidence across application domains

The corpus supports several recurrent empirical conclusions.

First, explicit temporal-spatial attention tends to outperform single-axis or purely convolutional alternatives in domains where long-range dependency matters. STAtten improves spiking transformer backbones on sequential CIFAR, ImageNet, and neuromorphic datasets while preserving the claimed complexity $\mathcal{O}(TND^2)$ [2409.19764]. In EEG, simultaneous spatial-temporal attention outperforms spatial-only, temporal-only, and sequential spatial-temporal variants on SEED and SEED-IV [2110.06553]. In video, searched spatiotemporal attention cells exceed non-local blocks while using less compute [2007.12034].

Second, structured or factorized attention often outperforms naïve full flattening. DSAN’s Multi-Space Attention and switch-attention are explicitly motivated as remedies for over-averaging in large flattened spatial-temporal token spaces; replacing them with flattened multi-head attention causes a severe degradation on Taxi-NYC [2006.08849]. STAtten uses local temporal blocks rather than global full space-time attention because distant spike correlations produce dead/silent neurons and memory overhead [2409.19764]. ST-SAN similarly keeps spatial structure explicit and outperforms a plain Transformer baseline that treats spatial-temporal information as a sequence [1912.07663].

Third, many reported “third axes” are not sample axes in the strict sense. TSA-SAN’s strongest ablation effect comes from DASG sample generation rather than its temporal-spatial attention module [2505.11083]. GSABT’s Top-$U$ sparse attention is over nodes, not samples [2412.19842]. STAN’s temporal component is built from stage-wise spatial features over “multiple moments,” not from attention across examples [2211.13940].

A plausible implication is that genuine sample attention remains underdeveloped relative to temporal and spatial attention. The literature more commonly addresses computational economy through token pruning [2411.06836], interpretability through frame weighting [1810.04511], or robustness through sample synthesis [2505.11083] than through explicit cross-sample attention.

## 8. Encyclopedia assessment

The literature represented here does not support a single canonical definition of Temporal-Spatial-Sample Attention. Instead, it supports a layered distinction.

In the strict sense, a method deserves the label only if it contains all three of the following: explicit temporal attention, explicit spatial attention, and explicit attention over a third sample or instance axis. Within the supplied papers, none provides an uncontested, fully general realization of that strict definition. STAtten is a clean temporal-spatial attention mechanism without sample attention [2409.19764]. STSAN, ST-SAN, AttentionNAS, and the EEG simultaneous spatial-temporal transformer are also temporal-spatial rather than temporal-spatial-sample models [2002.09693] [1912.07663] [2007.12034] [2110.06553]. GSABT, FedASTA, TAU, and DSAN pair spatial attention with non-attention temporal modeling or with only sample-adjacent mechanisms [2412.19842] [2405.13090] [2206.12126] [2006.08849]. TSA-SAN adds sample generation rather than sample attention [2505.11083].

In a broader, engineering-oriented sense, ST-SampleNet comes closest to a practical temporal-spatial-sample formulation by combining temporal transformer attention, spatial transformer attention, and learned region sampling as a third-axis selection mechanism [2411.06836]. But even there, the third axis is sampled region tokens rather than dataset samples.

This suggests that Temporal-Spatial-Sample Attention is presently better understood as a methodological design space than as a settled architecture family. The most stable components are temporal attention and spatial attention; the least stable is the sample axis, whose meaning varies from frame instance weighting [1810.04511], to grouped area construction [1810.10126], to client-level relational weighting [2405.13090], to sample generation [2505.11083], to region pruning [2411.06836]. Any rigorous use of the term therefore requires explicit specification of what “sample” denotes and whether it is truly attended over, merely selected, or only generated.

Source: https://www.emergentmind.com/topics/temporal-spatial-sample-attention