---
title: Alternating Spatio-Temporal Transformer (ASTT)
url: https://www.emergentmind.com/topics/alternating-spatio-temporal-transformer-astt
type: topic
---

# Alternating Spatio-Temporal Transformer (ASTT)

Searching arXiv for the specified papers and closely related context.
The Alternating Spatio-Temporal Transformer (ASTT) is a transformer architecture for spatio-temporal association in which feature refinement alternates between within-time-step relational modeling and across-time-step correspondence modeling. In the explicit formulation named in the literature, ASTT is introduced for associating apple fruitlets across stereo images collected on different days and from different camera poses, using alternating self-attention and cross-attention over tokens that combine shape and positional descriptors [2503.03200]. In a broader architectural sense, closely related tracker designs also alternate temporal query interaction and spatial feature interaction, although they do not formally use the ASTT name; the clearest example is the decoder in STTrack, where temporal attention over autoregressive queries is followed by spatial interaction and a spatio-temporal fusion stage [2403.10574].

## 1. Definition and problem formulation

ASTT is defined around a correspondence problem rather than a pure detection problem. In the fruitlet-association formulation, the task is to match fruitlets between two images \(I^t\) and \(I^{t+1}\), producing correspondences
\[
\mathcal{M} = \{(i,j)\mid f_i^t \leftrightarrow f_j^{t+1}\},
\]
with the constraint that each fruitlet may have at most one match and that some fruitlets may remain unmatched due to occlusion, fruit drop, or missed detections [2503.03200].

The motivating setting is unusually difficult because the objects are small, partially occluded, and observed under changing camera poses across days. The paper states that a fruitlet can be as small as 6 mm in diameter and may nearly double in size over a four-day period. Existing agricultural association methods are described as relying either on high-resolution point clouds or on temporally stable features, both of which are difficult to obtain for small fruitlets in orchard conditions [2503.03200].

Within that formulation, ASTT is not merely a temporal transformer applied to stacked frames. Its alternation is structural: self-attention models relations among entities within a single image/day, while cross-attention propagates information across the two days/images to support matching. This suggests that ASTT should be understood as an association architecture in which spatial context and temporal correspondence are iteratively co-refined, rather than as a generic temporal encoder.

## 2. Tokenization: shape and positional representations

In the explicit ASTT architecture, each detected fruitlet is represented by a shape descriptor and a position descriptor. The system first detects fruitlets and reconstructs a point cloud per fruitlet. Mask R-CNN provides segmentation masks and RAFT-Stereo provides disparities, which are fused into a fruitlet point cloud \(\mathcal{P}_i\) [2503.03200].

The shape pathway begins with point-cloud filtering through bilateral smoothing, depth discontinuity filtering, radial outlier removal, median distance filtering, and centering. The fruitlet is then centered, normalized so its largest principal component has unit length, and voxelized into \(\mathcal{V}_i\) at fixed voxel size \(v\). A MinkowskiEngine-based sparse 3D encoder-decoder learns a shape descriptor \(\mathbf{d}_i\). A decoder is used only during pre-training to reconstruct voxel occupancy; once pre-training is complete, the decoder is discarded. The paper states that this pre-training stabilizes and speeds up training [2503.03200].

The positional pathway is designed to encode both cluster-relative location and size/extent. PCA is applied to \(\mathcal{P}_i\), an enclosing bounding box is computed in the \(xy\)-plane, and the median \(z\) value is used rather than extrema because stereo depth can flatten small objects. The four 3D corners are transformed back into the original frame and passed through a two-layer MLP to produce the positional feature. The descriptor is given as
\[
p_i = \text{MLP}\left(\text{PCA}^{-1}\left( 
\begin{bmatrix} 
x_-, y_-, z_{\text{med}}\\
x_-, y_+, z_{\text{med}}\\
x_+, y_-, z_{\text{med}}\\
x_+, y_+, z_{\text{med}}
\end{bmatrix}
\right)\right),
\]
with
\[
\begin{bmatrix} x_-, x_+\\ y_-, y_+\\ z_{\text{med}} \end{bmatrix} \subset \text{PCA}(\mathcal{P}_i).
\]

Initial transformer tokens are then constructed by linearly projecting the shape descriptor and adding the positional descriptor:
\[
{}^{0}\mathbf{x}_i = \big[\text{Linear}(\mathbf{d}_i) + \mathbf{p}_i\big].
\]
The set of all tokens from both time steps is fed into the transformer encoder [2503.03200].

A central empirical conclusion follows directly from the ablation study: the positional descriptor is the most important component. Removing it drops F1 to 68.3%, whereas removing the shape descriptor yields 90.8% F1, and not pre-training the shape encoder yields 90.6% F1 [2503.03200]. The paper interprets this as evidence that spatial information is essential because fruitlets can look similar and point clouds can be incomplete, while shape becomes more important as the temporal gap increases.

## 3. Alternating attention mechanism

The core ASTT idea is a multi-layer transformer encoder with alternating self-attention and cross-attention. Each encoder layer contains layer normalization, a self-attention block, a cross-attention block, residual connections, feedforward networks, and dropout [2503.03200].

Self-attention operates within a single image/day and models relationships among fruitlets in the same cluster. Cross-attention operates across the two days/images and enables spatio-temporal matching by transferring information from one set of fruitlets to the other. The paper states that this design mirrors the matching problem: the model first refines each image’s internal relational structure, then compares across time, then refines again [2503.03200].

Given final-layer outputs \({}^{L}\mathbf{x}_i\), pairwise similarity scores are computed as
\[
\mathbf{S}_{ij} = \frac{ \text{Linear}({}^{L}\mathbf{x}_i^t)^\top \text{Linear}({}^{L}\mathbf{x}_j^{t+1}) }{ \sqrt{d} }, \quad \forall (i,j)\in \mathcal{A}\times \mathcal{B},
\]
where \(\mathcal{A}=\{0,\dots,M-1\}\) and \(\mathcal{B}=\{0,\dots,N-1\}\). Matchability is modeled separately through
\[
\sigma_i = \text{Sigmoid}(\text{Linear}({}^{L}\mathbf{x}_i)).
\]

This separation between similarity and matchability is architecturally important. It permits partial assignment, so the model does not force every fruitlet to match. In the orchard setting, that is necessary because true unmatched cases arise from fruit drop, occlusion, or failed detection. A common misconception is to treat ASTT as a standard bipartite matcher over a fully matchable set; the published formulation instead makes unmatched support explicit at both the score level and the loss level [2503.03200].

## 4. Assignment rule and optimization objectives

The final soft partial assignment matrix is computed using dual softmax:
\[
\mathbf{P}_{ij} = \sigma_i^t \sigma_j^{t+1} \text{Softmax}(\mathbf{S}(i,\cdot))_j \text{Softmax}(\mathbf{S}(\cdot,j))_i.
\]
A match is accepted if the assignment score is the best in both directions and exceeds a threshold tuned on validation data [2503.03200].

Training is divided into two stages. The first stage pre-trains the shape encoder-decoder with an auxiliary binary cross-entropy loss at each decoder layer \(\ell\):
\[
\mathcal{L}_{\text{aux}}^{\ell} = -\frac{1}{N_\ell}\sum_{i=1}^{N_\ell} \left[ {}^{\ell}y_i \log {}^{\ell}\hat{y}_i + (1-{}^{\ell}y_i)\log(1-{}^{\ell}\hat{y}_i) \right].
\]
The decoder loss averages these over decoder layers:
\[
\mathcal{L}_{\text{dec}}=\frac{1}{L}\sum_{\ell=1}^{L}\mathcal{L}_{\text{aux}}^{\ell}.
\]

The second stage trains the transformer encoder with the log-likelihood loss from LightGlue:
\[
\mathcal{L}_{\text{match}} = -\frac{1}{L}\sum_{\ell=1}^{L} \left( 
\frac{1}{|\mathcal{M}|}\sum_{(i,j)\in\mathcal{M}}\log {}^\ell \mathbf{P}_{ij}
+ \frac{1}{2|\overline{\mathcal{A}}|}\sum_{i\in\overline{\mathcal{A}}}\log\left(1-{}^\ell\sigma_i^{t}\right)
+ \frac{1}{2|\overline{\mathcal{B}}|}\sum_{j\in\overline{\mathcal{B}}}\log\left(1-{}^\ell\sigma_j^{t+1}\right)
\right).
\]
Here, \(\mathcal{M}\) is the set of true matches and \(\overline{\mathcal{A}}, \overline{\mathcal{B}}\) are the unmatched fruitlets in each image [2503.03200].

The implementation details are specific. The spatial encoder-decoder uses voxel size 0.03, four downsampling blocks, and four upsampling blocks, and is pre-trained for 200 epochs with ADAM, batch size 64, initial learning rate \(10^{-3}\), and 90% decay every 50 epochs. The transformer has four encoder layers with eight attention heads, trained for 200 epochs with ADAM, a 10-epoch warmup, and a learning rate starting at \(10^{-4}\) and linearly decaying [2503.03200].

## 5. Empirical results and observed operating regime

The ASTT evaluation uses stereo images collected over three years, 2021–2023, at the University of Massachusetts Amherst Cold Spring Orchard, including Fuji, Gala, and Honeycrisp apple varieties. Images were collected both by humans with a handheld illumination-invariant flash stereo camera and by a robot with the same camera mounted on a 7-DoF arm. A subset of 1350 images was manually labeled, yielding 4,362 cross-day image pairs spanning 1 to 7 days apart. The split is by cluster, with 2645 training pairs, 924 validation pairs, and 793 test pairs [2503.03200].

On that dataset, ASTT achieves **Precision: 92.9%**, **Recall: 92.2%**, and **F1-score: 92.4%**. The reported baselines are **ICP-Assoc** at 89.5% F1, **Desc-Assoc** at 86.4% F1, and **Loftr-Assoc** at 72.3% F1 [2503.03200].

The paper further reports that ASTT achieves the best precision and recall for pairs up to five days apart. For seven-day pairs, recall drops below ICP-Assoc, which the authors attribute to fewer training examples and larger growth-induced shape changes and displacement [2503.03200]. This suggests that the architecture is strongest in regimes where local spatial structure remains informative despite nontrivial appearance and pose change, but where the temporal gap has not yet induced extreme geometric drift.

The ablations clarify the operating regime. Positional information is decisive; shape contributes additional robustness; shape pre-training improves optimization. The resulting interpretation is not that ASTT eliminates the need for geometric structure, but that it leverages geometric structure more effectively through alternating intra-set and inter-set attention.

## 6. Relation to tracker-style spatio-temporal transformers

A separate line of work provides a useful comparison. STTrack, presented as “Autoregressive Queries for Adaptive Tracking with Spatio-Temporal Transformers,” does not formally define an ASTT module by name, but its decoder alternates between temporal query interaction and spatial feature interaction, making it the closest functional equivalent to an ASTT-style mechanism in single-object tracking [2403.10574].

STTrack is built from a HiViT-based spatial encoder, a temporal decoder, and a spatio-temporal information fusion module (STM). Spatial features are represented as
\[
f_z \in \mathbb{R}^{N_z \times D}, \quad f_x \in \mathbb{R}^{N_x \times D},
\]
with \(D=512\), and are processed as
\[
f_{zx}^0 = Concat(f_z, f_x), \qquad
f_{zx}^n = Encoder(f_{zx}^{n-1}), \; n=1,\dots,N, \qquad
f_{zx}^N = LN(f_{zx}^N).
\]
Temporal state is carried by learnable autoregressive target queries, with
\[
Q_{all} = Concat(Q_{pre}, Q_{cur}),
\]
and
\[
Q_{pre} = \begin{cases}
Concat(Q_1, \ldots, Q_{t-1}), & t < m \\
Concat(Q_{t-m+1}, \ldots, Q_{t-1}), & t \ge m.
\end{cases}
\]
This yields a sliding-window autoregressive process
\[
Q_1 \rightarrow Q_2 \rightarrow \cdots \rightarrow Q_t \rightarrow Q_{t+1}.
\]

Its temporal attention is explicitly asymmetric:
\[
\text{TA}(Q_{cur}, Q_{all}) = \text{Attention}(Q_{cur}, Q_{all}, Q_{all}),
\]
with \(n_h = 8\), \(d_m = 512\), and \(d_k=d_v=64\). The decoder then alternates this temporal interaction with multi-head attention over encoder features, followed by FFN layers. Final fusion is handled by STM through a parameter-free dot-product similarity
\[
S = f_x^N \odot f_{temporal}^\top.
\]

The relation between ASTT and STTrack is therefore structural rather than terminological. ASTT, in the strict sense, names the fruitlet-association architecture with alternating self-attention and cross-attention. STTrack, by contrast, is a tracker with autoregressive queries, temporal attention, and STM fusion. A plausible implication is that “alternating spatio-temporal transformer” can denote a broader design pattern—alternating temporal-state refinement and spatial-context interaction—even when the published module name differs. The STTrack results support the practical value of that pattern: the method reports **71.4% AUC** and **72.7% AUC** on LaSOT for STTrack-256 and STTrack-384, **AO 73.8%** and **76.0%** on GOT-10k, and runtime of **67.6 FPS** and **44.2 FPS**, respectively, with about **72M** parameters [2403.10574].

## 7. Significance, boundaries, and recurring points of confusion

ASTT occupies a specific methodological position at the intersection of transformer matching, geometric representation learning, and spatio-temporal association. The fruitlet-association formulation explicitly follows the spirit of feature matching transformers such as LightGlue and LoFTR, but adapts that strategy to stereo-derived 3D fruitlet representations and cluster-structured agricultural scenes [2503.03200]. Its contribution is therefore not only the use of attention, but the particular alternation of intra-day context formation and cross-day correspondence exchange on top of shape-and-position tokens.

Two points of confusion recur. First, ASTT is not synonymous with any transformer that processes temporal data. In the cited literature, the term names a specific architecture for fruitlet association, whereas tracker models such as STTrack are better described as ASTT-like rather than ASTT proper [2403.10574]. Second, ASTT is not a purely appearance-based matcher. The ablation evidence shows that positional structure is more important than the learned shape descriptor in the reported orchard benchmark, indicating that the model’s strength comes from contextualized geometric association rather than descriptor similarity alone [2503.03200].

Within those boundaries, the published evidence supports a clear interpretation. Alternation between self-attention and cross-attention is effective when objects are numerous, locally structured, partially ambiguous in appearance, and not always matchable. In such settings, ASTT provides a transformer-based mechanism for jointly modeling local spatial context, temporal correspondence, and explicit unmatched cases.

Source: https://www.emergentmind.com/topics/alternating-spatio-temporal-transformer-astt