Alternating Spatio-Temporal Transformer (ASTT)

• ASTT is a transformer model that alternates self-attention within a single image and cross-attention across images to effectively address spatio-temporal matching.
• It utilizes detailed shape and positional descriptors for fruitlet matching, with positional information proving critical for high F1 scores in challenging agricultural scenarios.
• Empirical results show superior precision and recall compared to baselines, and its design pattern informs broader tracking architectures like STTrack.

Searching arXiv for the specified papers and closely related context. The Alternating Spatio-Temporal Transformer (ASTT) is a transformer architecture for spatio-temporal association in which feature refinement alternates between within-time-step relational modeling and across-time-step correspondence modeling. In the explicit formulation named in the literature, ASTT is introduced for associating apple fruitlets across stereo images collected on different days and from different camera poses, using alternating self-attention and cross-attention over tokens that combine shape and positional descriptors (Freeman et al., 5 Mar 2025). In a broader architectural sense, closely related tracker designs also alternate temporal query interaction and spatial feature interaction, although they do not formally use the ASTT name; the clearest example is the decoder in STTrack, where temporal attention over autoregressive queries is followed by spatial interaction and a spatio-temporal fusion stage (Xie et al., 2024).

1. Definition and problem formulation

ASTT is defined around a correspondence problem rather than a pure detection problem. In the fruitlet-association formulation, the task is to match fruitlets between two images $I^t$ and $I^{t+1}$, producing correspondences

$$\mathcal{M} = \{(i,j)\mid f_i^t \leftrightarrow f_j^{t+1}\},$$

with the constraint that each fruitlet may have at most one match and that some fruitlets may remain unmatched due to occlusion, fruit drop, or missed detections (Freeman et al., 5 Mar 2025).

The motivating setting is unusually difficult because the objects are small, partially occluded, and observed under changing camera poses across days. The paper states that a fruitlet can be as small as 6 mm in diameter and may nearly double in size over a four-day period. Existing agricultural association methods are described as relying either on high-resolution point clouds or on temporally stable features, both of which are difficult to obtain for small fruitlets in orchard conditions (Freeman et al., 5 Mar 2025).

Within that formulation, ASTT is not merely a temporal transformer applied to stacked frames. Its alternation is structural: self-attention models relations among entities within a single image/day, while cross-attention propagates information across the two days/images to support matching. This suggests that ASTT should be understood as an association architecture in which spatial context and temporal correspondence are iteratively co-refined, rather than as a generic temporal encoder.

2. Tokenization: shape and positional representations

In the explicit ASTT architecture, each detected fruitlet is represented by a shape descriptor and a position descriptor. The system first detects fruitlets and reconstructs a point cloud per fruitlet. Mask R-CNN provides segmentation masks and RAFT-Stereo provides disparities, which are fused into a fruitlet point cloud $\mathcal{P}_i$ (Freeman et al., 5 Mar 2025).

The shape pathway begins with point-cloud filtering through bilateral smoothing, depth discontinuity filtering, radial outlier removal, median distance filtering, and centering. The fruitlet is then centered, normalized so its largest principal component has unit length, and voxelized into $\mathcal{V}_i$ at fixed voxel size $v$. A MinkowskiEngine-based sparse 3D encoder-decoder learns a shape descriptor $\mathbf{d}_i$. A decoder is used only during pre-training to reconstruct voxel occupancy; once pre-training is complete, the decoder is discarded. The paper states that this pre-training stabilizes and speeds up training (Freeman et al., 5 Mar 2025).

The positional pathway is designed to encode both cluster-relative location and size/extent. PCA is applied to $\mathcal{P}_i$, an enclosing bounding box is computed in the $xy$-plane, and the median $z$ value is used rather than extrema because stereo depth can flatten small objects. The four 3D corners are transformed back into the original frame and passed through a two-layer MLP to produce the positional feature. The descriptor is given as

$I^{t+1}$0

with

$I^{t+1}$1

Initial transformer tokens are then constructed by linearly projecting the shape descriptor and adding the positional descriptor: $I^{t+1}$2 The set of all tokens from both time steps is fed into the transformer encoder (Freeman et al., 5 Mar 2025).

A central empirical conclusion follows directly from the ablation study: the positional descriptor is the most important component. Removing it drops F1 to 68.3%, whereas removing the shape descriptor yields 90.8% F1, and not pre-training the shape encoder yields 90.6% F1 (Freeman et al., 5 Mar 2025). The paper interprets this as evidence that spatial information is essential because fruitlets can look similar and point clouds can be incomplete, while shape becomes more important as the temporal gap increases.

3. Alternating attention mechanism

The core ASTT idea is a multi-layer transformer encoder with alternating self-attention and cross-attention. Each encoder layer contains layer normalization, a self-attention block, a cross-attention block, residual connections, feedforward networks, and dropout (Freeman et al., 5 Mar 2025).

Self-attention operates within a single image/day and models relationships among fruitlets in the same cluster. Cross-attention operates across the two days/images and enables spatio-temporal matching by transferring information from one set of fruitlets to the other. The paper states that this design mirrors the matching problem: the model first refines each image’s internal relational structure, then compares across time, then refines again (Freeman et al., 5 Mar 2025).

Given final-layer outputs $I^{t+1}$3, pairwise similarity scores are computed as

$I^{t+1}$4

where $I^{t+1}$5 and $I^{t+1}$6. Matchability is modeled separately through

$I^{t+1}$7

This separation between similarity and matchability is architecturally important. It permits partial assignment, so the model does not force every fruitlet to match. In the orchard setting, that is necessary because true unmatched cases arise from fruit drop, occlusion, or failed detection. A common misconception is to treat ASTT as a standard bipartite matcher over a fully matchable set; the published formulation instead makes unmatched support explicit at both the score level and the loss level (Freeman et al., 5 Mar 2025).

4. Assignment rule and optimization objectives

The final soft partial assignment matrix is computed using dual softmax: $I^{t+1}$8 A match is accepted if the assignment score is the best in both directions and exceeds a threshold tuned on validation data (Freeman et al., 5 Mar 2025).

Training is divided into two stages. The first stage pre-trains the shape encoder-decoder with an auxiliary binary cross-entropy loss at each decoder layer $I^{t+1}$9: $$\mathcal{M} = \{(i,j)\mid f_i^t \leftrightarrow f_j^{t+1}\},$$0 The decoder loss averages these over decoder layers: $$\mathcal{M} = \{(i,j)\mid f_i^t \leftrightarrow f_j^{t+1}\},$$1

The second stage trains the transformer encoder with the log-likelihood loss from LightGlue: $$\mathcal{M} = \{(i,j)\mid f_i^t \leftrightarrow f_j^{t+1}\},$$2 Here, $$\mathcal{M} = \{(i,j)\mid f_i^t \leftrightarrow f_j^{t+1}\},$$3 is the set of true matches and $$\mathcal{M} = \{(i,j)\mid f_i^t \leftrightarrow f_j^{t+1}\},$$4 are the unmatched fruitlets in each image (Freeman et al., 5 Mar 2025).

The implementation details are specific. The spatial encoder-decoder uses voxel size 0.03, four downsampling blocks, and four upsampling blocks, and is pre-trained for 200 epochs with ADAM, batch size 64, initial learning rate $$\mathcal{M} = \{(i,j)\mid f_i^t \leftrightarrow f_j^{t+1}\},$$5, and 90% decay every 50 epochs. The transformer has four encoder layers with eight attention heads, trained for 200 epochs with ADAM, a 10-epoch warmup, and a learning rate starting at $$\mathcal{M} = \{(i,j)\mid f_i^t \leftrightarrow f_j^{t+1}\},$$6 and linearly decaying (Freeman et al., 5 Mar 2025).

5. Empirical results and observed operating regime

The ASTT evaluation uses stereo images collected over three years, 2021–2023, at the University of Massachusetts Amherst Cold Spring Orchard, including Fuji, Gala, and Honeycrisp apple varieties. Images were collected both by humans with a handheld illumination-invariant flash stereo camera and by a robot with the same camera mounted on a 7-DoF arm. A subset of 1350 images was manually labeled, yielding 4,362 cross-day image pairs spanning 1 to 7 days apart. The split is by cluster, with 2645 training pairs, 924 validation pairs, and 793 test pairs (Freeman et al., 5 Mar 2025).

On that dataset, ASTT achieves Precision: 92.9%, Recall: 92.2%, and F1-score: 92.4%. The reported baselines are ICP-Assoc at 89.5% F1, Desc-Assoc at 86.4% F1, and Loftr-Assoc at 72.3% F1 (Freeman et al., 5 Mar 2025).

The paper further reports that ASTT achieves the best precision and recall for pairs up to five days apart. For seven-day pairs, recall drops below ICP-Assoc, which the authors attribute to fewer training examples and larger growth-induced shape changes and displacement (Freeman et al., 5 Mar 2025). This suggests that the architecture is strongest in regimes where local spatial structure remains informative despite nontrivial appearance and pose change, but where the temporal gap has not yet induced extreme geometric drift.

The ablations clarify the operating regime. Positional information is decisive; shape contributes additional robustness; shape pre-training improves optimization. The resulting interpretation is not that ASTT eliminates the need for geometric structure, but that it leverages geometric structure more effectively through alternating intra-set and inter-set attention.

6. Relation to tracker-style spatio-temporal transformers

A separate line of work provides a useful comparison. STTrack, presented as “Autoregressive Queries for Adaptive Tracking with Spatio-Temporal Transformers,” does not formally define an ASTT module by name, but its decoder alternates between temporal query interaction and spatial feature interaction, making it the closest functional equivalent to an ASTT-style mechanism in single-object tracking (Xie et al., 2024).

STTrack is built from a HiViT-based spatial encoder, a temporal decoder, and a spatio-temporal information fusion module (STM). Spatial features are represented as

$$\mathcal{M} = \{(i,j)\mid f_i^t \leftrightarrow f_j^{t+1}\},$$7

with $$\mathcal{M} = \{(i,j)\mid f_i^t \leftrightarrow f_j^{t+1}\},$$8, and are processed as

$$\mathcal{M} = \{(i,j)\mid f_i^t \leftrightarrow f_j^{t+1}\},$$9

Temporal state is carried by learnable autoregressive target queries, with

$\mathcal{P}_i$0

and

$\mathcal{P}_i$1

This yields a sliding-window autoregressive process

$\mathcal{P}_i$2

Its temporal attention is explicitly asymmetric: $\mathcal{P}_i$3 with $\mathcal{P}_i$4, $\mathcal{P}_i$5, and $\mathcal{P}_i$6. The decoder then alternates this temporal interaction with multi-head attention over encoder features, followed by FFN layers. Final fusion is handled by STM through a parameter-free dot-product similarity

$\mathcal{P}_i$7

The relation between ASTT and STTrack is therefore structural rather than terminological. ASTT, in the strict sense, names the fruitlet-association architecture with alternating self-attention and cross-attention. STTrack, by contrast, is a tracker with autoregressive queries, temporal attention, and STM fusion. A plausible implication is that “alternating spatio-temporal transformer” can denote a broader design pattern—alternating temporal-state refinement and spatial-context interaction—even when the published module name differs. The STTrack results support the practical value of that pattern: the method reports 71.4% AUC and 72.7% AUC on LaSOT for STTrack-256 and STTrack-384, AO 73.8% and 76.0% on GOT-10k, and runtime of 67.6 FPS and 44.2 FPS, respectively, with about 72M parameters (Xie et al., 2024).

7. Significance, boundaries, and recurring points of confusion

ASTT occupies a specific methodological position at the intersection of transformer matching, geometric representation learning, and spatio-temporal association. The fruitlet-association formulation explicitly follows the spirit of feature matching transformers such as LightGlue and LoFTR, but adapts that strategy to stereo-derived 3D fruitlet representations and cluster-structured agricultural scenes (Freeman et al., 5 Mar 2025). Its contribution is therefore not only the use of attention, but the particular alternation of intra-day context formation and cross-day correspondence exchange on top of shape-and-position tokens.

Two points of confusion recur. First, ASTT is not synonymous with any transformer that processes temporal data. In the cited literature, the term names a specific architecture for fruitlet association, whereas tracker models such as STTrack are better described as ASTT-like rather than ASTT proper (Xie et al., 2024). Second, ASTT is not a purely appearance-based matcher. The ablation evidence shows that positional structure is more important than the learned shape descriptor in the reported orchard benchmark, indicating that the model’s strength comes from contextualized geometric association rather than descriptor similarity alone (Freeman et al., 5 Mar 2025).

Within those boundaries, the published evidence supports a clear interpretation. Alternation between self-attention and cross-attention is effective when objects are numerous, locally structured, partially ambiguous in appearance, and not always matchable. In such settings, ASTT provides a transformer-based mechanism for jointly modeling local spatial context, temporal correspondence, and explicit unmatched cases.