Adapt-STformer: Recurrent Transformer for Seq-VPR

• The paper introduces Adapt-STformer, which features a recurrent deformable transformer encoder that fuses spatial and temporal features for Seq-VPR.
• It achieves state-of-the-art retrieval accuracy with up to +17% recall improvement while cutting extraction time by 36% and memory usage by 35%.
• The framework leverages SeqGeM and SeqVLAD for robust global descriptor formation, making it well-suited for real-time, variable-length visual localization.

Adapt-STformer is a framework for sequential visual place recognition (Seq-VPR) that integrates a novel Recurrent Deformable Transformer Encoder (Recurrent-DTE) to address the demands of flexibility, computational efficiency, and low memory footprint in real-time visual localization tasks. Its design is aimed at overcoming the limitations of previous transformer-based approaches, specifically regarding inference speed, adaptability to variable sequence lengths, and resource consumption, while maintaining or surpassing state-of-the-art retrieval accuracy across diverse place recognition benchmarks.

1. Architectural Overview

Adapt-STformer processes an input sequence of RGB frames $\mathcal{S} = \{s_1, s_2, \dots, s_L\}$ by sequentially embedding, fusing, and aggregating spatiotemporal features with a three-stage architecture:

• Frame Encoding: Each input frame $s_t \in \mathbb{R}^{H \times W \times 3}$ passes through a compact convolutional transformer backbone (CCT384), yielding a frame-level embedding $f_t$ for $t=1,\dots,L$. The stack of embeddings forms $$F = [f_1, ..., f_L] \in \mathbb{R}^{L \times n \times D}$$, where $n$ is the token count and $D$ the embedding dimension.
• Recurrent-DTE Fusion: Sequential dependency is modeled with a shared Recurrent Deformable Transformer Encoder (DTE). At timestep $t$, the DTE receives as query $Q_t$ the previous output $\tilde{v}_{t-1}$ (with special initialization for $t=1$), and as key/value the current frame embedding $f_t$. Formally,

$$Q_1 = f_1 + \Delta \ \tilde{v}_t = \textsf{DTE}(Q_t = \tilde{v}_{t-1},\ K_t = V_t = f_t) \quad t \geq 2$$

where $\Delta$ is a learnable offset. This recurrence integrates temporal context efficiently.

• Aggregation and Global Descriptor Formation: All $\{\tilde{v}_t\}_{t=1}^L$ are stacked, permuted to treat tokens as batch items, and aggregated via SeqGeM (learnable mean pooling) followed by SeqVLAD (vector of locally aggregated descriptors) to form the final global sequence descriptor $V \in \mathbb{R}^{C \times D}$, with $C$ designating VLAD cluster centers.

2. Recurrent Deformable Transformer Encoder (Recurrent-DTE)

The Recurrent-DTE unifies temporal and spatial information fusion in a single, lightweight transformer encoder:

• Deformable Attention: Rather than dense global attention, each query attends to a subset of spatially localized offsets, reducing computational cost and emphasizing salient spatial regions. This sparsification avoids the quadratic complexity of standard self-attention.
• Temporal Recurrence: Information is fused iteratively, with each timestep refining its representation with the temporal "memory" from $\tilde{v}_{t-1}$ and the present frame’s encoding. Initialization uses the first frame's features combined with $\Delta$.
• Adaptability: Processing is naturally agnostic to sequence length $L$, as new frames are integrated sequentially without the need for token padding or static memory allocation. The same Recurrent-DTE module is reused at each timestep, confining memory and compute demands.

3. Sequence Aggregation Mechanisms

Adapt-STformer employs a two-stage aggregation after Recurrent-DTE fusion:

Stage	Function	Output Dimensionality
SeqGeM	Learnable mean pooling over time	$\mathbb{R}^{n \times D}$
SeqVLAD	Softly aggregates tokens into $C$ clusters	$\mathbb{R}^{C \times D}$

This hierarchical aggregation delivers compact, globally discriminative descriptors that are especially suited for retrieval in Seq-VPR settings.

4. Computational Efficiency and Flexibility

Key engineering advances underlying Adapt-STformer's efficiency:

• Recurrent Mechanism: Only the previous aggregate and current frame are needed at each step, minimizing memory and enabling variable-length sequences without architectural changes.
• Deformable Attention: Queries attend to a small set of keypoints; the DTE module is 71.3% faster than a standard non-deformable encoder for spatial attention, and the complete spatiotemporal processing is 88.1% faster than architectures that used separate temporal encoders.
• Aggregation Pipeline: SeqGeM accelerates sequence aggregation, leading to a further 56.8% speedup in this stage.

Overall, sequence extraction is 36% faster, and memory usage is 35% lower compared to the next-best baseline.

5. Experimental Results

Adapt-STformer was benchmarked on Nordland, Oxford (Easy and Hard splits), and NuScenes datasets. Empirical findings are summarized below:

Dataset	Recall Improvement	Extraction Time Reduction	Memory Usage Reduction
NuScenes	+17% (Recall)	36%	35%
Oxford-Hard	Statistically significant	36%	35%
Oxford-Easy	+3% @ Recall@1	–	–

In all cases, the method outperformed STformer, SeqVLAD, and related baselines, achieving not only higher recall (notably in harder scenarios) but also consistently lower computational cost, enabling real-time operation.

6. Comparison to Previous Seq-VPR Methods

Adapt-STformer departs from the common practice of using dual encoders (separate spatial and temporal transformers), instead favoring an integrated structure with the following benefits:

• Sequence Length Agnosticism: No fixed-length design, frame dropping, or padding required.
• Computational Scalability: Lowered complexity per frame; resources consumed are independent of the total sequence length.
• Single-shared Module: Simplifies architecture and reduces parameter count, enabling efficient deployment in constrained environments.

A plausible implication is that this design paradigm could see broader uptake in Seq-VPR and related video understanding domains, especially where compute and memory efficiency is paramount.

7. Practical Considerations and Limitations

Adapt-STformer achieves its efficiency and performance by leveraging deformable attention and recurrence. Remaining limitations include reliance on specific backbone choices (e.g., CCT384) and the possible sensitivity of VLAD aggregation to outlier features. Nonetheless, extensive ablations indicate that both the Recurrence and Deformable Attention components are essential to the observed recall and efficiency gains.

The method is applicable as a drop-in replacement in real-world Seq-VPR systems requiring variable sequence length processing under strict resource constraints, including robotics, automotive localization, and large-scale place retrieval scenarios. Continued investigation may further optimize backbone design or extend this approach to broader video understanding tasks.