Test-time Adaptation with Slot-Centric Models (2203.11194v3)

Abstract: Current visual detectors, though impressive within their training distribution, often fail to parse out-of-distribution scenes into their constituent entities. Recent test-time adaptation methods use auxiliary self-supervised losses to adapt the network parameters to each test example independently and have shown promising results towards generalization outside the training distribution for the task of image classification. In our work, we find evidence that these losses are insufficient for the task of scene decomposition, without also considering architectural inductive biases. Recent slot-centric generative models attempt to decompose scenes into entities in a self-supervised manner by reconstructing pixels. Drawing upon these two lines of work, we propose Slot-TTA, a semi-supervised slot-centric scene decomposition model that at test time is adapted per scene through gradient descent on reconstruction or cross-view synthesis objectives. We evaluate Slot-TTA across multiple input modalities, images or 3D point clouds, and show substantial out-of-distribution performance improvements against state-of-the-art supervised feed-forward detectors, and alternative test-time adaptation methods.

• The paper introduces a slot attention mechanism that compresses high-dimensional feature maps into compact slot vectors.
• It employs softmax-normalized attention and GRU-based iterative updates to refine slot representations effectively.
• The method improves model interpretability and resource efficiency, offering promising benefits for test-time adaptation.

Slot Attention for Feature Compression in Encoded Representations

The paper presents a methodological enhancement in feature representation compression through the Slot Attention mechanism. This approach aims to distill a set of feature vectors, $M \in \mathbb{R}^{N \times C}$, into a smaller set of slot vectors, $S \in \mathbb{R}^{P \times D}$, where $N$ and $P$ denote the number of tokens and slots, respectively, and $C$ and $D$ correspond to their dimensionalities.

Methodological Framework

The process begins by deriving an attention matrix, $A \in \mathbb{R}^{N \times P}$, between an encoded feature map $M$ and a set of learnable latent embeddings, $\hat{S} \in \mathbb{R}^{P \times D}$. This attention matrix is computed using the expression:

$A_{i,p} = \frac{\exp{(k(M_i) \cdot q(\hat{S_p})^T)}}{\sum_{p=0}^{P-1}\exp{(k(M_i) \cdot q(\hat{S_p})^T)}$

Where $k$, $q$, and $v$ represent linear transformations, $k$ and $v$ being applied to map $M$ to $\mathbb{R}^{N \times D}$, while $q$ transforms $\hat{S}$ within the same dimensional space. The distinctive feature of this approach is the application of softmax normalization over the slot axis, promoting a competitive mechanism where slot vectors selectively attend to specific features in $M$.

The extraction of slot vectors, $S$, from $M$ involves updating $\hat{S}$ through a Gated Recurrent Unit (GRU) operation:

$S = \mathrm{GRU}(\hat{S}, U)$

with $U$ determined by a weighted average of the transformed feature map $\hat{M}$ via the re-normalized attention matrix $\hat{A}$:

$$U = \hat{A}^T \hat{M}, \quad \hat{A}_{i,p} = \dfrac{A_{i,p}}{\sum_{i=0}^{N-1}A_{i,p}}$$

The iterative framework runs for three iterations, each reinforcing the approximation of $\hat{S}$ to the extracted slot vectors $S$, thereby refining the feature compression.

Implications and Future Directions

This paper underscores the potential of attention mechanisms in condensing complex feature representations into more manageable sets, which is critical for tasks demanding efficient and scalable models. The Slot Attention module not only compresses data but also facilitates interpretability by spotlighting feature-vector interactions.

Theoretically, this approach augments the repertoire of attention-based feature encoding strategies, laying groundwork for further exploration in areas of unsupervised and semi-supervised learning. Practically, its ability to streamline data preprocessing in resource-constrained environments marks a significant advancement.

Future developments could focus on optimizing the computational efficiency of the GRU-enhanced Slot Attention framework and exploring its applicability across diverse datasets and model architectures. Additionally, integrating feedback mechanisms to dynamically adjust the slot representations during model training could further enhance model robustness and flexibility in real-world applications.