SlotAdapt: Slot-Centric Adaptation Framework

• SlotAdapt is a slot-centric adaptation framework that refines scene decomposition via gradient descent and cross-view synthesis objectives.
• It integrates slot attention with pretrained diffusion models using adapter layers to enhance unsupervised object discovery, segmentation, and compositional editing.
• Empirical results show improved ARI, FG-ARI, and FID metrics across images, 3D point clouds, and video, highlighting its robustness under distribution shifts.

to=arxiv_search 天天中彩票中了json 彩神争霸怎么样ness? {"query":"(Prabhudesai et al., 2022) Slot-TTA Test-time Adaptation with Slot-Centric Models", "max_results": 5} SlotAdapt denotes a slot-centric adaptation framework within object-centric learning. In its earlier formulation, also described as Slot-TTA, it is a semi-supervised slot-centric scene decomposition model that adapts per test scene through gradient descent on reconstruction or cross-view synthesis objectives, with applications to images and 3D point clouds (Prabhudesai et al., 2022). In later work, the name refers to a method that combines slot attention with pretrained diffusion models by introducing adapters for slot-based conditioning, preserving the generative power of pretrained latent diffusion while reducing text-conditioning bias and supporting object discovery, segmentation, compositional generation, and controllable editing on real images; a subsequent thesis extends the same framework to video through Invariant Slot Attention and a Transformer-based temporal aggregator (Akan et al., 27 Jan 2025, Akan, 29 Sep 2025). Taken together, these sources suggest a shift from adapting slot-based decomposers at test time to adapting large frozen generative backbones through slot-conditioned interfaces.

1. Conceptual position within object-centric learning

SlotAdapt belongs to the broader class of object-centric learning methods that represent scenes through a set of slots. In this literature, slot attention is a widely adopted bottleneck that iteratively refines slot representations from low-level perceptual features. A persistent issue in such models is the need to predefine the number of slots, which presupposes prior knowledge of dataset complexity and ignores the fact that the number of objects varies across instances (Fan et al., 2024).

Within that context, SlotAdapt focuses less on discovering the correct slot cardinality and more on how slots interface with downstream inference or generation. The earlier Slot-TTA formulation emphasizes per-scene adaptation under distribution shift, while the diffusion-based SlotAdapt emphasizes compatibility with pretrained Stable Diffusion-style decoders without forcing slots into a text embedding manifold (Prabhudesai et al., 2022, Akan et al., 27 Jan 2025). This suggests that the defining property of SlotAdapt is not merely the presence of slots, but the use of slots as an adaptation interface: either for scene-specific refinement at inference time or for conditioning a frozen generative model.

A common misconception is to treat SlotAdapt as synonymous with any slot-based model. The comparison with Adaptive Slot Attention is useful here. AdaSlot introduces a discrete slot sampling module, a masked slot decoder, and a reconstruction-plus-slot-count-penalty objective to select the number of active slots dynamically (Fan et al., 2024). SlotAdapt, by contrast, is described in the supplied sources primarily in terms of adaptation mechanisms, guidance losses, and diffusion conditioning, not dynamic slot-number inference.

2. Slot-TTA: scene decomposition through per-scene adaptation

The 2022 formulation starts from a slot-centric encoder-decoder architecture. An input image $x \in \mathbb{R}^{H \times W \times 3}$ is processed by a convolutional backbone, producing a feature map of size $H' \times W' \times C$, which is flattened into $N = H' \cdot W'$ tokens $M \in \mathbb{R}^{N \times C}$. Linear projections $k(\cdot)$, $q(\cdot)$, and $v(\cdot)$ map these tokens to keys and values, and a set of $P$ learnable slot queries $\hat S \in \mathbb{R}^{P \times D}$ is iteratively updated by Slot Attention (Prabhudesai et al., 2022).

The core attention mechanism is explicitly competitive. Raw attention scores are computed as

$$A_{i p} = \frac{\exp\!\bigl(k(M_i)\cdot q(\hat S_p)^T\bigr)}{\sum_{p'=1}^P \exp\!\bigl(k(M_i)\cdot q(\hat S_{p'})^T\bigr)},$$

then normalized over tokens to give slot-wise weights

$H' \times W' \times C$0

with updates

$H' \times W' \times C$1

After three iterations, the final slots $H' \times W' \times C$2 are decoded into RGBA patches, and the reconstruction is formed by alpha compositing,

$H' \times W' \times C$3

Test-time adaptation is the distinguishing step. For each new scene, the decoder, and optionally the slot encoder’s final linear layers, are refined through gradient descent on a pixel-level reconstruction loss and a cross-view synthesis loss. With two views $H' \times W' \times C$4 and $H' \times W' \times C$5 and known relative camera pose $H' \times W' \times C$6,

$H' \times W' \times C$7

$H' \times W' \times C$8

and

$H' \times W' \times C$9

The backbone encoder and most of slot attention are frozen “to preserve object-centric inductive bias,” while a subset of parameters $N = H' \cdot W'$0 is updated for $N = H' \cdot W'$1 steps (Prabhudesai et al., 2022).

The reported hyperparameters are $N = H' \cdot W'$2, $N = H' \cdot W'$3, $N = H' \cdot W'$4 slots, learning rate $N = H' \cdot W'$5, adaptation steps per scene $N = H' \cdot W'$6, $N = H' \cdot W'$7, batch size $N = H' \cdot W'$8 scene with two views, and $N = H' \cdot W'$9 L2 weight decay on $M \in \mathbb{R}^{N \times C}$0, with KL divergence on slot assignments during pre-training to encourage sparsity (Prabhudesai et al., 2022).

3. Diffusion-based SlotAdapt for images

The later SlotAdapt reinterprets adaptation at the level of a pretrained generative backbone. Its stated goal is “to endow unsupervised slot-attention pipelines with the full generative power of large, pretrained latent diffusion models, while removing their built-in text-conditioning biases” (Akan et al., 27 Jan 2025). The model freezes the original UNet of a pretrained Stable Diffusion system, inserts new adapter cross-attention layers for slot conditioning, and adds a guidance loss aligning those adapters with slot-attention masks.

The high-level pipeline is as follows. The input image $M \in \mathbb{R}^{N \times C}$1 is processed by a visual backbone, specified as a CNN or DINOv2 in one summary and as a frozen DINOv2 ViT-B/14 in the later thesis summary, to yield patch-wise features $M \in \mathbb{R}^{N \times C}$2 (Akan et al., 27 Jan 2025, Akan, 29 Sep 2025). Slot Attention maps $M \in \mathbb{R}^{N \times C}$3 to slot vectors $M \in \mathbb{R}^{N \times C}$4 or $M \in \mathbb{R}^{N \times C}$5 together with encoder-side attention masks $M \in \mathbb{R}^{N \times C}$6. In parallel, the image is mapped to a latent variable through the pretrained Stable Diffusion VAE, noise is added, and the noised latent is denoised by a frozen UNet conditioned on both a register token and the slot adapters (Akan et al., 27 Jan 2025).

The adapter modules are inserted after every existing cross-attention in each down- and up-sampling block of the UNet. At block $M \in \mathbb{R}^{N \times C}$7, the feature map $M \in \mathbb{R}^{N \times C}$8 provides queries $M \in \mathbb{R}^{N \times C}$9, while the slots provide keys and values, $k(\cdot)$0 and $k(\cdot)$1. The adapter output is residual:

$k(\cdot)$2

followed by a feedforward network, and only adapter parameters are trained; the base UNet is frozen (Akan et al., 27 Jan 2025).

A second conditioning channel is the register token $k(\cdot)$3. One formulation states that SlotAdapt mean-pools either the slots or the image backbone features and prepends $k(\cdot)$4 to the text embedding sequence used by the original cross-attention layers. The later thesis summary defines

$k(\cdot)$5

and describes it as separating global scene/style information from per-object conditioning (Akan et al., 27 Jan 2025, Akan, 29 Sep 2025). This suggests that implementations varied, while preserving the same division between global context and per-slot object signals.

4. Objectives, guidance, and slot semantics

The diffusion objective follows latent DDPM training with slot-conditioned denoising. With clean latent $k(\cdot)$6, noisy latent $k(\cdot)$7, and $k(\cdot)$8, the model predicts noise via $k(\cdot)$9, and the loss is

$q(\cdot)$0

A self-supervised cross-attention guidance term then aligns encoder-side slot masks and decoder-side adapter masks:

$q(\cdot)$1

and the total objective is

$q(\cdot)$2

One summary reports a small guidance weight, “e.g. 0.025,” ramped in the first $q(\cdot)$3 K iterations, with $q(\cdot)$4 for the first $q(\cdot)$5 K and then a linear increase; the later thesis summary reports $q(\cdot)$6 with guidance warm-up after $q(\cdot)$7 k steps (Akan et al., 27 Jan 2025, Akan, 29 Sep 2025).

The slot attention backbone itself is iterative. Starting from flattened features $q(\cdot)$8 and randomly initialized slots, the updates take the form

$q(\cdot)$9

typically for $v(\cdot)$0 iterations (Akan et al., 27 Jan 2025). The later video-oriented formulation introduces Invariant Slot Attention, in which each slot carries pose parameters $v(\cdot)$1 and scale parameters $v(\cdot)$2, uses relative grids

$v(\cdot)$3

and updates pose by weighted moments of the attention map (Akan, 29 Sep 2025). That extension is used to disentangle object identity from pose before temporal aggregation.

The guidance loss is central to slot semantics. The stated purpose is to “align cross-attention from adapter layers with slot attention,” thereby enhancing alignment with objects in the input image “without using external supervision” (Akan et al., 27 Jan 2025). In the thesis summary, this is further described as encouraging each slot to cover whole objects rather than parts (Akan, 29 Sep 2025).

5. Empirical results across decomposition, segmentation, generation, and video

In the Slot-TTA setting, evaluation spans CLEVR-6, ObjectsRoom, and ObjectScans. Reported metrics are ARI, FG-ARI, and mIoU. On CLEVR-6, a feed-forward model without test-time adaptation obtains ARI $v(\cdot)$4, SlotAdapt with TTA reaches ARI $v(\cdot)$5 with “+18%,” and other TTA baselines such as self-supervised entropy minimization report ARI $v(\cdot)$6 (Prabhudesai et al., 2022). Qualitatively, the paper describes “diffuse, merged masks on occluded objects” before adaptation and “sharp object boundaries, correct number of slots even under heavy occlusion” after adaptation.

For the diffusion-based image model, the thesis summary reports unsupervised object segmentation results on COCO, VOC, and MOVi-E. On COCO, SlotAdapt+guidance achieves FG-ARI $v(\cdot)$7, $v(\cdot)$8, and $v(\cdot)$9; on VOC, FG-ARI $P$0 and $P$1; on MOVi-E, FG-ARI $P$2 and $P$3 (Akan, 29 Sep 2025). An earlier summary gives COCO/VOC segmentation in comparative form, stating “FG-ARI $P$4 (vs. DINOSAUR 34.3, SlotDiff 37.2, LSD 33.8)” and “$P$5: +3–10 pp improvements over prior” (Akan et al., 27 Jan 2025).

Generative metrics are also reported. For reconstruction quality, SlotAdapt records FID $P$6 and KID $P$7 versus SlotDiffusion FID $P$8 and LSD $P$9 in one summary; the thesis gives reconstruction FID/KID $\hat S \in \mathbb{R}^{P \times D}$0 versus $\hat S \in \mathbb{R}^{P \times D}$1 and $\hat S \in \mathbb{R}^{P \times D}$2 (Akan et al., 27 Jan 2025, Akan, 29 Sep 2025). In compositional generation, “by swapping, adding or removing individual slot tokens $\hat S \in \mathbb{R}^{P \times D}$3 at inference time,” the model performs object-level editing in complex COCO scenes, and compositional FID is reported as $\hat S \in \mathbb{R}^{P \times D}$4 versus SlotDiff $\hat S \in \mathbb{R}^{P \times D}$5 and LSD $\hat S \in \mathbb{R}^{P \times D}$6 in one summary, or $\hat S \in \mathbb{R}^{P \times D}$7 FID/KID versus $\hat S \in \mathbb{R}^{P \times D}$8 and $\hat S \in \mathbb{R}^{P \times D}$9 in the thesis (Akan et al., 27 Jan 2025, Akan, 29 Sep 2025).

The video extension adds a Transformer-based temporal aggregator over per-frame slots and evaluates on YouTube-VIS’19 and DAVIS’17. Reported video object segmentation results are YTVIS $$A_{i p} = \frac{\exp\!\bigl(k(M_i)\cdot q(\hat S_p)^T\bigr)}{\sum_{p'=1}^P \exp\!\bigl(k(M_i)\cdot q(\hat S_{p'})^T\bigr)},$$0, FG-ARI $$A_{i p} = \frac{\exp\!\bigl(k(M_i)\cdot q(\hat S_p)^T\bigr)}{\sum_{p'=1}^P \exp\!\bigl(k(M_i)\cdot q(\hat S_{p'})^T\bigr)},$$1, compared with $$A_{i p} = \frac{\exp\!\bigl(k(M_i)\cdot q(\hat S_p)^T\bigr)}{\sum_{p'=1}^P \exp\!\bigl(k(M_i)\cdot q(\hat S_{p'})^T\bigr)},$$2 for fully supervised SOLV, and DAVIS $$A_{i p} = \frac{\exp\!\bigl(k(M_i)\cdot q(\hat S_p)^T\bigr)}{\sum_{p'=1}^P \exp\!\bigl(k(M_i)\cdot q(\hat S_{p'})^T\bigr)},$$3, FG-ARI $$A_{i p} = \frac{\exp\!\bigl(k(M_i)\cdot q(\hat S_p)^T\bigr)}{\sum_{p'=1}^P \exp\!\bigl(k(M_i)\cdot q(\hat S_{p'})^T\bigr)},$$4 (Akan, 29 Sep 2025). For video generation, the same source reports on YTVIS PSNR $$A_{i p} = \frac{\exp\!\bigl(k(M_i)\cdot q(\hat S_p)^T\bigr)}{\sum_{p'=1}^P \exp\!\bigl(k(M_i)\cdot q(\hat S_{p'})^T\bigr)},$$5, SSIM $$A_{i p} = \frac{\exp\!\bigl(k(M_i)\cdot q(\hat S_p)^T\bigr)}{\sum_{p'=1}^P \exp\!\bigl(k(M_i)\cdot q(\hat S_{p'})^T\bigr)},$$6, LPIPS $$A_{i p} = \frac{\exp\!\bigl(k(M_i)\cdot q(\hat S_p)^T\bigr)}{\sum_{p'=1}^P \exp\!\bigl(k(M_i)\cdot q(\hat S_{p'})^T\bigr)},$$7, FID $$A_{i p} = \frac{\exp\!\bigl(k(M_i)\cdot q(\hat S_p)^T\bigr)}{\sum_{p'=1}^P \exp\!\bigl(k(M_i)\cdot q(\hat S_{p'})^T\bigr)},$$8, FVD $$A_{i p} = \frac{\exp\!\bigl(k(M_i)\cdot q(\hat S_p)^T\bigr)}{\sum_{p'=1}^P \exp\!\bigl(k(M_i)\cdot q(\hat S_{p'})^T\bigr)},$$9, and on DAVIS PSNR $H' \times W' \times C$00, SSIM $H' \times W' \times C$01, LPIPS $H' \times W' \times C$02, FID $H' \times W' \times C$03, FVD $H' \times W' \times C$04.

6. Limitations, related directions, and open questions

Several limitations recur across the SlotAdapt lineage. In the Slot-TTA formulation, the model “still struggles when objects have extremely similar appearance (slots collapse)” and requires known camera poses for cross-view losses (Prabhudesai et al., 2022). In the diffusion-based image model, reconstructed or edited images “sometimes exhibit slight global shifts relative to the source,” and under- or over-segmentation still occurs occasionally (Akan et al., 27 Jan 2025). The video extension adds that background reconstruction in edits can drift because of the stochastic sampling nature of diffusion, object addition remains challenging without anchor slots, and the approach is limited to relatively short, medium-resolution clips (Akan, 29 Sep 2025).

These limitations clarify several research directions already named in the sources. The Slot-TTA work proposes adapting which subset of parameters to fine-tune per scene and incorporating few-shot prompts, such as pointing to an object, to steer slot assignments (Prabhudesai et al., 2022). The diffusion-based work proposes explicit reconstruction-fidelity objectives, user prompts that map to specific slots, and adaptive slot splitting or merging (Akan et al., 27 Jan 2025). The thesis extends this list to dynamic slot merging or splitting, physical dynamics modeling, and interactive slot edits with physics or depth modules (Akan, 29 Sep 2025).

A further point of comparison concerns slot cardinality. AdaSlot shows that a model can start from a large pool of candidate slots $H' \times W' \times C$05, sample a binary mask $H' \times W' \times C$06 with Gumbel-Softmax, zero out dropped masks in the decoder, and optimize a reconstruction loss plus a sparsity penalty on the expected number of kept slots (Fan et al., 2024). Since later SlotAdapt summaries explicitly list “adaptive slot splitting/merging to handle variable object counts” as future work, a plausible implication is that SlotAdapt’s current strengths lie in adaptation and conditioning rather than in dynamically resolving slot number.

In that sense, SlotAdapt occupies a distinctive position in the slot-based literature. It treats slots not only as object-centric latent variables but as an interface layer for adapting perception and generation: first by refining a decomposition model on each test scene, and later by coupling slots to a frozen diffusion backbone through adapters, register tokens, and self-supervised attention guidance (Prabhudesai et al., 2022, Akan et al., 27 Jan 2025).