SmoothSA: Smoothing Slot Attention

• SmoothSA is an object-centric learning modification that improves Slot Attention by preheating cold-start queries and differentiating recurrence for videos.
• It incorporates a Transformer-based preheater and distinct iteration counts to inject sample-specific cues and reduce brittleness in object aggregation.
• Empirical evaluations show significant gains on object discovery, recognition, and reasoning benchmarks, validating its targeted corrections in SA pipelines.

Searching arXiv for the SmoothSA paper and closely related Slot Attention/object-centric learning context. SmoothSA is an object-centric learning modification to Slot Attention that addresses two mismatches in mainstream Slot Attention-based pipelines: the weak first aggregation caused by cold-start queries, and the use of homogeneous recurrent transforms for video frames whose query states are qualitatively different. Introduced in "Smoothing Slot Attention Iterations and Recurrences" (Zhao et al., 7 Aug 2025), SmoothSA augments standard Slot Attention with a preheating mechanism for image inputs and first video frames, and with differentiated iteration schedules across video recurrence. Its central claim is that Slot Attention should process fresh, uninformative queries differently from recurrent, already informative queries. In the reported experiments, SmoothSA is instantiated on top of DIAS$^i$ for images and RandSF.Q for videos, yielding gains on object discovery, object recognition, and downstream visual question answering benchmarks (Zhao et al., 7 Aug 2025).

1. Conceptual position within object-centric learning

Slot Attention (SA) is a standard mechanism in object-centric learning (OCL) for aggregating image features into a set of slot vectors via iterative refinement. In video settings, analogous refinement is performed recurrently across frames. SmoothSA targets two specific failure modes in this setup (Zhao et al., 7 Aug 2025).

First, standard SA begins from cold-start queries on images and on the first frame of a video. These initial queries are not derived from the current sample and therefore lack sample-specific cues. SmoothSA identifies this as a source of weak first-step aggregation, because the model must first discover what to attend to before it can perform accurate object aggregation (Zhao et al., 7 Aug 2025).

Second, in video OCL, first-frame and non-first-frame queries differ structurally. The first frame starts from cold-start queries, whereas later frames are initialized from previous slots and are therefore already informative. Standard video OCL nevertheless applies the same SA transform across all frames. SmoothSA treats this as a recurrence mismatch and replaces the homogeneous recurrence with an iteration schedule that reflects the difference between cold-start and recurrent query states (Zhao et al., 7 Aug 2025).

This suggests that SmoothSA is best understood not as a new object-centric architecture, but as a targeted correction to the initialization and recurrence assumptions built into mainstream Slot Attention usage.

2. Preheating cold-start queries

The first component of SmoothSA is preheating, which modifies the initialization stage for images and first video frames. Instead of feeding the initial cold-start queries $Q_1$ directly into Slot Attention, SmoothSA computes

$Q_1^* = \phi_p(Q_1, F_1),$

where $F_1$ denotes image features and $\phi_p$ is a preheater module (Zhao et al., 7 Aug 2025).

The preheated query is then used as the starting state for refinement: $S_1^{(0)} := \mathrm{sg}(Q_1^*),$ where $\mathrm{sg}(\cdot)$ denotes stop-gradient (Zhao et al., 7 Aug 2025). This design injects sample-specific feature information before the first SA iteration while partially disentangling the optimization of the preheater and the downstream aggregator.

The preheater is implemented as a single Transformer decoder block, but with its self-attention and cross-attention switched. The stated reason is architectural: interaction among cold-start queries is not especially useful before those queries have acquired object semantics, whereas cross-attention to features should happen earlier in order to inject sample-specific information (Zhao et al., 7 Aug 2025).

The preheater is trained by self-distillation internal to the OCL model. Its objective is

$$\arg \min_{C,\phi_n,\phi_p} \mathrm{MSE}\big(Q_1^*, \mathrm{sg}(S_1)\big),$$

where $C$ denotes query-initialization cues, $\phi_n$ is the query initializer, and $Q_1$0 is the final refined slot representation (Zhao et al., 7 Aug 2025). The target is the model’s own final slot representation rather than an external annotation. The paper characterizes this as a form of rigid self-distillation (Zhao et al., 7 Aug 2025).

A plausible implication is that SmoothSA treats the final slot state as a teacher signal for improving the initial query state, thereby shortening the effective refinement path required by standard SA.

3. Differentiated recurrence across video frames

The second component of SmoothSA concerns video recurrence. Standard video SA uses the same transformation on the first frame and on later frames. SmoothSA preserves the same SA parameters but changes the number of iterations used in each case (Zhao et al., 7 Aug 2025).

For the first frame, SmoothSA follows the standard multi-iteration pattern: $Q_1$1

$Q_1$2

$Q_1$3

$Q_1$4

For later frames, queries are predicted from previous slots,

$Q_1$5

but only a single SA iteration is applied: $Q_1$6

$Q_1$7

$Q_1$8

Thus SmoothSA uses 3 iterations on the first frame and 1 iteration on non-first frames (Zhao et al., 7 Aug 2025).

The argument is that recurrent queries already encode object identity and temporal context, and therefore do not require the same degree of refinement as cold-start queries. The paper further states that this change reduces computation on later frames by about $Q_1$9 compared with applying 3 iterations throughout (Zhao et al., 7 Aug 2025).

SmoothSA explicitly prefers shared SA weights with differentiated iteration counts over separate parameter sets for first and later frames. According to the reported ablation, shared weights preserve a general aggregation capability better than separate weights while still allowing frame-specific behavior through the schedule itself (Zhao et al., 7 Aug 2025).

4. Algorithmic formulation

SmoothSA applies slightly differently to images, first video frames, and later video frames, but the full workflow is compact.

For images and first video frames, the procedure is:

1. Initialize cold-start queries $Q_1^* = \phi_p(Q_1, F_1),$0 from cues $Q_1^* = \phi_p(Q_1, F_1),$1.
2. Preheat them via

$Q_1^* = \phi_p(Q_1, F_1),$2

1. Apply stop-gradient:

$Q_1^* = \phi_p(Q_1, F_1),$3

2. Run SA refinement for 3 iterations:

$Q_1^* = \phi_p(Q_1, F_1),$4

3. Train the preheater using MSE toward the final slots $Q_1^* = \phi_p(Q_1, F_1),$5 (Zhao et al., 7 Aug 2025).

For later video frames $Q_1^* = \phi_p(Q_1, F_1),$6, the procedure is:

4. Predict the current queries from previous slots:

$Q_1^* = \phi_p(Q_1, F_1),$7

5. Run only one SA iteration:

$Q_1^* = \phi_p(Q_1, F_1),$8

This compactly defines the “smoothed” iteration-and-recurrence schedule of SmoothSA (Zhao et al., 7 Aug 2025).

The paper presents SmoothSA as a modification rather than a replacement: it leaves Slot Attention itself intact, but alters the query condition entering the first frame and the recurrence depth used thereafter.

5. Empirical evaluation

SmoothSA is evaluated as an upgrade to two strong OCL systems: DIAS$Q_1^* = \phi_p(Q_1, F_1),$9 for images, producing SmoothSA$F_1$0, and RandSF.Q for videos, producing SmoothSA$F_1$1. The experiments use DINOv2 ViT-S/14 features as the encoder backbone (Zhao et al., 7 Aug 2025).

Benchmarks and metrics

The reported object discovery datasets are:

Setting	Dataset	Slots
Image OCL	ClevrTex	11
Image OCL	COCO	7
Image OCL	VOC	6
Video OCL	YTVIS HQ	7

The corresponding object discovery metrics are ARI, ARI$F_1$2, mBO, and mIoU (Zhao et al., 7 Aug 2025).

For object recognition, the OCL model is frozen and a 2-layer MLP is trained on slots for COCO image recognition and YTVIS video recognition, using top-1 classification and bounding-box regression R2 as metrics (Zhao et al., 7 Aug 2025).

For downstream reasoning, slots are combined with the Aloe multi-modal reasoning model on GQA for images and CLEVRER for videos (Zhao et al., 7 Aug 2025).

Reported results on object discovery

On ClevrTex, SmoothSA$F_1$3 reports ARI 76.8, ARI$F_1$4 80.8, mBO 60.0, and mIoU 58.1 (Zhao et al., 7 Aug 2025). On COCO, SmoothSA$F_1$5 reports ARI 26.2, ARI$F_1$6 42.1, mBO 33.2, and mIoU 31.7 (Zhao et al., 7 Aug 2025). On VOC, SmoothSA$F_1$7 reports ARI 30.6, ARI$F_1$8 34.3, mBO 45.3, and mIoU 44.1; the paper highlights about a 4-point ARI gain over prior state of the art (Zhao et al., 7 Aug 2025).

On YTVIS HQ, SmoothSA$F_1$9 reports ARI 44.1, ARI$\phi_p$0 61.5, mBO 41.1, and mIoU 40.6, outperforming compared video baselines including RandSF.Q (Zhao et al., 7 Aug 2025).

Reported results on recognition and reasoning

For COCO recognition, SmoothSA$\phi_p$1 + MLP achieves 0.73 top-1 and 0.64 bbox R2, compared with SPOT + MLP at 0.67 class top-1 and 0.62 bbox R2 (Zhao et al., 7 Aug 2025). For YTVIS recognition, SmoothSA$\phi_p$2 + MLP achieves 0.50 top-1 and 0.62 bbox R2, compared with SlotContrast + MLP at 0.40 top-1 and 0.53 bbox R2 (Zhao et al., 7 Aug 2025).

For GQA, SmoothSA$\phi_p$3 + Aloe reaches 56.7%, compared with SPOT + Aloe at 52.3% (Zhao et al., 7 Aug 2025). For CLEVRER, SmoothSA$\phi_p$4 + Aloe reaches 98.7% per-option and 96.9% per-question, compared with SlotContrast + Aloe at 97.2% per-option and 95.6% per-question (Zhao et al., 7 Aug 2025).

Taken together, these results indicate that the gains are not restricted to segmentation-style object discovery metrics; they transfer to recognition and reasoning tasks that depend on slot quality.

6. Ablations and interpretation

The ablation studies are central to the interpretation of SmoothSA, because the method is intentionally minimal and its justification depends on whether each modification has a distinct effect (Zhao et al., 7 Aug 2025).

Preheater design

On COCO, a Transformer decoder block preheater achieves 68.3, whereas using an SA module as preheater achieves 63.3 (Zhao et al., 7 Aug 2025). The switched ordering of self-attention and cross-attention is critical: with the switch, performance is 68.3; without it, performance drops to 49.6 (Zhao et al., 7 Aug 2025). This is one of the strongest pieces of evidence that early feature conditioning, rather than early query-query interaction, is the relevant design choice.

Applying stop-gradient to the preheated query yields 68.3, compared with 67.5 without stop-gradient, indicating a modest but consistent gain (Zhao et al., 7 Aug 2025).

The preheating loss weight also matters. Reported values are 59.7 at weight 10, 65.5 at 50, 68.3 at 100, and 67.4 at 200, with 100 the best among the tested choices (Zhao et al., 7 Aug 2025).

Transform differentiation

For recurrence, the paper compares shared and separate weights on YTVIS and reports 68.3 for shared weights versus 52.3 for separate weights (Zhao et al., 7 Aug 2025). This supports the claim that differentiated behavior should come from the schedule rather than from frame-specific parameter sets.

For iteration counts, the best schedule is 3+1 for both unconditional and conditional video OCL. On unconditional YTVIS, the reported values are 105.6 for 3+1, 97.4 for 1+1, and 103.4 for 3+3. On conditional MOVi-C, they are 136.3 for 3+1, 133.9 for 1+1, and 132.7 for 3+3 (Zhao et al., 7 Aug 2025). This directly supports the paper’s claim that first frames require stronger refinement and later frames benefit from lighter recurrence.

Qualitative interpretation

The qualitative analysis reports sharper object masks, cleaner slot-object assignment, and better consistency across frames in videos (Zhao et al., 7 Aug 2025). A discussion experiment further compares a model trained with preheating to one trained without it under reduced iteration counts at test time. The preheated model degrades only slightly, whereas the non-preheated model collapses much more quickly (Zhao et al., 7 Aug 2025).

This suggests that “smoothing” in SmoothSA refers not merely to improved metrics, but to a reduced brittleness of the iterative aggregation process itself.

7. Significance, scope, and related distinctions

SmoothSA’s significance lies in its diagnosis of two narrow but consequential mismatches in Slot Attention practice: mismatch between cold-start queries and the first aggregation step, and mismatch between first-frame and later-frame query conditions in video recurrence (Zhao et al., 7 Aug 2025). Rather than altering the slot representation, the assignment objective, or the backbone encoder, it retools the transition into Slot Attention and the recurrence schedule around it.

A common misconception would be to read SmoothSA as a general-purpose smoothing regularizer or as a broad optimization technique. In the paper, the term is more specific: it denotes smoothing Slot Attention iterations and recurrences, not loss smoothing, gradient smoothing, or generic temporal regularization (Zhao et al., 7 Aug 2025).

Another potential misunderstanding is to treat preheating as external supervision. The reported formulation is explicitly self-distilled: the supervision target is the model’s own final slot state, and no additional labels are introduced for that purpose (Zhao et al., 7 Aug 2025).

SmoothSA also does not abandon iterative refinement. On the contrary, it preserves multi-step refinement where cold-start conditions justify it and reduces iterations only when recurrent initialization already carries sufficient information. This makes the method compatible with the original logic of Slot Attention while revising where and how strongly that logic should be applied.

A plausible implication is that SmoothSA exemplifies a broader design principle in OCL: iterative inference mechanisms may benefit when the update schedule depends on the informativeness of the latent queries rather than being fixed across inputs and time. The reported results indicate that even a small intervention along these lines can improve not only discovery metrics but also the utility of learned slots in recognition and reasoning pipelines (Zhao et al., 7 Aug 2025).