---
title: Invariant Slot Attention (ISA)
url: https://www.emergentmind.com/topics/invariant-slot-attention-isa
type: topic
---

# Invariant Slot Attention (ISA)

Searching arXiv for Invariant Slot Attention and closely related slot-attention papers to ground the article.
arxiv_search(query="Invariant Slot Attention object discovery with slot-centric reference frames", max_results=5)
Invariant Slot Attention (ISA) is an object-centric learning method that extends Slot Attention by assigning each slot its own reference frame, so that a slot represents an object in object-relative rather than camera-relative coordinates. Its stated purpose is to improve unsupervised object discovery by reducing the entanglement of appearance or identity with pose, and by incorporating spatial symmetries directly into both the attention mechanism and the decoder through slot-centric translation, scaling, and optional rotation of positional encodings [2302.04973].

## 1. Conceptual setting and motivation

ISA belongs to the object-centric learning literature, in which a scene is represented as a set of latent vectors, each intended to capture one object-like entity. The immediate precursor is Slot Attention, which maps a set of perceptual features to a set of exchangeable slots through iterative competitive attention, and is permutation-invariant with respect to input order and permutation-equivariant with respect to slot order [2006.15055].

The specific limitation targeted by ISA is that standard Slot Attention uses absolute image coordinates in both the attention module and the spatial broadcast decoder. In that formulation, a slot’s content depends on where the object appears, so appearance and pose become entangled. The ISA paper frames this as a failure to adequately capture spatial symmetries present in the visual world, with adverse consequences for sample efficiency, generalization to unseen positions, scales, and orientations, and consistency of learned object representations [2302.04973].

The core intervention is the introduction of slot-centric reference frames. Each slot is augmented with per-object pose variables—translation, scale, and optionally rotation—and positional encodings are transformed relative to the slot’s estimated pose. In the paper’s terminology, this is meant to make the slot latent approximately invariant to per-object pose transformations, while the pose parameters themselves are equivariant to those transformations. A common misconception is that ISA is a fully exact group-equivariant architecture; the method is instead presented as a practical mechanism for approximate, learned invariance in cluttered natural scenes, where 3D projection, occlusion, and background structure violate exact symmetry assumptions [2302.04973].

## 2. Baseline Slot Attention and the point of departure

Slot Attention provides the architectural substrate from which ISA is derived. In the standard formulation, an encoder converts an image into a set of feature vectors, slots are initialized from a shared Gaussian distribution, and iterative attention updates bind features to slots by competitive routing over multiple rounds [2006.15055].

ISA inherits this iterative slot-binding structure but changes how spatial information enters it. The baseline equations cited by the ISA paper are
\[
\mathrm{keys} = f\left(\mathcal{K}(\mathrm{inputs}) + g(\mathrm{abs\_grid})\right)
\]
and
\[
\mathrm{values} = f\left(\mathcal{V}(\mathrm{inputs}) + g(\mathrm{abs\_grid})\right),
\]
where \(\mathrm{abs\_grid}\) denotes the absolute coordinate grid scaled to \([-1,1]\), and \(\mathcal{K}, \mathcal{V}, \mathcal{Q}, g\) are linear maps. Queries are computed from slots, and dot products between queries and keys are normalized over slots so that slots compete for each input token [2302.04973].

This baseline already has an important symmetry property: slots are exchangeable rather than index-specialized. However, the same literature also emphasizes that absolute positional embeddings break translation equivariance, and that the original model’s invariance is primarily permutation symmetry over sets rather than invariance to spatial transformations [2006.15055]. ISA should therefore be understood not as a replacement for slot exchangeability, but as an extension that adds per-object spatial canonicalization on top of the standard slot-competition mechanism [2302.04973].

## 3. Slot-centric reference frames and relative positional encoding

The defining mechanism of ISA is the replacement of absolute positional encoding by slot-relative positional encoding. For translation and scale, each slot \(k\) carries a position \(S_p^k \in \mathbb{R}^2\) and scale \(S_s^k \in \mathbb{R}^2\), and ISA defines
\[
\mathrm{rel\_grid}^k = (\mathrm{abs\_grid} - S_p^k)\ /\ S_s^k.
\]
For translation, rotation, and scale jointly, ISA uses
\[
\mathrm{rel\_grid}^k = (S_r^k)^{-1}(\mathrm{abs\_grid} - S_p^k)\ /\ S_s^k,
\]
where \(S_r^k \in \mathbb{R}^{2\times 2}\) is a per-slot rotation matrix [2302.04973].

These transformed coordinates are inserted directly into the attention computation. Instead of one key and one value per input token, ISA constructs slot-specific keys and values:
\[
\mathrm{keys}^k = f\left(\mathcal{K}(\mathrm{inputs}) + g(\mathrm{rel\_grid}^k)\right),
\]
\[
\mathrm{values}^k = f\left(\mathcal{V}(\mathrm{inputs}) + g(\mathrm{rel\_grid}^k)\right).
\]
The same image token is therefore encoded differently depending on which slot is querying it. This slot-conditioned key/value construction is the paper’s central architectural change: cross-attention becomes pose-relative on a per-slot basis rather than globally absolute [2302.04973].

Pose parameters are estimated from the attention masks themselves. Position is computed as the weighted centroid
\[
S_p = \frac{\sum_{n=1}^{N} \mathrm{attn}_n \cdot \mathrm{abs\_grid}_n}
{\sum_{n=1}^{N} \mathrm{attn}_n},
\]
and scale from the weighted variance
\[
S_s = \sqrt{ \frac{ \sum_{n=1}^{N} (\mathrm{attn}_n + \epsilon)\cdot(\mathrm{abs\_grid}_n - S_p)^2 }
{ \sum_{n=1}^{N} (\mathrm{attn}_n + \epsilon) } }.
\]
If rotation is enabled, the paper uses a weighted PCA heuristic,
\[
v_1^k, v_2^k = \mathrm{WeightedPCA}(\mathrm{abs\_grid}, \mathrm{attn}^k),
\]
followed by post-processing to produce \(S_r^k\) [2302.04973].

The decoder is modified analogously. Instead of broadcasting a slot over a global grid with absolute coordinates, ISA uses the slot’s relative grid:
\[
(R,G,B,\alpha) = D_{\phi}(\mathrm{SB} + h(\mathrm{rel\_grid})),
\]
where \(\mathrm{SB}\) is the spatially broadcast slot tensor. The decoder thus reconstructs each object in slot-relative coordinates, and the paper argues that this encourages weight sharing across positions and scales [2302.04973].

## 4. Architecture, variants, and optimization details

ISA preserves the standard three-part object-discovery scaffold: an encoder, a slot-attention module, and a slot-wise decoder. The encoder is either a shallow 4-layer CNN for simpler synthetic settings or a ResNet-34 with group normalization for more difficult datasets such as CLEVRTex and Waymo. The Slot Attention block runs for 3 iterations, followed by one extra pass to compute final pose statistics without updating the slot latent vectors. Typical settings use 11 slots, with 4 slots on Tetrominoes and 5 slots on MultiShapeNet [2302.04973].

The paper distinguishes several symmetry configurations. **ISA-T** handles translation, **ISA-TS** handles translation and scale, and **ISA-TSR** additionally includes rotation. In appendix pseudocode, a scaling constant \(\delta = 5\) is introduced so that
\[
\text{rel\_grid}^k = \big[(S_r^k)^{-1}(\text{abs\_grid} - S_p^k)\big] / (S_s^k \times \delta),
\]
with the stated purpose of numerical stability when scales are small [2302.04973].

Two implementation details are emphasized as especially consequential. First, gradients must flow through pose estimation: stopping gradients through \(S_p, S_s, S_r\) strongly hurts performance. Second, although ISA requires per-slot keys and values and therefore increases the number of key/value embeddings from \(N\) to \(N\times K\), the paper states that the overhead is negligible in the common regime of roughly ten slots [2302.04973].

Optimization is also specified explicitly. Training uses Adam, with learning rate \(4\times10^{-4}\) for most datasets and \(2\times10^{-4}\) for Waymo depth targets, 50k warm-up steps for most datasets and 5k on Tetrominoes, cosine decay, batch size 64, 500k steps for most datasets, 300k for Waymo, and 50k for Tetrominoes [2302.04973].

A further subtlety is symmetry breaking. The ISA paper notes that encoder backbones, especially stronger ones such as ResNet, can leak global position information through padding artifacts. To mitigate this, some experiments append pose parameters to the slot vector before the GRU (“-Append”), explicitly breaking symmetry when needed. This suggests that ISA is not committed to maximal symmetry preservation in all components; rather, it uses slot-centric reference frames as the principal symmetry-inducing mechanism while permitting controlled departures when exact invariance is too restrictive in practice [2302.04973].

## 5. Empirical behavior, ablations, and limitations

ISA is evaluated on Tetrominoes, Objects Room, CLEVR, MultiShapeNet, CLEVRTex, and Waymo Open, using FG-ARI for object discovery, ARI for Objects Room including background, MSE for reconstruction quality, and FG-mIoU in the appendix for CLEVRTex [2302.04973].

On Tetrominoes, **ISA-T reaches the same performance as Slot Attention with 2× to 4× fewer samples**, and generalizes better when trained on objects appearing only on the left side of the image and tested on all positions. On Objects Room, the reported validation ARI values are \(85.5 \pm 6.6\) for **ISA-TS** and \(84.3 \pm 4.6\) for **ISA-TSR**. On CLEVR, the paper states that vanilla Slot Attention and ISA variants all achieve near-perfect segmentation and decoding, so gains are limited on that saturated benchmark [2302.04973].

On MultiShapeNet, the picture is more differentiated. On the “all data” setup, **ISA-TS** obtains \(69.8 \pm 1.1\) FG-ARI and **ISA-TSR** obtains \(77.7 \pm 5.5\). On a controlled split with exactly four objects and five slots, **ISA-TS** reaches \(86.5 \pm 1.1\) while **ISA-TSR** reaches \(80.7 \pm 6.4\). The paper interprets this as evidence that the heuristic rotation estimator can help in some regimes but can also destabilize discovery when orientation ambiguity is high [2302.04973].

CLEVRTex is the most detailed synthetic benchmark in the paper. With the original CNN backbone, ISA-T and ISA-TS each yield **10%+ absolute FG-ARI improvement** over Slot Attention. With a stronger ResNet-34 baseline, **ISA-TS** achieves \(92.9 \pm 0.4\) FG-ARI and **ISA-TSR** achieves \(93.3 \pm 0.7\); on CAMO the numbers are \(86.2 \pm 0.8\) and \(87.0 \pm 1.7\), and on OOD they are \(84.4 \pm 0.8\) and \(84.9 \pm 1.2\). The paper further notes that **ISA-TS outperforms the prior state of the art on the CLEVRTex OOD set** without ImageNet or background-model pretraining [2302.04973].

On the real-world Waymo Open dataset, the main reported result is that **ISA-TS improves FG-ARI by 12% over Slot Attention** in an RGB-only, single-frame setting. The paper simultaneously reports that adding rotation hurts, with **ISA-TSR about 5% worse than ISA-TS on Waymo**, and observes that the models often segment visually coherent structures such as buildings, trees, traffic signs, and lanes that are not rewarded by FG-ARI because only cars and pedestrians count toward the benchmark masks [2302.04973].

The ablation results are central to the interpretation of ISA. A decoder-only variant that uses relative coordinates only in the decoder but not in the attention mechanism underperforms full ISA on Tetrominoes and Waymo, indicating that encoder-side symmetry handling matters. On CLEVRTex with a ResNet backbone, the stop-gradient ablation drops from \(93.3 \pm 0.7\) FG-ARI to \(73.4 \pm 10.5\), showing that differentiating through pose estimation is crucial. These results support the view that ISA’s gains do not arise merely from a pose-aware decoder; they depend on inserting slot-relative coordinates into the binding process itself [2302.04973].

ISA’s limitations are also explicit. Rotation handling is heuristic and sometimes unreliable; exact symmetry assumptions do not hold in many 3D scenes; backgrounds and “stuff” do not fit cleanly into object-centric pose frames; and, in practice, invariance may need to be intentionally broken. This directly addresses another recurrent misconception: ISA is not a guarantee of canonical object representations under arbitrary transformations, but a slot-centric inductive bias whose effectiveness depends on how closely the data match the assumed symmetries [2302.04973].

## 6. Place within the slot-attention literature

ISA is best understood as one branch of a broader research program around Slot Attention. The original Slot Attention paper established exchangeable slots, competitive attention normalized over slots, weighted aggregation, iterative GRU-based refinement, and slot-wise decoding as the basic object-centric template [2006.15055]. ISA keeps that template and changes the spatial reference frame.

Subsequent work has explored orthogonal failure modes. "Attention Normalization Impacts Cardinality Generalization in Slot Attention" argues that the original weighted-mean aggregation discards slot occupancy information and harms generalization to changing numbers of slots or objects; it proposes replacing
\[
u_k = \frac{\sum_{n=1}^N \gamma_{n,k} v(x_n)}{\sum_{n=1}^N \gamma_{n,k}}
\]
with
\[
u_k = \frac{1}{N}\sum_{n=1}^N \gamma_{n,k} v(x_n),
\]
a scaled weighted sum that preserves assignment-mass information and improves cardinality robustness [2407.04170]. "Unlocking Slot Attention by Changing Optimal Transport Costs" takes a different direction, arguing that vanilla Slot Attention is too symmetry-preserving over slots and cannot break ties; its MESH module is designed to be exclusively multiset-equivariant rather than set-equivariant, so that equal or nearly equal slots can differentiate in dynamic scenes [2301.13197].

Other papers address still different desiderata. "Identifiable Object-Centric Representation Learning via Probabilistic Slot Attention" provides slot identifiability guarantees up to slot permutation and affine transformation through a probabilistic mixture prior, but it is not an ISA method by name [2406.07141]. "Learning Global Object-Centric Representations via Disentangled Slot Attention" separates scene-dependent extrinsic factors from scene-independent prototype-based intrinsic factors in order to support object identification across scenes; it is closely related in aim to ISA, but it achieves scene-independence through global prototype selection rather than slot-centric coordinate transforms [2410.18809]. "Slot Attention with Re-Initialization and Self-Distillation" focuses on redundancy, fragmented bindings, and internal consistency across iterations, again addressing a problem adjacent to, but distinct from, geometric invariance [2507.23755].

This constellation of results suggests that “invariance” in slot-based object-centric learning is not a single property. ISA addresses spatial symmetries through slot-centric reference frames [2302.04973]; other work addresses count robustness [2407.04170], tie-breaking under slot symmetry [2301.13197], identifiability up to equivalence classes [2406.07141], globally invariant object identity across scenes [2410.18809], or iterative binding stability [2507.23755]. A plausible implication is that ISA is most precisely characterized not as the universal solution to slot invariance, but as the canonical method for injecting per-object translation-, scale-, and optionally rotation-relative coordinates into both the binding and generation stages of Slot Attention.

Source: https://www.emergentmind.com/topics/invariant-slot-attention-isa