Invariant Slot Attention (ISA)

• Invariant Slot Attention (ISA) is an object-centric method that assigns each slot its own reference frame to decouple appearance from pose.
• ISA incorporates relative positional encoding by transforming absolute coordinates via per-object translation, scaling, and optional rotation.
• Empirical results demonstrate ISA’s enhanced object segmentation and generalization, improving performance with fewer training samples.

Searching arXiv for Invariant Slot Attention and closely related slot-attention papers to ground the article. arxiv_search(query="Invariant Slot Attention object discovery with slot-centric reference frames", max_results=5) Invariant Slot Attention (ISA) is an object-centric learning method that extends Slot Attention by assigning each slot its own reference frame, so that a slot represents an object in object-relative rather than camera-relative coordinates. Its stated purpose is to improve unsupervised object discovery by reducing the entanglement of appearance or identity with pose, and by incorporating spatial symmetries directly into both the attention mechanism and the decoder through slot-centric translation, scaling, and optional rotation of positional encodings (Biza et al., 2023).

1. Conceptual setting and motivation

ISA belongs to the object-centric learning literature, in which a scene is represented as a set of latent vectors, each intended to capture one object-like entity. The immediate precursor is Slot Attention, which maps a set of perceptual features to a set of exchangeable slots through iterative competitive attention, and is permutation-invariant with respect to input order and permutation-equivariant with respect to slot order (Locatello et al., 2020).

The specific limitation targeted by ISA is that standard Slot Attention uses absolute image coordinates in both the attention module and the spatial broadcast decoder. In that formulation, a slot’s content depends on where the object appears, so appearance and pose become entangled. The ISA paper frames this as a failure to adequately capture spatial symmetries present in the visual world, with adverse consequences for sample efficiency, generalization to unseen positions, scales, and orientations, and consistency of learned object representations (Biza et al., 2023).

The core intervention is the introduction of slot-centric reference frames. Each slot is augmented with per-object pose variables—translation, scale, and optionally rotation—and positional encodings are transformed relative to the slot’s estimated pose. In the paper’s terminology, this is meant to make the slot latent approximately invariant to per-object pose transformations, while the pose parameters themselves are equivariant to those transformations. A common misconception is that ISA is a fully exact group-equivariant architecture; the method is instead presented as a practical mechanism for approximate, learned invariance in cluttered natural scenes, where 3D projection, occlusion, and background structure violate exact symmetry assumptions (Biza et al., 2023).

2. Baseline Slot Attention and the point of departure

Slot Attention provides the architectural substrate from which ISA is derived. In the standard formulation, an encoder converts an image into a set of feature vectors, slots are initialized from a shared Gaussian distribution, and iterative attention updates bind features to slots by competitive routing over multiple rounds (Locatello et al., 2020).

ISA inherits this iterative slot-binding structure but changes how spatial information enters it. The baseline equations cited by the ISA paper are

$$\mathrm{keys} = f\left(\mathcal{K}(\mathrm{inputs}) + g(\mathrm{abs\_grid})\right)$$

and

$$\mathrm{values} = f\left(\mathcal{V}(\mathrm{inputs}) + g(\mathrm{abs\_grid})\right),$$

where $\mathrm{abs\_grid}$ denotes the absolute coordinate grid scaled to $[-1,1]$, and $\mathcal{K}, \mathcal{V}, \mathcal{Q}, g$ are linear maps. Queries are computed from slots, and dot products between queries and keys are normalized over slots so that slots compete for each input token (Biza et al., 2023).

This baseline already has an important symmetry property: slots are exchangeable rather than index-specialized. However, the same literature also emphasizes that absolute positional embeddings break translation equivariance, and that the original model’s invariance is primarily permutation symmetry over sets rather than invariance to spatial transformations (Locatello et al., 2020). ISA should therefore be understood not as a replacement for slot exchangeability, but as an extension that adds per-object spatial canonicalization on top of the standard slot-competition mechanism (Biza et al., 2023).

3. Slot-centric reference frames and relative positional encoding

The defining mechanism of ISA is the replacement of absolute positional encoding by slot-relative positional encoding. For translation and scale, each slot $k$ carries a position $S_p^k \in \mathbb{R}^2$ and scale $S_s^k \in \mathbb{R}^2$, and ISA defines

$$\mathrm{rel\_grid}^k = (\mathrm{abs\_grid} - S_p^k)\ /\ S_s^k.$$

For translation, rotation, and scale jointly, ISA uses

$$\mathrm{rel\_grid}^k = (S_r^k)^{-1}(\mathrm{abs\_grid} - S_p^k)\ /\ S_s^k,$$

where $$\mathrm{values} = f\left(\mathcal{V}(\mathrm{inputs}) + g(\mathrm{abs\_grid})\right),$$0 is a per-slot rotation matrix (Biza et al., 2023).

These transformed coordinates are inserted directly into the attention computation. Instead of one key and one value per input token, ISA constructs slot-specific keys and values: $$\mathrm{values} = f\left(\mathcal{V}(\mathrm{inputs}) + g(\mathrm{abs\_grid})\right),$$1

$$\mathrm{values} = f\left(\mathcal{V}(\mathrm{inputs}) + g(\mathrm{abs\_grid})\right),$$2

The same image token is therefore encoded differently depending on which slot is querying it. This slot-conditioned key/value construction is the paper’s central architectural change: cross-attention becomes pose-relative on a per-slot basis rather than globally absolute (Biza et al., 2023).

Pose parameters are estimated from the attention masks themselves. Position is computed as the weighted centroid

$$\mathrm{values} = f\left(\mathcal{V}(\mathrm{inputs}) + g(\mathrm{abs\_grid})\right),$$3

and scale from the weighted variance

$$\mathrm{values} = f\left(\mathcal{V}(\mathrm{inputs}) + g(\mathrm{abs\_grid})\right),$$4

If rotation is enabled, the paper uses a weighted PCA heuristic,

$$\mathrm{values} = f\left(\mathcal{V}(\mathrm{inputs}) + g(\mathrm{abs\_grid})\right),$$5

followed by post-processing to produce $$\mathrm{values} = f\left(\mathcal{V}(\mathrm{inputs}) + g(\mathrm{abs\_grid})\right),$$6 (Biza et al., 2023).

The decoder is modified analogously. Instead of broadcasting a slot over a global grid with absolute coordinates, ISA uses the slot’s relative grid: $$\mathrm{values} = f\left(\mathcal{V}(\mathrm{inputs}) + g(\mathrm{abs\_grid})\right),$$7 where $$\mathrm{values} = f\left(\mathcal{V}(\mathrm{inputs}) + g(\mathrm{abs\_grid})\right),$$8 is the spatially broadcast slot tensor. The decoder thus reconstructs each object in slot-relative coordinates, and the paper argues that this encourages weight sharing across positions and scales (Biza et al., 2023).

4. Architecture, variants, and optimization details

ISA preserves the standard three-part object-discovery scaffold: an encoder, a slot-attention module, and a slot-wise decoder. The encoder is either a shallow 4-layer CNN for simpler synthetic settings or a ResNet-34 with group normalization for more difficult datasets such as CLEVRTex and Waymo. The Slot Attention block runs for 3 iterations, followed by one extra pass to compute final pose statistics without updating the slot latent vectors. Typical settings use 11 slots, with 4 slots on Tetrominoes and 5 slots on MultiShapeNet (Biza et al., 2023).

The paper distinguishes several symmetry configurations. ISA-T handles translation, ISA-TS handles translation and scale, and ISA-TSR additionally includes rotation. In appendix pseudocode, a scaling constant $$\mathrm{values} = f\left(\mathcal{V}(\mathrm{inputs}) + g(\mathrm{abs\_grid})\right),$$9 is introduced so that

$\mathrm{abs\_grid}$0

with the stated purpose of numerical stability when scales are small (Biza et al., 2023).

Two implementation details are emphasized as especially consequential. First, gradients must flow through pose estimation: stopping gradients through $\mathrm{abs\_grid}$1 strongly hurts performance. Second, although ISA requires per-slot keys and values and therefore increases the number of key/value embeddings from $\mathrm{abs\_grid}$2 to $\mathrm{abs\_grid}$3, the paper states that the overhead is negligible in the common regime of roughly ten slots (Biza et al., 2023).

Optimization is also specified explicitly. Training uses Adam, with learning rate $\mathrm{abs\_grid}$4 for most datasets and $\mathrm{abs\_grid}$5 for Waymo depth targets, 50k warm-up steps for most datasets and 5k on Tetrominoes, cosine decay, batch size 64, 500k steps for most datasets, 300k for Waymo, and 50k for Tetrominoes (Biza et al., 2023).

A further subtlety is symmetry breaking. The ISA paper notes that encoder backbones, especially stronger ones such as ResNet, can leak global position information through padding artifacts. To mitigate this, some experiments append pose parameters to the slot vector before the GRU (“-Append”), explicitly breaking symmetry when needed. This suggests that ISA is not committed to maximal symmetry preservation in all components; rather, it uses slot-centric reference frames as the principal symmetry-inducing mechanism while permitting controlled departures when exact invariance is too restrictive in practice (Biza et al., 2023).

5. Empirical behavior, ablations, and limitations

ISA is evaluated on Tetrominoes, Objects Room, CLEVR, MultiShapeNet, CLEVRTex, and Waymo Open, using FG-ARI for object discovery, ARI for Objects Room including background, MSE for reconstruction quality, and FG-mIoU in the appendix for CLEVRTex (Biza et al., 2023).

On Tetrominoes, ISA-T reaches the same performance as Slot Attention with 2× to 4× fewer samples, and generalizes better when trained on objects appearing only on the left side of the image and tested on all positions. On Objects Room, the reported validation ARI values are $\mathrm{abs\_grid}$6 for ISA-TS and $\mathrm{abs\_grid}$7 for ISA-TSR. On CLEVR, the paper states that vanilla Slot Attention and ISA variants all achieve near-perfect segmentation and decoding, so gains are limited on that saturated benchmark (Biza et al., 2023).

On MultiShapeNet, the picture is more differentiated. On the “all data” setup, ISA-TS obtains $\mathrm{abs\_grid}$8 FG-ARI and ISA-TSR obtains $\mathrm{abs\_grid}$9. On a controlled split with exactly four objects and five slots, ISA-TS reaches $[-1,1]$0 while ISA-TSR reaches $[-1,1]$1. The paper interprets this as evidence that the heuristic rotation estimator can help in some regimes but can also destabilize discovery when orientation ambiguity is high (Biza et al., 2023).

CLEVRTex is the most detailed synthetic benchmark in the paper. With the original CNN backbone, ISA-T and ISA-TS each yield 10%+ absolute FG-ARI improvement over Slot Attention. With a stronger ResNet-34 baseline, ISA-TS achieves $[-1,1]$2 FG-ARI and ISA-TSR achieves $[-1,1]$3; on CAMO the numbers are $[-1,1]$4 and $[-1,1]$5, and on OOD they are $[-1,1]$6 and $[-1,1]$7. The paper further notes that ISA-TS outperforms the prior state of the art on the CLEVRTex OOD set without ImageNet or background-model pretraining (Biza et al., 2023).

On the real-world Waymo Open dataset, the main reported result is that ISA-TS improves FG-ARI by 12% over Slot Attention in an RGB-only, single-frame setting. The paper simultaneously reports that adding rotation hurts, with ISA-TSR about 5% worse than ISA-TS on Waymo, and observes that the models often segment visually coherent structures such as buildings, trees, traffic signs, and lanes that are not rewarded by FG-ARI because only cars and pedestrians count toward the benchmark masks (Biza et al., 2023).

The ablation results are central to the interpretation of ISA. A decoder-only variant that uses relative coordinates only in the decoder but not in the attention mechanism underperforms full ISA on Tetrominoes and Waymo, indicating that encoder-side symmetry handling matters. On CLEVRTex with a ResNet backbone, the stop-gradient ablation drops from $[-1,1]$8 FG-ARI to $[-1,1]$9, showing that differentiating through pose estimation is crucial. These results support the view that ISA’s gains do not arise merely from a pose-aware decoder; they depend on inserting slot-relative coordinates into the binding process itself (Biza et al., 2023).

ISA’s limitations are also explicit. Rotation handling is heuristic and sometimes unreliable; exact symmetry assumptions do not hold in many 3D scenes; backgrounds and “stuff” do not fit cleanly into object-centric pose frames; and, in practice, invariance may need to be intentionally broken. This directly addresses another recurrent misconception: ISA is not a guarantee of canonical object representations under arbitrary transformations, but a slot-centric inductive bias whose effectiveness depends on how closely the data match the assumed symmetries (Biza et al., 2023).

6. Place within the slot-attention literature

ISA is best understood as one branch of a broader research program around Slot Attention. The original Slot Attention paper established exchangeable slots, competitive attention normalized over slots, weighted aggregation, iterative GRU-based refinement, and slot-wise decoding as the basic object-centric template (Locatello et al., 2020). ISA keeps that template and changes the spatial reference frame.

Subsequent work has explored orthogonal failure modes. "Attention Normalization Impacts Cardinality Generalization in Slot Attention" argues that the original weighted-mean aggregation discards slot occupancy information and harms generalization to changing numbers of slots or objects; it proposes replacing

$\mathcal{K}, \mathcal{V}, \mathcal{Q}, g$0

with

$\mathcal{K}, \mathcal{V}, \mathcal{Q}, g$1

a scaled weighted sum that preserves assignment-mass information and improves cardinality robustness (Krimmel et al., 2024). "Unlocking Slot Attention by Changing Optimal Transport Costs" takes a different direction, arguing that vanilla Slot Attention is too symmetry-preserving over slots and cannot break ties; its MESH module is designed to be exclusively multiset-equivariant rather than set-equivariant, so that equal or nearly equal slots can differentiate in dynamic scenes (Zhang et al., 2023).

Other papers address still different desiderata. "Identifiable Object-Centric Representation Learning via Probabilistic Slot Attention" provides slot identifiability guarantees up to slot permutation and affine transformation through a probabilistic mixture prior, but it is not an ISA method by name (Kori et al., 2024). "Learning Global Object-Centric Representations via Disentangled Slot Attention" separates scene-dependent extrinsic factors from scene-independent prototype-based intrinsic factors in order to support object identification across scenes; it is closely related in aim to ISA, but it achieves scene-independence through global prototype selection rather than slot-centric coordinate transforms (Chen et al., 2024). "Slot Attention with Re-Initialization and Self-Distillation" focuses on redundancy, fragmented bindings, and internal consistency across iterations, again addressing a problem adjacent to, but distinct from, geometric invariance (Zhao et al., 31 Jul 2025).

This constellation of results suggests that “invariance” in slot-based object-centric learning is not a single property. ISA addresses spatial symmetries through slot-centric reference frames (Biza et al., 2023); other work addresses count robustness (Krimmel et al., 2024), tie-breaking under slot symmetry (Zhang et al., 2023), identifiability up to equivalence classes (Kori et al., 2024), globally invariant object identity across scenes (Chen et al., 2024), or iterative binding stability (Zhao et al., 31 Jul 2025). A plausible implication is that ISA is most precisely characterized not as the universal solution to slot invariance, but as the canonical method for injecting per-object translation-, scale-, and optionally rotation-relative coordinates into both the binding and generation stages of Slot Attention.