Platonic Transformer: Equivariant Attention

• Platonic Transformers are Transformer models that incorporate finite symmetry groups to enforce equivariance in feature transformations and weight-sharing.
• They use lifted attention over multiple group reference frames and group convolutions to maintain standard computational graphs while enhancing geometric invariance.
• Empirical evaluations show improved accuracy and cost-efficiency in computer vision, 3D point cloud analysis, and molecular property prediction compared to conventional methods.

The Platonic Transformer is a Transformer architecture that achieves principled equivariance to translations and Platonic solid symmetry groups by lifting attention and feature transformations into multiple reference frames associated with finite groups drawn from rotational symmetries in two and three dimensions. The design preserves the exact computational graph and efficiency of standard Transformer attention, while inducing a structured weight-sharing scheme via group convolutions. Empirically, Platonic Transformers demonstrate competitive accuracy and superior cost-efficiency in computer vision, 3D point cloud analysis, and molecular property prediction tasks (Islam et al., 3 Oct 2025).

1. Platonic Solid and Finite Symmetry Groups

A Platonic solid in 3D is a regular convex polyhedron whose rotational symmetry group, $G$, is a finite subgroup of $\mathrm{SO}(3)$. The approach focuses on the following three Platonic groups:

• Tetrahedral group $T$: Isomorphic to $A_4$, $|T|=12$, generated by rotations such as $R_1$ (120° about a chosen vertex-opposite face) and $R_2$ (180° about midpoints of opposite edges).
• Octahedral (cubic) group $O$: $|O|=24$, generated by $S_1$ (90° around the $\mathrm{SO}(3)$0-axis) and $\mathrm{SO}(3)$1 (120° about the $\mathrm{SO}(3)$2-axis).
• Icosahedral group $\mathrm{SO}(3)$3: $\mathrm{SO}(3)$4, generated by $\mathrm{SO}(3)$5 (72° about a vertex-opposite face) and $\mathrm{SO}(3)$6 (180° about midpoints of opposite edges).

In 2D, the methodology leverages cyclic $\mathrm{SO}(3)$7 ($\mathrm{SO}(3)$8; rotations by $\mathrm{SO}(3)$9) and dihedral $T$0 ($T$1; includes reflections) groups. The theory builds on group representations $T$2, requiring orthogonality/unitarity for stability and using irreducible representations (irreps) to enforce Schur’s Lemma for equivariant linear maps.

2. Transformer Attention in Group Reference Frames

Each input token $T$3 with position $T$4 and features $T$5 is “lifted” to a function $T$6—effectively, $T$7 parallel feature copies:

• Scalars: Simply copied into each frame.
• Vectors: Expressed in the local basis via $T$8, $T$9.

Queries, keys, and values become group-indexed:

• $A_4$0
• $A_4$1
• $A_4$2

Positions also transform: $A_4$3. Rotary Position Embeddings (RoPE) are applied in parallel frames. The unnormalized attention score from frame $A_4$4 is:

$A_4$5

where $A_4$6 block-diagonalizes $A_4$7 rotations by learned frequencies.

The attention is computed by standard softmax over $A_4$8, providing weights $A_4$9 and output

$|T|=12$0

These operations, including RoPE and softmax, match standard Transformer computation. The only modification is the parallelization over $|T|=12$1 group frames, conceptualized as additional attention heads with shared weights.

3. Group Equivariance, Weight-Sharing, and Induced Structure

Linear layers acting on group-indexed features must be equivariant under the left-regular group action $|T|=12$2:

• $|T|=12$3
• Equivariance requirement: $|T|=12$4

Equivariant linear maps are group convolutions:

$|T|=12$5

with $|T|=12$6 a learnable kernel. This enforces identical complexity and parameter count as a non-equivariant linear layer of size $|T|=12$7 via reshaping. Weight-sharing across output-input frame pairs with the same relative group element yields equivariance.

For a global rotation/reflection $|T|=12$8 applied to the data, outputs transform equivariantly:

$|T|=12$9

4. Connection to Dynamic Group Convolution

Omitting the softmax (i.e., adopting “linear” attention) and using fixed keys $R_1$0, Platonic Transformer attention reduces to a dynamic, content-adaptive convolution:

$R_1$1

with dynamic filters synthesized by the query:

$R_1$2

where $R_1$3 and $R_1$4 are linear functions of $R_1$5. Lifting to group frames produces a dynamic group convolution:

$R_1$6

This reveals that query tokens synthesize filters over relative displacements in a sparse Fourier basis, aligned with group-theoretic symmetries.

5. Linear-Time Convolutional Variant

The unnormalized dynamic convolution can be cast in matrix notation:

• $R_1$7, where $R_1$8 and $R_1$9
• Compute $R_2$0 in $R_2$1, then $R_2$2 in $R_2$3 for total $R_2$4 complexity in token count

This matches the complexity of Performer-style linear attention, demanding no approximation: dynamic convolution is exact without the softmax. The linear-time variant is advantageous for large token counts, though attention heads with softmax provide greater expressivity for orientation-specific patterns.

6. Empirical Evaluation Across Benchmarks

Platonic Transformers are evaluated in “Attention” (full softmax) and “Conv” (linear-time convolution) modes on diverse tasks:

Benchmark	Baseline	Platonic Group	Attention Mode	Conv Mode
CIFAR-10	RoPE (trivial group), 91.5% acc	C₄, D₄	~92.5% (no extra FLOPs)	~88.5%
ScanObjectNN	T(n) attention, 80.5% acc	Tetrahedral	81.3%	80.1%
QM9 (molecular)	T(n): μ=0.028, α=0.064 (MAE)	Tetra/Octahedral	μ=0.010–0.012, α=0.048–0.049	μ=0.012–0.014, α=0.047
OMol25	eSEN: 120 meV/3.37 meV (energy/atom)	Tetra/Octahedral	74 meV/2.63 meV	n/a

On QM9, Platonic variants match or outperform EquiformerV2 and are over $R_2$5 faster at inference than G-Hyena or TFN. For OMol25, Platonic Transformers achieve lower energy MAE under fixed computational budgets compared to eSEN, and approach MACE’s best result with 4× less compute. Cost-efficiency and scalability are central strengths in these empirical settings.

7. Architectural Summary and Significance

Platonic Transformers interpret each attention head as operating in parallel over $R_2$6 rotated frames, with all core operations—attention, RoPE, and linear layers—constrained to be equivariant under group symmetries by enforcing weight-sharing via group convolution. This design achieves exact (or approximate) $R_2$7-equivariance in unaltered Transformer architectures and computational complexity. The deep equivalence between RoPE-based attention and dynamic (group) convolution provides both modeling flexibility and efficiency: the softmax attention head delivers modeling power for non-symmetric data, while the linear-time convolutional mode scales to large inputs without approximation error. Across modalities, Platonic Transformers leverage geometric constraints for enhanced sample efficiency and state-of-the-art speed without additional computational cost (Islam et al., 3 Oct 2025).