Kuramoto Attention: Synchronizing Self-Attention on the Torus

Abstract: We introduce Kuramoto attention, a self-attention layer in which each hidden coordinate is an angle. The layer scores tokens by gated cosine similarity, attends over previous phase states, and updates each token by the tangent component of the attention-weighted circular mean. Because the values are the raw phase states, this update is exactly the Kuramoto coupling term $\sum_u A_{t,u}\sin(θ_u-θ_t)$, with the attention matrix acting as an adaptive, content-dependent coupling kernel. Equivalently, the gated score is a learned metric on the torus that selects which tokens couple, and the update pulls each token toward the circular mean of the tokens it selects, tightening their phase agreement. The same two ingredients, an invariant similarity score and an on-manifold mean, define such a layer on any compact group; the torus is the abelian case, where both are closed-form. The softmax weights solve an entropy-regularized phase-retrieval problem, and rotary position enters as a position-dependent phase drift in the score. On enwiki8 character-level language modeling, the layer trains as a functional LLM whose bits-per-character stays close to a strong matched RoPE+SwiGLU transformer: within $0.02$ BPC at one million parameters ($1.637\pm0.010$ versus $1.616\pm0.004$) and level on the median at five million ($1.448$ versus $1.452$ over five seeds) with the transformer ahead on the mean ($1.468$ versus $1.456$). These experiments establish that the constrained geometric structure is a viable LLM at this scale; the structure itself, and its synchronization reading, is the contribution. Ablations isolate the load-bearing components, and the result gives a compact bridge between self-attention and phase synchronization.

• The paper introduces Kuramoto attention, unifying self-attention with phase synchronization by mapping tokens to phase-valued oscillators on T^k.
• It employs a geometrically native update via an adaptive Kuramoto coupling and rotary position encodings to guide phase alignment.
• Empirical evaluations on language modeling demonstrate competitive bits-per-character scores and strong local synchronization without global phase collapse.

Kuramoto Attention: Synchronizing Self-Attention on the Torus

Introduction and Motivation

The paper "Kuramoto Attention: Synchronizing Self-Attention on the Torus" (2606.11585) proposes a novel self-attention mechanism whose hidden state is natively phase-valued, thereby treating each coordinate as an oscillator on the $k$-dimensional torus $T^k$. The design is geometrically grounded: the values correspond directly to oscillator phases, and the principal update step per layer enacts an exact adaptive Kuramoto coupling using an attention weight matrix as the coupling kernel. This construction tightly integrates the selection (attention) and synchronization (update) processes, forming a direct bridge between attention mechanisms and synchronization dynamics well-studied in physics.

Kuramoto Attention Layer Formulation

Each token is mapped to a vector of $k$ phases, $\theta \in \mathbb{R}^k / (2\pi\mathbb{Z})^k$, representing a bank of unit oscillators $e^{i\theta_j}$. The layer update comprises three stages: (1) computation of gated cosine similarities for content-based attention, (2) aggregation via the attention-weighted circular mean on the torus, and (3) a bounded, nonlinear feed-forward update in the phase domain.

The attention weights are generated by softmax over a rotary position-enhanced, metric-gated cosine similarity:

$$s_{t,u} = \frac{\tau}{\sqrt{k}} \sum_j g^q_{t,j} g^k_{u,j} \cos(\theta_{t,j} - \theta_{u,j} + \omega_j(t-u))$$

where the gates $g^q$, $g^k$ are learned via softplus activations and serve as a position-dependent, per-coordinate metric on $T^k$. The use of rotary embedding is interpreted as a per-coordinate phase drift, aligning with the notion of natural frequencies in Kuramoto models.

The value update is performed by moving the phase toward the attention-weighted circular mean:

$$a_{t,j} = \sum_u A_{t,u} \sin(\theta_{u,j} - \theta_{t,j}),$$

which is precisely the Kuramoto coupling term with the attention matrix $T^k$0 as a state-dependent coupling kernel.

Key claims:

• Selection and synchronization are unified: the same phase coherence quantity is employed for both gating neighbor selection and for driving dynamic phase synchronization.
• Geometry-native design: all principal computations remain natively on $T^k$1, with the only non-geometric component being the feed-forward SwiGLU residual.

  Figure 1: Local synchronization without global collapse — per-token, per-layer local order parameter remains high, indicating tight synchronization to attended neighbors, while the global order parameter decreases, verifying the absence of global phase collapse.

Theoretical Implications

The layer demonstrates that adaptive Kuramoto synchronization, with attention-based coupling, is functionally sufficient for competitive language modeling. The interpretability of the architecture is strengthened by the direct mapping between attention components and synchronization dynamics:

• The attention matrix—a softmax of cosine similarities—solves an entropy-regularized phase retrieval, acting as an adaptive kernel.
• Rotary position encodings correspond to learned natural frequencies, separating mechanism from content.

Crucially, the paper proves that the key geometric components (metric-gated similarity and the on-manifold mean value update) can be abstracted to any compact group, not just the abelian torus, laying groundwork for synchronization-driven attention on non-abelian structures (cf. Lohe models).

Figure 2: Learned natural frequencies ($T^k$2) evolve with depth, tracking a timescale hierarchy — early layers exhibit higher frequencies (local coupling), while deeper layers have lower frequencies (long-range dependencies).

Empirical Evaluation

On character-level language modeling (enwiki8), Kuramoto attention achieves BPC competitive with a RoPE+SwiGLU transformer:

• At 1M parameters: within $T^k$3 BPC of the baseline.
• At 5M parameters: median BPC matches the transformer (Kuramoto 1.448 vs transformer 1.452), with non-trivial variance due to outlier seeds.

Ablation studies highlight the criticality of the geometry-native synchronization machinery. Removing the on-manifold mean (replacing with a learned value projection) or the per-coordinate gates (metric) degrades BPC substantially (+0.25 and +0.09), whereas the non-geometric feed-forward block carries the highest individual load (+0.27 when removed). The model displays strong local synchronization without global phase collapse, as established by per-token and population order parameter dynamics.

Architectural Implications and Future Directions

• Multiplicative value path: Unlike standard dot-product attention, Kuramoto attention avoids additive cross-coordinate mixing in its value update. All mixing is via multiplicative gates; additive mixing occurs only in the feed-forward block. This architectural separation is empirically validated via targeted ablations.
• Normalization-free: The hidden state geometry ensures norm conservation (unit modulus per coordinate), obviating the need for explicit layer normalization techniques.
• Generalization: The architectural template extends naturally to any compact subgroup of the unitary group. Instantiations on $T^k$4 or other manifolds would address domains involving rotation or unitary quantum states.

Potential expansions include geometry-native (manifold-respecting) replacements for the feed-forward block and evaluation on larger scales and tasks.

Conclusion

Kuramoto attention provides a concrete instantiation of geometrically faithful, synchronization-driven self-attention, matching transformer performance at non-trivial parameter scales. Its design both demystifies the role of synchronization dynamics in attention and sets an extensible framework for manifold-valued and non-abelian architectures. The demonstrated interplay between metric-based selection and phase synchronization opens new avenues for principled, structure-aware attention mechanisms.