Fixed-Query Oscillator Attention

• Fixed-query oscillator attention is a mechanism that replaces input-dependent queries with fixed learned anchors on a sphere, encoding attention via cosine similarity.
• It employs Kuramoto–Lohe dynamics to equilibrate free oscillators, resulting in almost-global convergence guarantees and a physical interpretation of attention normalization.
• Empirical results demonstrate improved performance on tasks like keyword spotting and subject-verb agreement, highlighting its potential as a hardware-friendly alternative to softmax.

Fixed-query oscillator attention is an attention mechanism in which the query side is replaced by learned anchors fixed on a sphere, while input-dependent free oscillators evolve under Kuramoto–Lohe dynamics until they settle at positions that encode attention weights through cosine similarity. In this formulation, the nonlinear core of attention is no longer softmax but equilibration of a gradient flow on $\mathbb{S}^{d_{\mathrm{osc}}-1}$, followed by affine normalization at readout. The proposal is explicitly motivated by “energy-constrained physical substrates,” where exponentiation and in-loop global reduction are costly and lack a natural physical analog, whereas synchronization is native to oscillator arrays (Pasqualetti et al., 10 Jun 2026).

1. Definition and scope

In fixed-query oscillator attention, the “fixed-query” designation means that the query-like objects are not input-dependent token queries. Instead, they are learned anchors $r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$ that are fixed during inference. Inputs still determine pairwise couplings, but the query side of the mechanism is a fixed learned reference geometry on the sphere rather than a runtime-generated $q_i = W_Q e_i$ as in standard scaled dot-product attention (Pasqualetti et al., 10 Jun 2026).

The construction begins from the Lohe model for oscillators on the unit sphere,

$$\dot x_i = \Omega_i x_i + \left(I - x_i x_i^\top\right)\sum_{j=1}^n w_{ij}x_j,$$

with $\Omega_i=0$ in the fixed-query attention setting. For $d_{\mathrm{osc}}=2$, writing $x_i=(\cos\theta_i,\sin\theta_i)$ recovers the classical scalar Kuramoto model. The method then specializes the dynamics asymmetrically: anchor oscillators are learned and fixed, while free oscillators evolve under the forcing induced by those anchors. This asymmetry is central because the paper attributes the resulting uniqueness and almost-global convergence guarantees to the one-particle forced system rather than to a mutually coupled many-oscillator network (Pasqualetti et al., 10 Jun 2026).

This makes the mechanism distinct from several adjacent synchronization-based attention proposals. In "Kuramoto Attention," the effective query is implicit in the current token phase state and learned query gates, so it is oscillator attention but not literally fixed-query in the sense of a shared fixed learned query template (Nunley, 10 Jun 2026). In "Selective Synchronization Attention," there is no explicit query variable at all; the operator is best described as query-free oscillator attention derived from pairwise synchronization compatibility (Hays, 16 Feb 2026). "Attention as Frustrated Synchronization" keeps a causal query-key score map but replaces the value pathway with harmonic synchronization dynamics, making it a synchronization-based alternative to standard value attention rather than a fixed-query design (Nunley, 17 Jun 2026).

2. Dynamical formulation on the sphere

Let the input sequence be $e_1,\dots,e_T \in \mathbb{R}^{d_{\rm model}}$. For one attention head, fixed-query oscillator attention defines positive couplings

$$w_{ij} = \sigma\!\left(\frac{(Fe_i)^\top (Ge_j)}{\sqrt{d_h}}\right),$$

where $F,G \in \mathbb{R}^{d_h \times d_{\rm model}}$ are learned projections, $r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$0 is the head dimension, and $r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$1 is any strictly positive function; the experiments use $r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$2, i.e. softplus (Pasqualetti et al., 10 Jun 2026).

These couplings weight the learned anchors:

$r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$3

The free oscillator state $r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$4 evolves according to

$r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$5

The tangent projector $r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$6 keeps the dynamics on the sphere. The settled state is

$r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$7

Software training and inference use this analytic fixed point directly rather than numerically integrating the ODE (Pasqualetti et al., 10 Jun 2026).

Attention weights are then read out through shifted cosine similarity:

$r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$8

The shift by $r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$9 guarantees nonnegative numerators because $q_i = W_Q e_i$0. Per-head outputs use standard value aggregation,

$q_i = W_Q e_i$1

followed by the usual multi-head combination with $q_i = W_Q e_i$2 (Pasqualetti et al., 10 Jun 2026).

For causal masking, the method sets $q_i = W_Q e_i$3 for $q_i = W_Q e_i$4 and restricts the denominator in the readout to $q_i = W_Q e_i$5. The paper also notes a software-side sharpening variant,

$q_i = W_Q e_i$6

but treats $q_i = W_Q e_i$7 as the hardware-native case (Pasqualetti et al., 10 Jun 2026).

3. Geometric and variational interpretation

The fixed-query dynamics are presented as a constrained gradient flow on the sphere. For each free oscillator, define the energy

$q_i = W_Q e_i$8

Because $q_i = W_Q e_i$9 must remain on $$\dot x_i = \Omega_i x_i + \left(I - x_i x_i^\top\right)\sum_{j=1}^n w_{ij}x_j,$$0, the Euclidean gradient must be projected onto the tangent space, yielding

$$\dot x_i = \Omega_i x_i + \left(I - x_i x_i^\top\right)\sum_{j=1}^n w_{ij}x_j,$$1

The equilibrium therefore solves

$$\dot x_i = \Omega_i x_i + \left(I - x_i x_i^\top\right)\sum_{j=1}^n w_{ij}x_j,$$2

whose solution is $$\dot x_i = \Omega_i x_i + \left(I - x_i x_i^\top\right)\sum_{j=1}^n w_{ij}x_j,$$3 whenever $$\dot x_i = \Omega_i x_i + \left(I - x_i x_i^\top\right)\sum_{j=1}^n w_{ij}x_j,$$4 (Pasqualetti et al., 10 Jun 2026).

This gives the mechanism a clear geometric decomposition. First, the digital front-end computes input-dependent positive couplings. Second, those couplings form a weighted anchor sum in $$\dot x_i = \Omega_i x_i + \left(I - x_i x_i^\top\right)\sum_{j=1}^n w_{ij}x_j,$$5. Third, the free oscillator relaxes to the unit vector maximally aligned with that sum. Fourth, cosine similarity to the anchors is affinely normalized into a row-stochastic attention distribution. The essential nonlinearity is therefore the normalization map $$\dot x_i = \Omega_i x_i + \left(I - x_i x_i^\top\right)\sum_{j=1}^n w_{ij}x_j,$$6, interpreted as physical equilibration rather than softmax arithmetic (Pasqualetti et al., 10 Jun 2026).

The same section of the literature makes clear that not all oscillator-attention proposals share this geometry. In "Kuramoto Attention," values are the raw phase states, the update is exactly $$\dot x_i = \Omega_i x_i + \left(I - x_i x_i^\top\right)\sum_{j=1}^n w_{ij}x_j,$$7 coordinatewise, and the query is state-tied rather than fixed (Nunley, 10 Jun 2026). In "Attention as Frustrated Synchronization," the update is instead a harmonic Kuramoto–Sakaguchi–Daido coupling law with a learned complex kernel and one-step delay, so the computation is located in structured departures from agreement rather than in convergence to a single spherical anchor mixture (Nunley, 17 Jun 2026).

4. Theoretical guarantees

The main theorem for fixed-query oscillator attention states that if

$$\dot x_i = \Omega_i x_i + \left(I - x_i x_i^\top\right)\sum_{j=1}^n w_{ij}x_j,$$8

then the gradient flow

$$\dot x_i = \Omega_i x_i + \left(I - x_i x_i^\top\right)\sum_{j=1}^n w_{ij}x_j,$$9

has exactly two equilibria,

$\Omega_i=0$0

The equilibrium $\Omega_i=0$1 is asymptotically stable with basin of attraction

$\Omega_i=0$2

while $\Omega_i=0$3 is unstable. The proof uses the Lyapunov function $\Omega_i=0$4 and LaSalle’s invariance principle, giving almost-global convergence from every initial condition except the antipodal unstable equilibrium (Pasqualetti et al., 10 Jun 2026).

Two finite-time failure modes are isolated. The first is a degenerate weighted anchor sum, where $\Omega_i=0$5 is very small. If anchors are drawn i.i.d. uniformly from the sphere, the paper gives

$\Omega_i=0$6

and

$\Omega_i=0$7

for some absolute constant $\Omega_i=0$8. The second is initialization near the unstable antipode. If $\Omega_i=0$9 is uniform on the sphere, then the probability of starting within angle $d_{\mathrm{osc}}=2$0 of $d_{\mathrm{osc}}=2$1 decays exponentially in $d_{\mathrm{osc}}=2$2 for fixed $d_{\mathrm{osc}}=2$3 (Pasqualetti et al., 10 Jun 2026).

These guarantees are specific to the one-free-oscillator forced system used in fixed-query attention. The paper explicitly does not extend them to generic mutually coupled multi-oscillator Lohe networks, where multistability is common. This distinction matters because several neighboring synchronization models, including Kuramoto Attention and FSN, rely on token-token coupling rather than fixed anchors (Nunley, 10 Jun 2026).

5. Empirical behavior

The reported experiments span keyword spotting, subject-verb agreement, WikiText-2, and TinyStories. The paper’s headline pattern is task-dependent: at the minimal hardware configuration $d_{\mathrm{osc}}=2$4, oscillator attention outperforms softmax on low-complexity bidirectional tasks, while on causal language modeling softmax remains better but the gap decreases monotonically as $d_{\mathrm{osc}}=2$5 grows (Pasqualetti et al., 10 Jun 2026).

Task	Softmax	Fixed-query oscillator attention
Keyword spotting	$d_{\mathrm{osc}}=2$6	$d_{\mathrm{osc}}=2$7
Subject-verb agreement, hard sentences	$d_{\mathrm{osc}}=2$8	$d_{\mathrm{osc}}=2$9
WikiText-2 PPL, $x_i=(\cos\theta_i,\sin\theta_i)$0	$x_i=(\cos\theta_i,\sin\theta_i)$1	$x_i=(\cos\theta_i,\sin\theta_i)$2
WikiText-2 PPL, $x_i=(\cos\theta_i,\sin\theta_i)$3	$x_i=(\cos\theta_i,\sin\theta_i)$4	$x_i=(\cos\theta_i,\sin\theta_i)$5
TinyStories PPL, $x_i=(\cos\theta_i,\sin\theta_i)$6	$x_i=(\cos\theta_i,\sin\theta_i)$7	$x_i=(\cos\theta_i,\sin\theta_i)$8
TinyStories PPL, $x_i=(\cos\theta_i,\sin\theta_i)$9	$e_1,\dots,e_T \in \mathbb{R}^{d_{\rm model}}$0	$e_1,\dots,e_T \in \mathbb{R}^{d_{\rm model}}$1

On hard subject-verb agreement sentences, the gain is $e_1,\dots,e_T \in \mathbb{R}^{d_{\rm model}}$2 percentage points, and the paper reports zero training failures for oscillator attention versus one catastrophic failure in five runs for softmax. On keyword spotting, oscillator attention improves by $e_1,\dots,e_T \in \mathbb{R}^{d_{\rm model}}$3 percentage point at $e_1,\dots,e_T \in \mathbb{R}^{d_{\rm model}}$4. By contrast, on causal language modeling the perplexity gap falls from $e_1,\dots,e_T \in \mathbb{R}^{d_{\rm model}}$5 to $e_1,\dots,e_T \in \mathbb{R}^{d_{\rm model}}$6 on WikiText-2 and from $e_1,\dots,e_T \in \mathbb{R}^{d_{\rm model}}$7 to $e_1,\dots,e_T \in \mathbb{R}^{d_{\rm model}}$8 on TinyStories as $e_1,\dots,e_T \in \mathbb{R}^{d_{\rm model}}$9 increases from $$w_{ij} = \sigma\!\left(\frac{(Fe_i)^\top (Ge_j)}{\sqrt{d_h}}\right),$$0 to $$w_{ij} = \sigma\!\left(\frac{(Fe_i)^\top (Ge_j)}{\sqrt{d_h}}\right),$$1 (Pasqualetti et al., 10 Jun 2026).

Ablations indicate that the attention geometry itself carries much of the computation on the bidirectional tasks. With frozen $$w_{ij} = \sigma\!\left(\frac{(Fe_i)^\top (Ge_j)}{\sqrt{d_h}}\right),$$2, keyword spotting changes from $$w_{ij} = \sigma\!\left(\frac{(Fe_i)^\top (Ge_j)}{\sqrt{d_h}}\right),$$3 to $$w_{ij} = \sigma\!\left(\frac{(Fe_i)^\top (Ge_j)}{\sqrt{d_h}}\right),$$4, and subject-verb agreement changes from $$w_{ij} = \sigma\!\left(\frac{(Fe_i)^\top (Ge_j)}{\sqrt{d_h}}\right),$$5 to $$w_{ij} = \sigma\!\left(\frac{(Fe_i)^\top (Ge_j)}{\sqrt{d_h}}\right),$$6. The paper interprets this as evidence that the oscillator attention geometry is the main computational element in those settings (Pasqualetti et al., 10 Jun 2026).

The ODE verification experiments compare RK45 integration to the analytic fixed point. On TinyStories models, the fraction of tokens converged to error $$w_{ij} = \sigma\!\left(\frac{(Fe_i)^\top (Ge_j)}{\sqrt{d_h}}\right),$$7 at $$w_{ij} = \sigma\!\left(\frac{(Fe_i)^\top (Ge_j)}{\sqrt{d_h}}\right),$$8 rises from $$w_{ij} = \sigma\!\left(\frac{(Fe_i)^\top (Ge_j)}{\sqrt{d_h}}\right),$$9 at $F,G \in \mathbb{R}^{d_h \times d_{\rm model}}$0 to $F,G \in \mathbb{R}^{d_h \times d_{\rm model}}$1 at $F,G \in \mathbb{R}^{d_h \times d_{\rm model}}$2, while apparent antipodal and degenerate failures both decrease with oscillator dimension. Extending $F,G \in \mathbb{R}^{d_h \times d_{\rm model}}$3 from $F,G \in \mathbb{R}^{d_h \times d_{\rm model}}$4 to $F,G \in \mathbb{R}^{d_h \times d_{\rm model}}$5 at $F,G \in \mathbb{R}^{d_h \times d_{\rm model}}$6 raises convergence from $F,G \in \mathbb{R}^{d_h \times d_{\rm model}}$7 to $F,G \in \mathbb{R}^{d_h \times d_{\rm model}}$8, consistent with the asymptotic theorem (Pasqualetti et al., 10 Jun 2026).

6. Relation to adjacent notions of “fixed-query” and “oscillator attention”

The literature around attention and synchronization uses overlapping terminology, but the mechanisms are not interchangeable. The closest direct relatives of fixed-query oscillator attention are other oscillator-based self-attention layers on phase or manifold-valued states; more distant relatives either use “fixed-query” in a different sense or are not oscillator models at all.

Method	Relation to fixed-query oscillator attention	Key distinction
Kuramoto Attention (Nunley, 10 Jun 2026)	Strong oscillator-attention relative	Query is implicit and state-tied, not fixed across tokens
Selective Synchronization Attention (Hays, 16 Feb 2026)	Adjacent oscillator operator	Query-free synchronization kernel, not query-based
Frustrated Synchronization Network (Nunley, 17 Jun 2026)	Synchronization-based alternative	Keeps query-key score map, replaces value pathway with frustrated coupling
Tempo-invariant audio attention (Dorfer et al., 2018)	Only partially related	Soft frame-wise input weighting, explicitly not oscillator or fixed-query transformer attention
NAC (Razzaq et al., 11 Dec 2025)	Contrasting dynamical attention	First-order relaxation of logits, not oscillatory
DS-Det (Cao et al., 26 Jul 2025)	Different use of “fixed-query”	Fixed number of object queries in DETR-like decoders, not oscillator dynamics

"Attention as a Perspective for Learning Tempo-invariant Audio Queries" uses a softmax attention pathway $F,G \in \mathbb{R}^{d_h \times d_{\rm model}}$9 that outputs scalar weights $r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$00 for spectrogram frames and multiplies each frame by its weight before the audio CNN. The paper explicitly states that this is not oscillator attention, not periodic attention, and not a fixed external query vector in the transformer sense (Dorfer et al., 2018).

"NAC" reformulates attention-logit computation as the solution of a scalar first-order ODE,

$r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$01

with exact solution

$r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$02

The paper explicitly characterizes this as content-conditioned exponential relaxation toward a learned attractor, not as an oscillator mechanism (Razzaq et al., 11 Dec 2025).

"DS-Det" uses “fixed-query” to refer to a fixed-size set of detector queries in DETR-like decoders. Its notion of an “oscillator-like” problem is interpretive: the paper itself names the conflict between self-attention and cross-attention Recurrent Opposing inTeractions (ROT), not oscillator dynamics in the mathematical sense (Cao et al., 26 Jul 2025).

7. Limitations and research directions

The proposal is explicit about its scope. Its main objective is not to replace softmax in software, where softmax is described as “simpler, faster, and better understood,” but to provide a mathematically grounded blueprint for accurate attention on physical substrates. The intended deployment is hybrid: a digital front-end computes couplings and anchor positions, a physical oscillator network performs the equilibration, and a digital back-end reads out attention weights and continues the transformer computation (Pasqualetti et al., 10 Jun 2026).

Several constraints are intrinsic to that goal. Couplings are required to be positive; the geometry is spherical; the query side is fixed by learned anchors; and attention is read out through cosine similarity with affine normalization. The paper explicitly attributes the residual language-modeling gap to the limited expressivity of fixed learned query anchors relative to fully input-dependent softmax queries, especially when $r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$03 is small (Pasqualetti et al., 10 Jun 2026).

Hardware maturity is also uneven across dimensions. The $r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$04 case is described as the physically mature setting, with possible substrates including mechanical oscillators, Josephson junctions, MEMS resonators, and electrical LC tanks. Higher-dimensional oscillator substrates remain an open hardware question. The observed scaling of the perplexity gap with $r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$05 is empirical rather than theoretically proved; the paper reports fitted power laws for the gap but explicitly does not claim a theorem that oscillator attention converges to softmax as $r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$06 grows (Pasqualetti et al., 10 Jun 2026).

The future directions named in the paper are correspondingly concrete: using nonzero $r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$07 to encode positional information, identifying the effective rank of softmax attention required by a task and matching it with $r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$08, developing higher-dimensional physical oscillator substrates, and exploring biologically plausible learning rules such as Hebbian or STDP updates for couplings instead of backpropagation. Taken together, these directions suggest that fixed-query oscillator attention is best understood not as a universal replacement for standard attention, but as a physically grounded synchronization-based attention family whose central research problem is the trade-off between hardware realizability and representational flexibility (Pasqualetti et al., 10 Jun 2026).