---
title: Fixed-Query Oscillator Attention
url: https://www.emergentmind.com/topics/fixed-query-oscillator-attention
type: topic
---

# Fixed-Query Oscillator Attention

Fixed-query oscillator attention is an attention mechanism in which the query side is replaced by learned anchors fixed on a sphere, while input-dependent free oscillators evolve under Kuramoto–Lohe dynamics until they settle at positions that encode attention weights through cosine similarity. In this formulation, the nonlinear core of attention is no longer softmax but equilibration of a gradient flow on $\mathbb{S}^{d_{\mathrm{osc}}-1}$, followed by affine normalization at readout. The proposal is explicitly motivated by “energy-constrained physical substrates,” where exponentiation and in-loop global reduction are costly and lack a natural physical analog, whereas synchronization is native to oscillator arrays [2606.12059].

## 1. Definition and scope

In fixed-query oscillator attention, the “fixed-query” designation means that the query-like objects are not input-dependent token queries. Instead, they are learned anchors $r_j \in \mathbb{S}^{d_{\mathrm{osc}}-1}$ that are fixed during inference. Inputs still determine pairwise couplings, but the query side of the mechanism is a fixed learned reference geometry on the sphere rather than a runtime-generated $q_i = W_Q e_i$ as in standard scaled dot-product attention [2606.12059].

The construction begins from the Lohe model for oscillators on the unit sphere,
$$
\dot x_i = \Omega_i x_i + \left(I - x_i x_i^\top\right)\sum_{j=1}^n w_{ij}x_j,
$$
with $\Omega_i=0$ in the fixed-query attention setting. For $d_{\mathrm{osc}}=2$, writing $x_i=(\cos\theta_i,\sin\theta_i)$ recovers the classical scalar Kuramoto model. The method then specializes the dynamics asymmetrically: anchor oscillators are learned and fixed, while free oscillators evolve under the forcing induced by those anchors. This asymmetry is central because the paper attributes the resulting uniqueness and almost-global convergence guarantees to the one-particle forced system rather than to a mutually coupled many-oscillator network [2606.12059].

This makes the mechanism distinct from several adjacent synchronization-based attention proposals. In "Kuramoto Attention," the effective query is implicit in the current token phase state and learned query gates, so it is oscillator attention but not literally fixed-query in the sense of a shared fixed learned query template [2606.11585]. In "Selective Synchronization Attention," there is no explicit query variable at all; the operator is best described as query-free oscillator attention derived from pairwise synchronization compatibility [2602.14445]. "Attention as Frustrated Synchronization" keeps a causal query-key score map but replaces the value pathway with harmonic synchronization dynamics, making it a synchronization-based alternative to standard value attention rather than a fixed-query design [2606.18694].

## 2. Dynamical formulation on the sphere

Let the input sequence be $e_1,\dots,e_T \in \mathbb{R}^{d_{\rm model}}$. For one attention head, fixed-query oscillator attention defines positive couplings
$$
w_{ij} = \sigma\!\left(\frac{(Fe_i)^\top (Ge_j)}{\sqrt{d_h}}\right),
$$
where $F,G \in \mathbb{R}^{d_h \times d_{\rm model}}$ are learned projections, $d_h$ is the head dimension, and $\sigma$ is any strictly positive function; the experiments use $\sigma(x)=\log(1+e^x)$, i.e. softplus [2606.12059].

These couplings weight the learned anchors:
$$
h_i = \sum_{j=1}^T w_{ij} r_j \in \mathbb{R}^{d_{\mathrm{osc}}}.
$$
The free oscillator state $z_i \in \mathbb{S}^{d_{\mathrm{osc}}-1}$ evolves according to
$$
\dot z_i = \left(I-z_i z_i^\top\right)\sum_{j=1}^T w_{ij}r_j
= \left(I-z_i z_i^\top\right) h_i.
$$
The tangent projector $I-z_i z_i^\top$ keeps the dynamics on the sphere. The settled state is
$$
z_i^* = \frac{h_i}{\|h_i\|}.
$$
Software training and inference use this analytic fixed point directly rather than numerically integrating the ODE [2606.12059].

Attention weights are then read out through shifted cosine similarity:
$$
\alpha_{ij}=
\frac{1+z_i^{*\top}r_j}
{\sum_{l=1}^{T}\left(1+z_i^{*\top}r_l\right)}.
$$
The shift by $1$ guarantees nonnegative numerators because $z_i^{*\top}r_j \in [-1,1]$. Per-head outputs use standard value aggregation,
$$
o_i^{(h)}=\sum_{j=1}^{T}\alpha_{ij}^{(h)}v_j^{(h)},
\qquad
v_j^{(h)}=W_V^{(h)}e_j,
$$
followed by the usual multi-head combination with $W_O$ [2606.12059].

For causal masking, the method sets $w_{ij}=0$ for $j>i$ and restricts the denominator in the readout to $l \le i$. The paper also notes a software-side sharpening variant,
$$
\alpha_{ij}\propto (1+z_i^{*\top}r_j)^p,\qquad p\ge 1,
$$
but treats $p=1$ as the hardware-native case [2606.12059].

## 3. Geometric and variational interpretation

The fixed-query dynamics are presented as a constrained gradient flow on the sphere. For each free oscillator, define the energy
$$
V(z_i)=-z_i^\top h_i.
$$
Because $z_i$ must remain on $\mathbb{S}^{d_{\mathrm{osc}}-1}$, the Euclidean gradient must be projected onto the tangent space, yielding
$$
\dot z_i = (I-z_i z_i^\top)h_i.
$$
The equilibrium therefore solves
$$
\max_{z_i\in \mathbb{S}^{d_{\mathrm{osc}}-1}} z_i^\top h_i,
$$
whose solution is $h_i/\|h_i\|$ whenever $h_i\neq 0$ [2606.12059].

This gives the mechanism a clear geometric decomposition. First, the digital front-end computes input-dependent positive couplings. Second, those couplings form a weighted anchor sum in $\mathbb{R}^{d_{\mathrm{osc}}}$. Third, the free oscillator relaxes to the unit vector maximally aligned with that sum. Fourth, cosine similarity to the anchors is affinely normalized into a row-stochastic attention distribution. The essential nonlinearity is therefore the normalization map $h_i \mapsto h_i/\|h_i\|$, interpreted as physical equilibration rather than softmax arithmetic [2606.12059].

The same section of the literature makes clear that not all oscillator-attention proposals share this geometry. In "Kuramoto Attention," values are the raw phase states, the update is exactly $\sum_u A_{t,u}\sin(\theta_u-\theta_t)$ coordinatewise, and the query is state-tied rather than fixed [2606.11585]. In "Attention as Frustrated Synchronization," the update is instead a harmonic Kuramoto–Sakaguchi–Daido coupling law with a learned complex kernel and one-step delay, so the computation is located in structured departures from agreement rather than in convergence to a single spherical anchor mixture [2606.18694].

## 4. Theoretical guarantees

The main theorem for fixed-query oscillator attention states that if
$$
h_i=\sum_{j=1}^T w_{ij}r_j,\qquad \|h_i\|>0,
$$
then the gradient flow
$$
\dot z_i=(I-z_i z_i^\top)h_i,\qquad z_i\in\mathbb{S}^{d_{\mathrm{osc}}-1},
$$
has exactly two equilibria,
$$
z_i^*=\frac{h_i}{\|h_i\|}\quad\text{and}\quad -z_i^*.
$$
The equilibrium $z_i^*$ is asymptotically stable with basin of attraction
$$
\mathbb{S}^{d_{\mathrm{osc}}-1}\setminus\{-z_i^*\},
$$
while $-z_i^*$ is unstable. The proof uses the Lyapunov function $V(z_i)=-z_i^\top h_i$ and LaSalle’s invariance principle, giving almost-global convergence from every initial condition except the antipodal unstable equilibrium [2606.12059].

Two finite-time failure modes are isolated. The first is a degenerate weighted anchor sum, where $\|h_i\|$ is very small. If anchors are drawn i.i.d. uniformly from the sphere, the paper gives
$$
\mathbb{E}\bigl[\|h_i\|^2\bigr]=\|w_i\|^2,
$$
and
$$
\Pr\bigl(\|h_i\|^2\le \varepsilon \|w_i\|^2\bigr)
\le
\exp\!\bigl(-c\,(d_{\mathrm{osc}}-1)(1-\varepsilon)^2\bigr)
$$
for some absolute constant $c>0$. The second is initialization near the unstable antipode. If $z_i(0)$ is uniform on the sphere, then the probability of starting within angle $\alpha$ of $-z_i^*$ decays exponentially in $d_{\mathrm{osc}}$ for fixed $\alpha\in(0,\pi/2)$ [2606.12059].

These guarantees are specific to the one-free-oscillator forced system used in fixed-query attention. The paper explicitly does not extend them to generic mutually coupled multi-oscillator Lohe networks, where multistability is common. This distinction matters because several neighboring synchronization models, including Kuramoto Attention and FSN, rely on token-token coupling rather than fixed anchors [2606.11585].

## 5. Empirical behavior

The reported experiments span keyword spotting, subject-verb agreement, WikiText-2, and TinyStories. The paper’s headline pattern is task-dependent: at the minimal hardware configuration $d_{\mathrm{osc}}=2$, oscillator attention outperforms softmax on low-complexity bidirectional tasks, while on causal language modeling softmax remains better but the gap decreases monotonically as $d_{\mathrm{osc}}$ grows [2606.12059].

| Task | Softmax | Fixed-query oscillator attention |
|---|---:|---:|
| Keyword spotting | $87.45 \pm 0.70\%$ | $88.45 \pm 0.48\%$ |
| Subject-verb agreement, hard sentences | $92.11 \pm 7.87\%$ | $97.38 \pm 0.33\%$ |
| WikiText-2 PPL, $d_{\mathrm{osc}}=2$ | $99.58$ | $110.67 \pm 0.51$ |
| WikiText-2 PPL, $d_{\mathrm{osc}}=32$ | $99.58$ | $102.56 \pm 0.37$ |
| TinyStories PPL, $d_{\mathrm{osc}}=2$ | $8.54$ | $10.93 \pm 0.05$ |
| TinyStories PPL, $d_{\mathrm{osc}}=32$ | $8.54$ | $9.11 \pm 0.02$ |

On hard subject-verb agreement sentences, the gain is $+5.27$ percentage points, and the paper reports zero training failures for oscillator attention versus one catastrophic failure in five runs for softmax. On keyword spotting, oscillator attention improves by $+1.00$ percentage point at $d_{\mathrm{osc}}=2$. By contrast, on causal language modeling the perplexity gap falls from $+11.09$ to $+2.98$ on WikiText-2 and from $+2.39$ to $+0.57$ on TinyStories as $d_{\mathrm{osc}}$ increases from $2$ to $32$ [2606.12059].

Ablations indicate that the attention geometry itself carries much of the computation on the bidirectional tasks. With frozen $W_V$, keyword spotting changes from $88.45 \pm 0.48\%$ to $88.26 \pm 0.17\%$, and subject-verb agreement changes from $97.38 \pm 0.33\%$ to $97.14 \pm 0.46\%$. The paper interprets this as evidence that the oscillator attention geometry is the main computational element in those settings [2606.12059].

The ODE verification experiments compare RK45 integration to the analytic fixed point. On TinyStories models, the fraction of tokens converged to error $<0.01$ at $T_{\max}=30$ rises from $84.3\%$ at $d_{\mathrm{osc}}=2$ to $98.3\%$ at $d_{\mathrm{osc}}=32$, while apparent antipodal and degenerate failures both decrease with oscillator dimension. Extending $T_{\max}$ from $30$ to $5000$ at $d_{\mathrm{osc}}=2$ raises convergence from $84.3\%$ to $98.7\%$, consistent with the asymptotic theorem [2606.12059].

## 6. Relation to adjacent notions of “fixed-query” and “oscillator attention”

The literature around attention and synchronization uses overlapping terminology, but the mechanisms are not interchangeable. The closest direct relatives of fixed-query oscillator attention are other oscillator-based self-attention layers on phase or manifold-valued states; more distant relatives either use “fixed-query” in a different sense or are not oscillator models at all.

| Method | Relation to fixed-query oscillator attention | Key distinction |
|---|---|---|
| Kuramoto Attention [2606.11585] | Strong oscillator-attention relative | Query is implicit and state-tied, not fixed across tokens |
| Selective Synchronization Attention [2602.14445] | Adjacent oscillator operator | Query-free synchronization kernel, not query-based |
| Frustrated Synchronization Network [2606.18694] | Synchronization-based alternative | Keeps query-key score map, replaces value pathway with frustrated coupling |
| Tempo-invariant audio attention [1809.05689] | Only partially related | Soft frame-wise input weighting, explicitly not oscillator or fixed-query transformer attention |
| NAC [2512.10282] | Contrasting dynamical attention | First-order relaxation of logits, not oscillatory |
| DS-Det [2507.19807] | Different use of “fixed-query” | Fixed number of object queries in DETR-like decoders, not oscillator dynamics |

"Attention as a Perspective for Learning Tempo-invariant Audio Queries" uses a softmax attention pathway $h$ that outputs scalar weights $a_t$ for spectrogram frames and multiplies each frame by its weight before the audio CNN. The paper explicitly states that this is not oscillator attention, not periodic attention, and not a fixed external query vector in the transformer sense [1809.05689].

"NAC" reformulates attention-logit computation as the solution of a scalar first-order ODE,
$$
\frac{da_t}{dt}=-\omega_\tau([\mathbf q;\mathbf k])\,a_t+\phi([\mathbf q;\mathbf k]),
$$
with exact solution
$$
a_t = a_0 e^{-\omega_\tau t} + \frac{\phi}{\omega_\tau}(1-e^{-\omega_\tau t}).
$$
The paper explicitly characterizes this as content-conditioned exponential relaxation toward a learned attractor, not as an oscillator mechanism [2512.10282].

"DS-Det" uses “fixed-query” to refer to a fixed-size set of detector queries in DETR-like decoders. Its notion of an “oscillator-like” problem is interpretive: the paper itself names the conflict between self-attention and cross-attention Recurrent Opposing inTeractions (ROT), not oscillator dynamics in the mathematical sense [2507.19807].

## 7. Limitations and research directions

The proposal is explicit about its scope. Its main objective is not to replace softmax in software, where softmax is described as “simpler, faster, and better understood,” but to provide a mathematically grounded blueprint for accurate attention on physical substrates. The intended deployment is hybrid: a digital front-end computes couplings and anchor positions, a physical oscillator network performs the equilibration, and a digital back-end reads out attention weights and continues the transformer computation [2606.12059].

Several constraints are intrinsic to that goal. Couplings are required to be positive; the geometry is spherical; the query side is fixed by learned anchors; and attention is read out through cosine similarity with affine normalization. The paper explicitly attributes the residual language-modeling gap to the limited expressivity of fixed learned query anchors relative to fully input-dependent softmax queries, especially when $d_{\mathrm{osc}}$ is small [2606.12059].

Hardware maturity is also uneven across dimensions. The $d_{\mathrm{osc}}=2$ case is described as the physically mature setting, with possible substrates including mechanical oscillators, Josephson junctions, MEMS resonators, and electrical LC tanks. Higher-dimensional oscillator substrates remain an open hardware question. The observed scaling of the perplexity gap with $d_{\mathrm{osc}}$ is empirical rather than theoretically proved; the paper reports fitted power laws for the gap but explicitly does not claim a theorem that oscillator attention converges to softmax as $d_{\mathrm{osc}}$ grows [2606.12059].

The future directions named in the paper are correspondingly concrete: using nonzero $\Omega_i$ to encode positional information, identifying the effective rank of softmax attention required by a task and matching it with $d_{\mathrm{osc}}$, developing higher-dimensional physical oscillator substrates, and exploring biologically plausible learning rules such as Hebbian or STDP updates for couplings instead of backpropagation. Taken together, these directions suggest that fixed-query oscillator attention is best understood not as a universal replacement for standard attention, but as a physically grounded synchronization-based attention family whose central research problem is the trade-off between hardware realizability and representational flexibility [2606.12059].

Source: https://www.emergentmind.com/topics/fixed-query-oscillator-attention