Oscillatory-Multiplexed Attention

• Oscillatory-Multiplexed Attention is a synchronization-based paradigm that maps tokens to natural frequencies and phases to enforce content-dependent sparsity.
• It replaces standard dot-product attention with a Kuramoto-inspired operator, using multi-band decomposition across independent synchronization heads.
• By unifying semantic and positional cues in the frequency domain, it enhances computational efficiency and supports biologically plausible learning regimes.

Oscillatory-Multiplexed Attention (OMA) is a biologically grounded attention paradigm that replaces conventional dot-product self-attention in neural sequence models with a synchronization-based operator derived from the Kuramoto model of coupled oscillators. Each representational token is mapped into a learnable “natural frequency” and phase. The key innovation is that selective pairwise attention (“coupling”) emerges only for token pairs whose frequency mismatch allows them to phase-lock, enforcing a content- and position-dependent sparsity. Multi-band decomposition is realized via independent “synchronization heads,” each operating on distinct frequency subspaces, in analogy to neural oscillatory bands in biological systems. This multiplexed mechanism unifies semantic and positional information into the frequency domain, enables efficient computation, and supports biologically plausible learning regimes (Hays, 16 Feb 2026, Hays, 21 Feb 2026).

1. Mathematical Foundations and Operator Formulation

OMA is instantiated via the Selective Synchronization Attention (SSA) operator. Formally, for each input token $i$, two $d$-dimensional vectors are obtained by learned linear transformations: $$\omega_i = W_\omega x_i \in \mathbb{R}^d, \quad \theta_i = W_\theta x_i \in \mathbb{R}^d,$$ where $\omega_i$ encodes the token’s natural frequency and $\theta_i$ its phase. For synchronization head $h$, these projections are performed in a $d=D/H$-dimensional subspace.

Pairwise dynamics are governed by the Kuramoto model’s steady-state phase-locking condition. The phase difference $\phi_{ij} = \theta_i - \theta_j$ satisfies: $$\sin \phi_{ij}^{*} = \frac{\Delta\omega_{ij}}{K r J_{ij}}, \quad \Delta\omega_{ij} = \|\omega_i - \omega_j\|_2,$$ where $K$ is a global learnable coupling, $J_{ij} = \exp(-\alpha \|\omega_i - \omega_j\|^2)$ is a per-head frequency-dependent Gaussian kernel, and $r$ is the emergent global order parameter. A valid solution and thus nonzero attention exists only if $|\Delta\omega_{ij}| \leq K r J_{ij}$, implementing phase-locking sparsity.

The synchronization strength—serving as attention coefficient—is: $$a_{ij} = S_{ij} = \begin{cases} J_{ij} \sqrt{1 - \left(\frac{\Delta\omega_{ij}}{K r J_{ij}}\right)^2}, & \text{if } \Delta\omega_{ij} \leq K r J_{ij} \ 0, & \text{otherwise} \end{cases}$$ The order parameter $r$ is computed globally: $$r = \frac{1}{d} \sum_{\ell=1}^d \left| \frac{1}{N} \sum_{j=1}^N e^{i \theta_j^{(\ell)}} \right|$$ where $N$ is the sequence length.

2. Frequency-Multiplexed Multi-Head Decomposition

OMA generalizes the multi-head paradigm by assigning each “head” to an independent frequency band:

• Each head $h$ comprises its independent projections $W_\omega^{(h)}, W_\theta^{(h)}, \alpha_h$ and thus distinct coupling profiles $J^{(h)}, S^{(h)}$.
• Learnable per-head coupling bandwidth $\alpha_h$ determines the scale of frequency selectivity, such that each head effectively operates on a different segment of the input’s natural frequency spectrum (Hays, 16 Feb 2026).
• This mechanism mimics the theta, alpha, beta, gamma multiplexing observed in biological neural systems.
• Frequencies of tokens converge under training: tokens with overlapping contextual or semantic content acquire proximate frequencies, while those more dissimilar spread out, naturally encoding both semantic and positional relations.

OMA thereby eliminates the need for explicit positional encodings, since semantic and positional similarity are reflected directly in pairwise frequency proximity (Hays, 16 Feb 2026). Gradient descent on sequence-prediction loss implicitly drives frequency clustering as required for synchronization.

3. Algorithmic Procedure and Computational Properties

SSA, as the prototypical OMA implementation, processes each attention head through the following steps:

1. Linear projections generate $\omega_i$, $\theta_i$, and value vectors $v_i = W_V x_i$.
2. Compute all $O(N^2)$ pairwise frequency differences $\Delta\omega_{ij}$ and couplings $J_{ij}$.
3. Compute the global order parameter $r$ from phases.
4. Calculate threshold $\tau_{ij} = K r J_{ij}$, clamp the ratio, and determine valid synchronizations.
5. Evaluate $S_{ij}$, zeroing non-phase-locked pairs.
6. (Optional) Apply top-$k$ sparsification to rows.
7. Row-normalize $S$, then aggregate output: $$y_i = \sum_j S_{ij} v_j / (\sum_j S_{ij} + \varepsilon)$$.

Dense OMA exhibits $O(N^2 d)$ complexity per head—as with standard attention. However, the sparsity constraint ensures that only token pairs within the phase-locking window produce nonzero weights. With top-$k$ truncation or in regimes where the frequency spectrum is wide, the effective cost drops to $O(N k d)$, with $k \ll N$ (Hays, 16 Feb 2026).

A macroscopic order parameter $r$ enforces an attention regime that can shift dynamically between globally coherent (broad synchronization) and locally selective (sharp attention bands) according to task and training regime.

4. Integration into Neural Architectures and Empirical Properties

OMA is instantiated architecturally in the Oscillatory Synchronization Network (OSN) by substituting standard multi-head dot-product attention with multi-frequency synchronization heads. The OSN block is defined as: $$z = X + \text{Dropout}(\text{MFSH}(\text{LayerNorm}(X))) \ Y = z + \text{Dropout}(\text{FFN}(\text{LayerNorm}(z)))$$

Parameter count increases only marginally, by $H$ per-head $\alpha_h$ parameters and a single global $K$. SSA’s initialization exhibits:

• Perfect self-synchronization ($S_{ii}=1$) on the diagonal;
• Highly structured, non-uniform off-diagonal patterns, determined by the random frequency and phase initialization;
• Head diversity (different heads exhibit distinct coupling/synchronization matrices due to initial frequency separation).

At initialization, the order parameter $r$ typically concentrates near $0.85$, indicating considerable phase coherence. Training drives $r$ downward and sharpens off-diagonal selectivity, manifesting a curriculum from diffuse global attention toward structured, task-induced synchronization patterns (Hays, 16 Feb 2026).

5. Attention Multiplexing via Oscillatory Synchronization and Sparse Dynamics

OMA realizes attention multiplexing through the formation of dynamically synchronized clusters:

• Each token acts as an oscillator with frequency $\omega_i$; phase-locking occurs selectively for those within mutual coupling range.
• Clusters of tokens with proximate frequencies synchronize, effectively acting as attention “heads,” each head’s population and composition governed by the learned frequency spectrum and the underlying data.
• The local coupling kernel $C(\omega_i, \omega_j)$ enforces that strong communication is restricted to tokens within a similar frequency band, yielding adaptive, content-based sparsity (Hays, 21 Feb 2026).

Dynamical heads are thus not static pre-allocated subspaces but emergent, changing with each input sequence, enabling self-organizing, data-driven attention multiplexing. Attention can be interpreted as routing occurring only among coherent phase clusters, with cross-cluster interaction suppressed due to phase incoherence and resulting from the nonlinear synchrony constraint.

6. Biological and Computational Significance

OMA draws explicit inspiration from biological mechanisms in neural systems, mirroring oscillatory multiplexing observed in the brain. Its use of closed-form Kuramoto steady states grounds its operator in mathematical neuroscience. The intrinsic sparsity via phase-locking threshold precludes the need for ad hoc attention masking or pruning heuristics.

Further, OMA supports unification of positional and semantic coding, aligning with evidence that both are intertwined in cortical oscillatory patterns. In architectures implementing dual dynamical regimes—such as Hebbian-Oscillatory Co-Learning—OMA’s order parameter $r$ also gates slower-timescale plasticity, allowing learning consolidation only under sufficiently synchronized network states, further tying functional computation to biological principles (Hays, 21 Feb 2026).

7. Convergence, Stability, and Emergent Properties

The coupled oscillator/attention system admits a composite Lyapunov function ensuring global convergence: $$V(W, \boldsymbol\theta) = -\frac{K}{2N}\sum_{i,j}\cos(\theta_i - \theta_j) + \frac{\lambda}{2}\|W\|_F^2$$ Fast phase dynamics strictly reduce the oscillatory component, while slow Hebbian adaptation reduces the weight component. Borkar’s two-timescale theorem and LaSalle’s invariance principle guarantee almost sure convergence to stable equilibrium points, corresponding to structurally consistent and phase-locked network states (Hays, 21 Feb 2026).

Empirically, OMA/SSA-based architectures demonstrate:

• Strong architectural inductive biases at initialization, with non-uniform, structured attention;
• Dynamically emergent, head-diverse communication patterns;
• Adaptive sparsity and control over computational cost via emergent phase thresholds;
• Robust integration with gradient-based learning frameworks and drop-in compatibility with Transformer blocks.

The unified oscillatory-multiplexed attention paradigm thus establishes a principle for attention rooted in synchronization dynamics, with mathematically and biologically principled emergent structures (Hays, 16 Feb 2026, Hays, 21 Feb 2026).