Harmonic Oscillator Recurrent Network

• HORN is a continuous-time recurrent network that models each unit as a damped harmonic oscillator with second-order dynamics.
• It utilizes oscillatory transient responses, resonance, and fading memory to effectively encode temporal information in sequential tasks.
• Empirical studies demonstrate HORN’s ability to achieve competitive performance through both digital simulations and analog hardware, despite precision challenges.

Searching arXiv for relevant HORN and related continuous-time attention papers. First, I’ll look up the core HORN and oscillator-attention papers by title/id, then related context like ContiFormer. A Harmonic Oscillator Recurrent Network (HORN) is a recurrent neural architecture in which each unit is modeled as a damped harmonic oscillator rather than as a first-order hidden state updated by a conventional nonlinear recurrence. In its canonical form, the hidden dynamics are continuous-time, second-order, and explicitly parameterized by natural frequencies, damping factors, recurrent couplings, and input couplings. Recent work has developed two closely related lines of research: a direct HORN formulation and analog-electronic implementation for sequential classification, and an oscillator-based continuous-time attention mechanism for irregular time series that the authors explicitly identify as “exactly the kind of building block you’d want in a Harmonic Oscillator Recurrent Network (HORN)” (Carvalho et al., 4 Sep 2025, Shende et al., 12 Feb 2026).

1. Formal dynamical structure

In the full HORN model, the $i$-th node has scalar state $x_i(t)$ and obeys the damped, driven oscillator equation

$$\ddot{x}_{i}(t) + 2 \gamma_{i} \dot{x}_{i}(t) + \omega_{i}^2 x_{i}(t) = F(\mathbf{x},\dot{\mathbf{x}}, t),$$

where $x_i(t)\in\mathbb{R}$ is an oscillator amplitude, $\dot{x}_i(t)$ is a velocity-like state, $\omega_i>0$ is the natural angular frequency, $\gamma_i>0$ is the damping factor, and $F(\mathbf{x},\dot{\mathbf{x}},t)$ is a forcing term that includes recurrent coupling and external input. The unconstrained forcing term is

$$F(\mathbf{x},\dot{\mathbf{x}}, t) = \alpha\, \tanh\bigl( \mathbf{V}\mathbf{x} + \mathbf{W}\dot{\mathbf{x}} + \mathbf{I}\, s(t)\bigr),$$

with $\mathbf{V},\mathbf{W}\in\mathbb{R}^{n\times n}$, $x_i(t)$0, scalar input $x_i(t)$1, and global gain $x_i(t)$2 (Carvalho et al., 4 Sep 2025).

This formulation makes the HORN state intrinsically second-order. A node is characterized jointly by its amplitude $x_i(t)$3 and its velocity $x_i(t)$4, so the effective hidden state is $x_i(t)$5. In the absence of forcing,

$x_i(t)$6

whose solutions are exponentially decaying sinusoids with frequency $x_i(t)$7. The resulting recurrence is therefore not merely a discretized first-order state transition; it is the discretization of a mechanical-looking ODE with explicit oscillatory and dissipative structure.

The analog-electronic study uses a simplified HORN imposed by hardware constraints: only $x_i(t)$8 nodes, no self-feedback, no $x_i(t)$9 nonlinearity, and amplitude coupling only in the main text, with $$\ddot{x}_{i}(t) + 2 \gamma_{i} \dot{x}_{i}(t) + \omega_{i}^2 x_{i}(t) = F(\mathbf{x},\dot{\mathbf{x}}, t),$$0. Under these constraints and writing $$\ddot{x}_{i}(t) + 2 \gamma_{i} \dot{x}_{i}(t) + \omega_{i}^2 x_{i}(t) = F(\mathbf{x},\dot{\mathbf{x}}, t),$$1, the discretized “digital twin” becomes

$$\ddot{x}_{i}(t) + 2 \gamma_{i} \dot{x}_{i}(t) + \omega_{i}^2 x_{i}(t) = F(\mathbf{x},\dot{\mathbf{x}}, t),$$2

The paper notes a small typographical oddity in this equation—the sum is written with $$\ddot{x}_{i}(t) + 2 \gamma_{i} \dot{x}_{i}(t) + \omega_{i}^2 x_{i}(t) = F(\mathbf{x},\dot{\mathbf{x}}, t),$$3 instead of $$\ddot{x}_{i}(t) + 2 \gamma_{i} \dot{x}_{i}(t) + \omega_{i}^2 x_{i}(t) = F(\mathbf{x},\dot{\mathbf{x}}, t),$$4 in the coupling term—while interpreting it conceptually as standard linear recurrent coupling from other units’ amplitudes. The discretization uses a symplectic Euler scheme for stability of the stiff oscillator dynamics (Carvalho et al., 4 Sep 2025).

2. Oscillatory computation, memory, and recurrence

The central computational premise of HORN is that transient oscillatory dynamics, rather than fixed-point convergence, constitute the substrate of computation. In the formulation summarized by the analog-electronic study, amplitude $$\ddot{x}_{i}(t) + 2 \gamma_{i} \dot{x}_{i}(t) + \omega_{i}^2 x_{i}(t) = F(\mathbf{x},\dot{\mathbf{x}}, t),$$5 is analogous to a membrane-potential-like or rate-like variable, velocity $$\ddot{x}_{i}(t) + 2 \gamma_{i} \dot{x}_{i}(t) + \omega_{i}^2 x_{i}(t) = F(\mathbf{x},\dot{\mathbf{x}}, t),$$6 is a momentum-like variable, and the combination of intrinsic oscillation, damping, and recurrent forcing yields transient responses that evolve in both amplitude and phase (Carvalho et al., 4 Sep 2025).

Several functional consequences are emphasized. First, HORNs can encode information in both phase and amplitude. Second, they can exploit resonance and interference patterns: an oscillator responds more strongly to inputs matching its intrinsic frequency $$\ddot{x}_{i}(t) + 2 \gamma_{i} \dot{x}_{i}(t) + \omega_{i}^2 x_{i}(t) = F(\mathbf{x},\dot{\mathbf{x}}, t),$$7. Third, damped oscillators provide fading memory with multiple timescales, because memory traces decay naturally through damping while remaining spectrally structured. The analog-electronic paper attributes prior digital HORN advantages in learning speed and robustness relative to classical RNNs to phase-based encoding and synchronization, frequency-selective resonance, and fading memory with multiple timescales (Carvalho et al., 4 Sep 2025).

This dynamical organization distinguishes HORN from standard recurrent architectures in several precise ways. Classical RNNs are typically discrete-time and first-order, with updates of the form

$$\ddot{x}_{i}(t) + 2 \gamma_{i} \dot{x}_{i}(t) + \omega_{i}^2 x_{i}(t) = F(\mathbf{x},\dot{\mathbf{x}}, t),$$8

whereas HORN begins with continuous-time, second-order oscillator dynamics and only then discretizes them. Standard RNNs often operate near fixed points or chaotic attractors; HORN enforces oscillatory local dynamics in each node. Standard recurrent models encode time by hidden-state accumulation, whereas HORN introduces explicit phase and resonance structure through $$\ddot{x}_{i}(t) + 2 \gamma_{i} \dot{x}_{i}(t) + \omega_{i}^2 x_{i}(t) = F(\mathbf{x},\dot{\mathbf{x}}, t),$$9 and $x_i(t)\in\mathbb{R}$0. The model parameters also admit direct physical interpretation—frequencies, damping, couplings, and input gains—which is significant for hardware mapping and for continuous-time analysis.

A common misconception is that oscillatory recurrence is merely a stylistic reparameterization of a conventional RNN. The cited formulations suggest otherwise: the hidden state is explicitly second-order, the local dynamics are mechanically constrained, and the computational use of resonance and transient interference is built into the architecture rather than added post hoc. A plausible implication is that HORN imposes a strong inductive bias toward temporally structured signals, especially where periodicity, smoothness, or decaying correlations are natural.

3. Closed-form oscillator banks and the HORN-like attention viewpoint

The 2026 oscillator-attention work does not present itself as a HORN paper in name, but it makes the connection explicit by replacing ContiFormer’s Neural ODE dynamics with linear damped, driven harmonic oscillators and stating that this is “exactly the kind of building block you’d want in a Harmonic Oscillator Recurrent Network (HORN).” In that formulation, each scalar key or value coordinate satisfies

$x_i(t)\in\mathbb{R}$1

with learnable damping $x_i(t)\in\mathbb{R}$2, natural frequency $x_i(t)\in\mathbb{R}$3, and forcing term $x_i(t)\in\mathbb{R}$4. In first-order state-space form, with $x_i(t)\in\mathbb{R}$5 and $x_i(t)\in\mathbb{R}$6,

$x_i(t)\in\mathbb{R}$7

and the exact solution is

$x_i(t)\in\mathbb{R}$8

Because the matrix exponential is available in closed form in the underdamped, critically damped, and overdamped regimes, the continuous-time trajectories are analytic rather than solver-dependent (Shende et al., 12 Feb 2026).

The same paper represents queries by a truncated sinusoidal expansion,

$x_i(t)\in\mathbb{R}$9

and defines continuous-time attention logits by the time-averaged inner product

$\dot{x}_i(t)$0

The softmax weights then follow the standard Transformer scaling. Because both queries and keys are expressed in sinusoids and exponentially damped sinusoids, the attention integrals can be evaluated analytically. This yields a physical interpretation in which attention is a resonance phenomenon: the key oscillator acts as a linear filter with transfer function

$\dot{x}_i(t)$1

and query–key interaction measures spectral overlap. When query energy is concentrated near $\dot{x}_i(t)$2, the inner product and therefore the logit are large; when frequencies are misaligned, the oscillatory terms average out and the attention is small (Shende et al., 12 Feb 2026).

Relative to ContiFormer (Chen et al., 2024), the oscillator formulation replaces numerical ODE solves with closed-form propagation and closed-form attention kernels. The paper gives numerical complexity

$\dot{x}_i(t)$3

for the numerical continuous-attention formulation, and

$\dot{x}_i(t)$4

for the closed-form oscillator layer. With $\dot{x}_i(t)$5, $\dot{x}_i(t)$6, and $\dot{x}_i(t)$7, the dominant $\dot{x}_i(t)$8 cost ratio is approximately $\dot{x}_i(t)$9, described as a “three orders of magnitude” reduction in the $\omega_i>0$0 term. The same work also states a Harmonic Approximation Theorem showing that a shared undamped oscillator bank on a fixed frequency grid can approximate continuous key trajectories arbitrarily well on compact intervals, and therefore can approximate the induced attention logits and weights arbitrarily well (Shende et al., 12 Feb 2026).

From a recurrent perspective, this oscillator-attention layer is continuous-time and state-space structured: $\omega_i>0$1 with token-specific initial conditions and forcing. The paper therefore proposes an explicit HORN-style abstraction: a global oscillator bank parameterized by $\omega_i>0$2, a state $\omega_i>0$3, and irregularly timed inputs that update initial conditions or driving terms. This suggests that HORN can be interpreted broadly as a family of continuous-time recurrent models built from superpositions of damped harmonic modes, with either attention-based or recurrent readout.

4. Architecture, readout, and analog-electronic realization

In the proof-of-concept analog implementation, the network architecture is deliberately small. The recurrent core contains $\omega_i>0$4 damped harmonic oscillator units, with all-to-all amplitude coupling except self-couplings, scalar input $\omega_i>0$5, and a 10-class output readout. In the digital simulation for sequential MNIST, one pixel is presented per integration step, so $\omega_i>0$6, and the continuous-time frequency $\omega_i>0$7 is interpreted as radians per pixel. The input sequence is a row-wise serialization of the $\omega_i>0$8 grayscale image, yielding 784 steps; the hardware experiment prepends a registration pulse and zero-pads the sequence to length $\omega_i>0$9 (Carvalho et al., 4 Sep 2025).

Classification uses an affine terminal-time readout

$\gamma_i>0$0

with $\gamma_i>0$1, $\gamma_i>0$2, and softmax on the logits. The supervised loss is the standard cross-entropy

$\gamma_i>0$3

with $\gamma_i>0$4. The digital model is trained by backpropagation through time in PyTorch. The paper explicitly states 66 trainable parameters in total: 16 in the recurrent core implemented in analog hardware and 50 in the readout (Carvalho et al., 4 Sep 2025).

The analog system is implemented on an anabrid Model-1 hybrid analog-digital computer. Each node is realized as a two-integrator cascade implementing

$\gamma_i>0$5

so that, equivalently,

$\gamma_i>0$6

The experimental setup includes a digital control computer, an ADALM2000 signal generator for producing analog waveforms, the Model-1 analog computer implementing the oscillator network, and a TeensyLogger recording inputs and outputs (Carvalho et al., 4 Sep 2025).

Parameter transfer from the digital twin to the analog twin is governed by a scaling factor $\gamma_i>0$7, where the experiment duration is $\gamma_i>0$8, the sample length is $\gamma_i>0$9, the sample duration is $F(\mathbf{x},\dot{\mathbf{x}},t)$0, and the machine integration factor is $F(\mathbf{x},\dot{\mathbf{x}},t)$1. The mapping is

$F(\mathbf{x},\dot{\mathbf{x}},t)$2

The implemented digital model uses homogeneous oscillator parameters

$F(\mathbf{x},\dot{\mathbf{x}},t)$3

Hardware operation is constrained by effective precision $F(\mathbf{x},\dot{\mathbf{x}},t)$4 in machine units and a state range $F(\mathbf{x},\dot{\mathbf{x}},t)$5. To avoid clipping, the input weights $F(\mathbf{x},\dot{\mathbf{x}},t)$6 are rescaled per sample so that the maximum nodal amplitude stays just within this interval (Carvalho et al., 4 Sep 2025).

5. Empirical behavior and benchmark results

On sequential MNIST, the simplified 4-node digital HORN achieves 59.24% test accuracy. The analog system is evaluated on 10,000 test samples by comparing the recorded analog state trajectories against the digital twin with four custom metrics: mismatch at decision time, area difference over the sequence, average phase difference, and temporal correlation. The paper reports that, for the majority of samples, these error metrics are small, indicating high fidelity of the analog dynamics relative to the digital twin. Nevertheless, direct reuse of the digital readout on analog-generated trajectories yields only 28.39% agreement with the predictions of the digital model (Carvalho et al., 4 Sep 2025).

The same study analyzes this discrepancy geometrically. At readout time, the trained digital affine readout partitions the 4D HORN state space into decision regions for the ten digit classes. The paper reports that some classes—specifically 0, 3, 7, and 9—occupy very small volumes in this state space and lie near decision boundaries. Because the analog hardware has effective precision of approximately $F(\mathbf{x},\dot{\mathbf{x}},t)$7 in machine units, while the floating-point digital twin has much higher precision, small state perturbations can cross decision boundaries even when the underlying analog dynamics closely match the digital ones. The authors therefore attribute the low agreement primarily to precision mismatch rather than to failure of the analog oscillator dynamics (Carvalho et al., 4 Sep 2025).

When the analog HORN is instead used as a reservoir and a new linear readout is trained on the analog states, the lost task performance is recovered. With a linear SVM trained on digital HORN states, performance reaches about 74% accuracy. With a linear SVM trained on analog HORN states, the reported performance is ~75.50% vs 73.75% for analog versus digital, respectively. The paper interprets this as evidence that the analog transient dynamics preserve the task-relevant information, even when the original digital readout is too brittle to transfer directly (Carvalho et al., 4 Sep 2025).

The oscillator-attention line of work reports a different empirical profile, focused on irregular time series. On event prediction benchmarks including Synthetic, Neonate, Traffic, MIMIC, StackOverflow, and BookOrder, the oscillator model matches or slightly trails ContiFormer in log-likelihood and accuracy while being much faster. One example given is Traffic log-likelihood: ContiFormer $F(\mathbf{x},\dot{\mathbf{x}},t)$8, OsciFormer $F(\mathbf{x},\dot{\mathbf{x}},t)$9, with accuracy almost equal. On synthetic event sequences, log-likelihood is reported as around $$F(\mathbf{x},\dot{\mathbf{x}}, t) = \alpha\, \tanh\bigl( \mathbf{V}\mathbf{x} + \mathbf{W}\dot{\mathbf{x}} + \mathbf{I}\, s(t)\bigr),$$0 versus $$F(\mathbf{x},\dot{\mathbf{x}}, t) = \alpha\, \tanh\bigl( \mathbf{V}\mathbf{x} + \mathbf{W}\dot{\mathbf{x}} + \mathbf{I}\, s(t)\bigr),$$1, with equal accuracy of approximately $$F(\mathbf{x},\dot{\mathbf{x}}, t) = \alpha\, \tanh\bigl( \mathbf{V}\mathbf{x} + \mathbf{W}\dot{\mathbf{x}} + \mathbf{I}\, s(t)\bigr),$$2. On long-context UCR/UEA multivariate classification, OsciFormer reaches average accuracy 64.5%, equal to the best baseline reported, and on MI reaches 91.8% ± 0.2. On the HR benchmark for irregular heart-rate prediction, the reported RMSE is $$F(\mathbf{x},\dot{\mathbf{x}}, t) = \alpha\, \tanh\bigl( \mathbf{V}\mathbf{x} + \mathbf{W}\dot{\mathbf{x}} + \mathbf{I}\, s(t)\bigr),$$3 for OsciFormer, compared with Transformer $$F(\mathbf{x},\dot{\mathbf{x}}, t) = \alpha\, \tanh\bigl( \mathbf{V}\mathbf{x} + \mathbf{W}\dot{\mathbf{x}} + \mathbf{I}\, s(t)\bigr),$$4, RFormer $$F(\mathbf{x},\dot{\mathbf{x}}, t) = \alpha\, \tanh\bigl( \mathbf{V}\mathbf{x} + \mathbf{W}\dot{\mathbf{x}} + \mathbf{I}\, s(t)\bigr),$$5, and ContiFormer out of memory. On an irregular synthetic binary classification task, OsciFormer achieves equidistant encoding 100%, event-based encoding 99.83% (±0.32), and per-epoch time 0.56 min, whereas ContiFormer reaches event-based 99.93% with per-epoch time 3.83 min (Shende et al., 12 Feb 2026).

These two empirical programs address different questions. The analog HORN study evaluates whether oscillatory recurrent dynamics can survive transfer into physical analog hardware. The oscillator-attention study evaluates whether damped harmonic oscillator parameterizations can replace Neural ODE dynamics in continuous-time attention for irregular sequences. Taken together, they support the view that oscillator-based recurrence is not limited to a single architecture family, but can serve as a reusable continuous-time computational core.

6. Limitations, assumptions, and open directions

The principal modeling limitation is linearity in the continuous-time core. In the oscillator-attention formulation, key and value dynamics are linear damped oscillators, even though the overall network remains nonlinear through projections and feed-forward layers. The paper states that highly nonlinear or chaotic dynamics, such as logistic-map long-horizon forecasting, are hard to capture; adding more modes reduces but does not eliminate this mismatch. The same study also notes that the universal approximation theorem assumes a fixed frequency grid, whereas practical grids are learned or log-uniform but still global; very localized or time-varying frequencies may therefore require many modes. Finally, the method remains quadratic in sequence length because it retains Transformer-style $$F(\mathbf{x},\dot{\mathbf{x}}, t) = \alpha\, \tanh\bigl( \mathbf{V}\mathbf{x} + \mathbf{W}\dot{\mathbf{x}} + \mathbf{I}\, s(t)\bigr),$$6 attention, so the speedups are specifically against NODE-based continuous models rather than against linear-time state-space models or approximate attention schemes (Shende et al., 12 Feb 2026).

The analog HORN implementation exposes a different set of constraints. The proof-of-concept uses only four nodes, homogeneous frequencies and damping, no self-feedback, and no $$F(\mathbf{x},\dot{\mathbf{x}}, t) = \alpha\, \tanh\bigl( \mathbf{V}\mathbf{x} + \mathbf{W}\dot{\mathbf{x}} + \mathbf{I}\, s(t)\bigr),$$7 nonlinearity. The anabrid Model-1 provides limited integrators, adders, and programmable coefficients, which constrains network size and topology. Effective precision is limited to $$F(\mathbf{x},\dot{\mathbf{x}}, t) = \alpha\, \tanh\bigl( \mathbf{V}\mathbf{x} + \mathbf{W}\dot{\mathbf{x}} + \mathbf{I}\, s(t)\bigr),$$8 in machine units and the dynamic range to $$F(\mathbf{x},\dot{\mathbf{x}}, t) = \alpha\, \tanh\bigl( \mathbf{V}\mathbf{x} + \mathbf{W}\dot{\mathbf{x}} + \mathbf{I}\, s(t)\bigr),$$9, making readout transfer brittle. Parameter transfer from software to hardware requires careful scaling and per-sample rescaling of input weights, and all training is performed in simulation rather than in situ on the analog substrate (Carvalho et al., 4 Sep 2025).

Several practical design points recur across the literature summarized here. For HORN-like oscillator banks on irregular time series, the oscillator-attention paper reports that a log-uniform frequency initialization over $\mathbf{V},\mathbf{W}\in\mathbb{R}^{n\times n}$0 in normalized time works well, that optimal initial damping lies in $\mathbf{V},\mathbf{W}\in\mathbb{R}^{n\times n}$1, and that $\mathbf{V},\mathbf{W}\in\mathbb{R}^{n\times n}$2 oscillator modes yields essentially identical predictive performance to larger settings, with accuracy saturating around $\mathbf{V},\mathbf{W}\in\mathbb{R}^{n\times n}$3. It also states that most learned coordinates end up underdamped, where the resonance interpretation is strongest (Shende et al., 12 Feb 2026). In the analog HORN context, the results suggest that precision-aware or noise-aware training, low-precision-compatible readouts, or in situ learning could be important for future hardware realizations (Carvalho et al., 4 Sep 2025).

A second common misconception is that successful analog transfer requires exact prediction agreement with a digital twin. The analog HORN results argue against that interpretation: dynamical agreement can be high while classification agreement is low if the readout is geometrically brittle. Conversely, the reservoir-computing experiment indicates that analog trajectories can preserve sufficient information even when direct reuse of floating-point readout parameters fails. This suggests that future HORN research may need to treat the oscillator core and the decoder as a coupled system whose precision assumptions should be co-designed rather than transferred independently.

In the broader sense implied by both papers, HORN denotes more than a specific four-node recurrent classifier. It denotes a class of continuous-time recurrent systems whose hidden state is a superposition of damped harmonic oscillators, with inputs modulating forcing or initial conditions, and with outputs read either by a terminal linear classifier, by a reservoir readout, or by a continuous-time attention mechanism that aggregates oscillator states through resonance. Under that view, the defining features of HORN are the oscillator state-space backbone, the use of damping and frequency as learnable inductive biases, and the exploitation of transient oscillatory dynamics as the primary computational resource.