CAWN: Continuous Acoustic Wave Network

• Continuous Acoustic Wave Network is an autoregressive model that substitutes self-attention with complex-valued wave processes to mix sequence information.
• It employs a Resonance Layer that projects tokens into multi-headed complex phasors, using causal phase accumulation and frequency-dependent retention for long-context signal preservation.
• The architecture achieves linear time complexity with a constant-size phase state, enabling efficient processing of millions of tokens without quadratic memory growth.

Continuous Acoustic Wave Network (CAWN) is an autoregressive language-model architecture that replaces Transformer self-attention with a continuous complex-valued “wave” process. Rather than forming an $L \times L$ attention matrix, CAWN maps hidden states into multi-headed complex phasors, mixes the sequence through a causal $\mathcal{O}(L)$ phase-accumulation recurrence, and uses selective phase resonance with frequency-dependent retention to preserve useful signals over extremely long contexts. In its reported 150M-parameter instantiation, trained through a continuous streaming loop on a 100-Billion-token corpus and evaluated at a 5-Billion-token milestone, the model retrieves targeted information across 2,000,000 tokens while plateauing at 8.72 GB of Peak VRAM under chunked prefill (Čugalj et al., 5 Apr 2026).

1. Architectural definition and motivation

CAWN was introduced to address two limitations identified in prevailing sequence models. The first is the $\mathcal{O}(L^2)$ cost of self-attention, where key–value caches and pairwise token–token interactions scale unfavorably with context length. The second is signal degradation in linear-time alternatives such as State Space Models, which compress the past into a decaying continuous state and may wash out long-range information. CAWN’s response is a fully continuous sequence-mixing mechanism in $\mathbb{C}$, in which context is represented as overlapping multi-frequency waves whose constructive and destructive interference carries grammar and semantics (Čugalj et al., 5 Apr 2026).

The architecture is organized around a Resonance Layer. In the reported CAWN-150M configuration, each of the $N=16$ layers processes $X \in \mathbb{R}^{L \times D}$ with $D=896$ through two parallel paths. The acoustic path consists of block attention residual routing, RMSNorm, a Temporal Syntax Cache, projection into complex phasors, causal phase accumulation in $\mathbb{C}$, and a Depth-wise Harmonic Convolution plus SwiGLU “Ear,” followed by a residual add. In parallel, an FFN path applies block attention residual routing, RMSNorm, a standard MLP / GELU FFN, and another residual add. Layers are grouped into blocks of four, with Block Attention Residuals providing depth-wise state routing while preserving strictly linear time-axis computation (Čugalj et al., 5 Apr 2026).

This formulation makes CAWN neither an attention approximation nor a conventional recurrent kernel. The past is encoded as a set of complex harmonic oscillators whose phases rotate deterministically and whose amplitudes are modulated by tokens. A plausible implication is that the model’s memory is organized spectrally rather than through explicit token-pair routing.

2. Resonance Layer and complex-domain sequence mixing

Within a Resonance Layer, the block-routed hidden state at time $t$, after RMSNorm and temporal convolution, is denoted $x_t$. For each token, head, and harmonic, CAWN projects four quantities: a positive amplitude, a base phase, a head-level input gate, and a head-plus-frequency retention gate. In the 150M model, the architecture uses $\mathcal{O}(L)$0 Acoustic Heads and $\mathcal{O}(L)$1 fixed harmonics, yielding $\mathcal{O}(L)$2 complex channels (Čugalj et al., 5 Apr 2026).

The projections are

$\mathcal{O}(L)$3

$\mathcal{O}(L)$4

$\mathcal{O}(L)$5

$\mathcal{O}(L)$6

Each token contributes a complex phasor $\mathcal{O}(L)$7, scaled by $\mathcal{O}(L)$8. The actual recurrence is implemented through real and imaginary pushes,

$\mathcal{O}(L)$9

$\mathcal{O}(L^2)$0

followed by a causal phase-accumulation update over flattened channel index $\mathcal{O}(L^2)$1. With fixed rotation angle

$\mathcal{O}(L^2)$2

the complex state satisfies

$\mathcal{O}(L^2)$3

with boundary condition $\mathcal{O}(L^2)$4 (Čugalj et al., 5 Apr 2026).

This recurrence is causal and linear in sequence length. A token injected at step $\mathcal{O}(L^2)$5 contributes a component whose phase at step $\mathcal{O}(L^2)$6 is shifted by $\mathcal{O}(L^2)$7, so relative distance is encoded by deterministic rotation rather than explicit attention weights. After accumulation, the real and imaginary parts are concatenated,

$\mathcal{O}(L^2)$8

Although the state is conceptually complex, implementation is explicit real-imaginary arithmetic. The phase accumulator runs inside a fused Triton kernel in float32, with phase-state clamping

$\mathcal{O}(L^2)$9

and gradients through the accumulator are also clamped to $\mathbb{C}$0 to avoid NaNs. The authors describe this as “true-complex phase accumulation,” because the model maintains a continuous complex state with explicit rotations rather than approximate kernel tricks or DFT matrices (Čugalj et al., 5 Apr 2026).

3. Selective Phase Resonance, retention, and local syntax separation

In CAWN, “resonance” denotes the sum of overlapping waves from all past tokens. Alignment between a new token’s projected phase and components already present in $\mathbb{C}$1 yields constructive interference; misalignment yields destructive interference. The architecture makes this resonance selective through two gates. The input gate $\mathbb{C}$2 controls whether token $\mathbb{C}$3 injects energy into head $\mathbb{C}$4, and the retention gate $\mathbb{C}$5 controls how much prior state survives at each harmonic frequency (Čugalj et al., 5 Apr 2026).

Frequency-Dependent Retention is implemented by the static bias schedule $\mathbb{C}$6, which decays from $\mathbb{C}$7 for low-frequency channels to $\mathbb{C}$8 for high-frequency channels, together with an initial projection bias of $\mathbb{C}$9 before the sigmoid. This engineering choice yields low-frequency channels with $N=16$0, described as near-perfect global memory, and high-frequency channels with $N=16$1, described as short-term “scratchpads.” The memory system is therefore explicitly multi-timescale: low frequencies accumulate information across very long durations, while high frequencies decay more rapidly and capture local patterns (Čugalj et al., 5 Apr 2026).

The input gate is made aggressively sparse through Hard-Threshold Gating via Straight-Through Estimation. The forward pass applies a hard threshold

$N=16$2

with $N=16$3, linearly annealed from $N=16$4 to $N=16$5 during the first 5% of training. The final computation uses the classic STE form

$N=16$6

and the pre-sigmoid input is shifted by $N=16$7, biasing $N=16$8 toward near zero unless the model strongly opens the gate. The stated purpose is to protect long-term phase memory from noise, especially when training data include injected garbage-token segments (Čugalj et al., 5 Apr 2026).

CAWN reserves the global phase state primarily for global semantics and long-range context, while local syntax is handled by the Temporal Syntax Cache. This module applies a causal depth-wise Conv1D over time with kernel size $N=16$9, asymmetric padding of two steps to the left, elementwise clamp to $X \in \mathbb{R}^{L \times D}$0, and SiLU activation. The result $X \in \mathbb{R}^{L \times D}$1 then feeds the head and harmonic projections. The separation between Temporal Syntax Cache and phase accumulation effectively decouples local grammar from global semantic resonance (Čugalj et al., 5 Apr 2026).

4. Harmonic post-processing and depth-wise routing

After phase accumulation, the state $X \in \mathbb{R}^{L \times D}$2 is reshaped so that the $X \in \mathbb{R}^{L \times D}$3 harmonics form a 1D spatial dimension and the $X \in \mathbb{R}^{L \times D}$4 real-plus-imaginary components act as channels. CAWN then applies a Depth-wise Harmonic Convolution, also described as the “Cochlea,” in which each channel has its own 1D filter across neighboring harmonics. This stage allows adjacent frequencies to interact, suppresses correlated noise and destructive interference across frequency bands, and prepares the representation for nonlinear projection back into hidden space (Čugalj et al., 5 Apr 2026).

The harmonic convolution is followed by the SwiGLU “Ear”: $X \in \mathbb{R}^{L \times D}$5

$X \in \mathbb{R}^{L \times D}$6

The resulting vector in $X \in \mathbb{R}^{L \times D}$7 is added to the local residual stream. In effect, the acoustic path first mixes context continuously in complex harmonic space, then performs frequency-domain filtering, then re-enters the standard hidden-state domain (Čugalj et al., 5 Apr 2026).

Depth-wise routing across layers is handled by Block Attention Residuals. CAWN groups layers into blocks of four, maintains an “active partial stream” within each block, archives the stream at block boundaries into a list of block states $X \in \mathbb{R}^{L \times D}$8, and resets the active stream to zero. For routing, each archived state is normalized by RMSNorm to form keys, a single learned pseudo-query vector $X \in \mathbb{R}^{L \times D}$9 produces scalar logits per block and time step, and a softmax over the depth dimension yields a weighted sum of previous block states at the same time index. Because this attention runs only over depth, not over sequence positions, its per-token cost is $D=896$0 in the number of blocks and does not alter CAWN’s $D=896$1 time-axis complexity (Čugalj et al., 5 Apr 2026).

The residual-severing design addresses a specific deep-network pathology identified in the paper: early information can be diluted when residual streams are repeatedly summed layer after layer. The archived block states provide what the paper describes as access to “pristine” earlier states without requiring time-axis attention.

5. Complexity, memory profile, and long-context behavior

CAWN’s sequence-length complexity is linear. Per token, the architecture performs head-and-harmonic projections, a fixed number of trigonometric operations and multiplications per harmonic channel, depth-wise harmonic convolution, and small MLPs, but never constructs an $D=896$2 matrix. The paper summarizes this as compute $D=896$3 and forward-memory $D=896$4 for activations, plus a fixed-size phase state in streaming or inference settings (Čugalj et al., 5 Apr 2026).

A central claim of the architecture is $D=896$5 state passing in context length during autoregressive generation. To generate token $D=896$6, the model needs only the previous phase state $D=896$7, the last two local states for the Temporal Syntax Cache, and the model weights. There is no growing key–value cache. For extremely long prefills, the paper uses chunked prefill: split the input into chunks of length 32,768, run chunk 1 and keep only its final phase state, initialize chunk 2 with that phase state, and continue. Under this procedure, PyTorch sees a fixed context length per forward pass while only a constant-size phase state is carried across calls (Čugalj et al., 5 Apr 2026).

Empirically, the reported memory behavior is specific. On H100 (80GB), peak VRAM grows linearly up to about 32k tokens and then flatlines at 8.72 GB when chunked prefill is engaged. Contexts up to 2 million tokens are processed with VRAM remaining at 8.72 GB. On an 8GB GPU, the paper reports 2.82 GB for a Transformer versus 2.26 GB for CAWN at 1,024 tokens, 3.89 GB versus 2.64 GB at 2,048 tokens, Transformer OOM at 4,096 tokens, and CAWN at 4.91 GB for 8,192 tokens. On H100, CAWN handles 37,831 tokens in an unchunked pass at 7.34 GB, after which chunked prefill takes over (Čugalj et al., 5 Apr 2026).

The long-context retrieval demonstration uses a Targeted Semantic Retrieval protocol: three semantic targets are inserted early in a stream, followed by high-entropy garbage tokens, and the model is later queried autoregressively for those targets.

Context length	Retrieval outcome	Peak VRAM
Standard context $D=896$8	All three tokens retrieved	$D=896$9 GB
19k, 37.8k, 100k, 1M	100% retrieval for all three	Up to 8.72 GB
2M	“Red” and “Blue” correct; “Green” fails (“I”)	8.72 GB

Up to 1,000,000 tokens distance, the paper reports 100% retrieval accuracy for all three targets. At 2,000,000 tokens, “Green” fails, which the authors hypothesize may be due to destructive interference for its spectral bands or cumulative numerical error. This suggests that the architecture’s long-context behavior is strong but not numerically trivial at extreme horizons (Čugalj et al., 5 Apr 2026).

6. Training protocol, contextual denoising, and reported performance

The reported prototype is CAWN-150M with $\mathbb{C}$0, $\mathbb{C}$1 Resonance Layers in 4 blocks, $\mathbb{C}$2 heads, $\mathbb{C}$3 harmonics, $\mathbb{C}$4 complex channels, and FFN expansion factor $\mathbb{C}$5. Embeddings and output LM head are weight-tied. Dropout is $\mathbb{C}$6 on wave projections, and layer outputs are scaled with a depth-aware factor $\mathbb{C}$7. The training corpus is an English blend of approximately 100B tokens: 50% FineWeb-Edu PDFs, 30% DCLM, and 20% standard FineWeb-Edu, tokenized with the LLaMA-2 BPE tokenizer at vocabulary size 32k. Training uses infinite streaming through a PyTorch IterableDataset, AdamW, bfloat16 mixed precision for most of the network, float32 for the phase accumulator, native sequence length $\mathbb{C}$8, and gradient accumulation $\mathbb{C}$9 steps times micro-batch size $t$0, for effective batch size $t$1 on 24GB RTX 3090. The complex phase state $t$2 is persisted across micro-batches even though the gradient graph spans only $t$3 (Čugalj et al., 5 Apr 2026).

The paper also specifies explicit stability guards. If cross-entropy loss is NaN or $t$4, the step is skipped. If the global gradient norm exceeds $t$5, gradients are zeroed and the scheduler advances to protect phase state. This training regime is paired with deliberate data augmentation: large segments of garbage tokens are inserted into the stream and followed by meaningful queries, while training still uses autoregressive cross-entropy on all tokens. The stated effect is that the model learns to close $t$6 gates on noise and preserve the phase accumulator, yielding explicitly learned associative recall within noise rather than zero-shot generalization alone (Čugalj et al., 5 Apr 2026).

On WikiText-103 validation, CAWN perplexity decreases monotonically with steps: approximately $t$7 at $t$8k steps, approximately $t$9 at $x_t$0k steps, and approximately $x_t$1 at $x_t$2k steps, reported as approximately $x_t$3B tokens. The paper compares this to Pythia-160M at step $x_t$4 and $x_t$5B tokens with perplexity $x_t$6, stating that CAWN matches that parity around $x_t$7k steps and later improves to approximately $x_t$8. Fully saturated baselines trained on much larger token counts remain lower in absolute perplexity: GPT-2 Small at approximately $x_t$9 and SmolLM at approximately $\mathcal{O}(L)$00 (Čugalj et al., 5 Apr 2026).

Zero-shot reasoning is evaluated with the EleutherAI LM Evaluation Harness. At approximately $\mathcal{O}(L)$01B tokens and $\mathcal{O}(L)$02k steps, CAWN-150M reports PIQA $\mathcal{O}(L)$03 accuracy and ARC-Easy $\mathcal{O}(L)$04. The paper compares these numbers with Pythia-160M at $\mathcal{O}(L)$05B tokens, PIQA $\mathcal{O}(L)$06 and ARC-Easy $\mathcal{O}(L)$07, and SmolLM-135M trained on 600B tokens, PIQA $\mathcal{O}(L)$08 and ARC-Easy $\mathcal{O}(L)$09. The reported interpretation is that CAWN shows emergent reasoning beyond random guess and outperforms a parameter-matched early-stage Transformer baseline, while remaining below heavily trained Transformer baselines in absolute terms (Čugalj et al., 5 Apr 2026).

7. Relation to adjacent architectures, terminological scope, and limitations

Relative to self-attention, CAWN does not compute pairwise similarities $\mathcal{O}(L)$10 or aggregate value vectors through an attention matrix. Instead, it carries context as a superposition of harmonic components in a global complex state whose phases rotate deterministically. Relative to State Space Models and linear RNN-style models such as Mamba, RWKV, Griffin, and Hyena, the distinguishing feature is explicit complex phase rotation with frequency-dependent retention: low-frequency channels act as long-memory components and high-frequency channels as short-term scratchpads. The paper also locates CAWN closer in spirit to RoPE or Fourier features than to standard recurrent kernels, but dynamic and recurrent rather than static embedding (Čugalj et al., 5 Apr 2026).

The term “Continuous Acoustic Wave Network” is not unique to language modeling. Separate literatures describe physical acoustic-wave networks and related constructs: symmetry-broken cross-shape metamaterial networks for quasi-lossless routing along arbitrary pathways (Dai et al., 2016), resonant 3-port and 4-port acoustic networks with phase-controlled splitting, combining, and routing (Richoux et al., 2019), the continuous boostlet transform as a representation system for acoustic waves in $\mathcal{O}(L)$11D space–time organized by the Poincaré group and isotropic dilations (Zea et al., 2024), and acoustic neural networks implemented through passive waveguide-based computation with constrained recurrent and hierarchical architectures (Kalthoff et al., 26 Nov 2025). In that broader context, the “acoustic” in CAWN for language modeling denotes a wave-based complex-state metaphor and mechanism rather than a physical metamaterial device.

The reported limitations are explicit. Experiments are currently at 150M parameters and around 5B processed tokens, with planned but untested scaling to $\mathcal{O}(L)$12–$\mathcal{O}(L)$13B parameters and $\mathcal{O}(L)$14T tokens. The model depends on custom Triton code and autograd for complex kernels, which introduces engineering overhead and possible portability issues. On a consumer GPU and up to 16k context, FlashAttention-based Llama or Pythia baselines achieve approximately $\mathcal{O}(L)$15–$\mathcal{O}(L)$16 tokens/s, whereas CAWN reports approximately $\mathcal{O}(L)$17 tokens/s, although the paper notes that Transformer throughput degrades as key–value caches grow while CAWN throughput is flat in context length. Extremely long horizons still show occasional failures, as in the 2M-token “Green” retrieval miss. The architecture is currently text-only, and the contribution of the noise-injection strategy to gating behavior has not been isolated through ablations (Čugalj et al., 5 Apr 2026).

These constraints place CAWN in a specific research position: a fully specified linear-time language-model architecture with explicit complex-domain sequence mixing, strong empirical evidence for long-context state compression, and unresolved questions around scaling, systems integration, and numerical behavior at extreme horizons.