Spectral-Window Hybrid: Efficient Sequence Modeling

• Spectral-Window Hybrid (SWH) is a neural architecture that integrates global FFT-based spectral convolution with local sliding-window attention for sequence modeling.
• It employs FFT to capture long-range, decaying dependencies while using chunked attention to achieve high-resolution local interactions.
• SWH scales near-linearly with sequence length, demonstrating improved generalization, lower perplexity, and faster convergence over traditional Transformers.

The Spectral-Window Hybrid (SWH) is a neural architecture for sequence modeling that combines global spectral convolution with local sliding-window attention in parallel to achieve efficient, expressive modeling across extremely long contexts. SWH decouples sequence modeling into two computational streams—one capturing long-range, decaying dependencies via Fast Fourier Transform (FFT)-accelerated convolutions, and another achieving content-sensitive, high-resolution modeling within a fixed local window via chunked attention. This hybrid approach eliminates the $\mathcal{O}(T^2)$ complexity bottleneck of global self-attention in conventional Transformers, enabling near-linear scaling with sequence length $T$ while maintaining strong representational fidelity at both global and local scales (Khasia, 4 Jan 2026).

1. Motivation and Architectural Rationale

Transformers achieve strong results on local and moderate-length retrieval tasks through global self-attention, but their $\mathcal{O}(T^2)$ compute and memory overheads severely constrain long-context applications. Structured State-Space Models (SSMs) offer more efficient scaling, but generally lack the content-adaptive precision of attention-based models. SWH addresses these challenges by executing two operations in parallel: a global spectral branch for long-horizon, pattern-based modeling—using parameterized causal convolution with exponentially damped kernels—and a local windowed attention branch to recover precise, token-level dependencies within bounded contexts.

This separation allows the model to combine continuous, spectral decay with fine-grained, content-aware interactions, each optimized for computational efficiency, and then aggregate them for final prediction. Such decoupling is central to attaining scalability with competitive accuracy across both synthetic and real-world sequence modeling benchmarks (Khasia, 4 Jan 2026).

2. Global Branch: Causal Spectral Convolution via FFT

The global branch in SWH models long-range, decaying dependencies by convolving projections of the input sequence with a family of parameterized damped harmonic oscillator kernels,

$$K_{t,c} = \exp(-|\alpha_c| t)\cos(\omega_c t), \quad t = 0, \ldots, T-1, \; c = 0, \ldots, D-1$$

where $\alpha \in \mathbb{R}^D$ and $\omega \in \mathbb{R}^D$ are learnable per-channel decay and frequency parameters. For an input $X \in \mathbb{R}^{B \times T \times D}$, projected to $U = X W_{\mathrm{conv}}$, the global output is the causal 1D convolution $Y_{\mathrm{spec}} = U * K$.

Naïve convolution here is $\mathcal{O}(T^2 D)$, but by the Convolution Theorem, this operation is efficiently computed in the frequency domain. The sequence and kernel are zero-padded to $2T$, FFT-transformed, multiplied elementwise, inverse transformed, and then cropped:

$$\begin{aligned} \hat U &= \text{Pad}(U, 2T),\quad \hat K = \text{Pad}(K, 2T)\ \tilde U &= \mathcal{F}(\hat U),\quad \tilde K = \mathcal{F}(\hat K)\ \tilde H &= \tilde U \odot \tilde K\ H_{\mathrm{spec}} &= \mathcal{F}^{-1}(\tilde H)\ Y_{\mathrm{spec}} &= \text{Crop}_{0:T-1}(H_{\mathrm{spec}}) \end{aligned}$$

This implementation achieves time complexity $\mathcal{O}(T \log T \cdot D)$ and space $\mathcal{O}(T D)$. The global branch models slow, content-independent decays and trends characteristic of long-range dependencies.

3. Local Branch: Chunked Sliding-Window Attention

The local branch provides high-fidelity modeling of short-range, content-dependent interactions via chunked multi-head sliding-window attention. The input is projected to queries, keys, and values, with rotary positional embeddings applied. The sequence is padded to a multiple of $W$ (window size), split into $N = \lceil T/W \rceil$ non-overlapping chunks, and for each chunk, a two-chunk sliding context is constructed.

For chunk $n$, each query attends to a concatenation of the previous chunk (zero-padded for $n=1$) and the current chunk. Block-causal masks $M^{(n)}$ ensure proper autoregressive conditioning, with softmax masking invalid positions. The per-chunk output is concatenated and linearly projected:

$$S_{i,j}^{(n)} = \frac{Q_{i}^{(n)} (K^{(n)}_{\mathrm{ctx}})_j^T}{\sqrt{D/H}} + M_{i,j}^{(n)}$$

$$O^{(n)} = \mathrm{Softmax}(S^{(n)}) V_{\mathrm{ctx}}^{(n)}, \quad Y_{\mathrm{local}} = \mathrm{Concat}_{n=1}^N (O^{(n)}) W_{\mathrm{local}}$$

Here, each query attends to at most $2W$ positions, yielding time $\mathcal{O}(TWD)$ and space $\mathcal{O}(TW H)$. This branch recovers the content-adaptive retrieval capability of standard attention mechanisms over local neighborhoods.

4. Aggregation and Output Processing

After both branches, the outputs $$Y_{\mathrm{spec}}, Y_{\mathrm{local}} \in \mathbb{R}^{B \times T \times D}$$ are fused through an elementwise summation, with the spectral output normalized via RMSNorm, followed by a final linear projection:

$$Y = (\mathrm{RMSNorm}(Y_{\mathrm{spec}}) + Y_{\mathrm{local}}) W_{\mathrm{out}}, \quad W_{\mathrm{out}} \in \mathbb{R}^{D \times D}$$

This produces per-token embeddings that integrate decaying global context with locally precise dependencies, enabling downstream prediction or classification tasks.

5. Computational Complexity

A summary of asymptotic compute and memory requirements:

Method	Time Complexity	Space Complexity
Global Attention	$\mathcal{O}(T^2 D)$	$\mathcal{O}(T^2)$
Spectral Branch	$\mathcal{O}(T \log T D)$	$\mathcal{O}(T D)$
Window Attention	$\mathcal{O}(T W D)$	$\mathcal{O}(T W H)$
SWH Total	$\mathcal{O}(T D (\log T + W))$	$\mathcal{O}(T D)$

By holding $W$ constant and much smaller than $T$, SWH scales near-linearly, becoming feasible for extreme long-horizon context modeling with nearly constant VRAM usage (Khasia, 4 Jan 2026).

6. Empirical Evaluation

Synthetic Sequence Tasks

On associativity, induction, sorting, length generalization, and needle-in-a-haystack tasks, SWH matches Transformer baselines for in-distribution regimes and surpasses them on length extrapolation, due to the continuous parameterization of its spectral kernel. Example accuracies for D=128, L=2, H=4, and 3,000 training steps:

Method	Associative	Induction	Sorting	LenGen×4	Needle
Standard Transformer	0.80	0.81	0.97	0.02	0.00
SWH	0.86	0.81	0.98	0.05	0.05

SWH demonstrates improved length generalization due to its spectral convolution (Khasia, 4 Jan 2026).

Language Modeling

On FineWeb-Edu (125M parameters, 12 layers, D=768, H=12, block size 1024), SWH achieves consistently lower perplexity than Transformers over 16,000 validation steps, indicating accelerated convergence and improved modeling of real-world texts.

Computational Efficiency

Inference latency and VRAM benchmarks (T = 512–4096) demonstrate that SWH scales as $\mathcal{O}(T)$, realizing $\sim$60% inference speedup at T=4096 versus Transformers, with nearly flat VRAM usage.

7. Practical Implementation and Applications

Key implementation details: 12 layers, $D=768$, $H=12$, window size $W=256$, block size $T=1024$, AdamW optimizer with $\text{lr}=6 \times 10^{-4}$, batch size 64, 8× gradient accumulation, mixed BF16/FP16 (FFT in FP32), FFT padding to $2T$, and bias terms disabled except for spectral input projections. Code is publicly available at https://github.com/VladimerKhasia/SWH.

Applications

• Long-document language modeling (e.g., books, code, legal documents)
• Time-series forecasting featuring slow decay and short-term spikes
• Speech and music modeling requiring co-existing global and local dynamics

These use cases leverage SWH’s ability to model both broad, slow-decay dynamics and sharp local transitions in a computationally tractable fashion.

8. Strengths, Limitations, and Future Directions

Strengths:

• Eliminates the global $\mathcal{O}(T^2)$ attention bottleneck, achieving near-linear scaling in $T$.
• Retains high local retrieval fidelity via sliding-window attention.
• Demonstrates strong length extrapolation, real-world perplexity gains, and accelerated training convergence on realistic language tasks.

Limitations:

• The global spectral branch applies a fixed-form damped-oscillator kernel; consequently, it may underfit global, content-specific interaction structures.
• Zero-padding to length $2T$ for FFT increases per-batch compute and memory usage.
• The local window size $W$ imposes a trade-off between local recall and per-step computational cost.

A plausible implication is that extending the global kernel to capture richer, input-adaptive global patterns could further enhance expressivity, while advanced padding or streaming FFT methods might reduce overhead.

SWH constitutes a principled hybrid architecture that leverages FFT-based spectral convolution and sliding-window attention to enable efficient, scalable sequence modeling across diverse domains requiring both global and local context awareness (Khasia, 4 Jan 2026).