Caracal: Causal Architecture via Spectral Mixing

Abstract: The scalability of LLMs to long sequences is hindered by the quadratic cost of attention and the limitations of positional encodings. To address these, we introduce Caracal, a novel architecture that replaces attention with a parameter-efficient, $\mathcal{O}(L \log L)$ Multi-Head Fourier (MHF) module. Our contributions are threefold: (1) We leverage the Fast Fourier Transform (FFT) for sequence mixing, inherently addressing both bottlenecks mentioned above. (2) We apply a frequency-domain causal masking technique that enforces autoregressive capabilities via asymmetric padding and truncation, overcoming a critical barrier for Fourier-based generative models. (3) Unlike efficient models relying on hardware-specific implementations (e.g., Mamba), we uses standard library operators. This ensures robust portability, eliminating common deployment barriers. Evaluations demonstrate that Caracal performs competitively with Transformer and SSM baselines, offering a scalable and simple pathway for efficient long-sequence modeling. Code is available in Appendix.

• The paper introduces a Multi-Head Fourier module that replaces quadratic attention with efficient FFT-based token mixing, reducing complexity from O(L²) to O(L log L).
• The study reports competitive accuracy and throughput on long-context tasks, achieving a threefold speedup over traditional Transformer baselines at L=8192.
• The architecture’s hardware-universal design eliminates explicit positional encodings by enforcing causal masking in the frequency domain and integrating local sliding window attention.

Caracal: Efficient Causal Sequence Modeling via Spectral Mixing

Motivation and Context

The Caracal architecture (2605.00292) addresses critical inefficiencies in large-scale sequence modeling, particularly with respect to context length scalability and computational cost induced by standard attention mechanisms. Transformers suffer from $\mathcal{O}(L^2)$ scaling and dependence on explicit positional encodings—both of which constitute bottlenecks in long-context autoregressive LM training and inference. Recent paradigms have attempted mitigation either through sparse and adaptive attention enhancements, or by substituting attention via SSMs and spectral methods. However, SSMs like Mamba require hardware-specific custom kernels for efficiency, limiting portability and extensibility, whereas prior Fourier-based models are fundamentally non-causal and fail to support generative tasks due to the absence of effective frequency-domain causal masking.

Caracal proposes a paradigm shift by introducing a highly efficient, hardware-agnostic, and causal sequence modeling mechanism that elegantly leverages FFT-based token mixing. It achieves competitive accuracy and throughput with a minimalistic and transparent architecture, significantly reducing complexity without sacrificing autoregressive capability.

Architectural Innovations

Multi-Head Fourier (MHF) Module

Caracal replaces the global Masked Multi-Head Attention of the Transformer backbone with its Multi-Head Fourier (MHF) module, removing explicit positional encodings. The MHF block integrates a shallow 1D causal convolution to instill local inductive bias, followed by gated content and frequency-domain mixing pipelines. The gating stream is generated dynamically from the input and modulates the FFT representation to yield a content-aware token mixing operation. Autoregressive causality in the frequency domain is enforced through asymmetric sequence padding and post-iFFT truncation—the "pad-FFT-multiply-iFFT-truncate" pipeline efficiently realizes a causal convolution equivalent in $\mathcal{O}(L \log L)$.

All mixing operations remain strictly parallel along the sequence dimension, preserving the algorithmic clarity and hardware universality. Empirical and theoretical analysis confirms that the MHF module inherently captures global positional information via its spectral basis, obviating the need for explicit positional encodings.

Figure 1: Comparison of a standard Transformer model (left) and the Caracal model (right), showing the replacement of Masked Multi-Head Attention with the Multi-Head Fourier (MHF) module and removal of positional encodings.

Hybrid Sliding Window Attention

To maximize linguistic consistency and precision, Caracal selectively retains a small proportion of attention layers, restricted to a fixed-size sliding window. These layers provide fine-grained local context refinement, enabling the architecture to combine both global spectral mixing and local pointwise matching. Importantly, positional encodings are omitted even within retained attention layers due to the global context propagation from MHF modules. The hybridization ratio is empirically optimized and found to peak at a 2:1 frequency of MHF to SWA layers.

Theoretical Analysis

Caracal’s MHF module exploits the equivalence between convolution in the time domain and multiplication in the frequency domain. By generating the convolutional kernel dynamically from the input, Caracal achieves data-dependent mixing analogous to attention, but with enforced Toeplitz structure and causal constraints—only $\mathcal{O}(L)$ degrees of freedom per sequence as opposed to $\mathcal{O}(L^2)$ in attention. This shift invariant inductive bias is optimal for modeling global symmetries and periodicities.

The architecture’s causal masking in the frequency domain circumvents the inefficiencies and structural barriers faced by prior FFT-based models—no intermediate weight matrix is needed, and causality is strictly ensured by input sequence manipulation and result truncation.

Experimental Results

Scaling and Effectiveness

Caracal demonstrates competitive accuracy on common sense reasoning, language modeling, and long-context retrieval benchmarks relative to Transformer (Llama), SSMs (Mamba, Mamba2), and hybrid architectures (Jamba). Notably, it achieves robust scaling across parameter counts, maintaining consistent performance improvements with increasing model size.

On information extraction and long-context retrieval (SWDE, FDA), Caracal matches the performance of other sub-quadratic architectures but trails the dense pointwise retrieval of full attention models. The trade-off underscores the efficiency-responsiveness Pareto frontier for global sequence modeling.

Efficiency

Caracal achieves near-linear computational scaling, with $\mathcal{O}(L \log L)$ complexity, and remains performant even at $L=8192$ context lengths. Token throughput and total training time results show Caracal performing comparably or better than linear-time SSM baselines. For example, at maximum tested context, Caracal offers a threefold speedup over attention-based Transformer baselines.

Figure 2: Efficiency comparison across varying context lengths $L \in \{256, \dots, 8192\}$ highlighting near-linear scaling and high throughput at long sequence lengths.

Ablation and Parameter Studies

Ablation studies confirm the importance of local priors (SWA layers and pre-convolution) for fine-grained linguistic performance, while explicit positional encodings provide no significant gain. The hierarchical two-stage gating mechanism in MHF enhances intra-head communication and outperforms simpler alternatives. Parameter sweeps for the MHF/SWA ratio indicate excessive attention density degrades performance, validating the architecture's emphasis on spectral mixing.

Practical and Theoretical Implications

Caracal refines the landscape for efficient sequence modeling by providing an accessible, implementationally simple, and hardware-independent architecture with strong scaling and competitive accuracy. It offers a solution for applications requiring long-context modeling but limited by energy, hardware, or deployment complexity constraints. Theoretical implications include the demonstration of dynamic spectral mixing as a principled replacement for attention, blending data-dependency and shift-invariant bias efficiently. The architectural clarity facilitates extensibility and customization—beneficial for both academic research and production deployment.

Failure modes predominantly relate to fine-grained retrieval, as spectral mixing inherently dilutes exact token match resolution. Ongoing development may integrate hierarchical mixing or multiscale gating to bridge this gap without reverting to quadratic cost.

Conclusion

Caracal establishes a practically viable, theoretically sound, and empirically validated foundation for causal sequence modeling via spectral mixing. The approach mitigates attention’s quadratic complexity, eliminates reliance on positional encodings, and achieves content-aware mixing in an algorithmically elegant, parallel, and hardware-universal manner. It opens new directions for efficient and scalable sequence models suitable for both research and deployment, pending further large-scale exploration and retrieval-focused refinements.