Phase-Associative Memory: Sequence Modeling in Complex Hilbert Space

Abstract: We present Phase-Associative Memory (PAM), a recurrent sequence model in which all representations are complex-valued, associations accumulate in a matrix state $S_{t}$ $\in$ $\mathbb{C}^{d \times d}$ via outer products, and retrieval operates through the conjugate inner product $K_t^* \cdot Q_t / \sqrt{d}$. At $\sim$100M parameters on WikiText-103, PAM reaches validation perplexity 30.0, within $\sim$10\% of a matched transformer (27.1) trained under identical conditions, despite $4\times$ arithmetic overhead from complex computation and no custom kernels. We trace the experimental path from vector-state models, where holographic binding fails due to the $O(1/\sqrt{n})$ capacity degradation of superposed associations, to the matrix state that resolves it. The competitiveness of an architecture whose native operations are complex-valued superposition and conjugate retrieval is consistent with recent empirical evidence that semantic interpretation in both humans and LLMs exhibits non-classical contextuality, and we discuss what this implies for the choice of computational formalism in language modeling.

• The paper introduces a novel recurrent sequence model that encodes all representations in a complex Hilbert space for effective associative memory using quantum logic principles.
• It leverages interleaved ComplexGatedUnit and PAM layers to preserve both phase and magnitude, enabling precise retrieval via conjugate inner products.
• Empirical results on WikiText-103 show competitive perplexity compared to transformers, despite higher arithmetic overhead, highlighting key trade-offs in performance.

Phase-Associative Memory: Complex Hilbert Space Sequence Modeling

Introduction and Motivation

The paper introduces the Phase-Associative Memory (PAM) architecture, a recurrent sequence model that places all core representations—embeddings, activations, associations—in complex Hilbert space, using complex-valued tensors throughout the model. Central to PAM is the accumulation of associative structure in a complex-valued matrix state, with retrieval via conjugate inner products, aligning its operational semantics with formal quantum logic. The motivation arises from increasing empirical and theoretical evidence that semantic interpretation, both in human cognition and in LLMs, exhibits non-classical contextuality. Prior work (e.g., quantification of contextuality violations in language-based CHSH tests) suggests that real-valued vector spaces and classical attention mechanisms may merely approximate a richer, intrinsically non-Boolean structure naturally expressed in complex space.

The paper situates PAM within both the historical and technical landscape: from quantum mechanics’ challenge to classical separability and contextuality, through the persistent limitations of LLMs (such as intractable hallucination and the failure of mechanistic interpretability), to the classical result that associative memory capacity in real-valued vector-space superposition suffers $O(1/\sqrt{n})$ degradation. The choice to encode sequence structure in a complex matrix state and to use the conjugate inner product for retrieval is justified not by analogy to quantum computation, but by the formal equivalence between quantum measurement in contextual systems and the algebraic operations of PAM.

Architectural Design and Complex-Valued Computation

The PAM model architecture consists of complex-valued embeddings, repeated blocks each interleaving a ComplexGatedUnit (CGU) for channel mixing with a Phase-Associative Memory (PAM) layer for sequence mixing, and a tied complex-valued output head. Every operation in the core computation path is complex-valued and intrinsically phase-preserving. Representation of complex numbers uses split-real tensors of shape $[\ldots, d, 2]$, supporting complex linear maps, RMS normalization of magnitudes, and the modReLU activation, which thresholds magnitudes without corrupting phase.

The CGU operates as a SwiGLU-style gating mechanism within the complex domain, using both magnitude and phase for parameterized gating and rotation. Each block applies CGU and PAM with residual connections, scaling the updates with learned coefficients. Logits for output are calculated as the complex inner product between the final activation and the tied embedding matrix.

The PAM module itself replaces both attention and recurrent backbones. Each PAM head maintains a complex matrix state $S_t^{(h)} \in \mathbb{C}^{d \times d}$, updated by accumulating the outer product of the value and the conjugate key, modulated by learned, input-conditioned decay and state-protect gates. Retrieval is executed as the matrix-vector product $S_t \tilde{Q}_t$, where $\tilde{Q}_t$ is the scaled query. The core retrieval function performs destructive and constructive interference via the conjugate inner product: phase alignment between stored keys and queries is essential for effective retrieval, a property unobtainable in real-valued systems with only non-negative inner products.

Empirical Results

PAM was trained with approximately 100 million parameters on WikiText-103, using a comparable training regime to a matched standard transformer baseline (of similar parameter count and hyperparameters). On validation metrics, PAM achieves 30.0 perplexity, compared to the transformer’s 27.1, despite incurring a $4\times$ overhead in arithmetic operations from complex-valued computation and leveraging a first-generation unoptimized PyTorch implementation with no custom CUDA kernels. Per-token throughput is correspondingly lower (23k vs. 96k tokens/sec on a single A100), but only due to computational overhead, not algorithmic inefficiency.

Figure 1: Validation perplexity trajectories for interleaved PAM, sequential PAM, and transformer baselines, demonstrating that interleaving channel and sequence mixing yields improved convergence.

Several ablations and architectural variations were evaluated:

• When both query and key vectors were normalized to unit magnitude (thus removing the magnitude degree of freedom and retaining only phase), text generation quickly devolved into catastrophic repetition, establishing that both magnitude and phase dynamics are required for effective conjugate retrieval.
• Supplementing PAM with sparse local attention did not improve performance at this scale, supporting the hypothesis that the complex-valued recurrence in PAM already captures the majority of associations that attention would.
• Sequential block arrangements (16 CGU followed by 16 PAM layers) and RoPE removal both degraded validation perplexity, confirming the structural necessity of the interleaved pattern for PAM.

Critically, the fixed-size matrix state in PAM ensures $O(1)$ per-token inference cost (ignoring the quadratic factor in the relatively small $d$), in contrast to the transformer’s KV-cache which grows linearly in context length. At sequence length 2048, the transformer’s per-layer cache is $\sim56\times$ larger in state than PAM’s.

Theoretical and Practical Implications

The design of PAM bridges findings in quantum logic—specifically, that any system with contextual observables is appropriately modeled in a complex Hilbert space—with the operational demands of modern LLMs. The paper posits that transformers, and to some extent any classical architecture, must consume additional parameters and computation to embed the non-classical, phase-dependent correlations of natural language into real-valued vector spaces.

Notably, the conjugate inner product in PAM is not only computationally attractive but theoretically justified: it allows encoding, retrieval, and interference directly over the association matrix, exploiting both magnitude and phase to enhance selectivity and suppress crosstalk from superposed associations. This is a significant address to limitations of linear attention, where inability to suppress non-matching keys impairs associative recall.

The paper further connects to mechanistic interpretability failures in both deep neural networks and neuroscience, arguing that these may reflect inherent limitations of decompositional paradigms when underlying structures are non-separable and contextual by construction. If true, architectures like PAM may provide representations that are not only more efficient, but also naturally aligned with the true causal and semantic structure of language.

Comparative Analysis

Empirical comparison with a matched transformer confirms that PAM’s perplexity on WikiText-103 is within $\sim$10% of the baseline, a gap smaller than the computational expense ratio, and striking given the novelty and lack of hardware optimization for complex-valued networks. For further context, the 124M-parameter GPT-2 baseline historically achieves a similar perplexity (∼31), despite different data splits and evaluation protocols.

Interpretation of the convergence curves and epoch-by-epoch performance demonstrates that PAM’s more expensive associative mechanism converges nearly as quickly and narrows its gap to the transformer with further training, suggesting that the model makes efficient use of its parameters.

Future Directions

Several pertinent open questions emerge:

• Will the perplexity gap to transformers shrink, remain stable, or widen at higher parameter and data scales or under hardware-specific complex optimizations?
• Does the phase structure in PAM’s representations yield information-theoretic or interpretational advantages—e.g., more robust context-dependence, resistance to adversarial prompting, or enhanced compositionality?
• How do PAM’s representations differ from attention- or SSM-based models with respect to formal quantum contextuality, as measured by operational tests like CHSH violations in semantic interpretation?

Further work, including direct tests of associative retrieval at long context with PAM’s matrix memory, mechanistic probes of contextual dependence, and hardware optimizations, will likely clarify its competitive profile and theoretical salience.

Conclusion

Phase-Associative Memory offers a sequence model in which the entire stack—embedding, association, retrieval—lives natively in complex Hilbert space, aligned with formal quantum frameworks for contextual and semantic interpretation. Empirically, PAM obtains validation perplexity close to a conventional transformer baseline, despite significantly higher arithmetic complexity and non-optimized implementation. The results support several strong claims:

• Matrix-state recurrence in complex space resolves superposition capacity bottlenecks of vector-state models.
• Both phase and magnitude are essential for capacity and selectivity; pure-phase or magnitude normalization induces catastrophic collapse.
• Supplementation with standard attention provides no benefits at this scale—the model’s recurrence subsumes it.
• The architecture provides a test-bed for operationalizing quantum-contextual formalisms in practical language modeling.

Whether such architectures will find their place at frontier scale, and whether they enable more interpretable or robust models, will depend both on ongoing empirical scaling and deeper theoretical work at the formal boundary between computation, context, and meaning.