Complex-Valued Phase-Coherent Transformer

Abstract: Complex-valued Transformers have largely inherited softmax attention from real-valued architectures. However, row-normalised token competition is not necessarily aligned with phase-preserving computation. In this paper, we introduce the Phase-Coherent Transformer (PCT), which applies a real-valued, element-independent, smooth gate to L2-normalised complex query-key similarities. PCT replaces token competition with token-non-competing attention and is designed to preserve phase information across layers. Across mid-scale benchmarks spanning long-range memory, hierarchical long-range reasoning, positional retrieval, phase-based memory and superposition, and image classification, PCT shows strong generalisation across task categories. Under parameter-fair comparison, PCT consistently outperforms both the standard softmax Transformer and its direct complex-valued counterpart. Moreover, even on tasks traditionally considered difficult for complex-valued neural networks, such as NIAH and LRA-Text, PCT remains competitive with Multiscreen, the strongest real-valued NN baseline in our comparison. Experiments introducing gates that deliberately violate the PCT conditions show that the design is not incidental: smooth gates that preserve negatively aligned phase components remain strong, whereas gates that delete such components collapse on long-range retrieval, and gates whose outputs become excessively large suffer clear performance degradation. PCT also shows no depth-related accuracy collapse across the tested depth range. These results support introducing multi-layer phase-coherent structure into attention as a promising design principle for achieving generalisation in complex-valued Transformers.

• The paper presents a novel attention mechanism that preserves phase information by replacing softmax with a sigmoid-gated cosine similarity measure.
• It employs complex linear projections with L2 normalization on queries and keys to ensure phase coherence and enhance generalization across tasks.
• Empirical tests demonstrate superior performance on benchmarks like memory, image classification, and algorithmic reasoning while reducing hyperparameter sensitivity.

Complex-Valued Phase-Coherent Transformer: Technical Essay

Motivation and Context

Complex-valued neural networks (CvNNs) have historically found application predominantly in domains with intrinsically complex-valued data (e.g., PolSAR, wireless communication, MRI, audio spectrograms). Real-valued neural networks (RvNNs), supported by a rich ecosystem and empirical results, have long dominated general-purpose domains such as language and vision. The adoption of Transformers in CvNNs has, until now, largely consisted of porting real-valued architectures—retaining the softmax-based, row-normalized attention mechanism. This practice imposes token competition and does not inherently exploit or preserve the phase information carried by complex-valued weights and activations.

Recent developments undermine softmax-attention’s universality. In RvNNs, the Multiscreen attention primitive has shown clear advantages on long-range dependency tasks. In the CvNN regime, the preservation of phase coherence across layers emerges as an architectural desideratum, incompatible with the token-competition enforced by softmax normalization. These observations motivate the search for a fundamentally different attention paradigm in complex-valued Transformers—one that eliminates unnecessary constraints on the phase information propagation.

Phase-Coherent Transformer (PCT): Design and Principles

The central proposition of this paper is the Phase-Coherent Transformer (PCT), in which the standard softmax attention is replaced by a real-valued, smooth, element-independent gating function (e.g., sigmoid) applied to the real part of the cosine similarity between L2-normalized complex queries and keys. Critically, the attention weights in PCT are not row-normalized, eliminating token competition and thereby enabling phase-preserving computations across layers.

Key stages in PCT’s architecture:

• Q, K, V projections: Implemented as native complex linear maps, with L2 normalization imposed on queries and keys.
• Attention score: Computed as the real part of the inner product between L2-normalized queries and keys.
• Gate function: Typically a sigmoid, applied element-wise and bias-initialized to $-\log N$ (with $N$ the sequence length).
• No row normalization: Each token computes its attention independently, removing coupling.

Comparative baselines include both complex- and real-valued Transformers with softmax and various alternative gating mechanisms (e.g., screening attention, hard thresholds) under parameter-matched conditions ($$\mathrm{dim}_\text{real} = 1.41\times \mathrm{dim}_\text{complex}$$ to ensure equal scalar budgets).

Empirical Results

Across an extensive suite of benchmarks—including long-range memory, positional retrieval (NIAH), algorithmic reasoning (LRA-ListOps), image classification (FFT-MNIST), and phase-sensitive memory and superposition (multi-pitch, phase-memory)—PCT exhibits several strong empirical properties:

• Superior Generalization: In all tested tasks, PCT matches or outperforms both vanilla real-valued and complex-valued Transformer baselines, as well as the strongest real-side non-softmax baseline (real_screen).
• Phase-Preserving Computation: PCT and its close variants (e.g., complex_screen) solve even phase-challenging tasks (NIAH, long-range Copy) deterministically, while competitors employing softmax collapse to chance performance.
• Robustness to Hyperparameters: PCT uniquely maintains top performance across wide learning rate and batch size sweeps, in contrast to other designs needing fine-tuned settings.
• Depth Scalability: No evidence of performance degradation up to depth 20; PCT exhibits flat or near-flat accuracy as model depth increases—a significant departure from prior CvNN depth-scaling concerns.
• Task Robustness: On both synthetic and real data benchmarks (RadioML, MusicNet), PCT is consistently competitive, though there remain edge cases (e.g., real RadioML) where a real-side screening mechanism marginally outperforms all complex-side variants.

Theoretical Framework and Isolation Experiments

The core theoretical insight is formalized as a two-level phase coherence requirement for complex-valued attention:

1. Per-layer phase coherence (L1): Each layer must preserve phase relationships through the absence of token competition (elementwise, row-independent gating).
2. Multi-layer/cascade phase coherence (L2): The stack must be jointly stable under phase perturbations, which requires the gate to be both bounded and smooth with a non-vanishing gradient over the operational range (essential for maintaining signal integrity over depth).

These requirements are instantiated in four formal gate conditions:

• C1: Real-valued output.
• C2: Bounded output within the operational range.
• C3: Smoothness and non-zero gradient over the operational range.
• C4: Element independence.

Systematic experiments with counterexample gates (e.g., relu, clamped relu, cubic) demonstrate:

• Anti-phase deletion (C3 violation) leads to catastrophic performance collapse regardless of C2 satisfaction.
• Unbounded gate output (C2 violation) in the operational range degrades, but does not fully destroy, performance (“partial collapse”).
• Row normalization (C4 violation) reproduces failures to propagate phase over depth.

Isolation experiments, conducted in a 2×2 grid over C2 and C3, directly confirm their necessity for reliable long-range generalization. The framework’s results are machine-checked in Lean for the primary theorems.

Implications and Future Directions

Practically, these results imply that complex-valued Transformers using appropriately constrained (smooth, bounded, and non-competitive) attention gates deliver robust, generalizable performance across a spectrum of tasks, rivaling or surpassing state-of-the-art real-valued architectures—even on tasks not tailored to or benefiting from phase information. This expands the potential of CvNNs as general-purpose sequence and reasoning models.

Theoretically, the isolation of phase-coherence as the critical design axis, and the supporting formal (and mechanized) proofs, provide a roadmap for the principled exploration of alternative attention mechanisms in neural sequence models, both complex and real-valued.

While PCT appears to subsume relevant inductive biases for phase-preserving computation, open questions remain—particularly concerning the marginal advantage of screening-style attention on certain physical-complex tasks (e.g., real RadioML). Further exploration of the interaction between phase coherence, attention selectivity, and the statistics of target domains is warranted.

Conclusion

The Phase-Coherent Transformer architecture advances the design of complex-valued sequence models by providing a theoretically justified and empirically validated method for propagating phase information across deep stacks without inducing token competition. The four-condition gate framework isolates the minimal requirements for robust, generalizable attention. PCT consistently achieves or matches top-tier results across a comprehensive set of benchmarks and exhibits depth scalability absent in prior CvNNs. The empirical and mathematical findings establish phase-coherent attention as a principled and practically effective direction for complex-valued transformer development.