---
title: Complex-Valued Vision Transformer (kViT)
url: https://www.emergentmind.com/topics/complex-valued-vision-transformer-kvit
type: topic
---

# Complex-Valued Vision Transformer (kViT)

A Complex-Valued Vision Transformer (kViT) is an architecture designed to perform end-to-end learning directly on complex-valued data, such as MRI k-space, utilizing all data components and preserving phase information throughout the model pipeline. Unlike standard Vision Transformers (ViT) that operate on real-valued image data post-magnitude reconstruction, kViT models process raw frequency-domain (k-space) samples and natively employ complex-valued layers, attention, and normalization, thereby aligning the model structure with the physical properties of the input modality [2306.09827], [2601.18392].

## 1. Core Principles and Motivation

The motivation for kViT arises from the inherent complex-valued nature of many scientific imaging modalities, notably MRI, where k-space data contains both magnitude and phase crucial to accurate modeling and diagnosis. Traditional pipelines discard phase information and rely on local operations that are incongruent with the global, non-local structure of k-space. kViT architectures reframe the problem by operating directly on complex-valued signals without projecting into $\mathbb{R}^2$, thus preserving spectral integrity and enabling direct-from-scanner AI analysis [2601.18392].

## 2. Complex-Valued k-Space Patch Embedding

In kViT, the initial input is a complex-valued k-space array $X \in \mathbb{C}^{H \times W}$. Rather than extracting local grid patches, the kViT leverages a physics-informed radial patching strategy:

- Each k-space sample $X(u_i, v_i)$ is mapped to polar coordinates $(r_i, \theta_i)$.
- The full $H\times W$ grid is sorted by $r_i$ and partitioned into $N$ concentric “rings” (radial patches) of equal cardinality, each containing $P$ samples (where $N = HW/P$).
- Each patch $P_n$ comprises all $X(u, v)$ such that $R_{(n-1)P+1} \leq \sqrt{u^2+v^2} < R_{nP}$.
- Every radial patch (complex vector of size $P$) is flattened and embedded using a complex weight matrix $W \in \mathbb{C}^{d \times P}$ and bias $b \in \mathbb{C}^d$: 
  \[
  \mathbf e_n = W x_n + b = (W_r x_n^{(r)} - W_i x_n^{(i)} + b_r) + i (W_r x_n^{(i)} + W_i x_n^{(r)} + b_i)
  \]
This radial scheme better reflects the spectral locality and energy distribution of MRI acquisition compared to Cartesian patches, and is critical for maintaining robustness under high acceleration (undersampling) [2601.18392].

## 3. Complex-Valued Transformer Building Blocks

The kViT replaces all core components of the standard ViT with their complex-valued analogues as defined in [2306.09827]:

- **Complex Linear Layers:** Given $x \in \mathbb{C}^n$, $y = W x + b$ with $W$ and $b$ complex. Real and imaginary parts are parameterized separately for implementation. Initialization uses the complex Glorot (Xavier) scheme by sampling magnitude $|W_{jk}|$ from Rayleigh$(\sigma)$ with $\sigma^2 = 2/(fan_{in} + fan_{out})$.
- **Complex Attention:** Multi-head self-attention operates with complex queries, keys, and values ($Q, K, V \in \mathbb{C}^{N \times d}$). The standard variant (“CAtt”) computes attention as:
  \[
  CAtt(Q, K, V) = \mathrm{softmax}(\Re(Q K^H) / \sqrt{d})\, V
  \]
  Here, $K^H$ denotes Hermitian (conjugate transpose) and softmax is computed over the real part. Several alternative attention mechanisms—absolute, phase-preserving, and real/imaginary split—are possible.
- **Complex Layer Normalization:** Each complex vector $z \in \mathbb{C}^d$ is treated as a $2\times d$ real matrix (stacking real and imaginary parts), estimated with mean $\mu$, $2\times2$ covariance covariances, and whitened accordingly. A learnable affine transform with positive-definite scaling is applied.
- **Complex Feed-Forward Networks:** Use two stacked complex linear layers separated by complex-valued nonlinearities (e.g., modReLU, zReLU, or split ReLU on real/imag components).
- **Positional Encoding:** Patch positions are injected via either (a) learnable complex positional embeddings $\phi_n \in \mathbb{C}^d$ or (b) rotary position encoding (RoPE) adapted to the complex domain. The RoPE rotates each $(2i,2i+1)$ pair by $\theta_n = n \omega_i$:
  \[
  \begin{pmatrix}
    \cos(n\omega_i) & -\sin(n\omega_i) \\
    \sin(n\omega_i) &  \cos(n\omega_i)
  \end{pmatrix}
  \]

## 4. Model Architecture and Training Details

The typical architecture follows a transformer encoder paradigm, stacking $L$ complex transformer layers, each containing:

1. Complex LayerNorm
2. Complex Multi-Head Attention (using radial-patched tokens)
3. Complex LayerNorm (post-attention)
4. Complex Feed-Forward Network with residual connections

Classification models prepend a learnable complex class token $x_{cls} \in \mathbb{C}^d$, which is used as the summary representation for downstream decision layers (taking either $|x_{cls}|$ or $\Re(x_{cls})$ as input to a final real-valued softmax head).

Key hyperparameters from [2601.18392], [2306.09827]:

| Dataset/Task              | Layers | Heads | Embedding $d$ | Patch Tokens $N$ | MLP dim | Dropout | Batch Size | Params (M) | VRAM (GB) |
|---------------------------|--------|-------|---------------|------------------|---------|---------|------------|------------|-----------|
| fastMRI Prostate/Knee     | 6      | 16    | 256           | 16               | 768     | 0.1     | 64         | 4.9        | 0.96      |
| In-house MIL (brain)      | 2      | 4     | 48            | --               | 92      | --      | 1          | --         | 0.52      |

The models are trained via AdamW (lr=$1\mathrm{e}{-4}$), with cross-entropy loss and early stopping (patience=15). Data are subjected to acceleration factors $R\in\{2,4,\ldots,24\}$, applying cutout augmentation as necessary to enhance generalization [2601.18392], [2306.09827].

## 5. Efficiency, Robustness, and Empirical Results

kViT models demonstrate significant computational and statistical advantages over real-valued baselines:

- **Memory Efficiency:** Attention operates over $N=16$ tokens (radial patches), yielding $O(N^2 d)$ complexity, as opposed to $O(196^2 d)$ for standard $14\times14$ ViTs. On fastMRI Prostate (batch size 64), kViT required 0.96 GB VRAM vs. 10.6 GB for ResNet50 and 3.7 GB for ViT-Tiny; in multi-instance learning, kViT used 0.52 GB versus ViT-Tiny 11.7 GB and EfficientNet-B0 35.4 GB—a 68$\times$ reduction [2601.18392].
- **Overfitting and Generalization:** Complex-valued transformers exhibit reduced overfitting relative to doubled-real (“stacked R/I”) counterparts; in MusicNet, real ViT overfits by epoch 10, while complex variants remain stable [2306.09827].
- **Performance and Robustness:** On fastMRI Prostate test data, kViT achieves AUROC $78.2\pm 2.2$ at full sampling, and $77.0\pm 3.0$ under $16\times$ undersampling, outperforming or matching real-valued models. ViT-Tiny drops precipitously to $56.3\pm6.2$ under the same conditions. In-house multi-instance learning (MIL) shows kViT maintains AUPRC around 54 even under aggressive undersampling, whereas ViT-Tiny degrades to 43 [2601.18392].
- **Ablation Findings:** Using fewer or more rings (radial patches) affects AUROC non-monotonically (optimum at 16), and removing RoPE or augmentation reduces accuracy. Discarding phase produces severe collapse in performance (AUROC from 0.685 to 0.551 at $16\times$ acceleration), confirming the necessity of preserving phase.

## 6. Key Implementation Considerations and Required Modules

Implementation draws upon standardized complex-valued NN recipes (e.g., Trabelsi et al. ICLR 2018) and the architectural utilities from [2306.09827]:

- **Complex Linear Layer**: Trabelsi-form complex FC, forward and Wirtinger-based backward pass.
- **Complex Multi-Head Attention**: CAtt preferred; variant selection (AAtt, APAtt, RI Att) possible.
- **Complex LayerNorm**: Covariance-based normalization with learnable positive-definite scaling.
- **Complex Feed-Forward**: Two layers, nonlinearities selected from modReLU/zReLU/split ReLU.
- **Residuals, Dropout, Optimizer**: Standard transformer scaffolding; Adam on split real/imag parameters suffices. No specialized optimizer is required.

These core modules enable operation throughout on raw complex-valued data, thus avoiding the loss of representational fidelity incurred by mapping $\mathbb{C}\to\mathbb{R}^2$, while yielding both accuracy and significant gains in robustness and computational efficiency [2306.09827], [2601.18392].

## 7. Significance and Domain-Specific Impact

kViT establishes a framework for domain-aligned, resource-efficient analysis of frequency-domain data, particularly in MRI where phase and spectral locality are essential. The architecture achieves SOTA robustness under aggressive undersampling, sustains classification accuracy competitive with real-valued baselines, and demonstrates substantial reductions in memory and computational load [2601.18392]. The use of physics-informed patching and complex-valued operations provides an architectural blueprint for extending Transformers to other complex-data-intensive domains, including remote sensing and scientific imaging. These advances suggest kViT may become the preferred paradigm for direct-from-scanner AI analysis, providing a pathway for improved fidelity and efficiency in complex-valued signal processing.

Source: https://www.emergentmind.com/topics/complex-valued-vision-transformer-kvit