Hrrformer: Efficient HRR Self-Attention

• Hrrformer is an efficient self-attention model that recasts standard Transformer attention using HRR and FFT-based binding/unbinding to handle ultra-long sequences.
• It reduces the quadratic complexity of traditional attention (O(T²H)) to a subquadratic O(TH log H), enabling sequence lengths over 100,000.
• Empirical benchmarks show Hrrformer delivers competitive accuracy with significant improvements in speed and memory efficiency, especially in raw byte tasks like malware analysis.

Hrrformer is an efficient self-attention model that re-casts the standard Transformer’s attention mechanism using Holographic Reduced Representations (HRR). Designed for domains demanding extremely long sequence lengths, such as raw malware byte analysis, Hrrformer maintains the high-level query-key-value matching structure of classic attention, but replaces explicit all-pairs dot-products with associative vector binding and unbinding in a fixed-dimensional space. This approach yields fast and memory-efficient attention, making it viable for sequence lengths $T \geq 100,000$—beyond the practical limits of conventional Transformer architectures (Alam et al., 2023).

1. Motivation: Addressing Quadratic Complexity

The canonical self-attention mechanism computes, for each of $T$ sequence positions, attention scores against all $T$ keys, resulting in $\mathcal{O}(T^2 H)$ floating-point operations and $\mathcal{O}(T^2)$ memory footprint. For very long sequences, such costs are prohibitive; on modern GPUs, $T \to 100,000$ is out of reach for deep learning practitioners. Hrrformer retains the “query matches key, returning a weighted sum of values” logic, but achieves subquadratic efficiency by dispensing with explicit $T \times T$ attention matrices. Instead, it uses HRR-based associative operations with FFT-based acceleration, obtaining per-layer time complexity $\mathcal{O}(T H \log H)$ and space complexity $\mathcal{O}(T H)$.

2. Holographic Reduced Representations for Attention

HRR encodes entities as high-dimensional vectors. Given $x$, $T$0, HRR “binding” is defined as:

$T$1

where $T$2 denotes the complex FFT and $T$3 is pointwise multiplication. An “unbinding” vector $T$4 is:

$T$5

Under i.i.d. Gaussian assumptions, binding and unbinding facilitate reliable retrieval of associated content: $$( x \circledast y + a \circledast b )^{Tx^{-1}} \approx 1$$T$6x$T$70$ otherwise. In contrast to classic HRR theory—which assumes random vectors—Hrrformer operates with learned network representations for queries, keys, and values, further employing softmax normalization to suppress noise from non-randomness.

3. Hrrformer Attention Pass: Workflow Overview

Queries ($T$8), keys ($T$9), and values ($T$0) are the standard linear projections from Transformer hidden states, with $T$1. Hrrformer executes attention as follows:

1. Key–value superposition: Compute a single superposed vector by binding each key to its value:

$T$2

2. Unbinding per query: For each query $T$3, approximate retrieval by unbinding $T$4 with $T$5:

$T$6

3. Cosine similarity scoring: For each $T$7, compute attention logits:

$T$8

4. Noise denoising via softmax: Normalize weights for all positions:

$T$9

5. Weighted sum of values: Compute the output as in conventional attention:

$\mathcal{O}(T^2 H)$0

All binding/unbinding operations are performed with FFTs, granting $\mathcal{O}(T^2 H)$1 cost per vector, and backpropagation traverses these steps analogously to convolutional networks.

4. Computational Efficiency and Scaling Comparison

A direct comparison between standard attention and Hrrformer reveals the efficiency gains:

Model	Time Complexity	Space Complexity
Transformer (dot-product attn)	$\mathcal{O}(T^2 H)$2	$\mathcal{O}(T^2 H)$3
Hrrformer	$\mathcal{O}(T^2 H)$4	$\mathcal{O}(T^2 H)$5

For $\mathcal{O}(T^2 H)$6, Hrrformer achieves both asymptotic and practical improvements, enabling training and inference on raw byte sequences at lengths unreachable by prior models (Alam et al., 2023).

5. Architecture Details and Training Protocols

A single Hrrformer attention layer replaces the standard attention block, while residual connections, layer-norm, and positionwise feed-forward components are retained identically. Example hyperparameters include $\mathcal{O}(T^2 H)$7, 4 or 8 heads, embedding dimension $\mathcal{O}(T^2 H)$8–$\mathcal{O}(T^2 H)$9, and MLP hidden size $\mathcal{O}(T^2)$0–$\mathcal{O}(T^2)$1. Notably, a single Hrrformer attention layer is often sufficient for competitive accuracy, reducing overall parameter count, memory usage, and training time. Dropout is fixed at $\mathcal{O}(T^2)$2; Adam optimizer is used with initial learning rate $\mathcal{O}(T^2)$3 decayed to $\mathcal{O}(T^2)$4 during training, with batch sizes chosen to fully utilize $\mathcal{O}(T^2)$5 GB GPUs. Hrrformer demonstrates rapid convergence, typically within $\mathcal{O}(T^2)$6 the number of epochs required by competing efficient-attention architectures.

6. Empirical Performance Benchmarks

On the Long Range Arena (LRA) suite (sequence lengths $\mathcal{O}(T^2)$7–$\mathcal{O}(T^2)$8), Hrrformer realizes state-competitive accuracy and substantial efficiency improvements:

Model	Accuracy	Throughput	Mem (GB)	Epochs
Transformer (6L×200)	54.4 %	10 ex/s	22	200
Luna-256 (6L×200)	61.95 %	24 ex/s	3.2	200
Hrrformer (1L×20)	59.97 %	684 ex/s	0.66	20
Hrrformer (3L×20)	60.83 %	246 ex/s	0.93	20

Throughput measured on byte-level text task with GPU memory at peak. Hrrformer (1L×20) is approximately $\mathcal{O}(T^2)$9 point below Luna-256 in accuracy but requires only $T \to 100,000$0 the epochs, $T \to 100,000$1–$T \to 100,000$2 less GPU memory, and is $T \to 100,000$3–$T \to 100,000$4 faster per epoch, yielding up to $T \to 100,000$5 wall-clock speedup.

On EMBER raw-byte malware classification ($T \to 100,000$6 up to $T \to 100,000$7), Hrrformer scales to $T \to 100,000$8, whereas most competing models run out-of-memory or out-of-time for $T \to 100,000$9. At $T \times T$0, Hrrformer achieves $T \times T$1 test accuracy, surpassing all previous approaches.

7. Limitations and Prospects for Further Development

HRR theory assumes that queries, keys, and values are “random-like” vectors, whereas they are learned in practice. Hrrformer addresses noise due to non-randomness via softmax denoising, but residual noise persists. On certain long-sequence tracking tasks (e.g., Path-X LRA at $T \times T$2), Hrrformer exceeds the $T \times T$3 GB memory ceiling. The current binding/unbinding operations are global; potential directions include integrating local/global mixtures, learning binding kernels, or exploring hybridization with state-space models. The core symbol-manipulation concept of “bind all key–value pairs into one superposition, then unbind with each query” demonstrates that HRR-style mechanisms can substitute for dot-product attention, yielding dramatic cost savings while maintaining strong performance, especially in raw byte modeling regimes at $T \times T$4 (Alam et al., 2023).