---
title: 'Hrrformer: Efficient HRR Self-Attention'
url: https://www.emergentmind.com/topics/hrrformer
type: topic
---

# Hrrformer: Efficient HRR Self-Attention

Hrrformer is an efficient self-attention model that re-casts the standard Transformer’s attention mechanism using Holographic Reduced Representations (HRR). Designed for domains demanding extremely long sequence lengths, such as raw malware byte analysis, Hrrformer maintains the high-level query-key-value matching structure of classic attention, but replaces explicit all-pairs dot-products with associative vector binding and unbinding in a fixed-dimensional space. This approach yields fast and memory-efficient attention, making it viable for sequence lengths $T \geq 100,000$—beyond the practical limits of conventional Transformer architectures [2305.19534].

## 1. Motivation: Addressing Quadratic Complexity

The canonical self-attention mechanism computes, for each of $T$ sequence positions, attention scores against all $T$ keys, resulting in $\mathcal{O}(T^2 H)$ floating-point operations and $\mathcal{O}(T^2)$ memory footprint. For very long sequences, such costs are prohibitive; on modern GPUs, $T \to 100,000$ is out of reach for deep learning practitioners. Hrrformer retains the “query matches key, returning a weighted sum of values” logic, but achieves subquadratic efficiency by dispensing with explicit $T \times T$ attention matrices. Instead, it uses HRR-based associative operations with FFT-based acceleration, obtaining per-layer time complexity $\mathcal{O}(T H \log H)$ and space complexity $\mathcal{O}(T H)$.

## 2. Holographic Reduced Representations for Attention

HRR encodes entities as high-dimensional vectors. Given $x$, $y \in \mathbb{R}^H$, HRR “binding” is defined as:
$$
x \circledast y := \mathcal{F}^{-1}\bigl(\mathcal{F}(x) \circ \mathcal{F}(y)\bigr)
$$
where $\mathcal{F}$ denotes the complex FFT and $\circ$ is pointwise multiplication. An “unbinding” vector $y^{-1}$ is:
$$
y^{-1} := \mathcal{F}^{-1}\left( \frac{1}{\mathcal{F}(y)} \right)
$$
Under i.i.d. Gaussian assumptions, binding and unbinding facilitate reliable retrieval of associated content: $(
x \circledast y + a \circledast b
)^Tx^{-1} \approx 1$ when $x$ matches, and nearly $0$ otherwise. In contrast to classic HRR theory—which assumes random vectors—Hrrformer operates with learned network representations for queries, keys, and values, further employing softmax normalization to suppress noise from non-randomness.

## 3. Hrrformer Attention Pass: Workflow Overview

Queries ($Q$), keys ($K$), and values ($V$) are the standard linear projections from Transformer hidden states, with $Q,K,V \in \mathbb{R}^{T \times H}$. Hrrformer executes attention as follows:

1. **Key–value superposition:** Compute a single superposed vector by binding each key to its value:
   $$
   \beta = \sum_{i=1}^T k_i \circledast v_i
   $$
2. **Unbinding per query:** For each query $q_t$, approximate retrieval by unbinding $\beta$ with $q_t$:
   $$
   \hat{v}_t = q_t^{-1} \circledast \beta
   $$
3. **Cosine similarity scoring:** For each $v_t$, compute attention logits:
   $$
   a_t = \cos( v_t, \hat{v}_t )
   $$
4. **Noise denoising via softmax:** Normalize weights for all positions:
   $$
   w_t = \frac{ \exp(a_t) }{ \sum_i \exp(a_i) }
   $$
5. **Weighted sum of values:** Compute the output as in conventional attention:
   $$
   \mathrm{Attention}(Q,K,V)_t = \sum_{i=1}^T w_i v_i
   $$
All binding/unbinding operations are performed with FFTs, granting $O(H \log H)$ cost per vector, and backpropagation traverses these steps analogously to convolutional networks.

## 4. Computational Efficiency and Scaling Comparison

A direct comparison between standard attention and Hrrformer reveals the efficiency gains:

| Model                | Time Complexity | Space Complexity |
|----------------------|----------------|-----------------|
| Transformer (dot-product attn) | $O(T^2 H)$         | $O(T^2)$         |
| Hrrformer            | $O(T H \log H)$ | $O(T H)$        |

For $T \gg H$, Hrrformer achieves both asymptotic and practical improvements, enabling training and inference on raw byte sequences at lengths unreachable by prior models [2305.19534].

## 5. Architecture Details and Training Protocols

A single Hrrformer attention layer replaces the standard attention block, while residual connections, layer-norm, and positionwise feed-forward components are retained identically. Example hyperparameters include $H=256$, 4 or 8 heads, embedding dimension $256$–$512$, and MLP hidden size $512$–$1024$. Notably, a single Hrrformer attention layer is often sufficient for competitive accuracy, reducing overall parameter count, memory usage, and training time. Dropout is fixed at $0.1$; Adam optimizer is used with initial learning rate $10^{-3}$ decayed to $10^{-5}$ during training, with batch sizes chosen to fully utilize $32$ GB GPUs. Hrrformer demonstrates rapid convergence, typically within $1/10$ the number of epochs required by competing efficient-attention architectures.

## 6. Empirical Performance Benchmarks

On the Long Range Arena (LRA) suite (sequence lengths $1K$–$16K$), Hrrformer realizes state-competitive accuracy and substantial efficiency improvements:

| Model                   | Accuracy  | Throughput  | Mem (GB) | Epochs |
|-------------------------|-----------|-------------|----------|--------|
| Transformer (6L×200)    | 54.4 %    | 10 ex/s     | 22       | 200    |
| Luna-256 (6L×200)       | 61.95 %   | 24 ex/s     | 3.2      | 200    |
| Hrrformer (1L×20)       | 59.97 %   | 684 ex/s    | 0.66     | 20     |
| Hrrformer (3L×20)       | 60.83 %   | 246 ex/s    | 0.93     | 20     |

Throughput measured on byte-level text task with GPU memory at peak. Hrrformer (1L×20) is approximately $1$ point below Luna-256 in accuracy but requires only $1/10$ the epochs, $5$–$10\times$ less GPU memory, and is $28$–$684\times$ faster per epoch, yielding up to $280\times$ wall-clock speedup.

On EMBER raw-byte malware classification ($T$ up to $131,000$), Hrrformer scales to $T=131,000$, whereas most competing models run out-of-memory or out-of-time for $T \geq 16,000$. At $T=16,000$, Hrrformer achieves $91.03\,\%$ test accuracy, surpassing all previous approaches.

## 7. Limitations and Prospects for Further Development

HRR theory assumes that queries, keys, and values are “random-like” vectors, whereas they are learned in practice. Hrrformer addresses noise due to non-randomness via softmax denoising, but residual noise persists. On certain long-sequence tracking tasks (e.g., Path-X LRA at $T=16,000$), Hrrformer exceeds the $32$ GB memory ceiling. The current binding/unbinding operations are global; potential directions include integrating local/global mixtures, learning binding kernels, or exploring hybridization with state-space models. The core symbol-manipulation concept of “bind all key–value pairs into one superposition, then unbind with each query” demonstrates that HRR-style mechanisms can substitute for dot-product attention, yielding dramatic cost savings while maintaining strong performance, especially in raw byte modeling regimes at $T \gtrsim 10^5$ [2305.19534].

Source: https://www.emergentmind.com/topics/hrrformer