Hrrformer Architecture

• Hrrformer Architecture is a Transformer variant that redefines self-attention using HRR's circular convolution and correlation, enhancing computational efficiency.
• It achieves sub-quadratic time (O(TH log H)) and linear space complexity, enabling practical processing of much longer sequences than traditional models.
• Empirical evaluations show rapid convergence (~20 epochs) and superior performance on tasks such as malware classification and long-range benchmarks.

The Hrrformer is a Transformer-family architecture in which the conventional self-attention mechanism is re-cast in terms of Holographic Reduced Representations (HRR), a neuro-symbolic method utilizing circular convolution and correlation. By replacing pairwise dot-product attention with HRR-based superposition and unbinding operations, the Hrrformer achieves sub-quadratic time and space complexity—$\mathcal{O}(T H \log H)$ time and $\mathcal{O}(T H)$ space—enabling practical attention at sequence lengths well beyond the feasible limits of standard Transformers. Despite approximate attention from HRR, the Hrrformer retains high accuracy on benchmarks and demonstrates rapid convergence and efficiency for domains requiring very long context windows, most notably malware sequence classification (Alam et al., 2023).

1. Query, Key, and Value Computation in Hrrformer

Token embeddings $x_t\in\mathbb{R}^H$ are projected positionwise into queries $q_t$, keys $k_t$, and values $v_t$ via learned matrices $W^Q,W^K,W^V\in\mathbb{R}^{H\times H}$:

$Q = X W^Q,\quad K = X W^K,\quad V = X W^V,$

where $X\in\mathbb{R}^{T\times H}$ represents the stacked input sequence. Multi-head attention divides $H$ into $\mathcal{O}(T H)$0 heads of $\mathcal{O}(T H)$1 dimensions per head, applying these projections per head identically. These projections preserve the canonical query–key–value paradigm of the original Transformer.

2. Holographic Reduced Representation Attention

Hrrformer attention replaces dot products and softmax weighting with two operators originating from Plate’s HRR: circular convolution (binding) and circular correlation (unbinding), both efficiently implemented via FFT.

• Binding (Circular Convolution): Given $\mathcal{O}(T H)$2,

$\mathcal{O}(T H)$3

where $\mathcal{O}(T H)$4 is the complex FFT, $\mathcal{O}(T H)$5 its inverse, and $\mathcal{O}(T H)$6 denotes elementwise multiplication.

• Unbinding (Circular Correlation): For $\mathcal{O}(T H)$7,

$\mathcal{O}(T H)$8

with $\mathcal{O}(T H)$9 denoting elementwise division.

For suitably random $x_t\in\mathbb{R}^H$0 and $x_t\in\mathbb{R}^H$1 (IID zero-mean Gaussians with variance $x_t\in\mathbb{R}^H$2), HRR ensures near-orthogonality in high dimensions:

$x_t\in\mathbb{R}^H$3

3. HRR-based Self-Attention Workflow

The Hrrformer’s attention implements the Transformer’s high-level querying logic via superposition and HRR unbinding:

1. Key–Value Superposition: Encode all key–value pairs in a single vector,

$x_t\in\mathbb{R}^H$4

2. Query Unbinding: For each position $x_t\in\mathbb{R}^H$5, extract an approximate value via,

$x_t\in\mathbb{R}^H$6

where $x_t\in\mathbb{R}^H$7 is the correlation inverse of $x_t\in\mathbb{R}^H$8.

1. Softmax Denoising: Compute $x_t\in\mathbb{R}^H$9 for all $q_t$0 as attention logits, stack into vector $q_t$1, and obtain normalized weights $q_t$2. Although $q_t$3 contains uniform noise, the softmax operation is invariant to addition of a constant offset, effectively “denoising” the signal.
2. Weighted Value Output: The output at position $q_t$4 is given by

$q_t$5

Each convolution/correlation is $q_t$6 via FFT; total attention is $q_t$7 in time and $q_t$8 in space, since only $q_t$9 and the $k_t$0 arrays are retained.

4. Layer Design and Integration

A single Hrrformer encoder layer adopts a pre-layer normalization (pre-norm) Transformer form:

• Layer structure:
  • LayerNorm$k_t$1
  • Projections: $k_t$2
  • HRR attention (produces $k_t$3)
  • Residual: $k_t$4
  • LayerNorm$k_t$5
  • 2-layer MLP (GELU or ReLU nonlinearity, linear back to $k_t$6); output $k_t$7
  • Residual: $k_t$8

For the Long Range Arena (LRA) Image task, three layers are used; for other LRA tasks, up to six layers; for EMBER malware classification, a single layer suffices. Multi-head attention splits $k_t$9, and outputs are concatenated and linearly projected to $v_t$0.

5. Implementation and Hyperparameters

Key details include:

• FFT batching: Binding/unbinding operations are computed via

$v_t$1

batchwise across all tokens and heads.

• Numerical stability: A small $v_t$2 (e.g., $v_t$3) is added when dividing in $v_t$4 to prevent division by zero in the frequency domain.
• Softmax: No special processing outside conventional masking for padding.
• Hyperparameter ranges:
  • LRA: $v_t$5 or $v_t$6, $v_t$7–$v_t$8 heads, MLP-dim $v_t$9–$W^Q,W^K,W^V\in\mathbb{R}^{H\times H}$0, dropout 0.1, Adam optimizer, initial lr $W^Q,W^K,W^V\in\mathbb{R}^{H\times H}$1 decayed to $W^Q,W^K,W^V\in\mathbb{R}^{H\times H}$2 over 20 epochs.
  • EMBER: $W^Q,W^K,W^V\in\mathbb{R}^{H\times H}$3, $W^Q,W^K,W^V\in\mathbb{R}^{H\times H}$4, MLP-dim $W^Q,W^K,W^V\in\mathbb{R}^{H\times H}$5, single layer, 10 training epochs.

6. Efficiency, Approximation, and Scaling Analysis

Comparisons with standard transformer attention include:

Property	Standard Attention	Hrrformer/HRR Attention
Time complexity	$W^Q,W^K,W^V\in\mathbb{R}^{H\times H}$6	$W^Q,W^K,W^V\in\mathbb{R}^{H\times H}$7
Space complexity	$W^Q,W^K,W^V\in\mathbb{R}^{H\times H}$8	$W^Q,W^K,W^V\in\mathbb{R}^{H\times H}$9
Exactness	Exact	Approximate, denoised
Typical convergence	$Q = X W^Q,\quad K = X W^K,\quad V = X W^V,$0200 epochs	$Q = X W^Q,\quad K = X W^K,\quad V = X W^V,$120 epochs

HRR attention is approximate and introduces uniform noise to $Q = X W^Q,\quad K = X W^K,\quad V = X W^V,$2, but given softmax “denoising,” the empirical impact is limited. For $Q = X W^Q,\quad K = X W^K,\quad V = X W^V,$3, computational savings are substantial. On LRA benchmarks, the Hrrformer achieves convergence in $Q = X W^Q,\quad K = X W^K,\quad V = X W^V,$420 epochs, in contrast to $Q = X W^Q,\quad K = X W^K,\quad V = X W^V,$5200 for most sparse or linear-efficient alternatives.

7. Empirical Evaluation

Findings from (Alam et al., 2023):

• Long Range Arena (LRA, $Q = X W^Q,\quad K = X W^K,\quad V = X W^V,$6 up to 16K):
  • Single-layer Hrrformer achieves $Q = X W^Q,\quad K = X W^K,\quad V = X W^V,$7 average accuracy; multilayer, $Q = X W^Q,\quad K = X W^K,\quad V = X W^V,$8.
  • Performance nearly matches Luna-256 ($Q = X W^Q,\quad K = X W^K,\quad V = X W^V,$9) and exceeds the original Transformer ($X\in\mathbb{R}^{T\times H}$0).
  • Per-epoch speed is $X\in\mathbb{R}^{T\times H}$1 samples/sec (vs $X\in\mathbb{R}^{T\times H}$2 for Luna-256), $X\in\mathbb{R}^{T\times H}$310x faster, with $X\in\mathbb{R}^{T\times H}$479% less GPU memory usage.
  • Total training time is up to $X\in\mathbb{R}^{T\times H}$5 faster due to both per-epoch speed and decreased epochs to convergence.
• EMBER Malware Classification ($X\in\mathbb{R}^{T\times H}$6 up to 131K):
  • Hrrformer scales to $X\in\mathbb{R}^{T\times H}$7; other full-attention models run out of memory by $X\in\mathbb{R}^{T\times H}$8K–$X\in\mathbb{R}^{T\times H}$9K.
  • At $H$0, achieves $H$1 accuracy, surpassing all baselines.
  • Runtime per epoch equals or surpasses the best (F-Net), partly attributed to $H$2.

In summary, the Hrrformer formulation—by recasting Transformer self-attention with HRR operators—matches the core query–key–value logic but with near-linear scaling in sequence length and substantial empirical speedup, maintaining near–state-of-the-art accuracy and data efficiency on both synthetic and real-world long-sequence tasks (Alam et al., 2023).