---
title: Holographic Reduced Representations (HRRs)
url: https://www.emergentmind.com/topics/holographic-reduced-representations-hrrs
type: topic
---

# Holographic Reduced Representations (HRRs)

Holographic Reduced Representations (HRRs) provide a vector-symbolic architecture for compositional reasoning, neuro-symbolic computation, and scalable memory encoding by leveraging high-dimensional vectors and algebraic binding operations, such as circular convolution. These methods enable the binding of role-filler pairs, storage of multiple symbolic associations, and approximate retrieval through unbinding. HRRs and their descendants integrate properties from symbolic logic and distributed representation, forming a central substrate for neuro-symbolic learning, differentiable reasoning, and efficient implementation of sequence and memory models.

## 1. Mathematical Foundation and Core Operations

The HRR model represents each symbol as a dense, high-dimensional real vector $z \in \mathbb{R}^d$, typically drawn as $z \sim \mathcal{N}(0, I/d)$. The core operations are:

- **Bundling (Superposition):** $x + y$ preserves similarity of $x$ and $y$ within the same vector space.
- **Binding (Association):** Circular convolution is employed as
  $$
  (x * y)_j = \sum_{k=0}^{d-1} x_k\, y_{(j-k) \bmod d}
  $$
  or equivalently, using the discrete Fourier transform (DFT),
  $$
  x * y = \mathcal{F}^{-1} \left( \mathcal{F}(x) \odot \mathcal{F}(y) \right)
  $$
- **Unbinding (Approximate Inverse):** Given $c = x * y$, retrieval of $y$ is performed as $c * x^*$, where $x^*$ is the involution defined by $x^*_j = x_{(-j)\bmod d}$. In the DFT, this is the complex conjugate.
- **Algebraic Properties:** Binding is commutative and associative; bundling is distributive over binding.

Numerical stability is addressed by projecting each vector to a unit-magnitude subspace in the Fourier domain:
$$
\pi(z) = \mathcal{F}^{-1}\left( \mathcal{F}(z) / |\mathcal{F}(z)| \right)
$$
ensuring invertibility and stable unbinding in learning setups [2109.02157][2312.15310].

## 2. Role in Neural and Neuro-Symbolic Architectures

HRRs serve as differentiable, constant-size substrates bridging symbolic representations and neural networks. In neuro-symbolic loss frameworks for classification, each class $n$ has a learnable key–value pair $(k_n, v_n)$, each projected to the unit-magnitude Fourier manifold. The target representation for class $n$ is $k_n * v_n$, and training minimizes
$$
\mathcal{L} = \sum_{i=1}^{B} \left\| k_{n_i} * v_{n_i} - \hat{y}_i \right\|_2
$$
with $\hat{y}_i$ being the tanh-activated model output [2312.15310].

At inference, unbinding by all possible keys retrieves $C$ candidate values, which are scored by cosine similarity to their respective $v_c$. This mechanism enables compositional generalization, as established in subitizing tasks, multi-label classification, and symbolic reasoning [2312.15310][2109.02157].

HRRs have been integrated into neural network architectures:
- As output layers with symbolic loss for multi-label classification, improving memory and compute efficiency over traditional approaches [2109.02157].
- As key-value associative arrays within RNN extensions such as Associative LSTM, where multiple permuted HRR traces are averaged to suppress interference [1602.03032].

## 3. Extensions: Complex, Generalized, and Non-Commutative HRRs

**Fourier Holographic Reduced Representation (FHRR):** Uses complex unit-modulus vectors, where binding is elementwise complex multiplication, $x * y$, and inversion is via complex conjugation. This construction preserves invertibility and similarity under binding [2405.09689][2201.11691].

**Generalized HRR (GHRR):** Extends FHRR to stacks of unitary matrices $A_j \in \mathrm{U}(m)$ per position:
$$
H = [A_1,\ldots,A_D]^\top, \quad H_1 \circledast H_2 = [A_{1j}A_{2j}]_{j=1}^D
$$
Binding is now elementwise matrix multiplication, supporting non-commutativity as a function of the diagonality of unitary factors $Q_j$. This allows GHRRs to interpolate smoothly between FHRR (fully commutative) and full tensor-product representations (fully non-commutative). Non-commutative binding is essential for representing nested or ordered structures without explicit permutations [2405.09689].

**Geometric HRRs:** By replacing convolution with geometric (Clifford) product, HRR-like properties are realized in a geometric algebra context, enabling strict invertibility and basis-free interpretation, though at the expense of exponential scaling in representation size [0710.2611].

## 4. Memory, Capacity, and Retrieval

HRRs encode multiple associations in a single superposed trace. For $N_{\mathrm{items}}$ key-value pairs,
$$
c = \sum_{k=1}^{N_{\mathrm{items}}} r_k * x_k
$$
Retrieval by unbinding with $r_j^*$ gives
$$
\hat{x}_j = x_j + \sum_{k \neq j} (r_j^{-1} * r_k) * x_k = x_j + \mathrm{noise}
$$
The retrieval noise grows linearly with $N_{\mathrm{items}}$. Capacity is thus limited by the trace dimension. Methods such as redundancy via multiple copies, as in Associative LSTM, reduce the variance of interference noise by a factor $1/N_\mathrm{copies}$, maintaining accuracy as more pairs are stored [1602.03032].

Empirical studies show that with complex-unit-magnitude projection, capacity for reliable retrieval becomes linear in $d$, e.g., $n_\mathrm{max} \approx 0.375 d$ at <3% error [2109.02157]. In generalized settings (GHRR), memorization capacity for bound vectors is restored to the linear regime even for complex, non-commutative bindings [2405.09689].

## 5. Applications in Sequence Encoding and Attention

In sequence processing:
- HRRs encode order via recursive binding, enabling shift-equivariant, similarity-preserving hypervector sequence representations. For a position HV $pos$ and symbol HV $e_a$,
  $$
  e_{a,i} = pos^i \odot e_a
  $$
  and summing over a local radius $R$ allows controlled similarity decay across positions. This encoding matches human word similarity data and supports fast, shift-equivariant comparisons [2201.11691].

In self-attention:
- HRR-based attention replaces quadratic dot-product interactions with binding (circular convolution) and collective unbinding, providing $O(TH \log H)$ complexity as opposed to $O(T^2H)$ in standard transformers. This construction enables efficient learning and inference on long-range sequences (up to $T=131{,}072$) with competitive accuracy and up to $280\times$ improved speed-to-convergence [2305.19534].

| Operation   | Formula                                  | Algebraic Properties  |
|-------------|------------------------------------------|-----------------------|
| Bundling    | $x + y$                                 | Preserves similarity  |
| Binding     | $x * y$ (circular convolution or FFT)   | Commutative, associative |
| Unbinding   | $c * x^*$                              | Approximate (HRR) or exact (FHRR, GHRR) |
| Projection  | $\pi(z) = \mathcal{F}^{-1}(\mathcal{F}(z)/|\mathcal{F}(z)|)$ | Ensures invertibility  |

## 6. Limitations and Theoretical Considerations

Standard HRRs are basis-dependent: convolution is not a geometric operation and lacks a basis-free interpretation [0710.2611]. The approximate inverse is exact only for unitary vectors; without projection, unbinding accumulates error rapidly under learning. Memorization and retrieval degrade linearly with the number of bound items unless redundancy or higher dimensions are introduced [1602.03032][2109.02157].

Generalizations (FHRR, GHRR) cure many structural issues:
- Complex unbinding is exact up to quantization noise.
- GHRR enables flexible commutativity control and obviates explicit permutations for positional encoding.
- Geometric HRR provides projective, basis-free analogues at the cost of exponential size [0710.2611].

A remaining limitation is that all finite-dimensional HRR-based architectures are fundamentally “distributed,” lacking fine-grained interpretability for individual vector elements.

## 7. Impact, Generalizations, and Future Directions

HRRs and descendants occupy a foundational position in the development of neuro-symbolic systems, differentiable memory, and efficient sequence modeling. Recent advances demonstrate their capacity to:
- Enable end-to-end differentiable, constant-size output layers for large discrete label spaces, with interpretability and significant resource savings [2109.02157].
- Scalably implement linear-cost multi-head self-attention architectures with near state-of-the-art accuracy for long sequences [2305.19534].
- Support variable and nested structures in both commutative (FHRR) and non-commutative (GHRR) settings, with improved decoding and control over similarity preservation [2405.09689][2201.11691].

Future directions include learning input-dependent binding factors for context-sensitive compositionality, integrating GHRR into deep pipelines, and applying these models to richer data types and reasoning tasks [2405.09689].

## References

- [0710.2611] Geometric Analogue of Holographic Reduced Representation
- [1602.03032] Associative Long Short-Term Memory
- [2109.02157] Learning with Holographic Reduced Representations
- [2201.11691] Recursive Binding for Similarity-Preserving Hypervector Representations of Sequences
- [2305.19534] Recasting Self-Attention with Holographic Reduced Representations
- [2312.15310] Towards Generalization in Subitizing with Neuro-Symbolic Loss using Holographic Reduced Representations
- [2405.09689] Generalized Holographic Reduced Representations

Source: https://www.emergentmind.com/topics/holographic-reduced-representations-hrrs