Holographic Embeddings (HolE)

• Holographic Embeddings (HolE) are a framework that encodes knowledge graph facts by composing entity vectors through circular correlation, balancing expressiveness with scalability.
• HolE leverages Fourier transforms to compute circular correlation efficiently in O(d log d) time, enabling fixed-width storage and reduced computational complexity.
• Empirical results on benchmarks like WN18 and FB15k show HolE’s competitive link prediction performance and its theoretical equivalence to the ComplEx model.

Holographic Embeddings (HolE) are a framework for learning compositional vector-space representations of knowledge graphs (KGs), designed to efficiently capture rich feature interactions among entities and relations while preserving scalability and practical optimization properties. HolE models facts as triples $(s, p, o)$—subject, predicate, object—and leverages circular correlation as a compositional operator, inspired by holographic associative memory mechanisms, to encode relational information. This approach achieves a balance between expressiveness, memory footprint, and computational complexity through a fixed-width embedding paradigm, making it distinct among knowledge-graph embedding techniques (Nickel et al., 2015).

1. Mathematical Formulation

Let $E$ and $P$ be the sets of entities and relations, respectively, and $d$ the embedding dimension. HolE assigns each entity $i \in E$ a real-valued vector $\mathbf{e}_i \in \mathbb{R}^d$ and each relation $k \in P$ a real-valued vector $\mathbf{r}_k \in \mathbb{R}^d$. To represent a KG fact $(s, p, o)$, HolE composes the subject and object embeddings via circular correlation: $$(\mathbf{h} \star \mathbf{t})_k = \sum_{i=0}^{d-1} h_i \, t_{(k+i)\bmod d}, \quad k = 0, \ldots, d-1,$$ where $E$0, $E$1.

The triple's score is given by

$E$2

This function quantifies the plausibility of a fact in the KG, with high scores corresponding to likely true triples (Nickel et al., 2015, Trouillon et al., 2017).

Circular correlation compresses the tensor product $E$3 into a vector $E$4 by summing over anti-diagonals, efficiently retaining critical pairwise interactions without quadratic memory or computation (Nickel et al., 2015).

2. Efficient Computation via Fourier Transforms

Circular correlation in HolE is evaluated via the Discrete Fourier Transform (DFT), using the equivalence: $E$5 where $E$6 is the DFT, $E$7 its inverse, $E$8 denotes complex conjugation, and $E$9 is the Hadamard product. This allows HolE to compute the correlation in $P$0 time, compared to $P$1 for explicit time-domain computation (Nickel et al., 2015, Trouillon et al., 2017).

An alternative—termed the spectral version (Editor's term)—is to learn frequency-domain embeddings directly. Each embedding can be represented in the Fourier domain, $P$2, subject to conjugate symmetry for real-valuedness. The scoring function becomes

$P$3

where all vectors are in $P$4 (Hayashi et al., 2017). In this form, scoring reduces to $P$5 if conjugate-symmetric embeddings are maintained, further reducing the effective cost.

3. Training Objectives and Optimization

Training of HolE embeddings proceeds via either regularized logistic loss or pairwise margin-based ranking loss:

• Logistic loss (binary labeling):

$P$6

where $P$7 are true/false labels, $P$8 is an $P$9 regularization parameter.

• Pairwise margin ranking loss:

$d$0

where $d$1 is the margin, $d$2 is the set of positive triples, and $d$3 consists of negative samples, typically generated by corrupting subjects or objects (Nickel et al., 2015, Trouillon et al., 2017, Hayashi et al., 2017).

Optimization is performed by stochastic gradient descent, with AdaGrad or Adam for adaptive learning rates. Early stopping is typically performed on a validation split.

4. Empirical Results and Comparisons

HolE has been benchmarked on several datasets:

• WN18 (WordNet subset): 40,943 entities, 18 relations, 151,442 triples.
• FB15k (Freebase subset): 14,951 entities, 1,345 relations, 592,213 triples.
• Countries dataset: relational reasoning across three levels of difficulty (Nickel et al., 2015).

HolE demonstrates superior or competitive link prediction performance (filtered Mean Reciprocal Rank, Hits@N) compared to TransE, TransR, ER-MLP, and RESCAL. On WN18, filtered MRR achieved by HolE is 0.938, outperforming RESCAL (0.890), ER-MLP (0.712), TransR (0.605), and TransE (0.495). On FB15k, HolE achieves 0.524 MRR, exceeding RESCAL (0.354), TransE (0.463), TransR (0.346), and ER-MLP (0.288) (Nickel et al., 2015).

In the synthetic symmetry/antisymmetry task, HolE and ComplEx recover symmetric and antisymmetric relations with sufficient embedding rank, though ComplEx requires roughly half the rank due to its complex-valued parameterization (Trouillon et al., 2017).

5. Relation to Complex Embeddings (ComplEx) and Theoretical Equivalence

HolE is theoretically equivalent to the ComplEx model under a frequency-domain reinterpretation. ComplEx represents entities and relations in $d$4 and scores triples as

$d$5

where $d$6 denotes complex-valued embeddings. The key insight is that real vectors have DFTs with conjugate symmetry; mapping HolE’s real embeddings via DFT yields a ComplEx instance with conjugate-symmetric embeddings, and the scoring functions coincide up to a scale factor: $d$7 for properly normalized frequency spaces (Hayashi et al., 2017, Trouillon et al., 2017). Conversely, any ComplEx instance with conjugate symmetry can be converted to an equivalent HolE model via the inverse DFT. The two methods are therefore indistinguishable when loss functions and hyperparameters coincide.

The implication is that ComplEx can be viewed as a generalization of HolE, relaxing the conjugate-symmetry constraint and thus doubling the available degrees of freedom per embedding when $d$8 is fixed (Hayashi et al., 2017).

6. Computational, Representational, and Practical Considerations

Memory and Complexity

Model	Embedding Type	Per-triple Scoring Time	Memory Complexity
HolE	$d$9	$i \in E$0	$i \in E$1
ComplEx	$i \in E$2	$i \in E$3	$i \in E$4
RESCAL/tensor	$i \in E$5	$i \in E$6	$i \in E$7

HolE achieves fixed-width $i \in E$8 storage for each entity/relation, efficient scoring and gradient computations via FFT ($i \in E$9). When embeddings are learned in the frequency domain, scoring matches ComplEx’s linear complexity, but requires explicit handling of conjugate symmetry (Nickel et al., 2015, Trouillon et al., 2017).

Modeling Properties

HolE’s circular correlation operator is non-commutative, capturing asymmetric relations—critical in real-world KGs. The first component of the correlation encodes the standard dot product, enhancing sensitivity to symmetric or similarity-based relationships (Nickel et al., 2015).

HolE embeddings are strictly real, facilitating integration with real-valued machine learning models and downstream tasks (Trouillon et al., 2017). ComplEx supports linear-time scoring and full expressiveness for arbitrary binary relations via relaxation of the conjugate symmetry.

Limitations and Extensions

The fixed partitioning of features by circular correlation limits potential expressiveness; learnable partitions or advanced compositional operators may further enhance performance. HolE has not been generalized to higher-arity relations or natively encodes facts about facts. Combining HolE with neural attention, deep architectures, or literal attributes remains an active research direction (Nickel et al., 2015).

7. Practical Implications and Guidelines

Empirical performance between HolE and ComplEx is indistinguishable given equivalent loss functions and tuning. The choice between models is dictated by application-specific constraints:

• HolE is advantageous when real-valued embeddings are required for compatibility with external models, or when FFT overhead is not a bottleneck.
• ComplEx is preferred in high-throughput environments, as it enables strictly linear time scoring at the cost of complex-valued parameters (Trouillon et al., 2017).

Recognizing the algebraic equivalence between HolE and ComplEx enables researchers to adopt the representation best aligned with computational infrastructure and integration requirements, while retaining the full expressive power required for link prediction and relational learning (Hayashi et al., 2017, Trouillon et al., 2017).

---

Principal references:

(Nickel et al., 2015) Holographic Embeddings of Knowledge Graphs (Trouillon et al., 2017) Complex and Holographic Embeddings of Knowledge Graphs: A Comparison (Hayashi et al., 2017) On the Equivalence of Holographic and Complex Embeddings for Link Prediction