Disentanglement with Holographic Reduced Representations

Abstract: Disentanglement, the separation of factors of variation in data using neural networks, remains a long-standing challenge in machine learning. Prior work has addressed this problem with variational autoencoders and generative adversarial networks that incorporate ideas from variational inference and information-theoretic constraints. In contrast to methods that rely on continuous representations, we propose a design that treats disentangled representations as symbolic structures, motivated by the compositional relationships among the concepts that make up samples from a distribution. However, learning discrete symbolic structures with neural networks while maintaining differentiability is difficult and often requires complex architectures. To address this, we introduce an unsupervised learning algorithm that uses holographic reduced representations (HRR) for neural disentanglement. We show that the HRR unbinding operation provides an inductive bias for separating factors and yields competitive results against baselines, as measured by latent traversals and disentanglement metrics. We complement these empirical findings with an information-theoretic analysis of the HRR unbinding channel. We prove that unbinding induces approximately independent symbol-value pairs and derive a per-slot capacity bound that quantifies how many distinct symbolic concepts can be reliably encoded, giving a quantitative account of the inductive bias toward disentanglement. The resulting representations differ from standard autoencoder-based models, in that their latent units are vectors that are summed together, rather than scalar dimensions of a low-dimensional latent vector. We show that this HRR representation is more robust to noise than other disentangled representations and maintains reconstruction quality across a range of SNRs.

• The paper proposes a novel neuro-symbolic method using HRRs to achieve unsupervised disentanglement of latent factors.
• It develops an HRR-VQ autoencoder architecture with rigorous theoretical analysis on mutual information and slot independence.
• Empirical results demonstrate superior disentanglement, noise robustness, and modular compositionality compared to leading baselines.

Disentanglement with Holographic Reduced Representations: Technical Summary and Analysis

Introduction and Motivation

The problem of unsupervised disentanglement—factorizing the generative sources of variation in data—remains central in representation learning, yet current approaches predominantly utilize variants of variational autoencoders (VAEs) and, to a lesser degree, discrete latent variable models and GANs. Such methods typically rely on continuous or unstructured discrete representations, limiting their ability to naturally encode compositional, modular semantics. The paper "Disentanglement with Holographic Reduced Representations" (2606.09725) proposes a neuro-symbolic approach, leveraging the algebraic and compositional properties of Holographic Reduced Representations (HRRs), a Vector Symbolic Architecture (VSA), as an inductive bias for latent factor separation within neural networks, without the need for supervision or curated data pairs.

HRRs for Symbolic and Compositional Latent Representations

The architectural core is the HRR-based latent space. In HRRs, each symbol is encoded as a random high-dimensional vector, and structured objects (e.g., role-filler pairs) are bound via circular convolution ('binding') and combined ('bundling') through vector addition. Notably, approximate unbinding (via inverse convolution) allows retrieval of the bound value vector given its symbol vector—a property that supports compositional, neuro-symbolic memory.

Critical to this work is the insertion of an HRR unbinding mechanism at the bottleneck of a convolutional autoencoder. The encoder produces a latent vector $z = \sum_{i=1}^m s_i \otimes v_i$, where $s_i$ are fixed random symbol vectors and $v_i$ are value vectors retrieved, denoised, and vector-quantized through a learned codebook. This structure enforces that each 'slot'—symbol-value pair—acts as a (statistically near-independent) channel for encoding a generative factor, as theoretically established by the HRR noise analysis.

The HRR-VQ mechanism is trained end-to-end using reconstruction, vector quantization, and statistically targeted regularization losses, with the HRR structure imposed by both the algebra of the latent space and architectural constraints.

Theoretical Analysis: Latent Channel Capacity and Slot Independence

A key theoretical contribution is an information-theoretic upper bound on the mutual information between input data $x$ and the symbolic latent $z$, formalized as:

$$I(x; z) \leq \min\left(m d \log\left(1 + \frac{1}{m}\right), m \log k\right)$$

Here, $m$ is the symbolic length (number of slots), $d$ is the dimensionality of the HRR vectors, and $k$ the codebook size. This bound quantifies the channel capacity under additive noise from unbinding and vector quantization, offering principled guidance for selecting architectural hyperparameters.

Further, the authors rigorously establish that, as $d \to \infty$, the cross-slot correlations (total correlation) induced by HRR cross-talk vanish. Thus, each slot can encode information near-independently—an exceptionally strong inductive bias for true disentanglement, aligning distinct generative factors to distinct slots without entanglement through the latent space.

Empirical Results: Disentanglement Quality and Robustness

Comprehensive empirical results support the theoretical claims. The HRR model is benchmarked against strong baselines ($s_i$0-VAE, $s_i$1-TCVAE, VQ-VAE, QLAE) across standard datasets (Shapes3D, Falcor3D, Isaac3D, MPI3D-C) and assessed using InfoMEC (information-theoretic mutual information metrics) and DCI (Disentanglement, Completeness, Informativeness).

Strong findings include:

• Superior Disentanglement: The HRR model attains the highest aggregate scores on InfoM (modularity) and InfoC (compactness) on 5/6 metrics, showing 5–10% relative improvement over next-best baselines in the majority of settings. The slot-structured architecture directly contributes to higher modularity by enforcing factor alignment per slot.
• Noise Robustness: The HRR and VQ-VAE models show robust reconstructive performance under substantial latent Gaussian noise perturbations. HRR is unique in achieving both top disentanglement and top-tier noise robustness—an asset for deployment in uncontrolled, physical-world environments.
• Compositionality and Modularity: Latent component swap and cumulative slot interpolation experiments demonstrate that HRR slots correspond to distinct generative factors. Swapping a slot only affects the associated factor, and cumulative swaps effect smooth, modular transitions, with minimal spurious factor propagation—substantiating the theoretical finding of approximate slot independence.

Implementation Details and Regularization

The architecture is a CNN-based autoencoder, incorporating frozen randomly initialized symbol vectors, a learned quantized codebook, and a single-layer denoising network for value vectors. The total objective function combines binary cross-entropy reconstruction, VQ and commitment terms, and statistical regularizers enforcing HRR-conformant variance and mean statistics for the latent, value, and codebook vectors. Backpropagation through quantization utilizes the straight-through estimator. Notably, model parameter count is kept close to baseline autoencoders.

Limitations and Design Choices

Notable limitations include:

• The empirical comparison omits recent (computationally expensive) baselines requiring substantial architectural or optimization differences.
• The latent-to-source ratio is limited to 1.5 due to the unbinding noise scaling with the number of slots.
• Larger latent spaces result in modest increases in parameter count.
• Results on compositional generalization to previously unseen factor combinations remain an open empirical area.

Ablation studies further demonstrate that training stability and performance degrade when deviating from specific choices: learned vs frozen symbols, static vs learned codebooks, or omitted denoising networks.

Implications and Prospects

This work constitutes, to date, the strongest evidence that the symbolic, compositional bias of VSAs—specifically HRRs—enables superior, robust disentanglement without supervision, in contrast to the continuous or 'flat' discrete alternatives. These findings open several avenues:

• Model Identifiability: The established SNR- and codebook-limited capacity of HRR latents could inform identifiability guarantees and help design next-generation neuro-symbolic models with provable properties.
• Physical AI: Demonstrated robustness to additive noise has strong implications for embodied AI, where latent-space perturbations due to sensor or channel noise are unavoidable.
• Compositional Generalization: The modular slot-wise binding structure offers a template for models capable of out-of-distribution reasoning over novel combinations of known factors, a major shortcoming of previous disentanglement methods.

Potential research directions include formal analysis of identifiability, study of slot-to-factor alignment under latent overparameterization, and methods for efficiently scaling HRR capacities. Further, examining the transfer and adaptation of compositional HRR representations across domains and tasks could impact interpretability, controlled generation, and systematic generalization.

Conclusion

The approach presented in "Disentanglement with Holographic Reduced Representations" (2606.09725) advances the unsupervised learning of disentangled latent representations by harnessing the compositional algebra of HRRs as an inductive bias, both theoretically and empirically. The HRR-based symbolic latent formulation achieves competitive or superior disentanglement scores, robust noise tolerance, and interpretable modularity, outperforming established baselines. The proposed framework substantiates the value of VSAs for representation learning and signals a promising direction for robust, neuro-symbolic, and compositional AI.