Biology-inspired joint distribution neurons based on Hierarchical Correlation Reconstruction allowing for multidirectional neural networks (2405.05097v5)

Abstract: Biological neural networks seem qualitatively superior (e.g. in learning, flexibility, robustness) to current artificial like Multi-Layer Perceptron (MLP) or Kolmogorov-Arnold Network (KAN). Simultaneously, in contrast to them: biological have fundamentally multidirectional signal propagation \cite{axon}, also of probability distributions e.g. for uncertainty estimation, and are believed not being able to use standard backpropagation training \cite{backprop}. There are proposed novel artificial neurons based on HCR (Hierarchical Correlation Reconstruction) allowing to remove the above low level differences: with neurons containing local joint distribution model (of its connections), representing joint density on normalized variables as just linear combination of $(f_\mathbf{j})$ orthonormal polynomials: $\rho(\mathbf{x})=\sum_{\mathbf{j}\in B} a_\mathbf{j} f_\mathbf{j}(\mathbf{x})$ for $\mathbf{x} \in [0,1]^d$ and $B\subset \mathbb{N}^d$ some chosen basis. By various index summations of such $(a_\mathbf{j}){\mathbf{j}\in B}$ tensor as neuron parameters, we get simple formulas for e.g. conditional expected values for propagation in any direction, like $E[x|y,z]$, $E[y|x]$, which degenerate to KAN-like parametrization if restricting to pairwise dependencies. Such HCR network can also propagate probability distributions (also joint) like $\rho(y,z|x)$. It also allows for additional training approaches, like direct $(a\mathbf{j})$ estimation, through tensor decomposition, or more biologically plausible information bottleneck training: layers directly influencing only neighbors, optimizing content to maximize information about the next layer, and minimizing about the previous to remove noise, extract crucial information.

• The paper introduces HCR to model joint distributions via polynomial parametrization, enabling efficient capture of complex nonlinear interactions.
• The methodology supports multidirectional propagation of gradients and probabilities, enhancing the handling of higher-order dependencies and missing data.
• A focus on optimizing model complexity suggests tensor decomposition as a promising future direction to manage high-dimensional challenges.

Understanding Hierarchical Correlation Reconstruction in Neural Networks

Introduction to Hierarchical Correlation Reconstruction (HCR)

Hierarchical Correlation Reconstruction (HCR) is a theoretical framework for modeling neurons in neural networks. It goes beyond the traditional approach by allowing the neurons to model entire joint distributions of connected variables, as opposed to merely capturing dependencies between single-layer inputs and outputs.

Key Concepts and Implementation

HCR models joint distribution through polynomial parametrization and operates under the assumption that variables have been normalized. This normalization helps in mapping inputs to a nearly uniform distribution, simplifying the computational processes:

• Polynomial Basis and Parametrization: The model represents the joint distribution as a sum of products of polynomial functions. This is akin to expanding the inputs into a polynomial space, allowing the capture of complex, nonlinear interactions among them.
• Efficient Coefficient Estimation: The use of orthonormal polynomials enables efficient computation of coefficients by simple averaging over data points. This represents an economical way to capture relationships within the data.

Multidirectional Propagation and Practical Uses

One of the significant advancements proposed by HCR is the ability of neurons to propagate information in multiple directions:

• Backpropagation Through Conditional Distributions: Unlike traditional methods which typically propagate gradients unidirectionally, HCR facilitates multidirectional propagation both of gradients and probability distributions.
• Potential Applications: This feature could be instrumental in networks handling tasks where feedback or reciprocal interactions among variables/inputs are necessary, such as dynamic systems modeling and certain types of reinforcement learning scenarios.

Handling of Higher-Order Interactions

HCR allows for the inclusion and modeling of higher-order dependencies without significant computational overhead:

• Model Flexibility: By including higher-order moments (e.g., skewness, kurtosis within its parameters), HCR can describe more complex distributions, which are often encountered in real-world datasets.
• Handling Missing and Incomplete Data: The structure of HCR implies that missing data can be managed more gracefully, allowing the network to continue functioning even if some inputs are missing.

Technical Challenges and Future Directions

Despite its promising approach, implementing HCR in practical applications presents several challenges:

• Optimization of Polynomial Basis: Selecting and optimizing the polynomial basis is crucial for performance but is non-trivial. It often involves trade-offs between model complexity and computational feasibility.
• High Dimensional Data: As the dimensionality of the data increases, the number of coefficients required to model the joint distributions grows exponentially.
• Tensor Decomposition and Model Reduction: Exploring tensor decomposition methods could provide a pathway to managing the complexity by approximating the high-dimensional tensors with simpler, lower-order components.

Conclusion and Prospective Insights

HCR represents an interesting shift towards incorporating biologically-inspired mechanisms in artificial neural networks. By modeling joint distributions and allowing for bidirectional data flow, HCR has the potential to enhance neural network architectures notably.

This approach could lead to more robust models that better mimic human cognitive processes, improving both the interpretability and efficiency of neural networks. However, considerable research and experimentation remain necessary to optimize these models for practical use and to explore their full potential in various AI applications.