"Efficient Complexity": a Constrained Optimization Approach to the Evolution of Natural Intelligence

Abstract: A fundamental question in the conjunction of information theory, biophysics, bioinformatics and thermodynamics relates to the principles and processes that guide the development of natural intelligence in natural environments where information about external stimuli may not be available at prior. A novel approach in the description of the information processes of natural learning is proposed in the framework of constrained optimization, where the objective function represented by the information entropy of the internal states of the system with the states of the external environment is maximized under the natural constraints of memory, computing power, energy and other essential resources. The progress of natural intelligence can be interpreted in this framework as a strategy of approximation of the solutions of the optimization problem via a traversal over the extrema network of the objective function under the natural constraints that were examined and described. Non-trivial conclusions on the relationships between the complexity, variability and efficiency of the structure, or architecture of learning models made on the basis of the proposed formalism can explain the effectiveness of neural networks as collaborative groups of small intelligent units in biological and artificial intelligence.