Metaheuristic Design of Feedforward Neural Networks: A Review of Two Decades of Research (1705.05584v1)

Abstract: Over the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era.

• The paper presents a comprehensive review of metaheuristic methods applied to optimizing feedforward neural network architectures beyond traditional gradient-based techniques.
• It details the evolution of strategies including genetic algorithms, particle swarm, and ant colony optimization to enhance network design and performance.
• The study highlights multiobjective and neuroevolution frameworks that balance training error with model complexity for improved robustness and adaptability.

Insights into Two Decades of Metaheuristic Design for Feedforward Neural Networks

The paper "Metaheuristic Design of Feedforward Neural Networks: A Review of Two Decades of Research" provides a comprehensive literature synthesis on the development and optimization strategies for feedforward neural networks (FNNs) using metaheuristic methods. Over the last two decades, numerous approaches have been explored to enhance the performance and generalization capacity of FNNs, particularly through the lens of optimization techniques that go beyond traditional gradient-descent methods, such as backpropagation.

Key Concepts and Historical Context

Feedforward neural networks are a class of artificial neural networks characterized by unidirectional signal flow and are widely regarded for their universal function approximation capabilities. Traditional optimization of these networks primarily depended on gradient-based methods, which, while successful in many applications, are often limited by issues such as local minima and sensitivity to hyperparameters. The emergence of metaheuristic algorithms has provided alternative paths that address some of these limitations by leveraging bio-inspired and evolution-based search techniques to improve the robustness and flexibility of neural network training.

Metaheuristic Approaches

Metaheuristics such as genetic algorithms (GA), particle swarm optimization (PSO), and ant colony optimization (ACO), among others, have been applied to enhance FNNs by optimizing not only the weights but also architectures, activations, and learning environments. This has involved the transformation of FNN components into optimization problem formulations that can leverage the exploratory power of these algorithms. The methods are divided into single-solution based approaches, population-based algorithms, and hybrid or memetic strategies that combine multiple techniques to balance exploration and exploitation.

Evolutionary Strategies and Neural Architecture

The paper highlights innovative methodologies for evolving neural network structures, components, and learning rules within the metaheuristic framework. Evolving neural architectures, for instance, involves using genetic representations to encode structural traits, facilitating evolutionary adaptation that traditionally cannot be easily achieved with local optimization methods. Furthermore, neuroevolution paradigms such as EPNet and NEAT have provided additional layers of adaptability by co-evolving the network topology and function parameters concurrently.

Multiobjective Optimization

The multiobjective metaheuristic framework is also explored as a mechanism to tackle complex neural network design challenges. Here, the objectives often extend beyond simple training error minimization to incorporate network complexity and generalization metrics. Algorithms like NSGA-II have been employed to derive Pareto-optimal solutions that offer balanced trade-offs among competing objectives and enhanced network generalization.

Implications and Future Directions

Metaheuristic design methodologies have paved the way for robust, versatile, and adaptable neural network solutions capable of tackling the increasingly complex demands posed by modern data environments, including those inherent in big data and non-stationary contexts. As the field evolves, future research could explore integrating these methods with emerging technologies like quantum computing and the Internet of Things (IoT), or explore handling high-dimensional and heterogeneous data spaces. The ongoing challenge lies in refining these metaheuristic strategies to further enhance the computational efficiency and adaptive capabilities of FNNs in real-world applications.