Training Neural Networks is NP-Hard in Fixed Dimension (2303.17045v2)

Abstract: We study the parameterized complexity of training two-layer neural networks with respect to the dimension of the input data and the number of hidden neurons, considering ReLU and linear threshold activation functions. Albeit the computational complexity of these problems has been studied numerous times in recent years, several questions are still open. We answer questions by Arora et al. [ICLR '18] and Khalife and Basu [IPCO '22] showing that both problems are NP-hard for two dimensions, which excludes any polynomial-time algorithm for constant dimension. We also answer a question by Froese et al. [JAIR '22] proving W[1]-hardness for four ReLUs (or two linear threshold neurons) with zero training error. Finally, in the ReLU case, we show fixed-parameter tractability for the combined parameter number of dimensions and number of ReLUs if the network is assumed to compute a convex map. Our results settle the complexity status regarding these parameters almost completely.

• The paper demonstrates the NP-hardness of training two-layer neural networks in fixed dimensions, challenging the feasibility of polynomial-time solutions.
• It establishes W[1]-hardness for networks with ReLU activations or using two linear threshold neurons to achieve zero training error, illustrating the inherent complexity even in small architectures.
• A fixed-parameter tractability result under convex mapping shows that imposing convexity can enable efficient training methods despite the overall NP-hard nature of the problem.

Analyzing the Computational Complexity of Training Two-Layer Neural Networks with a Fixed Dimension

The paper, "Training Neural Networks is NP-Hard in Fixed Dimension," authored by Vincent Froese and Christoph Hertrich, rigorously investigates the parameterized complexity of training two-layer neural networks with ReLU and linear threshold activations. Notably, the paper addresses certain intricate aspects of computational complexity linked to fixed-dimensional neural networks, providing significant resolutions to longstanding open questions in the field.

The researchers focus on two primary questions initially posed by Arora et al. [ICLR '18] and further discussed by Khalife and Basu [IPCO '22] regarding whether polynomial-time solutions exist for training these neural networks when the dimension remains constant. The authors demonstrate that the problem is NP-hard even in the context of two dimensions, decisively concluding that no polynomial-time algorithm can resolve these problems under the constraints stated, given the fixed dimension.

Insights and Findings

1. NP-Hardness in Fixed Dimensions: The paper asserts the NP-hard status of training two-layer neural networks in fixed dimensions. Specifically, it reveals that with two dimensions, the complexity remains challenging, foreclosing any polynomial-time algorithm expectations for constant dimensions.
2. W[1]-Hardness with Respect to ReLU: For networks utilizing ReLU, the authors establish W[1]-hardness when the network implements four ReLUs or two linear threshold neurons achieving a zero training error. This demonstrates the inherent complexity and computationally demanding nature of training even with seemingly small architectures, tying the problem closer to the fixed-parameter complexity classes.
3. Fixed-Parameter Tractability under Convexity: A positive result shows fixed-parameter tractability when considering the combined parameters of dimension and ReLUs, assuming the network computes a convex map. This insight implies that certain constraints or assumptions, such as convexity, significantly contribute to resource-efficient solutions in deep learning model training.

Implications

The findings of this paper have substantial theoretical and practical implications for the domain of neural network training:

• Algorithm Design:

The NP-hardness result directs future endeavors in algorithmic development toward meta-heuristic methods or approximation algorithms when engaging with two-layer network training scenarios particularly under constant dimensions.

• Model Complexity and Real-World Applications:

The facets of computational limits discovered underscore a critical complexity barrier, prompting a reevaluation of model architectures and training methodologies in real-world machine learning applications where dimensional constraints are non-negotiable.

• Future Research Directions:

Research could diverge towards exploring practical approximation or heuristic techniques that can complement the lack of polynomial-time solvability or further extend the results to multi-layer networks and other, non-linear activation functions considering practical and complex real-world constraints.

Summarizing, Froese and Hertrich have substantially enhanced the theoretical landscape regarding the computational complexity involved in training neural networks, particularly under fixed input dimensions. The absence of polynomial-time solutions reinforces the need for innovative approximation strategies or the adoption of assumptions that facilitate feasible computational efforts. This work, therefore, lays a foundational understanding and catalyzes the exploration of alternative models or solutions that can circumvent these inherent computational barriers.