Connectivity of loss-landscape features in the full parameter space for the XOR network

Determine the connectivity structure among loss-landscape features—specifically wells, channels, trenches, barriers, plateaus, and rims—surrounding the zero-loss solution in the full nine-dimensional parameter space of weights and biases for the XOR network with sigmoid activation and two hidden neurons (two inputs and one output). Ascertain to what extent and in what manner these features connect in the complete parameter space beyond two-parameter cross-sections.

Background

The paper analyzes the loss landscape of the minimal XOR neural network (two inputs, two hidden neurons with sigmoid activations, and one output), presenting numerous two-parameter cross-sections that display distinct geometric features such as wells, channels, trenches, barriers, plateaus, and rims near the zero-loss solution.

While the cross-sections reveal complex local structures and their influence on backpropagation dynamics, the global arrangement of these features in the full nine-dimensional space of weights and biases remains uncharted. The authors explicitly defer a paper of how these features are connected in the complete high-dimensional landscape, identifying it as future work.