Complexity of Linear Regions in Self-supervised Deep ReLU Networks

Abstract: There has been growing interest in studying the complexity of Rectified Linear Unit (ReLU) based activation networks. Recent work investigates the evolution of the number of piecewise-linear partitions (linear regions) that are formed during training. However, current research is limited to examining the complexity of models trained in a supervised way. Self-Supervised Learning (SSL) differs in that it directly optimises the representation space using a loss function to enhance the model's performance across multiple downstream tasks. This study investigates the local distribution of linear regions produced by SSL models. We demonstrate that the evolution of linear regions correlates with the representation quality by utilising SplineCam to extract two-dimensional polytopes near the data distribution. We track the number, area, eccentricity, and boundaries of regions throughout training. The study compares supervised, contrastive, and self-distillation methods over two standard benchmark datasets, MNIST and FashionMNIST. The analysis of the experimental results shows that self-supervised methods create substantially fewer regions to achieve comparable accuracy to supervised models. Contrastive methods rapidly expand regions over time, whereas self-distillation methods tend to consolidate by merging neighbouring regions. Lastly, we can detect representation collapse early within the geometric space of linear regions. Our analysis suggests that polytopal metrics can serve as reliable indicators of representation quality and model performance.

• The paper reveals that self-supervised networks consolidate linear regions, achieving comparable accuracy with fewer partitions than supervised methods.
• It employs a geometric framework with SplineCam to measure volume, eccentricity, and boundary count, providing precise characterization of network representation.
• The study demonstrates early detection of representation collapse by monitoring rapid declines in region count before conventional feature metrics change.

Complexity of Linear Regions in Self-supervised Deep ReLU Networks

Motivation and Background

The piecewise-linear partitioning induced by ReLU activations is central to understanding the complexity and generalization capacity of deep neural networks. Prior research has focused on supervised networks, establishing the relationship between the number of linear regions and network expressivity. Self-supervised learning (SSL), which has gained prominence due to its efficacy in obtaining transferable representations, optimizes the representation space without explicit labels, raising open questions about how SSL objectives shape the geometric partitioning of the input space.

Geometric Framework and Definitions

The study leverages SplineCam for exact extraction and characterization of linear regions generated by deep ReLU networks. Each region is defined by a unique activation pattern across neurons, resulting in a convex polytope where the network implements a single affine transformation. The geometric properties—volume, eccentricity, and boundary count—are measured to provide insight into the structural behavior of representation spaces for different learning paradigms.

Figure 1: Illustration of the partitioning mechanism of ReLU networks in the input space and the emergence of linear regions.

Volume quantifies the spatial dominance of a region; eccentricity measures anisotropy, reflecting the proximity to activation boundaries; and boundary count signals the complexity of the partitioning. These geometric metrics serve as proxies for robustness and generalization.

Figure 2: Each linear region is characterized by geometric descriptors essential for evaluating representation quality.

Experimental Analysis: Supervised vs. Self-supervised Partitioning

The experiments span both two-dimensional synthetic data and canonical high-dimensional datasets (MNIST, FashionMNIST). For supervised learning, networks with contrastive and classification objectives form denser, highly partitioned spaces, evidenced by large region counts and smaller, more eccentric regions. In SSL, contrastive methods (SimCLR, MoCo) still maintain partitioning pressure but exhibit reduced complexity, while self-distillation methods (SimSiam, BYOL) consolidate regions, yielding fewer, larger, and more isotropic partitions.

Figure 3: Self-supervised methods produce fewer linear regions than supervised counterparts while achieving comparable accuracy across the training epochs.

Figure 4: Density of linear regions near the data manifold ('moons' dataset) visualizes distinct partitioning between methods.

Notably, contrastive self-supervised objectives, which utilize negative samples, force finer partitioning, creating smaller regions with greater density near the data distribution. Conversely, self-distillation methods rapidly merge regions due to the absence of these repulsive constraints, increasing the risk of representation collapse.

Evolution of Region Metrics Through Training

Region count trajectories demonstrate that supervised methods exhibit early rapid partitioning followed by stabilization and merging, matching the data distribution more finely over time. SSL methods, particularly self-distillation, show early region contraction then stalling complexity growth, further reinforcing their tendency for consolidation. The geometric distributions of area, eccentricity, and boundary count evolve over epochs, signaling underlying representation dynamics.

Figure 5: Distributions of area, eccentricity, and boundary count across training epochs for multiple methods on MNIST.

Figure 6: Comparable geometric evolution for FashionMNIST, confirming qualitative trends across benchmarks.

SSL models are empirically shown to require fewer regions for similar accuracy, with the supervised methods achieving 0.8557 accuracy using 11,349 regions (SupCon) while SimCLR reaches 0.8351 with only 4,297. The reduction in region complexity correlates with increased generalization and robustness, as extensively documented in related work.

Early Detection of Representation Collapse via Geometric Metrics

The study provides a strong claim that representation collapse is detectable earlier in the geometric space than in standard feature-space metrics. Simulation of collapse (removal of prediction head in SimSiam) reveals that region count drops precipitously well before feature standard deviation is significantly affected, offering a more sensitive diagnostic for collapse phenomena in SSL regimes.

Figure 7: Region count declines faster than representation standard deviation, indicating early detection of collapse.

Figure 8: Visualization of region evolution during collapse; the network transitions from dense partitioning to merged regions.

Final Geometry and Partitioning Across Methods

At completion (epoch 100), supervised models remain densely partitioned, while self-supervised contrastive methods exhibit somewhat finer partitioning than self-distillation, which produces fewer, voluminous regions. The geometric space, visualized on both MNIST and FashionMNIST, displays consistent trends: supervision drives denser, smaller regions; self-supervision, especially self-distillation, fosters consolidation and isotropy.

Figure 9: Supervised models (MNIST) form highly dense regions; SSL models consolidate into larger volumes.

Figure 10: SSL and supervised trends on FashionMNIST replicate MNIST findings.

Implications and Prospective Directions

The geometric analysis of linear regions extends the toolkit for evaluating representation quality, offering direct metrics that correlate with performance, robustness, and generalization. The identification of early collapse via region count opens new avenues for regularization and architectural design, allowing practitioners to optimize the polytopal space directly.

Practically, reduced region complexity portends improved transferability and out-of-distribution sensitivity. Theoretically, the findings reinforce the notion that geometric structure, rather than sheer expressivity, is foundational for effective representation learning. For continual learning, methods that modulate region partitioning—transitioning from consolidation (self-distillation) to expansion (contrastive)—appear promising for balancing capacity and stability.

Conclusion

This study rigorously explores the geometric complexity of linear regions in self-supervised ReLU networks. SSL, and particularly self-distillation, achieves comparable accuracy with substantially fewer linear regions, demonstrating efficient partitioning. Geometric diagnostics provide robust tools for monitoring representation quality and detecting collapse. Future research should explore direct optimization of polytopal metrics and their integration into regularization frameworks to further enhance robustness and transferability of deep representations.