A Global Geometric Analysis of Maximal Coding Rate Reduction

Abstract: The maximal coding rate reduction (MCR$^2$) objective for learning structured and compact deep representations is drawing increasing attention, especially after its recent usage in the derivation of fully explainable and highly effective deep network architectures. However, it lacks a complete theoretical justification: only the properties of its global optima are known, and its global landscape has not been studied. In this work, we give a complete characterization of the properties of all its local and global optima, as well as other types of critical points. Specifically, we show that each (local or global) maximizer of the MCR$^2$ problem corresponds to a low-dimensional, discriminative, and diverse representation, and furthermore, each critical point of the objective is either a local maximizer or a strict saddle point. Such a favorable landscape makes MCR$^2$ a natural choice of objective for learning diverse and discriminative representations via first-order optimization methods. To validate our theoretical findings, we conduct extensive experiments on both synthetic and real data sets.

• The paper rigorously characterizes the local and global optima of the MCR² objective, revealing that local maximizers lie in low-dimensional, orthogonal subspaces while global maximizers ensure maximally diverse representations.
• The study shows every critical point is either a local maximizer or a strict saddle, enabling gradient descent methods to efficiently converge.
• Experimental validations on synthetic and real datasets confirm that features learned via MCR² are compact and discriminative, supporting its use in deep network design.

A Global Geometric Analysis of Maximal Coding Rate Reduction

The paper presents a comprehensive theoretical study of the Maximal Coding Rate Reduction (MCR²) objective, which has been established as an effective method for learning structured and compact deep representations. The authors aim to fill the gap in the theoretical understanding of the MCR² problem by characterizing its local and global optima, as well as analyzing the global optimization landscape of the objective function. This analysis is essential for validating the use of MCR² in the design of deep network architectures and understanding the convergence properties of optimization methods applied to this objective.

Objective and Formulation of MCR²

MCR² aims to learn deep representations that are both compact within classes and discriminative across classes. The objective function is designed to maximize the difference between the coding rate of the entire dataset and the sum of the coding rates within each class. Formally, this can be expressed as:

$$F(\mathbf{Z}) = R(\mathbf{Z}) - \sum_{k=1}^K \frac{m_k}{m} R_c(\mathbf{Z}_k) - \frac{\lambda}{2} \|\mathbf{Z}\|_F^2,$$

where $\mathbf{Z}$ is the matrix of all features, $\mathbf{Z}_k$ are the features of each class, $R(\cdot)$ denotes the coding rate, and $R_c(\cdot)$ is the class-specific coding rate. The parameters $\lambda$ and $\epsilon$ control the regularization and coding precision, respectively.

Theoretical Contributions

Characterizing Local and Global Optima

The authors rigorously derive the conditions under which the local and global optima of the MCR² problem can be explicitly characterized. They demonstrate that:

1. Local Optimizers: Each local maximizer corresponds to features that lie in low-dimensional subspaces, which are orthogonal across different classes. This ensures within-class compactness and between-class discriminability.
2. Global Optimizers: The global optimizers not only satisfy the conditions of local maximizers but also ensure that the total dimension across all classes is maximized. This leads to what the authors term as "maximally diverse representations".

The explicit characterizations confirm that the MCR² objective's optima reflect desirable properties for representation learning in deep networks.

Optimization Landscape Analysis

A significant portion of the paper is dedicated to the analysis of the optimization landscape of the MCR² objective:

1. Strict Saddle Property: The authors prove that every critical point of the MCR² objective is either a local maximizer or a strict saddle point. This property implies that simple gradient-based optimization methods can effectively find local maximizers, as they will almost surely escape saddle points.
2. Implications for Gradient Descent: Given the benign landscape, gradient descent (GD) with random initialization is shown to converge to a local maximizer efficiently, thus making the MCR² a practical objective for training deep networks.

Experimental Validation

The theoretical findings are validated through extensive experiments on both synthetic and real datasets:

1. Synthetic Data: Experiments demonstrate that GD applied to the MCR² objective converges to local or global optima with properties matching the theoretical predictions.
2. Real Data: The use of MCR² in training deep networks shows that the learned features exhibit the compact and discriminative properties elucidated in the theory. Networks trained on MNIST and CIFAR-10 datasets validate that classes are well-separated in the feature space, supporting the theoretical claims.

Practical and Theoretical Implications

The presented analysis of the MCR² objective has several significant implications:

1. Model Interpretability: The theoretical insights into the structure of local and global optima enhance the interpretability of models trained using the MCR² principle.
2. Design of Network Architectures: Understanding the benign optimization landscape supports the use of MCR² in the construction of "white-box" network architectures, such as ReduNet.
3. Future Directions: The paper opens avenues for exploring advanced optimization techniques tailored to the MCR² objective and extending the analysis to more complex settings, such as sparse rate reduction objectives used in transformer-like architectures.

In conclusion, this paper provides a foundational understanding of the MCR² objective, rigorously demonstrating its effectiveness and efficiency for representation learning in deep networks. The results not only validate the empirical success of MCR² but also lay the groundwork for more principled and efficient approaches to optimizing representation learning objectives in practical applications.