Are we ready for a new paradigm shift? A Survey on Visual Deep MLP (2111.04060v6)

Abstract: Recently, the proposed deep MLP models have stirred up a lot of interest in the vision community. Historically, the availability of larger datasets combined with increased computing capacity leads to paradigm shifts. This review paper provides detailed discussions on whether MLP can be a new paradigm for computer vision. We compare the intrinsic connections and differences between convolution, self-attention mechanism, and Token-mixing MLP in detail. Advantages and limitations of Token-mixing MLP are provided, followed by careful analysis of recent MLP-like variants, from module design to network architecture, and their applications. In the GPU era, the locally and globally weighted summations are the current mainstreams, represented by the convolution and self-attention mechanism, as well as MLP. We suggest the further development of paradigm to be considered alongside the next-generation computing devices.

• The paper presents an extensive survey highlighting the evolution and potential of visual deep MLP architectures compared to CNNs and Transformers.
• It examines modular design innovations like axial and local window token-mixing to address high computational costs and resolution challenges.
• Empirical insights reveal that while MLP-based models are promising, their success hinges on optimized pre-training and advancements in computing hardware.

Survey on Visual Deep MLP as a New Paradigm in Computer Vision

The paper "Are we ready for a new paradigm shift? A Survey on Visual Deep MLP" authored by Liu et al. provides a comprehensive overview of the evolution of neural network architectures in computer vision, with a focus on the potential shift towards Multilayer Perceptrons (MLPs) as foundational blocks. With deep learning bridging the gap between theoretical and practical applications, this paper highlights the intricate balance between computational efficiency, data availability, and architectural innovation that has driven paradigm shifts in the field of computer vision from CNNs through Vision Transformers to potential MLP-based architectures.

Historical Context and Motivation

The historical perspective presented in the paper traces the trajectory from early neural networks like the perceptron and Boltzmann machine to the widespread adoption of CNNs in the 2010s, driven by their computational feasibility and impressive performance in automating feature extraction. The introduction of Transformer models to vision tasks extended this evolution by leveraging global attention mechanisms, marking another significant paradigm shift. In this continuum, the recent interest in Token-mixing deep MLP models like MLP-Mixer proposes yet another shift, raising the question of whether MLPs can indeed serve as the next dominant paradigm in the computer vision landscape.

Analysis of MLP Models and Novel Modular Designs

The core of the paper dissects the intrinsic differences between CNNs, Transformers, and proposed MLP architectures, specifically Token-mixing MLPs. The authors compare these approaches based on their ability to aggregate spatial information, computational complexity, and sensitivity to input resolution. They outline the challenges faced by MLP architectures—such as high computational cost and the risk of overfitting due to large parameter spaces—in achieving performance parity with CNNs and Transformers in both data-efficient and resolution-agnostic formats.

In response to these challenges, the paper reviews various MLP-like variants that have emerged, detailing innovations such as the axial and local window-based decomposition of Token-mixing operations to manage computational complexity and enable flexible input resolutions. These variations draw from CNN-like architectures to incorporate desirable properties such as local receptive fields while also attempting to retain global information capture capabilities analogous to Transformers.

Performance and Applicability in Visual Tasks

The empirical evaluation includes performance benchmarks across various tasks such as image classification on ImageNet, highlighting a gap in performance compared to state-of-the-art CNN and Transformer models. While MLP-based variants have started to bridge this gap, the authors note the critical importance of self-supervised pre-training and task-specific optimizations that could play pivotal roles in enhancing MLP utility across other areas such as object detection, semantic segmentation, and low-level vision tasks.

Implications and Future Directions

The paper's discussions suggest cautious optimism regarding the potential of deep MLPs as a new paradigm. The authors advocate for continued exploration in tailored architectural designs that integrate global and local information processing efficiently, as well as for advances in MLP-specific pre-training and optimization techniques. The synthesis of these factors is essential for facilitating effective learning and scalably deploying MLP-based networks across broader visual tasks. Furthermore, given the computational demands, the development of next-generation computing hardware could also catalyze the widespread adoption of such architectures.

In summary, while the paper provides a rigorous evaluation of the present landscape and potential trajectory for MLPs in vision tasks, it underscores the need for innovative algorithmic strategies and technological advancements to realize this theoretical potential practically, highlighting a sustained interplay between architecture and hardware evolution.