RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality (2112.11081v2)

Abstract: Compared to convolutional layers, fully-connected (FC) layers are better at modeling the long-range dependencies but worse at capturing the local patterns, hence usually less favored for image recognition. In this paper, we propose a methodology, Locality Injection, to incorporate local priors into an FC layer via merging the trained parameters of a parallel conv kernel into the FC kernel. Locality Injection can be viewed as a novel Structural Re-parameterization method since it equivalently converts the structures via transforming the parameters. Based on that, we propose a multi-layer-perceptron (MLP) block named RepMLP Block, which uses three FC layers to extract features, and a novel architecture named RepMLPNet. The hierarchical design distinguishes RepMLPNet from the other concurrently proposed vision MLPs. As it produces feature maps of different levels, it qualifies as a backbone model for downstream tasks like semantic segmentation. Our results reveal that 1) Locality Injection is a general methodology for MLP models; 2) RepMLPNet has favorable accuracy-efficiency trade-off compared to the other MLPs; 3) RepMLPNet is the first MLP that seamlessly transfer to Cityscapes semantic segmentation. The code and models are available at https://github.com/DingXiaoH/RepMLP.

• The paper introduces RepMLPNet, which integrates locality injection into fully-connected layers to capture local patterns for enhanced image recognition.
• The methodology employs a RepMLP Block that fuses convolutional and FC layers, achieving competitive ImageNet performance with reduced training resources.
• The study demonstrates the network’s versatility in tasks like semantic segmentation, paving the way for efficient and robust vision MLP innovations.

An Analysis of RepMLPNet: Advancements in Hierarchical Vision MLP

The paper "RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality" introduces a novel approach to multi-layer perceptron (MLP) architectures for vision tasks, specifically by proposing the RepMLPNet. This architecture leverages a novel methodology termed Locality Injection, aimed at merging the advantages of convolutional layers and fully-connected (FC) layers for improved image recognition performance.

The core motivation behind this work is to address the inherent limitations of FC layers in capturing local patterns, which are typically well-managed by convolutional layers due to their local receptive fields. By innovatively incorporating local priors into FC layers through the structural re-parameterization method known as Locality Injection, the paper suggests that it is possible to create a more balanced approach that capitalizes on the long-range modeling capabilities of FC layers without forsaking the locality strength of convolutional networks.

Central to the architecture is the RepMLP Block, a key component of the proposed RepMLPNet. This block uses a hierarchical design, extracting features with three FC layers, combined with Locality Injection, which fuses conv layers' parameters into FC layers. Consequently, RepMLPNet is distinguished from other MLPs by its ability to operate as a backbone model for downstream tasks, such as semantic segmentation, due to its capacity to produce feature maps at multiple levels.

In terms of empirical results, RepMLPNet demonstrates a favorable accuracy-efficiency trade-off on the ImageNet benchmark when compared to other MLP models. The architecture achieves competitive performance with fewer training resources, significantly less training time, and an enhanced balance of accuracy against computational cost. Notable is the demonstration of RepMLPNet's seamless transferability to complex tasks like Cityscapes semantic segmentation, a significant milestone for vision-centric MLP models.

The structural re-parameterization, specifically Locality Injection, is shown to be a versatile tool not limited to the proposed architecture. The work establishes that this methodology can enhance other MLP models by addressing their typical dependency on large datasets and extensive training epochs to learn effective inductive biases. For instance, when applied to ResMLP, Locality Injection results in performance improvements with marginal parameter overhead.

From a theoretical standpoint, the implications include the potential for extending the design space of MLP architectures for vision tasks, further generalizing the applicability of structural re-parameterization strategies within neural networks. Practically, the development of RepMLPNet pushes the boundary for applying MLPs to high-resolution visual tasks, previously a domain dominated by CNNs and Transformers. Moreover, this work provides a pathway for further innovations in designing efficient neural network architectures that balance computational efficiency with model performance.

In conclusion, the paper provides substantial advancements in the field of vision MLP architectures and presents compelling empirical evidence supporting the benefits of integrating locality into fully-connected layers. Future advancements may involve extending these principles to more specialized vision tasks or exploring more profound synergistic designs between MLP and convolutional structures for ubiquitous AI applications.