Dynamic Neural Networks: A Survey (2102.04906v4)

Abstract: Dynamic neural network is an emerging research topic in deep learning. Compared to static models which have fixed computational graphs and parameters at the inference stage, dynamic networks can adapt their structures or parameters to different inputs, leading to notable advantages in terms of accuracy, computational efficiency, adaptiveness, etc. In this survey, we comprehensively review this rapidly developing area by dividing dynamic networks into three main categories: 1) instance-wise dynamic models that process each instance with data-dependent architectures or parameters; 2) spatial-wise dynamic networks that conduct adaptive computation with respect to different spatial locations of image data and 3) temporal-wise dynamic models that perform adaptive inference along the temporal dimension for sequential data such as videos and texts. The important research problems of dynamic networks, e.g., architecture design, decision making scheme, optimization technique and applications, are reviewed systematically. Finally, we discuss the open problems in this field together with interesting future research directions.

• The paper introduces a comprehensive categorization of dynamic neural networks into sample-wise, spatial-wise, and temporal-wise types to boost accuracy and efficiency.
• The paper employs adaptive architectural adjustments and input-conditioned parameter modifications to enhance representation power and resource allocation.
• The paper discusses decision-making strategies and reinforcement learning approaches to overcome non-differentiable challenges in dynamic network training.

Dynamic Neural Networks: A Comprehensive Survey

Dynamic neural networks have emerged as a significant area of research within deep learning, offering adaptability in computational structures and weights in response to varying inputs. This paper provides an extensive overview of dynamic networks, categorizing them into three primary forms: sample-wise, spatial-wise, and temporal-wise dynamics. These networks differ from static models, which maintain fixed architectures and parameters, striving to enhance accuracy, computational efficiency, and adaptability.

Categories of Dynamic Networks

• Sample-wise Dynamic Networks: These networks adapt architectures or parameters per input sample. The paper delineates between models with dynamic architectures—adjusting network depth, width, or routing within SuperNets—and those with dynamic parameters, employing input-conditioned adjustments or predictions of network weights. Results highlight improved representation power and efficiency, achieved through methods like dynamic depth and width adaptations.
• Spatial-wise Dynamic Networks: Focused on image data, these networks perform adaptive computations across different spatial locations. They operate at varying granularity, from pixel-level dynamic architectures and parameter adjustments to region-level dynamic transformations and hard attention, finally addressing resolution-level adaptations. The research demonstrates significantly reduced computational redundancy while maintaining prediction accuracy.
• Temporal-wise Dynamic Networks: These models address sequence data like texts and videos, adapting computational efforts based on temporal context. Techniques such as dynamic hidden state updates, early exits, and temporal frame skipping contribute to improved efficiency in processing sequential data, notably improving RNN and video recognition tasks.

Decision Making and Training

The paper discusses various decision-making strategies in dynamic networks, such as confidence-based criteria and the application of policy networks or gating functions for adaptive computation. However, the non-differentiable nature of these decisions necessitates specific training strategies, including the use of reinforcement learning and reparameterization methods. Furthermore, dynamic networks are often trained with additional objectives that balance task performance with computational efficiency.

Applications and Potential

Dynamic networks find applications across numerous fields, including computer vision tasks such as object detection and image segmentation, natural language processing, and video recognition. The adaptability of these networks introduces potential for significant advancements in efficiency and computational resource allocation.

Challenges and Future Directions

Despite the progress, several challenges remain. The design of theoretically justified decision mechanisms and robust training methods to mitigate overconfidence are pertinent areas for future research. Additionally, architecture design tailored for dynamic adaptiveness calls for more innovative ideas. Furthermore, the gap between theoretical and practical computational efficiency poses implementation challenges that need addressing in future hardware and algorithmic designs.

In conclusion, dynamic neural networks represent a promising evolution in deep learning architectures with their versatile and efficient approach to input-dependent inference. However, further explorations into theoretical foundations, architectural innovation, and practical implementation are essential to harness their full potential.