Adaptive Dropout for Pruning Conformers

Abstract: This paper proposes a method to effectively perform joint training-and-pruning based on adaptive dropout layers with unit-wise retention probabilities. The proposed method is based on the estimation of a unit-wise retention probability in a dropout layer. A unit that is estimated to have a small retention probability can be considered to be prunable. The retention probability of the unit is estimated using back-propagation and the Gumbel-Softmax technique. This pruning method is applied at several application points in Conformers such that the effective number of parameters can be significantly reduced. Specifically, adaptive dropout layers are introduced in three locations in each Conformer block: (a) the hidden layer of the feed-forward-net component, (b) the query vectors and the value vectors of the self-attention component, and (c) the input vectors of the LConv component. The proposed method is evaluated by conducting a speech recognition experiment on the LibriSpeech task. It was shown that this approach could simultaneously achieve a parameter reduction and accuracy improvement. The word error rates improved by approx 1% while reducing the number of parameters by 54%.

• The paper demonstrates that adaptive dropout, using Gumbel-Softmax for retention probability estimation, prunes over 50% of parameters while enhancing performance.
• It employs adaptive dropout layers at key points in the Conformer model to enable simultaneous training and pruning, ensuring smoother convergence.
• This innovative method facilitates efficient speech recognition on resource-constrained devices and sets the stage for future research on dynamic model reconfiguration.

A Formal Analysis of Adaptive Dropout for Pruning Conformers

The paper "Adaptive Dropout for Pruning Conformers" presents a strategy for reducing the parameter count in Conformer-based speech recognition models through a novel adaptive dropout approach. The methodology described leverages adaptive dropout layers characterized by unit-wise retention probabilities for the dual purpose of training and pruning models concurrently. The underlying hypothesis is that units exhibiting lower retention probabilities can be considered redundant and are, thus, candidates for pruning.

The study introduces adaptive dropout layers at strategic points within the Conformer architecture, specifically targeting the hidden layer of the feed-forward network, query and value vectors of the self-attention component, and the input vectors of the LConv component. By integrating these layers, the model achieves a significant reduction in parameter count. The empirical evaluations, conducted on the LibriSpeech dataset, indicate that the proposed method achieves a 54% reduction in parameters while concurrently improving word error rates (WER) by approximately 1%.

In theoretical terms, the approach diverges from traditional methods by estimating retention probabilities using the Gumbel-Softmax technique and employing controlled regularization to dynamically adjust these probabilities during the training process. This addresses typical issues such as premature hard pruning decisions and ensures a smoother convergence process.

Numerical Results and Analysis

The experimental results as presented indicate compelling numerical success. The model achieves a recognition accuracy improvement on both ‘clean’ and ‘other’ test datasets compared to a baseline Conformer model with 111 million parameters, despite using only 50.1 million parameters post-pruning. Notably, other handcrafted compact models, such as shallower or narrower variants, generally showed increased WER compared to both the baseline and the proposed method. This underscores the efficacy of the adaptive dropout layers in maintaining or even enhancing model performance while significantly reducing computational complexity.

Practical Implications and Theoretical Insights

From a practical standpoint, the application of adaptive dropout layers could potentially influence the deployment of speech recognition architectures across devices with varying computational constraints. If further refined, this method might enable real-time processing on devices that were previously unsuited to handle such tasks due to hardware limitations.

Theoretically, the integration of adaptive dropout and unit-level pruning strategies could redefine best practices for model architecture optimization, providing insights into more effective utilization and allocation of computational resources within neural networks. The method also opens avenues for further research into dynamic model reconfiguration during training, which could influence broader applications beyond speech recognition.

Future Directions

Looking forward, the adaptive dropout method may be leveraged in exploring unsupervised or self-supervised tasks, especially in models where over-parameterization is an inherent characteristic. Moreover, the symbiosis between dropout and pruning—further reinforced by empirical outcomes—invites exploration into more complex architectures beyond the Conformer and could potentially set a precedent for adaptive neural network design.

In conclusion, the paper delineates an efficacious approach to pruning Conformer architectures, delivering clear performance improvements alongside a substantial reduction in model size. These insights are likely to cultivate a foundation for ongoing research in model efficiency and performance trade-offs, particularly in the field of neural architecture optimization.