EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation (2202.07959v3)

Abstract: We introduce EdgeFormer -- a parameter-efficient Transformer for on-device seq2seq generation under the strict computation and memory constraints. Compared with the previous parameter-efficient Transformers, EdgeFormer applies two novel principles for cost-effective parameterization, allowing it to perform better given the same parameter budget; moreover, EdgeFormer is further enhanced by layer adaptation innovation that is proposed for improving the network with shared layers. Extensive experiments show EdgeFormer can effectively outperform previous parameter-efficient Transformer baselines and achieve competitive results under both the computation and memory constraints. Given the promising results, we release EdgeLM -- the pretrained version of EdgeFormer, which is the first publicly available pretrained on-device seq2seq model that can be easily fine-tuned for seq2seq tasks with strong results, facilitating on-device seq2seq generation in practice.

• The paper introduces a novel encoder-favored Transformer that prioritizes encoder parameters to significantly enhance seq2seq performance on resource-constrained devices.
• It employs load-balanced, adaptive layers like Bias-LA, Adapter-LA, and Prefix-LA to optimize computational efficiency while mitigating the limitations of shared weighting.
• EdgeFormer sets a new benchmark by outperforming existing efficient models under strict limits of 2G FLOPS and 10 million parameters for on-device NLP.

EdgeFormer: Transforming On-Device Seq2Seq Efficiency

The paper, "EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation," presents a novel Transformer architecture designed to surmount the formidable challenges accompanying on-device sequence-to-sequence (seq2seq) tasks. By balancing parameter efficiency with computational and memory constraints, EdgeFormer sets the stage for effective on-device NLP applications without compromising performance.

The impetus for this research is evident: the prevalence of edge devices necessitates compact, efficient models that do not overtax limited computational and memory resources. EdgeFormer's innovation is rooted in two core principles—encoder-favored and load-balanced parameterization—paired with layer adaptation techniques. Notably, EdgeFormer follows the standard Transformer architecture, comprising a deep (12-layer) encoder and a shallow (2-layer) decoder, optimized for on-device environments.

Key Innovations and Methodology

1. Cost-Effective Parameterization: EdgeFormer emphasizes encoder-favored parameterization, directing most of the parameter budget to the encoder, thereby improving performance. The paper's analysis demonstrates that encoder parameterization yields more substantial gains than investing heavily in the decoder. Moreover, the interleaved decoder structure allows for shared parameterization across attention modules, optimizing the use of parameters while maintaining a structurally consistent network that's conducive to shared weighting strategies.
2. Layer Adaptation Approaches: To counteract the potential oversights of tied weights—where shared parameters might limit model specialization—the research integrates layer adaptation. This enhancement takes the form of Bias-LA, Adapter-LA, and Prefix-LA, each varying in complexity and adaptiveness while adding negligible computational overhead. Adapter-LA employs LoRA-style low-rank adaptation, whereas Prefix-LA introduces trainable prefix tokens, both significantly boosting task performance.
3. Performance Under Constraints: EdgeFormer is evaluated under strict resource constraints imposed for on-device inference: a computational ceiling of 2G FLOPS and a memory budget capped at 10 million model parameters. The model outperformed Universal Transformers and other parameter-efficient counterparts like DeLighT and Shapeshifter, setting a new benchmark for quality under given computational budgets. Through int8 quantization and vocabulary optimizations, EdgeFormer meets runtime requirements and maintains high performance with reduced resource demand.

Implications and Future Directions

The paper's contributions hold substantial practical and theoretical implications. Practically, EdgeFormer enables more efficient NLP deployment on edge devices, facilitating real-world applications such as real-time translation, personal assistants, and error correction on resource-constrained hardware. Theoretically, this research pushes forward the dialogue on optimal parameter utilization and efficient layer configuration in Transformers, questioning existing norms in neural network design.

Future advancements could explore differentiated load balancing, exploring how distinct model parameters might inherently benefit from variable reuse frequencies or adaptation techniques. Such insights could further refine our understanding of Transformer architectures and spur innovations in both model efficiency and effectiveness.

In releasing EdgeLM, the pre-trained version of EdgeFormer, the authors provide a significant resource to the community, underscoring the model's broad applicability to various seq2seq tasks. This release promises to stimulate further research and practical adaptations in on-device seq2seq modeling, reinforcing the integral role of hardware-aware innovations in NLP's ongoing evolution.