Addition is All You Need for Energy-efficient Language Models (2410.00907v2)

Abstract: Large neural networks spend most computation on floating point tensor multiplications. In this work, we find that a floating point multiplier can be approximated by one integer adder with high precision. We propose the linear-complexity multiplication L-Mul algorithm that approximates floating point number multiplication with integer addition operations. The new algorithm costs significantly less computation resource than 8-bit floating point multiplication but achieves higher precision. Compared to 8-bit floating point multiplications, the proposed method achieves higher precision but consumes significantly less bit-level computation. Since multiplying floating point numbers requires substantially higher energy compared to integer addition operations, applying the L-Mul operation in tensor processing hardware can potentially reduce 95% energy cost by element-wise floating point tensor multiplications and 80% energy cost of dot products. We calculated the theoretical error expectation of L-Mul, and evaluated the algorithm on a wide range of textual, visual, and symbolic tasks, including natural language understanding, structural reasoning, mathematics, and commonsense question answering. Our numerical analysis experiments agree with the theoretical error estimation, which indicates that L-Mul with 4-bit mantissa achieves comparable precision as float8_e4m3 multiplications, and L-Mul with 3-bit mantissa outperforms float8_e5m2. Evaluation results on popular benchmarks show that directly applying L-Mul to the attention mechanism is almost lossless. We further show that replacing all floating point multiplications with 3-bit mantissa L-Mul in a transformer model achieves equivalent precision as using float8_e4m3 as accumulation precision in both fine-tuning and inference.

• The paper introduces the L algorithm, which replaces floating-point multiplications with integer additions to significantly reduce energy usage.
• The method uses low-bit mantissa operations to achieve precision comparable to traditional FP8 methods while cutting energy costs in tensor operations.
• Empirical evaluations across transformer attention and various benchmarks confirm minimal performance loss with substantial energy savings.

Addition is All You Need for Energy-Efficient LLMs

The research paper titled "Addition is All You Need for Energy-Efficient LLMs" by Hongyin Luo and Wei Sun introduces a novel algorithm for floating-point multiplication, termed as linear-complexity multiplication (L). This method significantly reduces the computational resources needed for large neural network operations, particularly in the context of energy efficiency.

Core Contribution and Methodology

The primary contribution of this work is the L algorithm, which approximates the multiplication of floating-point numbers using integer addition operations. This algorithm is particularly effective in the reduction of computational overhead compared to traditional 8-bit floating point multiplications (FP8s). Notably, L achieves higher precision while consuming significantly less computational energy. The paper theoretically underpins the accuracy of L, showing that a 4-bit mantissa in L achieves comparable precision to FP8 multiplications (float8_e4m3), and a 3-bit mantissa outperforms float8_e5m2.

In practical terms, the L algorithm leverages simplifications in the mantissa multiplication and bypasses rounding operations—traditionally required in floating-point multiplications. This is achieved through a novel approach to bit-level operations, where signed, exponent, and mantissa calculations are streamlined to integer additions. The efficiency gain is highlighted by the fact that the L algorithm reduces the energy cost of element-wise floating-point tensor multiplications by up to 95% and by 80% in the context of dot products.

Practical and Theoretical Evaluation

The paper undertakes a rigorous theoretical error estimation for the L algorithm, evaluating it on a broad spectrum of tasks and datasets that include natural language understanding, structural reasoning, mathematics, and commonsense question answering. These practical evaluations are compared against theoretical expectations to validate the precision and efficiency claims. The results from these empirical tests align well with theoretical error estimates, affirming the robustness of the L algorithm.

Moreover, the L algorithm is applied within transformer-based models, primarily focusing on the attention mechanism—a significant computational bottleneck in LLMs. The complexity reduction from $O(N^2)$ to linear time operations directly translates to substantial energy savings without compromising model performance.

Implications and Future Directions

The results and methodologies suggested have substantial practical implications. Reducing the computational burden of LLM operations can lead to decreased energy consumption, which is pivotal given the escalating energy demands of AI applications. The computational efficiency brought by the L algorithm can facilitate the deployment of large-scale AI systems in resource-constrained environments, such as edge computing devices.

From a theoretical perspective, this approach opens new pathways for exploring efficient arithmetic implementations within neural networks. Further research can advance by integrating this method into comprehensive AI hardware architectures, potentially enhancing performance in domains reliant on high computational throughput.

Experimental Insights

Several notable findings arise from the experimentation conducted in the paper. For instance, replacing floating point multiplications in attention layers with the L algorithm results in negligible performance loss across varied benchmarks like MMLU, BBH, ARC-C, CSQA, PIQA, OBQA, and SIQA. On average, a performance difference of mere 0.07% is observed compared to standard bf16 precision. Such minimal divergence exemplifies the utility of L in maintaining high model performance while reducing the energy footprint.

Similarly, in the GSM8k benchmark for arithmetic reasoning, the L-based models achieve accuracy improvements over conventional floating-point methods, reinforcing the practical applicability of the L algorithm across different types of tasks.

Vision-Language Tasks and Further Experimentation

The paper also extends experimentation to vision-LLMs using benchmarks like VQAv2, VizWiz, and TextVQA. Here too, the L-based approach performs competently, further substantiating its generality and effectiveness across different domains.

Additionally, a focused paper on the number of mantissa bits in the L operation underscores its efficiency even with significantly fewer bits, making it suitable for low-bit computations without substantial loss in precision. This suggests potential for further exploration into optimized bit-level operations that may yield even higher efficiency.

Conclusion

The "Addition is All You Need" paper posits a significant advancement in energy-efficient AI computations through the introduction of the L algorithm. This not only contributes to theoretical developments in computational arithmetic but also shepherds practical advancements in deploying LLMs more sustainably. Future efforts may include hardware-level implementations of the L algorithm, optimizing it for a variety of applications and potentially setting new standards in efficient AI model computations.