AutoNumerics-Zero: Automated Discovery of State-of-the-Art Mathematical Functions (2312.08472v1)

Abstract: Computers calculate transcendental functions by approximating them through the composition of a few limited-precision instructions. For example, an exponential can be calculated with a Taylor series. These approximation methods were developed over the centuries by mathematicians, who emphasized the attainability of arbitrary precision. Computers, however, operate on few limited precision types, such as the popular float32. In this study, we show that when aiming for limited precision, existing approximation methods can be outperformed by programs automatically discovered from scratch by a simple evolutionary algorithm. In particular, over real numbers, our method can approximate the exponential function reaching orders of magnitude more precision for a given number of operations when compared to previous approaches. More practically, over float32 numbers and constrained to less than 1 ULP of error, the same method attains a speedup over baselines by generating code that triggers better XLA/LLVM compilation paths. In other words, in both cases, evolution searched a vast space of possible programs, without knowledge of mathematics, to discover previously unknown optimized approximations to high precision, for the first time. We also give evidence that these results extend beyond the exponential. The ubiquity of transcendental functions suggests that our method has the potential to reduce the cost of scientific computing applications.

• The paper's main contribution is the development of AutoNumerics-Zero, which evolves highly precise and efficient approximations of transcendental functions using an evolutionary algorithm.
• It employs symbolic regression and CMA-ES optimization to fine-tune floating-point computations, significantly outperforming traditional methods in both precision and speed.
• The approach reduces computational costs and demonstrates potential for general application, extending beyond the exponential function to other mathematical operations.

Overview

In the field of computational science, evaluating transcendental functions with high precision and efficiency is crucial. This paper introduces a method that automates the discovery of optimized mathematical function approximations. Through an evolutionary algorithm known as AutoNumerics-Zero, the paper demonstrates the ability to outperform classical methods developed by mathematicians in approximating transcendental functions, particularly focusing on the exponential function.

Evolutionary Approach to Function Approximation

The core idea is to utilize a simple evolutionary algorithm to derive programs that approximate functions with a predefined precision in both real-number and floating-point (float32) computations. The evolutionary algorithm begins with no prior mathematical knowledge and evolves from scratch, using basic computational operations permissible on contemporary hardware. This process contrasts starkly with traditional mathematical methods that require deep theoretical insights and aim for high precision using advanced theory.

AutoNumerics-Zero Algorithm

AutoNumerics-Zero harnesses symbolic regression, starting from a blank state, to optimize programs in terms of precision and computational efficiency, such as operation count and execution speed. Unlike traditional approaches, this technique does not just evolve the program structure symbolically but also fine-tunes the floating-point coefficients using Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a strategy that avoids assuming differentiable real-valued arithmetic which is untrue for float32 numbers.

Results and Contributions

The paper's key results showcase that evolving programmatic approximations can achieve far greater precision in mathematical calculations on both real numbers and for float32 computations. For the exponential function, which is a well-studied mathematical operation, the evolutionary search managed to discover programs that were significantly more precise and, in practical float32 computations, also considerably faster than any known baseline method. The remarkable efficiencies obtained are attributed to the generation of code that takes advantage of compiler and hardware characteristics imperceptible to conventional mathematical techniques.

Implications

The ability of AutoNumerics-Zero to outperform existing methods in approximating transcendental functions, and its success in handling both precision and speed, highlights a significant potential for reducing the computational costs in scientific applications. Additionally, while this paper primarily focused on the exponential function, preliminary evidence suggests the method's effectiveness may extend to other functions as well, indicating a wide range of applicability for the proposed algorithm.