Generalized Orders of Magnitude for Scalable, Parallel, High-Dynamic-Range Computation

Abstract: Many domains, from deep learning to finance, require compounding real numbers over long sequences, often leading to catastrophic numerical underflow or overflow. We introduce generalized orders of magnitude (GOOMs), a principled extension of traditional orders of magnitude that incorporates floating-point numbers as a special case, and which in practice enables stable computation over significantly larger dynamic ranges of real numbers than previously possible. We implement GOOMs, along with an efficient custom parallel prefix scan, to support native execution on parallel hardware such as GPUs. We demonstrate that our implementation of GOOMs outperforms traditional approaches with three representative experiments, all of which were previously considered impractical or impossible, and now become possible and practical: (1) compounding real matrix products far beyond standard floating-point limits; (2) estimating spectra of Lyapunov exponents in parallel, orders of magnitude faster than with previous methods, applying a novel selective-resetting method to prevent state colinearity; and (3) capturing long-range dependencies in deep recurrent neural networks with non-diagonal recurrent states, computed in parallel via a prefix scan, without requiring any form of stabilization. Our results show that our implementation of GOOMs, combined with efficient parallel scanning, offers a scalable and numerically robust alternative to conventional floating-point numbers for high-dynamic-range applications.

• The paper introduces GOOMs, leveraging complex logarithms to transform multiplication into addition, ensuring stable matrix multiplications over vast dynamic ranges.
• Using the LMME algorithm, GOOMs enable parallel, scalable computations with enhanced precision, outperforming traditional floating-point methods.
• Experiments demonstrate that GOOMs effectively compute long matrix product chains and Lyapunov exponents, and they facilitate stable RNN training without conventional stabilization.

Generalized Orders of Magnitude for Scalable, Parallel, High-Dynamic-Range Computation

Introduction

The paper introduces Generalized Orders of Magnitude (GOOMs) as an enhancement to conventional numerical systems, aimed at stable computation over expansive dynamic ranges. This is crucial for numerous domains such as deep learning, where sequences of compounding real numbers can lead to significant numerical errors. GOOMs extend traditional floating-point numbers, encapsulating them as a special case, thereby promising stability over larger dynamic ranges by leveraging complex logarithms.

Core Concept: Generalized Orders of Magnitude

GOOMs are defined as complex logarithms of real numbers, designed to represent exponential magnitudes stably. The extension simplifies complex operations such as matrix multiplications and scalar products by transforming multiplication into addition, inherently enhancing numerical stability. The paper formalizes GOOMs as elements of a set that involves complex logarithms which exponentiate to real numbers, addressing limitations of existing numerical representations.

Implementation

GOOMs are implemented for the PyTorch framework, utilizing complex data types like Complex64 and Complex128 to manage greater dynamic ranges than existing floating-point formats. This implementation facilitates computations that exceed the precision and range limits of traditional Float32 and Float64, offering an "edifice of GOOMs" that extends beyond these constraints.

• Architecture: GOOMs are realized as composite data types with existing floating-point formats for real and imaginary components, providing increased precision at lower magnitudes and greater dynamic ranges.
• LMME: The Log-Matrix-Multiplication-Exp algorithm (LMME) is introduced, transforming matrix multiplication into a log-sum-exp operation over complex-typed GOOMs, optimized for parallel execution.

Experiments and Results

Three primary experiments showcase the capabilities of GOOMs:

1. Matrix Product Chains:

GOOMs compute long chains of matrix products that traditional floats cannot handle due to catastrophic numerical errors. GOOMs maintained stability across one million steps without errors, demonstrating their increased dynamic range and precision.

Figure 1: Longest chain multiplying random normal square matrices without catastrophic numerical error on an Nvidia GPU.

1. Parallel Lyapunov Exponents: Using a novel parallel prefix scan method, GOOMs compute accurate spectra of Lyapunov exponents orders of magnitude faster than existing sequential methods, crucial for analyzing the stability of dynamical systems.
2. RNNs without Stabilization: A deep recurrent neural network leveraging non-diagonal recurrences computed in parallel using GOOMs shows successful training and task performance without conventional stabilization techniques, highlighting GOOMs' potential for advancing neural network architectures.

Implications and Future Directions

GOOMs provide a software-based solution to numerical instability, offering enhanced scalability and precision for complex applications in science and engineering. Their integration within existing platforms like PyTorch ensures compatibility and ease of adoption, suggesting broad applicability from deep learning to other high-performance computing domains.

Future work could explore optimized kernel developments for GOOM operations and investigate their application across additional AI models and complex scientific computations, potentially transforming numerical computing paradigms.

Conclusion

The paper positions GOOMs as a vital advancement in numerical computation, mitigating limitations inherent in traditional floating-point representations. Through robust implementation and testing within cutting-edge computational paradigms, GOOMs offer a framework for stable, high-precision calculations over previously inaccessible dynamic ranges.