---
title: Integer-Only Softmax Implementation
url: https://www.emergentmind.com/topics/integer-only-softmax-implementation
type: topic
---

# Integer-Only Softmax Implementation

Integer-only softmax refers to implementations of the softmax nonlinearity in which all arithmetic—max finding, subtractions, exponentiations, and normalization—is performed strictly with integer-valued operations, with no floating-point arithmetic, transcendental functions, or datatype conversions in the inference path. This paradigm is central to fully quantized transformer and vision models, particularly at low bit-widths, as it removes the floating-point bottleneck for non-linearities and enables deployment on highly efficient integer hardware such as edge CPUs, microcontrollers, FPGAs, and custom ASICs. Multiple research groups have proposed such techniques for both software and hardware domains, yielding significant gains in throughput, energy efficiency, and area while maintaining high fidelity.

## 1. Mathematical Principles and Integer Reformulation

The standard floating-point softmax over a vector $x=(x_1,\dots,x_d)$ is defined as:
\[
\mathrm{Softmax}(x)_i = \frac{e^{x_i}}{\sum_{j=1}^d e^{x_j}}
\]
For numerical stability, implementations apply max-subtraction:
\[
\mathrm{Softmax}(x)_i = \frac{e^{x_i-x_{\max}}}{\sum_{j=1}^d e^{x_j-x_{\max}}},\quad x_{\max}=\max_j x_j
\]
The integer-only reformulation quantizes $x$ to an integer tensor $q_x$ via scale $S$, optionally zero-point $z_x$, so $x\approx S \cdot q_x + z_x$ [2101.01321], [2511.21513], [2411.17847], [2405.17849], [2511.15369]. The exponentials are then approximated with bit-shifting, polynomials, lookup tables, or power-of-two logic:
- **Base replacement**: $e^{y}$ is replaced with $2^{y/\ln2}$ or similar, enabling use of integer shifts instead of evaluting $e^x$.
- **Piecewise or polynomial approximation**: Over a restricted interval (typ. $(-\ln2,0]$), $\exp(p)$ is approximated by a degree-2 polynomial (e.g., $a(p+b)^2+c$) [2101.01321], [2411.17847].
- **Bit-shifting or power-of-two**: When the scale is chosen to map each integer change to a power-of-two step, the "exponential" becomes a right-shift [2307.03493], [2103.09301].
- **LUT-based**: Lookup tables precalculate exponential values for small quantized ranges [2511.21513].

The normalization (division by sum) is then accomplished by integer division, sometimes accelerated with reciprocal LUTs or fixed-point reciprocals.

## 2. Integer-Only Softmax Algorithms and Key Implementations

Multiple families of integer-only softmax exist, all sharing the properties above but differing in the approximation and normalization. The following table summarizes core methods, listed in the literature:

| Approach              | Exponential Approximation                | Normalization Technique         |
|-----------------------|------------------------------------------|---------------------------------|
| I-BERT [2101.01321]   | Degree-2 poly on $(-\ln2,0]$ + right-shift | Integer sum + division          |
| ITA [2307.03493]      | Shift-based, $e^x \to 2^{x'}$ as shift   | Integer reciprocal and shifts   |
| Softermax [2103.09301]| Piecewise-linear $2^{x}$ via 4-segment LUT| Online max-tracking + LUT reciprocal|
| SoftmAP [2411.17847]  | Poly+Barrett on quantized logit intervals| Integer sum, scaling, shift     |
| IPTQ-ViT [2511.15369] | Degree-1 Taylor (bitshifts) of $2^x$     | Fixed-point product + shift     |
| I-LLM [2405.17849]    | Shift+linear interpolation               | Integer sum + IntDiv (scaled product/division)|
| IntAttention [2511.21513]| 32-entry LUT (UINT8)                  | Integer sum + scaling           |

#### Example: Second-order Polynomial + Shift (I-BERT)
- Quantize logits to INT32: $x\approx S q$.
- Decompose $x-q_{\max}$ as $-z\ln2 + p$, $z\in\mathbb{N}$, $p\in(-\ln2,0]$. Compute $z=\lfloor -q/q_{\ln2}\rfloor$.
- Approximate $\exp(p)\approx a(p+b)^2+c$ with $a=0.3585, b=1.353, c=0.344$.
- Output: $\exp(x)\approx\left[a(p+b)^2+c\right]\gg z$.

#### Example: Power-of-Two Shift (ITA)
- Quantize logits to INT8.
- Set scale so that $(x_i - x_{\max})/S$ is integer in $[-8,0]$.
- $e^{x_i - x_{\max}}\approx 2^{(x_{q,i} - M_q)/32}$.
- Normalization is integer division after sum of power-of-two weights.

#### Example: LUT-based (IntAttention)
- Max-subtract, clip to $\le c$.
- Index into a 32-entry LUT of $e^{-c \Delta/c}$, all in UINT8.
- After accumulation, normalize with scaling to [0,255].

## 3. Quantization Schemes and Numerical Management

All integer-only softmax methods rely on aggressive quantization of activations (most commonly to INT8 or even INT4) and specific scaling/zero-point strategies [2511.15369], [2101.01321]. The key themes are:
- **Dynamic range limitation**: Logit differences (after subtracting max) are clipped to a small interval (e.g., $[-7,0]$ or $[-15,0]$) to ensure exponentials remain computable with low-precision polynomials or LUTs [2405.17849], [2511.15369], [2411.17847].
- **Fixed-point or integer-only scales**: All multiplicative factors are reduced to integer shifts/adds, and any "real" multipliers are precomputed as dyadic fractions ($m/2^k$).
- **Accumulator widths**: Sums and division (normalization) require higher bit-widths (16–32 bits typical for sequence lengths <256) to avoid overflow [2411.17847], [2103.09301].

Static calibration on representative data is commonly used to fix quantization scales and zero-points, ensuring validity of approximations for the target model [2101.01321].

## 4. Hardware and Software Implementations

Integer-only softmax is tightly coupled to high-performance deployment on hardware accelerators, edge devices, and integer-only vector engines.

- **In-memory associative processors (SoftmAP)**: Implements every operation with word-parallel, bit-serial logic in CAM arrays, ensuring full locality and eliminating off-chip memory movements. Polynomial exp, Barrett division, and all component primitives are LUT-driven and mapped as primitive compare/write cycles [2411.17847].
- **ASIC and FPGA**: Softermax uses Q-formats for all signals, with approaches such as 4-segment piecewise linear LUTs for $2^x$, and single-pass normalization with running max and reciprocal lookup [2103.09301]. ITA implements the entire softmax in fixed-point streaming units feeding directly into MAC arrays [2307.03493].
- **Edge CPU (IntAttention)**: Implements 32-entry LUTs for efficient exp() and integer-only scaling for normalization, integrated into an INT8 pipeline that achieves 2x–3.7x speedup over FP16 and 61% energy reduction [2511.21513].
- **Software libraries**: Many methods are available as pluggable modules for PyTorch, TVM, or similar frameworks.

## 5. Accuracy Trade-offs and Error Analysis

Empirical results consistently show that integer-only softmax introduces minimal or negligible degradation in accuracy when quantization and approximation parameters are carefully tuned.

- **Error bounds**: For polynomial or shift approximations, the pointwise maximum error of the exponential surrogate on the target interval is often below $2\times10^{-3}$ and generally less than the quantizer's LSB [2101.01321], [2511.15369].
- **End-to-end model impact**: W8A8 and even W4A8 quantized transformers (ViT, BERT, LLAMA) show $<2\%$ absolute accuracy drop on ImageNet, GLUE, or C4 after integer-only softmax is deployed, and frequently no visible loss versus partial floating-point inference [2511.15369], [2405.17849], [2411.17847].
- **Latency and energy**: I-BERT, Softermax, and ITA consistently report 2–4x speedup and 2–3x energy efficiency improvement over float baselines, and SoftmAP reaches up to $300$–$1,305$x energy reduction versus A100/RTX3090 for softmax alone [2411.17847], [2103.09301], [2307.03493].

## 6. Implementation Details and Practical Guidance

Implementations generally follow a structured pipeline:

1. **Quantize logits**: Inputs are quantized to INT8 or lower, with dynamic or static scaling.
2. **Max-subtraction**: Compute integer max per row or token, subtract from all elements.
3. **Clipping**: Clamp lower tail of differences to avoid underflow in exponentials.
4. **Exponential approximation**: Use either bit-shift, small LUT, or low-degree poly on the quantized and clipped difference.
5. **Summation and normalization**: Accumulate outputs in large enough integer type, then normalize using integer multiplication and right-shift or fixed-point reciprocal trick.
6. **Output re-quantization**: Probabilities are projected to a suitable range ([0,255], [0,127], etc.) for subsequent integer multipliers.

Other considerations:
- Use int32 accumulators for large $d$ (sequence/vocab size).
- Precompute all constants (LUTs, polynomial parameters, scale multipliers).
- Hardware: Implement shift-add logic as fast combinational circuits; LUT sizes rarely exceed a few dozen entries.
- Validate quantization scales with a calibration set; empirically verify the leading logit differences stay within intended range.

## 7. Comparative Evaluation and Research Directions

Integer-only softmax has evolved through multiple research milestones:
- **I-BERT** pioneered pure-integer approximation of all nonlinearities (GELU, Softmax, LayerNorm) in transformer models [2101.01321].
- **ITA** and **Softermax** demonstrated streaming, energy-efficient, and area-efficient units that can be directly mapped onto integer ASIC/FPGAs [2307.03493], [2103.09301].
- **IPTQ-ViT** generalized bit-shifting exponentials to post-training quantization for ViTs, providing plug-and-play operators only requiring calibration [2511.15369].
- **I-LLM** bridged to ultra-low bit-width LLMs (W4A4), integrating integer-only softmax with companion normalization and activation modules for aggressive compression [2405.17849].
- **SoftmAP** combined software and associative in-memory hardware, using Barrett arithmetic and high-order poly approximations, achieving three orders-of-magnitude improvement in energy-delay product over high-end GPUs [2411.17847].
- **IndexSoftmax/IntAttention** addressed edge CPU and commodity hardware, integrating LUT-based integer softmax to restore the GEMMs as the bottleneck for further optimization [2511.21513].

Collectively, these advances now enable fully quantized, floating-point-free transformer and vision model inference with negligible loss in accuracy and transformative gain in computational efficiency.

---

**References**:  
- I-BERT: Integer-only BERT Quantization [2101.01321]  
- ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers [2307.03493]  
- Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers [2103.09301]  
- IPTQ-ViT: Post-Training Quantization of Non-linear Functions for Integer-only Vision Transformers [2511.15369]  
- I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models [2405.17849]  
- SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors [2411.17847]  
- IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference [2511.21513]

Source: https://www.emergentmind.com/topics/integer-only-softmax-implementation