Integer-Only Softmax Implementation

• Integer-only softmax is a technique that redefines the traditional floating-point softmax by using integer arithmetic for max finding, subtraction, exponential approximation, and normalization.
• It enables fully quantized transformer and vision models to run efficiently on edge hardware like CPUs, microcontrollers, FPGAs, and ASICs, significantly boosting throughput and reducing energy use.
• The approach leverages methods such as piecewise polynomial approximations, bit-shifting, and LUT-based exponentiation, while using dynamic range clipping and fixed-point scaling to maintain high model fidelity.

Integer-only softmax refers to implementations of the softmax nonlinearity in which all arithmetic—max finding, subtractions, exponentiations, and normalization—is performed strictly with integer-valued operations, with no floating-point arithmetic, transcendental functions, or datatype conversions in the inference path. This paradigm is central to fully quantized transformer and vision models, particularly at low bit-widths, as it removes the floating-point bottleneck for non-linearities and enables deployment on highly efficient integer hardware such as edge CPUs, microcontrollers, FPGAs, and custom ASICs. Multiple research groups have proposed such techniques for both software and hardware domains, yielding significant gains in throughput, energy efficiency, and area while maintaining high fidelity.

1. Mathematical Principles and Integer Reformulation

The standard floating-point softmax over a vector $x=(x_1,\dots,x_d)$ is defined as: $$\mathrm{Softmax}(x)_i = \frac{e^{x_i}}{\sum_{j=1}^d e^{x_j}}$$ For numerical stability, implementations apply max-subtraction: $$\mathrm{Softmax}(x)_i = \frac{e^{x_i-x_{\max}}}{\sum_{j=1}^d e^{x_j-x_{\max}}},\quad x_{\max}=\max_j x_j$$ The integer-only reformulation quantizes $x$ to an integer tensor $q_x$ via scale $S$, optionally zero-point $z_x$, so $x\approx S \cdot q_x + z_x$ (Kim et al., 2021, Zhong et al., 26 Nov 2025, Rakka et al., 2024, Hu et al., 2024, Kim et al., 19 Nov 2025). The exponentials are then approximated with bit-shifting, polynomials, lookup tables, or power-of-two logic:

• Base replacement: $e^{y}$ is replaced with $2^{y/\ln2}$ or similar, enabling use of integer shifts instead of evaluting $$\mathrm{Softmax}(x)_i = \frac{e^{x_i}}{\sum_{j=1}^d e^{x_j}}$$0.
• Piecewise or polynomial approximation: Over a restricted interval (typ. $$\mathrm{Softmax}(x)_i = \frac{e^{x_i}}{\sum_{j=1}^d e^{x_j}}$$1), $$\mathrm{Softmax}(x)_i = \frac{e^{x_i}}{\sum_{j=1}^d e^{x_j}}$$2 is approximated by a degree-2 polynomial (e.g., $$\mathrm{Softmax}(x)_i = \frac{e^{x_i}}{\sum_{j=1}^d e^{x_j}}$$3) (Kim et al., 2021, Rakka et al., 2024).
• Bit-shifting or power-of-two: When the scale is chosen to map each integer change to a power-of-two step, the "exponential" becomes a right-shift (İslamoğlu et al., 2023, Stevens et al., 2021).
• LUT-based: Lookup tables precalculate exponential values for small quantized ranges (Zhong et al., 26 Nov 2025).

The normalization (division by sum) is then accomplished by integer division, sometimes accelerated with reciprocal LUTs or fixed-point reciprocals.

2. Integer-Only Softmax Algorithms and Key Implementations

Multiple families of integer-only softmax exist, all sharing the properties above but differing in the approximation and normalization. The following table summarizes core methods, listed in the literature:

Approach	Exponential Approximation	Normalization Technique
I-BERT (Kim et al., 2021)	Degree-2 poly on $$\mathrm{Softmax}(x)_i = \frac{e^{x_i}}{\sum_{j=1}^d e^{x_j}}$$4 + right-shift	Integer sum + division
ITA (İslamoğlu et al., 2023)	Shift-based, $$\mathrm{Softmax}(x)_i = \frac{e^{x_i}}{\sum_{j=1}^d e^{x_j}}$$5 as shift	Integer reciprocal and shifts
Softermax (Stevens et al., 2021)	Piecewise-linear $$\mathrm{Softmax}(x)_i = \frac{e^{x_i}}{\sum_{j=1}^d e^{x_j}}$$6 via 4-segment LUT	Online max-tracking + LUT reciprocal
SoftmAP (Rakka et al., 2024)	Poly+Barrett on quantized logit intervals	Integer sum, scaling, shift
IPTQ-ViT (Kim et al., 19 Nov 2025)	Degree-1 Taylor (bitshifts) of $$\mathrm{Softmax}(x)_i = \frac{e^{x_i}}{\sum_{j=1}^d e^{x_j}}$$7	Fixed-point product + shift
I-LLM (Hu et al., 2024)	Shift+linear interpolation	Integer sum + IntDiv (scaled product/division)
IntAttention (Zhong et al., 26 Nov 2025)	32-entry LUT (UINT8)	Integer sum + scaling

Example: Second-order Polynomial + Shift (I-BERT)

• Quantize logits to INT32: $$\mathrm{Softmax}(x)_i = \frac{e^{x_i}}{\sum_{j=1}^d e^{x_j}}$$8.
• Decompose $$\mathrm{Softmax}(x)_i = \frac{e^{x_i}}{\sum_{j=1}^d e^{x_j}}$$9 as $$\mathrm{Softmax}(x)_i = \frac{e^{x_i-x_{\max}}}{\sum_{j=1}^d e^{x_j-x_{\max}}},\quad x_{\max}=\max_j x_j$$0, $$\mathrm{Softmax}(x)_i = \frac{e^{x_i-x_{\max}}}{\sum_{j=1}^d e^{x_j-x_{\max}}},\quad x_{\max}=\max_j x_j$$1, $$\mathrm{Softmax}(x)_i = \frac{e^{x_i-x_{\max}}}{\sum_{j=1}^d e^{x_j-x_{\max}}},\quad x_{\max}=\max_j x_j$$2. Compute $$\mathrm{Softmax}(x)_i = \frac{e^{x_i-x_{\max}}}{\sum_{j=1}^d e^{x_j-x_{\max}}},\quad x_{\max}=\max_j x_j$$3.
• Approximate $$\mathrm{Softmax}(x)_i = \frac{e^{x_i-x_{\max}}}{\sum_{j=1}^d e^{x_j-x_{\max}}},\quad x_{\max}=\max_j x_j$$4 with $$\mathrm{Softmax}(x)_i = \frac{e^{x_i-x_{\max}}}{\sum_{j=1}^d e^{x_j-x_{\max}}},\quad x_{\max}=\max_j x_j$$5.
• Output: $$\mathrm{Softmax}(x)_i = \frac{e^{x_i-x_{\max}}}{\sum_{j=1}^d e^{x_j-x_{\max}}},\quad x_{\max}=\max_j x_j$$6.

Example: Power-of-Two Shift (ITA)

• Quantize logits to INT8.
• Set scale so that $$\mathrm{Softmax}(x)_i = \frac{e^{x_i-x_{\max}}}{\sum_{j=1}^d e^{x_j-x_{\max}}},\quad x_{\max}=\max_j x_j$$7 is integer in $$\mathrm{Softmax}(x)_i = \frac{e^{x_i-x_{\max}}}{\sum_{j=1}^d e^{x_j-x_{\max}}},\quad x_{\max}=\max_j x_j$$8.
• $$\mathrm{Softmax}(x)_i = \frac{e^{x_i-x_{\max}}}{\sum_{j=1}^d e^{x_j-x_{\max}}},\quad x_{\max}=\max_j x_j$$9.
• Normalization is integer division after sum of power-of-two weights.

Example: LUT-based (IntAttention)

• Max-subtract, clip to $x$0.
• Index into a 32-entry LUT of $x$1, all in UINT8.
• After accumulation, normalize with scaling to [0,255].

3. Quantization Schemes and Numerical Management

All integer-only softmax methods rely on aggressive quantization of activations (most commonly to INT8 or even INT4) and specific scaling/zero-point strategies (Kim et al., 19 Nov 2025, Kim et al., 2021). The key themes are:

• Dynamic range limitation: Logit differences (after subtracting max) are clipped to a small interval (e.g., $x$2 or $x$3) to ensure exponentials remain computable with low-precision polynomials or LUTs (Hu et al., 2024, Kim et al., 19 Nov 2025, Rakka et al., 2024).
• Fixed-point or integer-only scales: All multiplicative factors are reduced to integer shifts/adds, and any "real" multipliers are precomputed as dyadic fractions ($x$4).
• Accumulator widths: Sums and division (normalization) require higher bit-widths (16–32 bits typical for sequence lengths <256) to avoid overflow (Rakka et al., 2024, Stevens et al., 2021).

Static calibration on representative data is commonly used to fix quantization scales and zero-points, ensuring validity of approximations for the target model (Kim et al., 2021).

4. Hardware and Software Implementations

Integer-only softmax is tightly coupled to high-performance deployment on hardware accelerators, edge devices, and integer-only vector engines.

• In-memory associative processors (SoftmAP): Implements every operation with word-parallel, bit-serial logic in CAM arrays, ensuring full locality and eliminating off-chip memory movements. Polynomial exp, Barrett division, and all component primitives are LUT-driven and mapped as primitive compare/write cycles (Rakka et al., 2024).
• ASIC and FPGA: Softermax uses Q-formats for all signals, with approaches such as 4-segment piecewise linear LUTs for $x$5, and single-pass normalization with running max and reciprocal lookup (Stevens et al., 2021). ITA implements the entire softmax in fixed-point streaming units feeding directly into MAC arrays (İslamoğlu et al., 2023).
• Edge CPU (IntAttention): Implements 32-entry LUTs for efficient exp() and integer-only scaling for normalization, integrated into an INT8 pipeline that achieves 2x–3.7x speedup over FP16 and 61% energy reduction (Zhong et al., 26 Nov 2025).
• Software libraries: Many methods are available as pluggable modules for PyTorch, TVM, or similar frameworks.

5. Accuracy Trade-offs and Error Analysis

Empirical results consistently show that integer-only softmax introduces minimal or negligible degradation in accuracy when quantization and approximation parameters are carefully tuned.

• Error bounds: For polynomial or shift approximations, the pointwise maximum error of the exponential surrogate on the target interval is often below $x$6 and generally less than the quantizer's LSB (Kim et al., 2021, Kim et al., 19 Nov 2025).
• End-to-end model impact: W8A8 and even W4A8 quantized transformers (ViT, BERT, LLAMA) show $x$7 absolute accuracy drop on ImageNet, GLUE, or C4 after integer-only softmax is deployed, and frequently no visible loss versus partial floating-point inference (Kim et al., 19 Nov 2025, Hu et al., 2024, Rakka et al., 2024).
• Latency and energy: I-BERT, Softermax, and ITA consistently report 2–4x speedup and 2–3x energy efficiency improvement over float baselines, and SoftmAP reaches up to $x$8–$x$9x energy reduction versus A100/RTX3090 for softmax alone (Rakka et al., 2024, Stevens et al., 2021, İslamoğlu et al., 2023).

6. Implementation Details and Practical Guidance

Implementations generally follow a structured pipeline:

1. Quantize logits: Inputs are quantized to INT8 or lower, with dynamic or static scaling.
2. Max-subtraction: Compute integer max per row or token, subtract from all elements.
3. Clipping: Clamp lower tail of differences to avoid underflow in exponentials.
4. Exponential approximation: Use either bit-shift, small LUT, or low-degree poly on the quantized and clipped difference.
5. Summation and normalization: Accumulate outputs in large enough integer type, then normalize using integer multiplication and right-shift or fixed-point reciprocal trick.
6. Output re-quantization: Probabilities are projected to a suitable range ([0,255], [0,127], etc.) for subsequent integer multipliers.

Other considerations:

• Use int32 accumulators for large $q_x$0 (sequence/vocab size).
• Precompute all constants (LUTs, polynomial parameters, scale multipliers).
• Hardware: Implement shift-add logic as fast combinational circuits; LUT sizes rarely exceed a few dozen entries.
• Validate quantization scales with a calibration set; empirically verify the leading logit differences stay within intended range.

7. Comparative Evaluation and Research Directions

Integer-only softmax has evolved through multiple research milestones:

• I-BERT pioneered pure-integer approximation of all nonlinearities (GELU, Softmax, LayerNorm) in transformer models (Kim et al., 2021).
• ITA and Softermax demonstrated streaming, energy-efficient, and area-efficient units that can be directly mapped onto integer ASIC/FPGAs (İslamoğlu et al., 2023, Stevens et al., 2021).
• IPTQ-ViT generalized bit-shifting exponentials to post-training quantization for ViTs, providing plug-and-play operators only requiring calibration (Kim et al., 19 Nov 2025).
• I-LLM bridged to ultra-low bit-width LLMs (W4A4), integrating integer-only softmax with companion normalization and activation modules for aggressive compression (Hu et al., 2024).
• SoftmAP combined software and associative in-memory hardware, using Barrett arithmetic and high-order poly approximations, achieving three orders-of-magnitude improvement in energy-delay product over high-end GPUs (Rakka et al., 2024).
• IndexSoftmax/IntAttention addressed edge CPU and commodity hardware, integrating LUT-based integer softmax to restore the GEMMs as the bottleneck for further optimization (Zhong et al., 26 Nov 2025).

Collectively, these advances now enable fully quantized, floating-point-free transformer and vision model inference with negligible loss in accuracy and transformative gain in computational efficiency.

---

References:

• I-BERT: Integer-only BERT Quantization (Kim et al., 2021)
• ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers (İslamoğlu et al., 2023)
• Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers (Stevens et al., 2021)
• IPTQ-ViT: Post-Training Quantization of Non-linear Functions for Integer-only Vision Transformers (Kim et al., 19 Nov 2025)
• I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit LLMs (Hu et al., 2024)
• SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors (Rakka et al., 2024)
• IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference (Zhong et al., 26 Nov 2025)