Floating-Point Neural Networks

Updated 5 July 2026

Floating-Point Neural Networks are a family of numerical representations that use tunable floating-point arithmetic to balance dynamic range, precision, and implementation cost.
They enable efficient AI inference and training by tailoring exponent, mantissa, and bias parameters to match tensor distributions and hardware constraints.
They drive advances in accelerator design and formal verification by addressing trade-offs in quantization, gradient stability, and error resilience.

Floating-point neural networks are neural networks whose parameters, activations, gradients, accumulators, or implementation semantics are treated explicitly under floating-point arithmetic rather than exact real arithmetic or purely fixed-point arithmetic. In the literature, the topic spans custom low-precision formats for inference and training, standard IEEE-754 half-precision training, floating-point-oriented accelerator design, software-level verification of actual IEEE-754 executions, and expressivity theory under finite-precision operations. A central theme is that floating-point structure exposes a tunable trade-off between dynamic range, precision, implementation cost, and semantic fidelity, and that this trade-off can be exploited at the level of tensor formats, hardware datapaths, verification algorithms, and approximation theory (Huang et al., 2021, Yun et al., 2023, Park et al., 2024, Manino et al., 27 Oct 2025).

1. Numerical representations and format families

The canonical generalized floating-point value used across this literature is

$x = (-1)^s \cdot 2^{e-\mathrm{bias}} \cdot (1.f),$

with a sign bit, biased exponent, and mantissa fraction. Several works depart from fixed IEEE-style layouts by making some of these components tensor-dependent or layer-dependent. In FFP8, an 8-bit number is parameterized by $(x,y,z,b)$ , where $x$ is the sign-bit width, $y$ the exponent-bit width, $z$ the fraction-bit width, and $b$ the exponent bias, with $x+y+z=8$ . Its decoded value is

$V = (-1)^s \times 2^{e-b} \times \left(1+\frac{f}{2^z}\right),$

and the sign bit may be omitted for nonnegative tensors such as ReLU outputs, allowing all eight bits to be reassigned to exponent and fraction fields (Huang et al., 2021).

Other format families modify the exponent window or tensor statistics rather than only the bit split. AdaptivFloat $n,e$ uses 1 sign bit, $e$ exponent bits, and $(x,y,z,b)$ 0 mantissa bits, eliminates hardware denormals, guarantees a unique zero representation, and uses a layer-specific integer expbias to shift the representable exponent range so that it surrounds a layer’s observed maximum magnitude (Tambe et al., 2019). S2FP8 retains a standard 1/5/2 FP8 encoding for the stored tensor but introduces two tensor statistics, a squeeze factor $(x,y,z,b)$ 1 and a shift factor $(x,y,z,b)$ 2, with

$(x,y,z,b)$ 3

so that the log-distribution is reshaped before FP8 rounding and inverted afterward (Cambier et al., 2020). Phoenix studies 8-bit custom floating-point formats with 1 sign bit, $(x,y,z,b)$ 4 exponent bits, and $(x,y,z,b)$ 5 mantissa bits, identifying M5E2 and M4E3 as effective operating points for CNN quantization and hardware efficiency (Wu et al., 2020).

A separate line of work treats standard IEEE formats themselves as the primary object. Pure FP16 training uses IEEE-754 half precision end-to-end, without a 32-bit master copy or mixed-precision fallback, and analyzes its behavior relative to FP32 in terms of machine epsilon, dynamic range, and classification-margin stability (Yun et al., 2023). Low-precision training schemes such as float[5,6], context-float[4,7], and float[6,0] show that the exponent/mantissa allocation can be combined with per-context scaling or power-of-two constraints to alter the effective range and arithmetic cost during training (Ortiz et al., 2018).

Format family	Bit structure or tunable fields	Distinctive mechanism
FFP8	$(x,y,z,b)$ 6 with $(x,y,z,b)$ 7	Configurable sign bit and exponent bias
AdaptivFloat	1 sign, $(x,y,z,b)$ 8 exponent, $(x,y,z,b)$ 9 mantissa	Layer-granular `expbias` and optimal clipping
S2FP8	Stored 1/5/2 FP8	Per-tensor squeeze $x$ 0 and shift $x$ 1
Context-float	1 sign, 4 exponent, 7 mantissa	Layer-wise power-of-two scale $x$ 2
Pure FP16	IEEE-754 half precision	No master copy or loss scaling for classification tasks

These formats share a common premise: tensor distributions are not uniform across a model. This suggests that floating-point neural networks are best understood not as a single datatype choice, but as a family of distribution-matched numerical representations.

2. Inference and training under reduced precision

For inference, a major result is that tensor-specific floating-point formats can substantially reduce memory demand while preserving accuracy without retraining. FFP8 chooses exponent width, fraction width, bias, and sign-bit presence by analyzing each tensor’s log-magnitude histogram, maximum magnitude, and value mass. In the reported inference flow, off-chip weights are stored in the selected FFP8 format, activations produced internally in FP32 are rounded to FFP8 before off-chip writeback, and the fused-multiply-add unit itself remains in FP32. On ImageNet, best-fit FFP8 reduces Top-1 loss relative to FP32 to $x$ 3 for VGG-16, $x$ 4 for ResNet-50, and $x$ 5 for MobileNetV2, whereas fixed FP8(1,4,3) yields $x$ 6, $x$ 7, and $x$ 8, respectively (Huang et al., 2021).

AdaptivFloat pursues the same inference objective through per-layer dynamic exponent shifting and clipping. For post-training quantization and quantization-aware retraining, it is reported to outperform block floating-point, uniform integer, IEEE-style float, and posit encodings at very low precision across Transformer, LSTM-Seq2Seq with attention, and ResNet-50. The paper reports, for example, Transformer BLEU of 27.3/27.7 at 8-bit AdaptivFloat<8,3> versus an FP32 baseline of 27.4, Seq2Seq WER of 13.11%/12.59% at 8-bit versus FP32 WER of 13.34%, and ResNet-50 Top-1 of 75.7/76.0 at 8-bit versus FP32 Top-1 of 76.2% (Tambe et al., 2019).

For training, several distinct strategies appear. Low-precision floating-point schemes for CIFAR-10 show that 12-bit float[5,6] reaches 74.20% ± 0.4% versus a float32 baseline of 75.60% ± 0.4%, while context-float[4,7] reaches 78.02% ± 0.3% and Power-of-Two training reaches 73.42% ± 0.3%; the same study associates float[5,6] and context-fixed with estimated training time of ~0.75 h and Power-of-Two with ~0.23 h, compared with ~2.0 h for float32 (Ortiz et al., 2018). S2FP8 reports out-of-the-box FP8 training for ResNet-50, Transformer, and NCF using the closed-form factors $x$ 9 and $y$ 0 rather than hand-tuned loss scaling or FP32-only layers; on ImageNet-1K, it reports ResNet-50 Top-1 of 75.2% versus FP32 76.2%, and on English→Vietnamese Transformer it reports BLEU 25.3 for both FP32 and S2FP8 (Cambier et al., 2020).

Pure FP16 training broadens the notion of floating-point neural networks beyond custom low-bit formats. On CIFAR-10 classification, every pure-FP16 run in the reported 33 configurations reduces training time by at least 29% relative to FP32, and the model-file size halves exactly when weights are stored in 16 rather than 32 bits. Reported results include AlexNet at 381 s and 68.9% in FP32 versus 270 s and 69.5% in FP16, VGG16 at 1445 s and 82.9% versus 812 s and 84.3%, and ResNet-34 at 1914 s and 76.6% versus 1058 s and 76.1% (Yun et al., 2023). Dynamic container adaptation extends this logic further: Quantum Mantissa and Quantum Exponent reduce footprint by $y$ 1, BitWave by $y$ 2, and Gecko improves these averages to $y$ 3 and $y$ 4, respectively (Nikolić et al., 2022).

A common misconception is that lowering precision is uniformly harmful. The reported results do not support that generalization: context-float[4,7] exceeds the float32 baseline on CIFAR-10, pure FP16 sometimes improves test accuracy on classification tasks, and AdaptivFloat can surpass FP32 by up to +0.3 BLEU and -0.75 WER in the reported inference settings (Ortiz et al., 2018, Yun et al., 2023, Tambe et al., 2019). At the same time, the same literature identifies failure modes such as gradient underflow in narrow formats, optimizer $y$ 5 choices that flush to zero in FP16, and abrupt accuracy collapse under untuned vanilla FP8 (Yun et al., 2023, Cambier et al., 2020).

3. Hardware realizations and accelerator architectures

Floating-point neural networks have motivated hardware designs that try to preserve floating-point range without incurring the full cost of conventional floating-point units. FFP8 emphasizes compatibility with existing floating-point pipelines: support requires a small format-register bank holding approximately three fields $y$ 6 and a combinational converter that extracts sign and exponent fields, bias-converts to FP32, and pads the fraction into the FP32 mantissa. The reported overhead is typically less than 2% area in a merged 8-bit→32-bit converter block, with negligible extra latency of one cycle and similarly small power cost (Huang et al., 2021).

AdaptivFloat integrates into a Hybrid Float-Integer PE in which weights and activations are stored in AdaptivFloat form while the MAC datapath accumulates in integer. For the reported 4-PE accelerator, HFINT8/30 system power is 56 mW versus 61 mW for the INT8/24/40 baseline, area is 7.9 mm $y$ 7 versus 6.9 mm $y$ 8, and both execute 100 LSTM × 256 steps in the same 81 $y$ 9s (Tambe et al., 2019). This arrangement preserves a floating-point-like tensor representation while avoiding a full floating-point adder in the accumulation path.

Several accelerator proposals instead redesign the multiplier itself. Phoenix observes that 8-bit floating-point quantization incurs less error than 8-bit fixed-point quantization and proposes a processor with FP8 multipliers, truncation, and fixed-point accumulation. For M4E3, the synthesized multiplier in TSMC 28 nm HPC+ has maximum clock approximately 2.86 GHz and area approximately 112 NAND2 equivalents, versus approximately 2.22 GHz and 285 NAND2 equivalents for the fixed-point counterpart; the full Phoenix core has 1.44 mm $z$ 0, 2048 MACs, 1.0 GHz clock, 2.048 TMAC/s peak, and 1.091 W power (Wu et al., 2020). ELMA uses an 8-bit tapered log-float with hybrid log-domain multiplication, linear-domain addition, and a 38-bit Kulisch accumulator; its 8-bit PE is synthesized at 0.96× the power and 1.12× the area of an int8/32 MAC, while the 16-bit variant is 0.59× the power and 0.68× the area of IEEE-754 float16 FMA (Johnson, 2018).

The same pressure toward efficient floating-point arithmetic appears in memory-centric designs and software emulation. A hybrid-domain FP compute-in-memory macro splits the mantissa computation into a digital sub-add path and an analog sub-multiplication path, reports a 78.85/51.39 ≈ 1.53× reduction in energy per FP8 MAC relative to a fully digital FP CIM baseline, and shows end-to-end changes of 76.01% → 75.68% on ResNet-50, 90.73% → 89.49% on BERT-base, and 0.3759 → 0.3709 on RetinaNet (Yi et al., 11 Feb 2025). HOBFLOPS uses hardware-synthesized bitslice-parallel software operators for arbitrary FP(E,M) on commodity processors; on Intel AVX512, HOBFLOPS16 reaches 1.97×10 $z$ 1 MAC/s versus 0.24×10 $z$ 2 for SoftFP16, an 8.2× speedup, and lower-precision HOBFLOPS8 reaches 2.25×10 $z$ 3 MAC/s (Garland et al., 2020).

A distinct hybrid scheme represents weights in floating-point and activations in fixed-point. For AlexNet, SqueezeNet, GoogLeNet, and VGG-16, an 8-bit weight format with 3-bit mantissa and 4-bit exponent recovers full accuracy across all four reported networks, reduces weight storage by up to 36%, and reduces hardware multiplier power by up to 50% (Lai et al., 2017). Taken together, these architectures indicate that floating-point neural networks do not require a single canonical datapath: exponent handling, mantissa multiplication, accumulation, and storage can be partitioned in multiple ways.

4. Verification, robustness, and software-level semantics

A separate branch of the field treats floating-point arithmetic not as an efficiency device but as the semantic object that must be verified. The motivating claim is that robustness or safety statements proved for real-valued networks can fail for the actual deployed floating-point implementation. One study shows that neglecting floating-point error leads to unsound verification that can be systematically exploited, reporting 82 adversarial examples found out of 320 total cases across MNIST and CIFAR-10 implementations, with the GPU-Winograd implementation yielding 32/64 successful cases (Jia et al., 2020). Another software-level study constructs NeuroCodeBench 2.0, a benchmark of 912 plain-C verification examples covering math functions, activations, layers, and full networks up to 170K parameters, and reports that eight state-of-the-art software verifiers solve an average of only about 11% correctly while producing around 3% incorrect verdicts (Manino et al., 27 Oct 2025).

This literature also argues that verification must model the actual execution environment, including nondeterminism across accelerators. NAO formalizes operator-level acceptance regions for FP32 computation, combining sound IEEE-754 worst-case bounds with empirical percentile profiles calibrated across A100, H100, RTX6000, and RTX4090. It reports that empirical thresholds are $z$ 4– $z$ 5 times tighter than theoretical bounds, that the optimistic runtime overhead is approximately 0.3% on Qwen3-8B, and that bound-aware adversarial attacks achieve 0% success under the empirical thresholds (Yao et al., 15 Oct 2025). This suggests that verification of floating-point neural networks may require a combination of bit-precise theory and hardware-calibrated tolerance models.

Robustness certification under floating-point execution makes the same semantic gap explicit for sensitivity-based certifiers. A Lipschitz-based framework derives layerwise floating-point deviation recurrences, certificate degradation terms, and sufficient conditions for the absence of overflow, then implements an executable certifier using arbitrary-precision rational arithmetic. On fully connected robust models, the reported per-instance certification time is 0.8–1.4 ms; the certified-robust fraction under FP is 95.5% on MNIST versus 95.7% under real arithmetic, 82.2% on Fashion MNIST versus 83.7%, and 18.9% on CIFAR-10 versus 46.1%, while float16 certification is described as almost vacuous because the deviation bounds become too large (Murray, 6 Mar 2026).

An objective controversy therefore runs through this area. Real-arithmetic verification offers mathematical tractability, but the cited works show that IEEE-754 rounding, reduction order, library implementation, and accelerator nondeterminism can invalidate or weaken those guarantees in deployment (Jia et al., 2020, Manino et al., 27 Oct 2025, Murray, 6 Mar 2026). Floating-point neural networks, in this sense, are also a verification problem.

5. Expressive power under finite-precision computation

Theoretical work has increasingly rejected the assumption that neural networks should be analyzed only as exact real-valued functions. One line studies floating-point operations with finite significand bits and either unbounded or bounded exponents, showing that ReLU and step networks retain memorization and universal approximation power under floating-point execution. For $z$ 6, a 3-layer step network with $z$ 7 parameters memorizes any $z$ 8 input/output pairs, a 4-layer ReLU network with $z$ 9 parameters does the same, and analogous universal-approximation constructions match the parameter counts of classical exact-arithmetic results up to constants (Park et al., 2024).

A stronger interval-theoretic result establishes an interval universal approximation theorem directly over IEEE-style floating-point domains. For every target function $b$ 0, if $b$ 1, there exists a floating-point network whose interval semantics exactly matches the direct image of $b$ 2 on every abstract box in $b$ 3. The same work derives corollaries on the existence of provably robust floating-point networks and on the interval-completeness of straight-line programs using only floating-point additions and multiplications; it also notes that, over floats, the identity activation becomes non-affine because of rounding effects (Hwang et al., 19 Jun 2025).

Automatic differentiation introduces an additional expressive dimension. Floating-point networks with AD are shown to represent arbitrary function values and arbitrary back-propagated gradients simultaneously under practical activation functions including ReLU, ELU, GeLU, Swish, Sigmoid, and $b$ 4. The main simultaneous value-and-gradient fitting theorem states that, under mild compatibility conditions, for any depth $b$ 5 there exists an $b$ 6-layer floating-point network $b$ 7 such that $b$ 8 and $b$ 9 for all inputs in the bounded floating-point domain (Park et al., 3 May 2026).

A further generalization studies arbitrary reduction orders and inexact activation implementations with bounded ulp error, rather than fixed left-to-right reduction and correctly rounded activations. The resulting distinguishability framework shows that first-layer ability to distinguish every pair of inputs is necessary for universal representability, and that a suitable distinguishability condition is also sufficient. Universal representability is then established for practical implementations of Sigmoid, $x+y+z=8$ 0, ReLU, ELU, SeLU, GeLU, Swish, Mish, and $x+y+z=8$ 1 under these more realistic execution semantics (Park et al., 27 May 2026).

These results collectively alter the theoretical status of floating-point neural networks. Finite precision is not treated merely as a perturbation of a real-valued model; rather, the network executed in floating-point arithmetic becomes a distinct computational system with its own universal approximation, memorization, interval semantics, and gradient representability properties.

6. Recurring design principles and misconceptions

Several design principles recur across otherwise different subfields. First, exponent allocation is often more valuable than uniform resolution when tensor magnitudes are heterogeneous. FFP8 adjusts exponent width and bias per tensor, AdaptivFloat shifts a layer-granular exponent window, context-float applies a per-context power-of-two scale, and S2FP8 reshapes the log-distribution through $x+y+z=8$ 2 and $x+y+z=8$ 3 before standard FP8 rounding (Huang et al., 2021, Tambe et al., 2019, Ortiz et al., 2018, Cambier et al., 2020). Second, accumulation is frequently separated from storage format: FFP8 computes in FP32, AdaptivFloat accumulates in integer, ELMA uses a Kulisch accumulator, and Phoenix preserves the full-precision product before truncation and fixed-point reduction (Huang et al., 2021, Tambe et al., 2019, Johnson, 2018, Wu et al., 2020).

A second recurring theme is that the adequacy of a floating-point neural network cannot be judged solely by nominal bit width. The cited studies repeatedly show that per-layer histograms, exponent distributions, optimizer constants, reduction order, and activation-library behavior materially affect outcomes. This suggests that “FP8,” “FP16,” or “FP32” are often incomplete descriptors unless accompanied by bias conventions, scaling rules, accumulation semantics, and implementation details.

Two misconceptions are especially persistent. The first is that low precision necessarily destroys accuracy. The reported evidence is more conditional: FFP8 keeps ImageNet losses within about 0.1–0.3%, context-float[4,7] exceeds the float32 baseline on CIFAR-10, pure FP16 sometimes improves classification accuracy, and AdaptivFloat can exceed the FP32 baseline on sequence tasks (Huang et al., 2021, Ortiz et al., 2018, Yun et al., 2023, Tambe et al., 2019). The second is that real-arithmetic proofs are automatically meaningful for deployed systems. The verification literature directly contradicts that claim by exhibiting floating-point counterexamples, incorrect verifier verdicts, and certifiers whose guarantees degrade sharply at lower precision (Jia et al., 2020, Manino et al., 27 Oct 2025, Murray, 6 Mar 2026).

A plausible implication is that floating-point neural networks should be viewed as a joint problem in numerical analysis, machine learning, computer architecture, and formal methods. The same finite-precision structure that enables compression, acceleration, and sometimes regularization also creates new proof obligations, new failure modes, and new theoretical objects.