Fused Multiply-Add (FMA) Datapaths

• Fused multiply-add (FMA) datapaths are hardware structures that compute a × b + c in a single pipeline, reducing latency, energy consumption, and cumulative rounding error.
• Architectural variants such as fixed-precision, mixed-precision, and configurable designs enable optimized performance across CPUs, GPUs, AI accelerators, and domain-specific architectures.
• Innovative pipeline organization, error modeling, and hardware–algorithm codesign ensure high-throughput, energy-efficient mixed-precision arithmetic for scientific computing and machine learning.

A fused multiply-add (FMA) datapath is a hardware arithmetic structure that computes the operation $z = a \times b + c$ in a single, tightly integrated pipeline, performing only one rounding at the end of the computation. The FMA reduces latency, saves energy, and minimizes rounding and normalization compared to separate multiply and add units. Modern FMA units are central to general-purpose CPUs, GPUs, AI accelerators, and domain-specific architectures. FMA functionality is crucial for high-throughput mixed-precision arithmetic in scientific computing, machine learning, and signal processing, where hardware implementations must balance speed, area, energy, and accumulated error across deep pipelines and long reduction chains (Bhola et al., 2024, Rout et al., 19 Nov 2025, Henry et al., 2019).

1. Architectural Variants of Fused Multiply-Add Datapaths

FMA datapaths are implemented in several forms, each optimized for a specific combination of precision, throughput, and error control.

• Fixed-Precision FMA: A single datapath computes $z = a \times b + c$ for operands of the same format (e.g., IEEE-754 FP32). In CPUs and GPUs, the multiplier directs full-width products to an adder, followed by a unique rounding stage. Architectures such as Intel AVX-512 and ARM SVE provide vectorized FMA pipelines for SIMD workloads. Specialized out-of-order schedulers dispatch FMAs alongside other floating-point operations, as evidenced in the Lagarto II processor with up to two 64-bit FMA operations per cycle (Lazo, 2021).
• Mixed-Precision FMA (MPFMA): Accepts low-precision input operands (such as FP16/BF16/FP8) with accumulation and final result in a higher-precision format (e.g., FP32/INT32), either through explicit internal widening or accumulation strategies. For example, NVIDIA Tensor Cores implement 16 parallel 4×4 mixed-precision FMAs, multiplying FP16 (or FP8) operands, producing a full-precision (FP32) product for accumulation, and then writing back to FP16 or FP32 (Bhola et al., 2024, Henry et al., 2019, Rout et al., 19 Nov 2025).
• Configurable and Hybrid FMA: Emerging designs offer precision-scalable, mixed-type multiply-accumulate units. These units implement precision scaling and representation flexibility (e.g., INT8, FP8, BF16), supporting both inference (narrow bit-width) and training (wide dynamic range) through hybrid reduction trees and tunable accumulator widths (Rout et al., 19 Nov 2025, Cuyckens et al., 9 Nov 2025, Johnson, 2018).

2. Pipeline Organization and Computational Block Structures

FMA datapaths are characterized by an integrated pipeline that eliminates redundant normalization and rounding points in long compute chains:

1. Operand Fetch and Decode: Inputs $a$, $b$, $c$ are decoded, exponents and mantissas are extracted, sign flags determined, and formats normalized as needed for mixed-precision operation.
2. Multiplication: Mantissas are multiplied; exponents are added with bias correction. For low-precision formats, multiplication is packed (e.g., two FP16 packed per 32-bit word) and mapped onto shared multiplier trees or LUTs (Rout et al., 19 Nov 2025).
3. Alignment: Products and $c$ are aligned to a common exponent, often using barrel shifters, sign extension, and comparator matrices to find maximum exponents for correct alignment, especially in the presence of multiple sum terms or variable-precision inputs (Rout et al., 19 Nov 2025).
4. Accumulation: Carry-save adder (CSA) trees or compressor trees are used to sum products and the $c$ operand. In advanced designs, the accumulator is widened to the highest required precision to avoid intermediate overflows and underflows (e.g., E8M25 for FP16 inputs with FP32 accumulation) (Henry et al., 2019, Rout et al., 19 Nov 2025). Modern frameworks (e.g., UFO-MAC) globally optimize the compressor tree using ILP for both depth and interconnect timing, fusing the accumulator input into the first stage for minimal added latency and area (Zuo et al., 2024).
5. Normalization and Rounding: The sum is normalized (LZC, exponent adjust) and rounded once using mode-selectable strategies (RNE, toward-zero, directed). Only after all partial products and $c$ are accumulated is final rounding performed, minimizing overall error (Lazo, 2021, Bhola et al., 2024).

A representative 4-stage FEDP pipeline for a GPGPU tensor core is: (1) Low-precision multiplication, (2) Exponent alignment, (3) CSA accumulation, (4) Final normalization and rounding (Rout et al., 19 Nov 2025).

3. Precision, Mixed-Precision Support, and Data Formats

The modern FMA landscape is dominated by mixed-precision arithmetic, essential for energy efficiency and throughput in AI/ML and scientific workloads:

• Low-Precision Input Formats: FP16, BF16, FP8, INT8, UINT4—supporting deep learning inference/training and low-power vector processing.
• High-Precision Accumulation: Often FP32, INT32, or customizable widths; accumulation always occurs in an extended domain to minimize overflow and minimize rounding error (Rout et al., 19 Nov 2025, Henry et al., 2019, Cuyckens et al., 9 Nov 2025).
• Packed Multiplication: Multiple operands per register/word (e.g., 2-way for FP16, 4-way for FP8) directly feed shared multiplication hardware for maximum resource utilization (Rout et al., 19 Nov 2025).
• Internal Widening: After low-precision multiplication, intermediate results are widened (e.g., FP16×FP16 into full E8M25) and addition is performed at high precision, with rounding only at final output (Bhola et al., 2024, Henry et al., 2019).

Decomposing FP32 operands into multiple BF16 components and using chained FMAs enables reconstruction of FP32-level accuracy synchronized with high-throughput BF16 hardware (Henry et al., 2019). Architectures supporting "procrastinated" normalization and Kulisch accumulation (exact partial-sum register banks indexed by exponents) further minimize cumulative rounding after long dot-products (Liguori, 2024, Johnson, 2018).

4. Error Models and Numerical Guarantees

Deterministic and probabilistic error analyses provide upper bounds and practical error estimates for long FMA chains under hardware constraints:

• Single FMA Deterministic Error: Each operation incurs a single relative error $\delta$, $|\delta| \leq u$ (unit roundoff). The total error after $n$ FMAs grows as $\gamma_n = n u/(1 - n u) \sim O(n u)$ (Bhola et al., 2024).
• Forward Error Bound:

$|\hat z - z| \leq u |a b + c|$

where $z = a b + c,\, \hat z = \text{fl}(a b + c)$.

• Backward Error Bound:

There exist $\Delta a, \Delta b, \Delta c$ with $|\Delta\star| \leq u|\star|$ such that $\hat z = (a+\Delta a)(b+\Delta b) + (c+\Delta c)$.

• Probabilistic Model: Treats each rounding error as an i.i.d. uniform random variable $\delta_i \sim U[-u,u]$, yielding accumulated error scaling as $O(u\sqrt{n})$ with high probability. Empirical results on mixed-precision GEMM show that probabilistic bounds are nearly an order of magnitude tighter than deterministic ones, and actual observed forward errors are significantly lower (e.g., deterministic bound $\simeq 10^2$, probabilistic bound $\simeq 10^1$, observed error $\simeq 10^{-2}$ in a large FP16+FP32 block-matmul) (Bhola et al., 2024).
• Implications for Hardware and Algorithms: Probabilistic bounds justify more aggressive use of mixed and low precision, adaptive block sizes, and support for dynamic switch-over to higher precision when modeled error becomes excessive.

5. Datapath Optimization, Area, and Throughput

State-of-the-art FMA datapaths are optimized for area, delay, and energy via hierarchical compressor/adder tree synthesis, resource sharing, and precision scaling:

• Compressor Tree Synthesis: UFO-MAC applies two-stage ILP for compressor assignment and interconnection, systematically fusing the accumulator into the first tree stage for FMA. Delay and area are minimized by partitioning the carry-propagate adder (CPA) according to the trapezoidal arrival profile of the tree, deploying an FDC delay model to optimize depth/fanout per bit (Zuo et al., 2024).
• Pipeline Depth and Latency: Pipelined FMAs achieve depths from 4 cycles (mixed-precision GPGPU tensor core, 306.6 MHz, 9.8–30.7 GFLOPS) (Rout et al., 19 Nov 2025) to 13 stages (superscalar out-of-order 64-bit engine) (Lazo, 2021). Kulisch or exponent-indexed accumulators accumulate partial sums for thousands of cycles, with reconstruction after the complete chain (Liguori, 2024, Johnson, 2018).
• Area and Energy Efficiency: FMA units built for low-precision (e.g., 8/38-bit log-float) can match or undercut integer MACs in energy and area (e.g., 0.96× power, 1.12× area of int8/32 MAC at 28 nm; float16 log-float at 0.59× power and 0.68× area of standard FMA) (Johnson, 2018). FPGA implementations show FEDP designs using up to 60% fewer LUTs/FFs than HardFloat baselines, with zero DSP blocks (Rout et al., 19 Nov 2025).
• Datapath Flexibility and Integration: Precision-scalable MACs for microscaling (MX) standards merge integer- and floating-point approaches (hybrid reduction tree, early-accumulation plus final normalization), supporting INT8, FP8, FP6, FP4 modes, and achieving up to 4,065 GOPS/W at 500 MHz in 8×8 tensor core arrays (Cuyckens et al., 9 Nov 2025).

6. Application-Specific FMAs and Algorithm-Hardware Codesign

FMAs enable novel algorithmic techniques and adaptive precision workflows:

• Compensated Algorithms: Compensation strategies for reciprocal square-root, reciprocal hypotenuse, and Givens rotations use multiple FMAs to calculate differential residuals (e.g., $1 - a y_0^2$) with $O(u^2)$ error, outperforming naive multi-step approaches (which accumulate $O(u)$ errors) (Borges, 2021).
• Iterative Refinement: Chained BF16 or FP16 FMA units perform low-precision factorizations and high-precision refinement on matrix solutions, with convergence and residuals matching double-precision solvers in key regimes (Henry et al., 2019).
• Hardware Algorithm Codesign: Error models (deterministic and probabilistic) motivate tailored block sizes, block-based reductions (GEMM, systolic arrays), adaptive precision scaling, and codesign methods that factor worst-case error vs. typical error concentrations (Bhola et al., 2024).

7. Comparative Summary of Key FMA Hardware Features

The table below summarizes representative datapath characteristics across several recent hardware and algorithmic design points:

Architecture	Input Formats	Accumulation	Pipeline Depth	Area/Resource Efficiency	Notable Innovations
Superscalar OOO FP	FP64	FP64	13 stages	45% area reduction (V2)	FMAC-only design, full IEEE-754
GPGPU FEDP Unit	FP8–FP16/BF16	FP32/INT32	4 stages	LUT/FF 40–60%↓ vs. HardFloat	Shared-CSAs, DSP-less, mixed-prec
BF16→FP32 FMA	BF16×BF16	FP32	2–3 pipeline	~9× smaller multiplier	Decomposition for FP32 accuracy
Log-float ELMA	Log8×Log8	38-bit Kulisch	4 stages	0.68× float16 FMA area	Log domain multiplication, exact add
MX Hybrid MAC	INT8/FP8/FP6/FP4	53-bit norm.	4 stages	1.13–3.21× energy gain vs. prior	Hybrid reduction, tunable mantissa
Tensor Core (NVIDIA)	FP16/FP8	FP32/FP16	~depth 4	2–4× IO/energy savings	Tiled systolic, block-level FMA

All figures and characteristics are quoted or paraphrased directly from the referenced works (Bhola et al., 2024, Rout et al., 19 Nov 2025, Henry et al., 2019, Cuyckens et al., 9 Nov 2025, Zuo et al., 2024, Lazo, 2021, Johnson, 2018, Liguori, 2024).

---

Fused multiply-add datapaths enable high-throughput, energy- and area-efficient realization of vectorized arithmetic workloads across a spectrum of application domains. Innovations in mixed-precision support, reduction tree synthesis, probabilistic error modeling, and hardware–algorithm codesign continue to shift architectural frontiers, making FMA both a practical building block and a central abstraction for contemporary compute accelerators.