---
title: Fully Quantized Training (FQT)
url: https://www.emergentmind.com/topics/fully-quantized-training-fqt
type: topic
---

# Fully Quantized Training (FQT)

Fully quantized training (FQT) refers to end-to-end neural network optimization in which all primary tensors—weights, activations, and gradients—are represented and operated on in low-precision (integer or floating-point microformat) domains throughout every stage of forward and backward propagation, including model updates. Unlike quantization-aware training (QAT), which typically restricts quantization to inference and forward activations, pure FQT eliminates floating-point shadow state and aims to maximize memory, energy, and compute efficiency by leveraging low-bitwidth arithmetic for every dataflow operation. This paradigm is increasingly critical for deployment on hardware accelerators, edge devices, and large-scale distributed training regimes, including those used for modern transformers and large language models.

## 1. Statistical Foundations and Convergence Properties

The fundamental statistical principle underlying FQT is that quantized gradients—produced by symmetric, stochastic, low-bit quantizers—act as unbiased but higher-variance estimators of their full-precision counterparts. Given a parameter set $\theta$ and quantized gradients $\widetilde\nabla$, the expectation $\mathbb{E}[\widetilde\nabla] = \nabla$ if stochastic rounding is used independently across elements. The variance, however, is strictly larger:
\[
\operatorname{Var}[\widetilde\nabla] = \operatorname{Var}[\nabla] + \sum_{\ell=1}^L \mathbb{E}\bigg[ \operatorname{Var}(Q_b(\nabla_\ell)) \bigg] \cdot \sum_{k=1}^\ell \|J^{(k,\ell)}\|_2^2
\]
where $Q_b$ is the layer-wise quantizer, $B=2^b-1$, and $J^{(k,\ell)}$ encodes Jacobian sensitivity. For per-tensor quantization, as one lowers the bitwidth $b$ by 1, $B$ halves and variance increases fourfold, magnifying the efficiency/accuracy trade-off [2010.14298].

From a non-convex SGD convergence perspective, if the quantizer is unbiased and variance is finite, standard results yield
\[
\min_{t<T} \mathbb{E}[\|\nabla f(\theta_t)\|^2] = O(\sigma/\sqrt{T})
\]
with $\sigma^2$ bounding the increased variance. Unbiasedness is critical; bias in quantized gradients will induce systematic error and limit attainable optima [2010.14298].

## 2. Quantizer Designs and Variance Reduction

Early FQT relied on per-tensor uniform (linear) quantization, which introduces excess variance when data are heavy-tailed or certain samples or channels dominate. Two principled designs address this:

- **Per-Sample Quantizer (PSQ)**: Scales each sample (row) in the gradient matrix individually, significantly reducing variance when many rows are low-range. Variance is bounded by $D/(4B^2)\sum_i R(G_i)^2$, where $R(G_i)$ is the range in row $i$ [2010.14298].
- **Block-Householder Quantizer (BHQ)**: Uses blockwise Householder reflections to spread outlier contributions and further minimize blockwise variance, achieving $O(\lambda_1^2/N)$ scaling in the pathological one-row-dominant case. Complexity grows only modestly [2010.14298].

Recent frameworks generalize these approaches to group quantization with power-of-two shift-based scaling (ShiftQuant), enabling highly memory-local group-wise quantization and integer-only accumulation (ShiftMM), with variance arbitrarily close to the per-element lower bound for a moderate number of groups [2411.10948]. Unbiased stochastic rounding remains an essential ingredient for achieving these variance guarantees and convergence stability.

## 3. End-to-End Integer and Floating-Point Microformat Pipelines

FQT implementations fall into several system-level patterns:

- **Fixed-point and Integer Quantization**: All weights, activations, gradients, errors, and updates—plus batch normalization (“WAGEUBN” framework)—are stored and operated on in 8-bit integers [1909.02384]. Constant quantizers (bespoke for gradients), shift quantizers (for errors), and layered scaling are used to retain critical signal variance. In hardware, this allows full pipeline acceleration with ≤0.5–1.0% loss for image classification [1909.02384].

- **Blockwise Floating-Point Formats**: Modern “microformats” such as NVFP4, mixing E2M1 FP4 values in blocks with E4M3 per-block scales (block size 16), enable full FQT of LLMs and transformers, with all GEMMs and gradient updates performed in FP4 with scale factors [2505.19115][2510.27527]. Forward pass uses round-to-nearest for stability; backward and update use stochastic rounding to ensure unbiasedness. Empirical results show BF16-comparable accuracy for Llama2-7B on 200B tokens [2505.19115].

- **Group-Shared Exponent (GSE) Formats**: For on-device LLM fine-tuning (GSQ-Tuning), one partitions tensors into non-overlapping groups, sharing a single exponent per group and representing mantissas as integers, allowing plug-in with LoRA-style adapters [2502.12913]. Integer-only training with shared exponent-mantissa layouts yields up to 11× area and 5× power reduction versus FP8, with sub-0.3% accuracy loss for language benchmarks.

- **Pseudo-Quantization Training (PQT)**: Instead of hard rounding, PQT injects blockwise controlled stochastic noise (e.g., Gaussian rounded to small discrete values) into parameter updates and casts to low-precision FP. This eliminates optimizer inconsistency and allows dynamic adaptation of bitwidth, providing stability and efficiency at scale with ≈1.4% throughput penalty for LLMs [2505.11170].

## 4. Implementation Practices and Theoretical Limits

In end-to-end FQT, key practices include:

- **Quantizer placement**: All heavy data-flow tensors (matrix multiply operands, normalization statistics, update accumulators) are quantized. First and last layers can, in principle, remain quantized, although for 1–2 bit settings some overhead is unavoidable [2010.14298][1808.05240].

- **Dynamic or per-block scaling**: Per-block or per-group scale parameters are necessary to contain quantization noise under heavy-tailed distributions, particularly in transformers and LLMs [2505.19115][2403.12422]. Block sizes of 16–32 balance dynamic range with implementation overhead [2505.19115][2403.12422].

- **Rounding strategies**: Stochastic rounding is used for gradients and in backward/update passes for unbiasedness; round-to-nearest is typically reserved for forward activations and weights, where bias is less detrimental [2505.19115].

- **Gradient quantization and variance control**: To safely push bitwidth to the 1–4b regime, aggressive variance control is critical. Activation Gradient Pruning (AGP) selectively prunes low-range gradient groups and reallocates bitwidth, controlling total error. In practice, 1-bit FQT with SCQ (sample-channel joint quantization) and AGP can close the gap to direct PSQ quantization by 6 percentage points on typical fine-tuning tasks [2408.14267].

- **Theoretical limits**: There is a lower bound where the average per-coordinate gradient magnitude falls below $\sqrt{3}\,\sigma_q$ (quantization noise). Below that threshold, quantized training fails to yield useful descent, and training stalls [2505.19115].

## 5. Empirical Results Across Architectures

FQT achieves highly competitive accuracy and dramatic resource savings in diverse settings:

- **ImageNet ResNet-50**: PTQ (8b) for gradients suffers ~1% accuracy loss versus QAT; novel PSQ/BHQ recover QAT accuracy at 8b, remain within 0.5% of QAT with BHQ at 5b; PTQ fails below 7b [2010.14298].
- **Transformers for NMT and Language Modeling**: FullyQT (8b) yields BLEU matching or exceeding FP32 on WMT tasks; memory and compute reduced by 4× versus float [1910.10485]. Jetfire achieves 1.42× end-to-end transformer block speedup with 1.49× activation memory saving, matching FP16 accuracy with blockwise quantization [2403.12422].
- **Large Language Models**: NVFP4 (FP4, block size 16, E4M3 scale) enables full Llama2-7B pre-training in FP4 with BF16-matching performance, and QAF phase closes the last <1% gap [2505.19115]. TetraJet-v2, via NVFP4 + OsciReset + OutControl, closes over half the gap to full-precision across 200B tokens for LLMs up to 370M [2510.27527].
- **Tiny, Embedded, and On-Device Training**: FQT on Cortex-M MCUs, with 8-bit quantization on all tensors, supports on-device transfer learning matching float32 within 1–2%; dynamic partial gradient updates yield up to 6.6× speedup [2407.10734].
- **Sub-8-Bit Integer Training**: ShiftQuant (groupwise GEMM-friendly integer quantization) with quantized L1 normalization achieves $<0.9\%$ accuracy drop on 4-bit ResNets and 0.6% on 6-bit Transformers, plus $1.85\times$ throughput improvement vs. FP16 on ARM CPUs [2411.10948].

## 6. Best Practices and Open Challenges

Key recommendations emerging from recent FQT literature include:

- **Ensure quantized gradients are unbiased (use stochastic rounding wherever possible)** [2010.14298][2505.19115][2408.14267].
- **Monitor dynamic range and select the finest quantizer scale that maintains accuracy loss below 0.4–1%**; for most vision/translation tasks, PSQ at 6–8b or BHQ at 5–6b is robust [2010.14298].
- **Leverage per-block/group scaling and blockwise floating/microformats (e.g. FP4/E2M1 with E4M3 scale) in large transformer/LLM training** [2505.19115][2510.27527][2411.10948].
- **Deploy AGP or similar variance reduction for 1–4b settings, especially with adaptive (Adam) optimizers** [2408.14267].
- **Warm-start from a pre-trained full-precision teacher for extremely low bitwidths, and ensure that initial parameters are close in $L^\infty$ or $L^1$ norm to the quantized grid** [2012.05529][1808.05240].

Open areas include stable FQT-from-scratch for large LLMs at sub-4b, optimally choosing block/group partitions for microformat scaling, and efficient support for higher-order optimization (Adam, LAMB) in pure integer or microformat hardware. Better integration of dynamic quantization range tracking to minimize hardware and data-movement overhead (e.g., in-hindsight range estimation for fast static quantization [2105.04246]) remains essential for highly resource-constrained settings.

## 7. Limitations and Future Directions

While FQT now delivers near-FP32 accuracy for a wide range of architectures and tasks, certain accuracy limits exist for extremely low-bit (≤2b) training without variance-control and bias-mitigation. A continuing challenge is to develop universal, robust algorithms for full-from-scratch training at extreme quantization that guarantee stability in nonconvex regimes and across diverse data distributions. Further, adapting and customizing FQT for non-GEMM-dominated architectures, online continual learning, and privacy-preserving deployment on edge hardware remain compelling research frontiers [2310.03675][2407.10734][2502.12913].

Source: https://www.emergentmind.com/topics/fully-quantized-training-fqt