---
title: 'TetraJet-v2: Efficient 4-Bit LLM Training'
url: https://www.emergentmind.com/topics/tetrajet-v2
type: topic
---

# TetraJet-v2: Efficient 4-Bit LLM Training

TetraJet-v2 is an end-to-end fully-quantized training (FQT) method for large language models (LLMs) that achieves near-lossless accuracy using the NVFP4 4-bit floating-point format for activations, weights, and gradients across all linear layers. The method addresses key obstacles in low-precision training—specifically, weight oscillation and structural outliers—by introducing an unbiased double-block quantization scheme, an oscillation suppression mechanism (OsciReset), and an outlier retention strategy (OutControl) [2510.27527].

## 1. NVFP4 Numerical Format and Double-Block Quantization

The NVFP4 format encodes each value in four bits using the E2M1 floating-point scheme: 1 sign bit, 2 exponent bits, and 1 mantissa bit, with the set of representable values $\{0, \pm0.5, \pm1, \pm1.5, \pm2, \pm3, \pm4, \pm6\}$. Data are grouped into inner blocks of 16 elements, each associated with an E4M3 (8-bit) scaling factor that spans roughly $[-448, 448]$. Encoding operates as follows:

- Given an outer block of 128 elements, compute a global scale $S_{global} = \max_i |X_i| / (448 \cdot 6)$.
- For each inner sub-block $k$ ($16k$ to $16k+15$), compute $S_{block_k} = \max_{i \in [16k,\, 16k+15]} |X_i/S_{global}| / 6$.
- Each element $X_i$ is quantized as $P_i = \text{round}_{\text{FP4}}(X_i / (S_{global} \cdot S_{block_{\lfloor i/16 \rfloor}}))$.
- Decoding reconstructs $X_i \approx P_i \cdot S_{global} \cdot S_{block_{\lfloor i/16 \rfloor}}$.

Rounding strategy is deterministic round-to-nearest (RTN) in the forward path for activations and weights; for gradients in the backward pass, stochastic rounding is used, ensuring unbiasedness: $\mathbb{E}[\text{round}_S(x)]=x$.

## 2. Quantization Across Linear Layers

TetraJet-v2 applies unbiased double-block NVFP4 quantization to every operand in the three matrix multiplications (MMs) per Transformer linear layer:

- **Forward:** $Y = X W^{\top}$, both $X$ and $W$ deterministically quantized (denoted $Q_{D}$): $\hat{X} = Q_{D}^{(1)}(X)$, $\hat{W} = Q_{D}^{(2)}(W)$, $\hat{Y} = \hat{X} \cdot \hat{W}^{\top}$.
- **Backward w.r.t. $X$:** $\nabla_X L = \nabla_Y W$, using stochastically quantized gradients and weights ($Q_S$): $\widehat{\nabla_Y} = Q_S^{(3)}(\nabla_Y)$, $\widehat{W} = Q_S^{(4)}(\hat{W})$, $\nabla_X \approx \widehat{\nabla_Y} \cdot \widehat{W}$.
- **Backward w.r.t. $W$:** $\nabla_W L = X^{\top} \nabla_Y$, with $X$ and $\hat{Y}$ stochastically quantized: $\widehat{X} = Q_S^{(5)}(X)$, $\hat{Y}$ reused from the forward pass, $\nabla_W \approx \widehat{X}^{\top} \cdot Q_S^{(6)}(\hat{Y})$.

Stochastic quantization in the backward path guarantees that gradient estimates remain unbiased in expectation, ensuring stability of stochastic gradient descent.

## 3. Oscillation Detection and OsciReset

Weight oscillation arises when quantized weights “hop” excessively between bins despite negligible true weight movement, impairing convergence. TetraJet-v2 defines:

- $dist_M(w) = \sum_{t=1}^{T_0} |w^{(t)} - w^{(t-1)}|$ (master weight movement)
- $dist_Q(w) = \sum_{t=1}^{T_0} |Q(w^{(t)}) - Q(w^{(t-1)})|$ (quantized weight movement)
- $OsciRisk(w) = dist_Q(w) / dist_M(w)$

Whenever $OsciRisk(w) \geq \tau_{osci}$ (typically $\tau_{osci} \approx 8-16$), the master weight is reset to the nearest representable quantized value, eliminating accumulated drift. OsciReset operates periodically (e.g., every 200 steps, after accumulating statistics over ~50 steps), and only after learning rate decays below a threshold. This mechanism is activated after $30$–$50\%$ of total training steps.

## 4. Outlier Channel Handling Via OutControl

A critical insight in TetraJet-v2 is the presence of persistent structural outliers: 5–10% of activation channels exhibiting anomalously large magnitudes. OutControl statically identifies outlier channels by ranking the channelwise $L_2$ norms over a small calibration set to select the top $p\%$ as $C_{out}$.

- **Forward:** The activation matrix $A \in \mathbb{R}^{N \times C}$ is split: $[A_{non}, A_{out}]$. Non-outlier activations are quantized and matrix-multiplied, while outliers are kept at higher precision (FP8/BF16):

  $Y = Q_D(A_{non}) \cdot Q_D(W^{\top}) + A_{out} \cdot Q_D(W^{\top})$

- **Backward:** The same split is applied to $\partial L/\partial X$ and $\partial L/\partial W$; non-outlier regions are stochastically quantized, outliers retained at high precision. Optionally, a Random Hadamard Transform can be applied to further reduce group-level outlier variance.

OutControl preserves the accuracy of critical channels at a negligible cost due to their small proportion.

## 5. Empirical Evaluation

TetraJet-v2 was evaluated on the OLMo-2 LLM family with 70M, 150M, and 370M non-embedding parameters, using the OLMo-2-Mix-1124 dataset (C4 and The Pile), up to 212B tokens (batch size 1024, sequence length 4096). AdamW optimizer and cosine decay with linear LR warm-up were employed. All activations, weights, and gradients in every linear layer are quantized to NVFP4.

Baselines included Quartet (MXFP4) and the NVIDIA NVFP4 recipe (which retains partial BF16 precision).

| Method            | Train PPL (70M/150M/370M) | Val PPL (70M/150M/370M) |
|-------------------|--------------------------|-------------------------|
| BF16 (full prec)  | 35.95 / 26.38 / 18.70    | 45.27 / 33.49 / 23.70   |
| Quartet (MXFP4)   | 40.77 / 29.25 / 20.76    | 51.23 / 36.89 / 26.16   |
| NVIDIA recipe     | 40.50 / 29.18 / 20.75    | 50.94 / 36.73 / 26.20   |
| TetraJet-v2-base  | 39.26 / 28.39 / 20.23    | 49.33 / 35.88 / 25.50   |
| TetraJet-v2-full  | 38.08 / 27.58 / 19.89    | 47.75 / 34.95 / 25.11   |

On downstream zero-shot tasks (370M, 200B tokens), TetraJet-v2 yields average accuracy 43.60, WikiText-103 PPL 18.06, and Pile PPL 12.81, closing 51.3% of the FP4→BF16 performance gap relative to prior state of the art.

## 6. Best Practices for Implementation

- Quantize all three matrix multiplications in each linear layer, using $1 \times 16$ row and $16 \times 1$ column grouping to match NVIDIA hardware optimally.
- Apply deterministic rounding in the forward path (activations, weights); use stochastic rounding for all gradients in backward passes.
- Limit the application of the Random Hadamard Transform to backward matrix multiplies ($\nabla_{X}$ and $\nabla_{W}$) to reduce group-level outlier effects.
- Deploy OsciReset after substantial learning rate decay, accumulating oscillation statistics every $\sim$50 steps and resetting every $\sim$200 steps, thresholded at $\tau_{osci} \approx 8$–$16$.
- Predetermine the outlier channel set (5–10%) at initialization, retaining only these channels in FP8/BF16 for both forward and backward computation.
- Integrate TetraJet-v2 as drop-in wrappers for standard linear modules in PyTorch or TensorFlow.

## 7. Synthesis and Significance

TetraJet-v2 integrates unbiased double-block NVFP4 quantization, weight oscillation suppression, and selective high-precision outlier retention to enable nearly lossless 4-bit pretraining for LLMs across all linear layers. This approach halves the performance gap between full-precision and prior FP4 methods, establishing a new baseline for scalable, efficient, low-precision language model training [2510.27527].

Source: https://www.emergentmind.com/topics/tetrajet-v2