BitNet b1.58 Ternary Training

Updated 5 July 2026

The paper introduces BitNet b1.58 as a quantization-aware training framework that constrains linear-layer weights to a ternary set using per-tensor absmean normalization and round-clip operations.
It employs higher-precision latent weights with STE-based backpropagation, enabling native low-bit training with 1.58-bit weights and 8-bit activations for improved inference efficiency.
Extensions include median-based scaling, gradual 16-bit to 1.58-bit training curricula, activation quantization adaptations, and sparse training variants to enhance robustness and cross-architecture performance.

Searching arXiv for the relevant BitNet b1.58 and related ternary-training papers. BitNet b1.58 ternary training denotes a quantization-aware training regime in which the linear-layer weights of a LLM are constrained in the forward pass to a ternary alphabet, typically $\{-1,0,+1\}$ , while optimization proceeds through higher-precision latent or shadow weights using a straight-through estimator or related surrogate gradient mechanism. In the BitNet b1.58 family, the defining formulation is per-tensor absmean ternarization of weights together with low-bit activation quantization, most commonly 8-bit activations, yielding a native low-bit training paradigm rather than post-training compression (Ma et al., 2024). Subsequent work has broadened this paradigm in several directions: small-model studies and median-based quantization variants (Nielsen et al., 2024), continual 16-bit-to-1.58-bit pre-training curricula (Nielsen et al., 17 Feb 2025), open 2B-scale and 4T-token native pretraining reports (Ma et al., 16 Apr 2025), activation-side extensions to 4-bit regimes (Wang et al., 2024, Wang et al., 25 Apr 2025), joint training with semi-structured sparsity (Zhang et al., 5 Mar 2026), and downstream adaptation via distillation (Wu et al., 15 Oct 2025). The term is therefore best understood not as a single algorithmic line, but as a family of training methods centered on ternary BitLinear computation.

1. Formal definition and parameterization

BitNet b1.58 is defined by ternary weights taking values in $\{-1,0,1\}$ , with the “1.58-bit” label referring to the information content of a 3-state alphabet, $\log_2 3 \approx 1.58$ (Ma et al., 2024). In the canonical formulation, a latent full-precision weight matrix $W$ is mapped to a ternary forward-pass tensor $\widetilde{W}$ by absmean quantization: $\widetilde{W} = \mathrm{RoundClip}\!\left(\frac{W}{\gamma + \epsilon}, -1, 1\right), \qquad \gamma = \frac{1}{nm}\sum_{ij}|W_{ij}|,$ with

$\mathrm{RoundClip}(x,a,b)=\max(a,\min(b,\mathrm{round}(x))).$

This formulation is explicitly given in the original BitNet b1.58 paper and is repeated in later work that treats BitNet b1.58 as the reference ternary model class (Ma et al., 2024, Malhotra et al., 1 Jun 2025).

A scaled variant is common in later implementations and expositions. There the ternary code remains $\{-1,0,1\}$ , but the effective quantized weight used in computation is written as $\alpha \cdot \mathrm{RoundClip}(W/(\alpha+\epsilon),-1,1)$ , with $\alpha=\mathrm{mean}(|W|)$ , so the deployed values become $\{-1,0,1\}$ 0 (Wu et al., 15 Oct 2025, Wang et al., 25 Apr 2025). This suggests two equivalent viewpoints: a discrete ternary code space and a scaled ternary compute representation.

Activations are not ternary in BitNet b1.58. The baseline regime is W1.58A8: ternary weights and 8-bit activations (Ma et al., 2024). Several papers restate this either with per-token absmax activation quantization or equivalent 8-bit integer clipping formulations (Nielsen et al., 2024, Ma et al., 16 Apr 2025, Zhang et al., 5 Mar 2026). The original BitNet b1.58 paper further specifies symmetric per-token activation scaling to $\{-1,0,1\}$ 1 without zero-point quantization (Ma et al., 2024).

2. Canonical training mechanism

The central training pattern is quantization-aware training from scratch rather than post-training quantization. The original BitNet b1.58 work is explicit that the model is “trained from scratch, with 1.58-bit weights and 8-bit activations” (Ma et al., 2024). In practical terms, this means that training maintains latent real-valued or 16-bit shadow weights, quantizes them into ternary weights for the forward pass, and updates the latent parameters through surrogate gradients.

Later studies state this more explicitly. “BitNet b1.58 Reloaded” describes BitLinear as a drop-in replacement for torch.nn.Linear, with forward computation consisting of activation normalization, activation quantization, shadow-weight ternarization, quantized matrix multiplication, and output dequantization (Nielsen et al., 2024). The same paper also states that both quantized activations and quantized weights are detached from the computation graph and that a straight-through estimator is used so gradients update the 16-bit shadow weights (Nielsen et al., 2024). This is a more implementation-specific account than the original BitNet paper, which implies the presence of latent full-precision weights but does not spell out the backward rule in detail (Ma et al., 2024).

A concise abstraction of the standard BitNet b1.58 training loop is:

maintain latent higher-precision weights $\{-1,0,1\}$ 2;
compute a per-tensor scale, usually $\{-1,0,1\}$ 3;
quantize $\{-1,0,1\}$ 4 to ternary $\{-1,0,1\}$ 5 or scaled ternary values;
quantize activations to 8-bit;
run the forward pass with quantized operands;
backpropagate through quantizers using STE-like approximations;
update the latent weights with a standard optimizer.

This pattern recurs across multiple BitNet-family papers and related extensions (Nielsen et al., 2024, Ma et al., 16 Apr 2025, Wang et al., 25 Apr 2025).

The following table summarizes the canonical baseline.

Component	Baseline BitNet b1.58 formulation	Reported in
Weight codebook	$\{-1,0,1\}$ 6	(Ma et al., 2024)
Weight scale	Per-tensor absmean	(Ma et al., 2024)
Activations	8-bit	(Ma et al., 2024)
Training regime	Quantization-aware training from scratch	(Ma et al., 2024)
Optimization state	Latent/shadow higher-precision weights	(Nielsen et al., 2024)
Backward through quantizer	STE-based	(Nielsen et al., 2024)

A common misconception is that BitNet b1.58 trains directly on immutable ternary parameters. The literature does not support that for standard BitNet training. The dominant recipe keeps latent continuous weights and only uses ternary projections in the forward path (Nielsen et al., 2024, Ma et al., 16 Apr 2025). Direct low-precision weight updates without a master copy were later explored as an alternative research direction, not as the canonical BitNet method (Zhao et al., 2024).

3. Architectures, scaling behavior, and baseline empirical claims

BitNet b1.58 was introduced in the context of decoder-only Transformer LLMs and positioned as a native low-bit alternative to FP16/BF16 Transformers of the same size and token budget (Ma et al., 2024). In the original study, the key empirical pattern was scale dependence: the gap to full precision narrowed as parameter count increased, and parity or slight advantage emerged around 3B parameters. Reported perplexities on the main comparison were 12.33 vs 12.87 at 700M, 11.25 vs 11.29 at 1.3B, and 10.04 vs 9.91 at 3B for LLaMA FP16 vs BitNet b1.58 respectively, with average zero-shot accuracy likewise reaching parity around 3B (Ma et al., 2024). The same paper also reports a 2T-token 3B experiment in which BitNet b1.58 3B achieved 74.34 average score versus 73.22 for StableLM-3B on the listed tasks (Ma et al., 2024).

The later “BitNet b1.58 2B4T Technical Report” extends the native-pretraining narrative to a 2B-parameter model trained on 4T tokens and describes the result as the first open-source native 1-bit LLM at that scale (Ma et al., 16 Apr 2025). The report states that BitNet b1.58 2B4T uses forward-pass absmean ternary quantization for weights and per-token absmax INT8 activation quantization, together with bf16 master weights for training and packed 1.58-bit weights for inference (Ma et al., 16 Apr 2025). It also describes a three-stage lifecycle of pretraining, supervised fine-tuning, and direct preference optimization, though the exact optimizer-side mechanics remain only partially specified there (Ma et al., 16 Apr 2025).

Beyond decoder-only pretraining, “When are 1.58 bits enough?” broadens the empirical picture across model families. It reports that BitNet-style 1.58-bit training can be near full precision or sometimes better on MLPs, GNNs, encoder-only transformers, and encoder-decoder transformers (Nielsen et al., 2024). The most notable claim is architectural heterogeneity: encoder-only transformers appear to require roughly doubled hidden size to match 16-bit loss, whereas encoder-decoder T5-style models were reported to outperform their 16-bit counterparts in that study (Nielsen et al., 2024). This suggests that the BitNet b1.58 scaling story is architecture-dependent rather than universal.

A second misconception is that BitNet b1.58 is only viable for very large decoder-only models. The literature supports a more differentiated view. The original parity claim is strongest at multi-billion scale for decoder-only LLMs (Ma et al., 2024), but smaller-model studies show viability in other architectures and tasks, albeit often with capacity or hyperparameter caveats (Nielsen et al., 2024, Nielsen et al., 2024).

4. Variants of the ternarization rule and training stability

The most direct variant of the baseline ternarizer is the mean-versus-median scaling change introduced in “BitNet b1.58 Reloaded” (Nielsen et al., 2024). That paper replaces the absmean-derived scale with an alternative based on the median of $\{-1,0,1\}$ 7: $\{-1,0,1\}$ 8 The rest of the BitLinear machinery remains unchanged (Nielsen et al., 2024). The authors argue that median scaling is more robust to outliers and weight updates, but they do not claim universal superiority. Instead, their empirical conclusion is that mean versus median should be treated as a hyperparameter (Nielsen et al., 2024).

That same study is notable for showing that small language and vision models do not inherit all of the optimization behavior reported for larger BitNet LLMs. Very high learning rates that had been associated with large-scale BitNet training did not transfer cleanly; for small LLMs, $\{-1,0,1\}$ 9 could be unstable or size-dependent and $\log_2 3 \approx 1.58$ 0 was catastrophic for both 16-bit and 1.58-bit training (Nielsen et al., 2024). In vision tasks, smaller learning rates such as $\log_2 3 \approx 1.58$ 1 or $\log_2 3 \approx 1.58$ 2 were often best (Nielsen et al., 2024). The paper also reports that small LLMs often required roughly doubled hidden size to match equal-architecture 16-bit baselines, whereas small vision models could match or exceed 16-bit at equal size (Nielsen et al., 2024).

The open 2B4T report adds several high-level stabilization choices. It states that BitNet b1.58 2B4T incorporates SubLN normalization, uses $\log_2 3 \approx 1.58$ 3 in the feed-forward block, removes biases from linear and normalization layers, and benefits from relatively larger learning rates than comparable full-precision models (Ma et al., 16 Apr 2025). It also describes a two-stage pretraining schedule with high initial learning rate and weight decay followed by a lower-rate curated-data stage, where weight decay is set to zero (Ma et al., 16 Apr 2025). Because exact optimizer internals are not fully disclosed there, these points should be read as reported recipe elements rather than a complete reproducible specification.

The following table summarizes selected stability-related modifications.

Modification	Purpose	Reported in
Median-based scale	Robustness to outliers; alternate ternarization threshold	(Nielsen et al., 2024)
SubLN	Training stability in quantized regimes	(Ma et al., 16 Apr 2025)
$\log_2 3 \approx 1.58$ 4 in FFN	Sparsity and 1-bit-friendly computation	(Ma et al., 16 Apr 2025)
Two-stage LR/WD schedule	Better large-scale native pretraining	(Ma et al., 16 Apr 2025)

5. Training curricula and alternative optimization regimes

One major line of work asks whether BitNet b1.58 training must begin in ternary mode from the first step. “Continual Quantization-Aware Pre-Training” studies a curriculum in which the model is first trained in 16-bit and only later transitioned into 1.58-bit quantization-aware training (Nielsen et al., 17 Feb 2025). In that work, the authors use OLMo 1B with BitLinear substitutions and compare full 16-bit training, full 1.58-bit training, and 16-to-1.58-bit transitions at 2K, 4K, and 6K steps under a total budget of 10K steps (Nielsen et al., 17 Feb 2025). Their central finding is that the mixed curriculum outperforms full 1.58-bit-from-scratch training, with the best tested transition occurring after 2K steps, corresponding to about 20% of total training (Nielsen et al., 17 Feb 2025).

The same paper also studies two practical concerns: whether to retain optimizer state at the transition point, and whether to phase in quantization gradually rather than switching abruptly. Retaining AdamW state reduces the magnitude of the immediate loss spike but yields similar eventual loss after recovery; gradual soft quantization likewise smooths the transition but produces little lasting improvement in final loss or downstream task performance (Nielsen et al., 17 Feb 2025). This suggests that the main benefit comes from the early 16-bit phase itself rather than from a sophisticated transition mechanism.

A more radical departure is “Direct Quantized Training of LLMs with Stochastic Rounding,” which asks whether low-bit LLMs can be trained without keeping full-precision master weights at all (Zhao et al., 2024). That work proposes direct low-precision weight updates with stochastic rounding instead of STE. It reports that training with only low-precision weights is feasible even for ternary constraints, but direct 1.58-bit training underperforms reproduced BitNet b1.58 baselines. On WikiText-2, the reproduced BitNet b1.58 model achieved loss 5.52 and perplexity 45.83, while direct 1.58-bit training reached loss 6.20 and perplexity 73.41; 8-bit direct quantized training came closer at loss 5.80 and perplexity 55.75 (Zhao et al., 2024). This reinforces the current status of standard BitNet training: latent higher-precision weights remain important for best ternary quality.

These curriculum and optimization alternatives clarify an important terminological point. “BitNet b1.58 ternary training” usually refers to a latent-weight QAT regime, but the broader research area now includes 16-to-1.58-bit curricula (Nielsen et al., 17 Feb 2025) and direct low-bit updates via stochastic rounding (Zhao et al., 2024). These should be distinguished from the original canonical recipe.

6. Extensions beyond baseline W1.58A8 training

A substantial portion of the subsequent literature concerns extending or adapting BitNet b1.58 rather than redefining it.

Activation-side extensions

BitNet a4.8 introduces a hybrid activation design for 1-bit LLMs: 4-bit quantization for attention and feed-forward inputs together with sparsification and 8-bit quantization of outlier-heavy intermediate states (Wang et al., 2024). It retains BitNet b1.58’s ternary-weight formulation and STE-based latent-weight training, but modifies the activation path. The paper reports that BitNet a4.8 achieves performance comparable to BitNet b1.58 with the same 100B-token training cost and activates only 55% of parameters (Wang et al., 2024).

BitNet v2 pushes further toward native 4-bit activations by introducing H-BitLinear, which applies an online Hadamard transform before activation quantization in attention output and FFN down-projection paths (Wang et al., 25 Apr 2025). Weight ternarization remains the same per-tensor absmean rule: $\log_2 3 \approx 1.58$ 5 The paper reports that BitNet v2 trained from scratch with 8-bit activations matches BitNet b1.58 performance, and that native 4-bit activation continue-training produces only minimal degradation at 1.3B, 3B, and 7B scales (Wang et al., 25 Apr 2025).

Sparse training

Sparse-BitNet studies joint training of 1.58-bit quantization with dynamic $\log_2 3 \approx 1.58$ 6 semi-structured sparsity (Zhang et al., 5 Mar 2026). Its main novelty is not the ternary quantizer itself, which is inherited from prior BitNet work, but a stable joint recipe based on dense BF16 master weights, masks computed from latent weights, quantize-then-mask ordering, and a “dual STE” that allows gradients to flow through both quantization and dynamic masking (Zhang et al., 5 Mar 2026). The paper reports that BitNet tolerates structured sparsity better than full precision, with smaller perplexity degradation at equivalent $\log_2 3 \approx 1.58$ 7 sparsity and speedups up to 1.30X using a custom sparse tensor core (Zhang et al., 5 Mar 2026).

Q-Sparse combines BitNet b1.58 with top- $\log_2 3 \approx 1.58$ 8 activation sparsification (Wang et al., 2024). In the BitNet-compatible formulation: $\log_2 3 \approx 1.58$ 9 The paper uses STE through the top- $W$ 0 masking operator and claims that sparsely activated BitNet b1.58 models outperform dense BitNet baselines at the same inference compute budget. It also reports an inference-optimal sparsity ratio of 61.25% for 1.58-bit models (Wang et al., 2024).

Distillation and downstream adaptation

BitNet Distillation addresses a different regime: adaptation of pretrained dense LLMs into ternary BitNet-style students for downstream tasks rather than pretraining from scratch (Wu et al., 15 Oct 2025). The method combines SubLN, continual pretraining warm-up, and logits plus MiniLM-style attention distillation. Weight ternarization follows BitNet-style absmean quantization,

$W$ 1

with INT8 activations and STE in the quantized forward path (Wu et al., 15 Oct 2025). The paper reports that naive direct ternary finetuning performs poorly, but the full pipeline nearly recovers FP16 task performance across 0.6B to 4B models while enabling up to 10x memory savings and 2.65x faster CPU inference (Wu et al., 15 Oct 2025).

Autonomous and decentralized training

MAGNET presents a decentralized system that integrates BitNet b1.58 training into a broader autonomous pipeline. It uses BitLinear with STE, mean-based ternarization

$W$ 2

and activation quantization $W$ 3 to train a 618.3M LLaMA-compatible model called Genkidama (Kim et al., 26 Mar 2026). The most concrete contribution is a 10-phase hyperparameter sweep that improved validation loss by 16.7% and a validated export pathway to bitnet.cpp for CPU inference (Kim et al., 26 Mar 2026). This suggests that BitNet training has become sufficiently modular to serve as a component in automated search pipelines.

7. Deployment consequences, robustness, and PTQ competition

The training literature is closely linked to deployment considerations because BitNet’s motivation is fundamentally systems-oriented. The original BitNet b1.58 paper reports substantial inference-side gains. For the 3B model, memory drops from 7.89 GB to 2.22 GB and latency from 5.07 ms/token to 1.87 ms/token, corresponding to 3.55× less memory and 2.71× faster inference in the reported setup (Ma et al., 2024). The same paper also cites 71.4× arithmetic-operations energy savings on 7nm chips relative to FP16 LLaMA under the adopted energy model (Ma et al., 2024). These system gains are one reason BitNet is framed as a native training recipe rather than merely a compression method.

A hardware-oriented implication is that true ternary models are not optimally executed as generic 2-bit models. Platinum, a LUT-based ASIC accelerator paper, treats BitNet b1.58 as a representative ternary-weight LLM and shows that direct ternary LUT execution is better matched to $W$ 4 structure than bit-serial decomposition (Shan et al., 26 Nov 2025). For BitNet b1.58-3B, Platinum reports up to 73.6×, 4.09×, and 2.15× speedups over SpikingEyeriss, Prosperity, and a 16-thread T-MAC CPU baseline, with substantial energy reductions, all within a 0.96 mm $W$ 5 chip area (Shan et al., 26 Nov 2025). This is not a training result, but it clarifies why training a model natively into a ternary representation matters.

A separate deployment-oriented strand concerns fault tolerance on ternary compute-in-memory hardware. ReTern does not introduce a new BitNet training algorithm; instead it studies training-free fault-aware mapping for pretrained BitNet b1.58 700M and 3B models on ternary compute-in-memory accelerators (Malhotra et al., 1 Jun 2025). The paper reiterates that BitNet b1.58 models are obtained with quantization-aware training and notes that their relatively low zero-weight sparsity—37.05% for 700M and 37.55% for 3B—makes compute-in-memory fault mitigation harder than in more sparse ternary CNNs (Malhotra et al., 1 Jun 2025). This provides an indirect training-related insight: the learned ternary weight distribution of BitNet b1.58 affects hardware robustness.

The main challenge to the premise that high-quality ternary models require native QAT comes from post-training quantization. CAT-Q argues that the field has overcommitted to the necessity of BitNet-style expensive QAT and proposes a PTQ scheme based on learnable modulation and softened ternarization (Wang et al., 25 Jun 2026). The paper reports that, on five zero-shot commonsense reasoning tasks, Qwen3-based CAT-Q models can match or outperform BitNet v1/v2 families of similar size while using only about 1M calibration tokens rather than the 100B training tokens reported for BitNet v1/v2 comparisons (Wang et al., 25 Jun 2026). However, the comparison is scale-matched rather than architecture-matched, and the paper also admits severe degradation on harder math and coding tasks under plain CAT-Q (Wang et al., 25 Jun 2026). A plausible implication is that BitNet-style native ternary training remains most compelling when broad task robustness is required, whereas PTQ can be competitive in benchmark-specific or calibration-rich settings.

8. Controversies, misconceptions, and open directions

Several recurring misconceptions can be resolved directly from the literature.

One misconception is that BitNet b1.58 is a post-training quantization method. The foundational papers explicitly position it as native low-bit QAT from scratch (Ma et al., 2024). PTQ work such as CAT-Q is framed precisely as a challenge to that training-heavy assumption, which would make little sense if BitNet itself were already PTQ (Wang et al., 25 Jun 2026).

A second misconception is that “1-bit” in BitNet means binary weights. In the b1.58 family, the weights are ternary $W$ 6, not binary $W$ 7, and the 1.58-bit terminology is tied to the three-state alphabet (Ma et al., 2024, Nielsen et al., 2024).

A third misconception is that BitNet training eliminates high-precision state during training. Standard BitNet does not. The common recipe maintains higher-precision latent or shadow weights and applies quantization in the forward path (Nielsen et al., 2024, Ma et al., 16 Apr 2025). Direct low-precision updates remain an alternative research direction with quality limitations at the ternary level (Zhao et al., 2024).

Controversies and unresolved issues remain. The exact reason ternary training sometimes acts as a structural regularizer is still debated. Hybrid Gated Flow, for example, argues that a ternary backbone can stabilize otherwise unstable differential attention and recovers much of the quality gap to dense baselines with a small gated low-rank FP16 correction path, reporting validation loss 0.9306 versus BitNet’s 1.0294 and an FP16 baseline’s 0.8490 in its TinyStories proxy regime (Pizzo, 5 Feb 2026). This suggests that the discreteness of ternary training may impose useful constraints, but larger-scale confirmation remains preliminary.

Another open question is how far activation precision can be reduced without architecture-specific remedies. BitNet a4.8 and BitNet v2 both imply that activation outliers, not weight ternarization alone, become the main bottleneck once weight precision is fixed at 1.58 bits (Wang et al., 2024, Wang et al., 25 Apr 2025). The field’s response has been heterogeneous: hybrid sparsification, Hadamard transforms, and continue-training into A4 regimes rather than naïve from-scratch A4 everywhere.

Finally, reproducibility remains uneven. Some papers provide equations and high-level schedules but omit optimizer internals, initialization, layer-exemption policies, or exact implementation details (Ma et al., 2024, Ma et al., 16 Apr 2025). Others provide more explicit pseudocode or hyperparameters but operate in altered settings such as small models or joint sparsity (Nielsen et al., 2024, Zhang et al., 5 Mar 2026). This suggests that the mature understanding of BitNet b1.58 ternary training is now distributed across a family of papers rather than encapsulated in any single source.

9. Synthesis

BitNet b1.58 ternary training is best characterized as a native low-bit Transformer training framework in which linear weights are quantized in the forward pass to a ternary alphabet, usually via per-tensor absmean normalization and round-clip ternarization, while higher-precision latent weights are updated by STE-based quantization-aware optimization (Ma et al., 2024, Nielsen et al., 2024). The canonical baseline is W1.58A8, but the research area has expanded into median-based ternarization (Nielsen et al., 2024), mixed-precision pretraining curricula (Nielsen et al., 17 Feb 2025), direct low-bit updates without master weights (Zhao et al., 2024), activation-side A4 extensions (Wang et al., 2024, Wang et al., 25 Apr 2025), sparse structured training (Zhang et al., 5 Mar 2026, Wang et al., 2024), downstream distillation pipelines (Wu et al., 15 Oct 2025), and large-scale open technical reports (Ma et al., 16 Apr 2025).

The central empirical picture is stable. Native ternary training is viable and can match or exceed full precision in certain regimes, especially at larger decoder-only scale or in some architecture-specific settings (Ma et al., 2024, Nielsen et al., 2024). Yet the method is not monolithic: performance depends on scale, architecture, activation treatment, and optimization curriculum. The dominant training recipe still relies on higher-precision latent state, so the major efficiency benefits remain primarily inference-oriented. At the same time, the deployment literature increasingly shows that true ternary models unlock execution strategies that generic 2-bit or 4-bit approximations do not, from CPU-native bitnet.cpp serving to ternary LUT accelerators and compute-in-memory deployment (Shan et al., 26 Nov 2025, Kim et al., 26 Mar 2026, Malhotra et al., 1 Jun 2025).

Taken together, the literature supports a precise interpretation. BitNet b1.58 ternary training is neither merely weight compression nor merely a single ternary quantizer. It is a family of quantization-aware training regimes organized around ternary linear-weight computation, with ongoing work focused on improving optimization stability, activation quantization, sparse structure, adaptation pipelines, and deployment realism.