BitParT: 1-Bit Transformer for Top Jet Tagging

• The paper presents BitParT, a 1-bit binary-weight Particle Transformer that achieves near-parity with full-precision ParT in top-quark jet tagging.
• It employs selective quantization by binarizing over 67% of its weights while retaining full-precision in critical self-attention modules to mitigate performance loss.
• Deployment implications include reduced memory footprint and power consumption, making it suitable for FPGA/ASIC-based trigger systems and large-scale offline pipelines.

BitParT is a 1-bit, binary-weight variant of the Particle Transformer designed for top-quark jet tagging in high-energy physics. It extends ultra-low-bit Transformer ideas associated with recent LLM quantization to a collider-physics setting in which latency, on-chip memory, bandwidth, and power are central constraints, particularly for FPGA- and ASIC-based trigger systems and for large-scale offline reconstruction pipelines. In the reported study, BitParT is constructed to reduce model size and computational cost while preserving most of the tagging performance of a full-precision custom ParT, thereby serving as a concrete example of extreme quantization for physics applications (Rai et al., 10 Aug 2025).

1. Scientific setting and motivation

Top tagging addresses the classification of boosted hadronic top-quark jets against generic quark- and gluon-initiated QCD jets. At the LHC, sufficiently energetic top quarks produce hadronic decay products that become highly collimated and are reconstructed as a single “fat” jet, making constituent-level discrimination essential. This task is important for standard-model measurements and for many beyond-the-standard-model searches (Rai et al., 10 Aug 2025).

Transformer architectures are well suited to this problem because they operate on jets represented as sets or sequences of reconstructed particles and can model all-to-all constituent interactions through self-attention. The Particle Transformer, or ParT, is identified as one of the strongest transformer-based jet taggers and provides the architectural starting point for BitParT. The motivation for quantization is not primarily algorithmic novelty in isolation, but deployment pressure: full-precision transformer models are demanding in memory bandwidth and compute, whereas collider applications often impose microsecond-to-nanosecond timing budgets, limited on-chip memory, and strict power envelopes (Rai et al., 10 Aug 2025).

Within that context, 1-bit quantization is introduced as a hardware-aligned compression strategy. The underlying argument is that replacing many floating-point multiplications by bitwise operations can lower latency and power, reduce memory footprint and bandwidth, and increase throughput. BitParT imports this logic from the BitNet line of ultra-low-bit Transformer work and tests whether such extreme quantization can be transferred to a physics classification task without catastrophic loss in physics performance. A plausible implication is that BitParT is positioned not merely as a compressed model, but as a deployment-oriented design exercise for real-time inference in collider environments.

2. Architectural lineage and selective quantization

BitParT is architecturally matched to a small custom ParT so that observed differences can be attributed to quantization rather than to changes in depth, width, or attention structure. Both models have approximately 1.23M trainable parameters and use the same number of transformer layers, heads, and embedding dimensions. The distinguishing modification is that the full-precision linear layers used in ParT are replaced, in selected locations, by 1-bit BitLinear layers in BitParT (Rai et al., 10 Aug 2025).

The overall backbone retains the two-stage ParT structure. First, a particle attention stage applies a stack of $L=3$ transformer encoder blocks to per-particle embeddings. Second, a CLS attention stage appends a learnable class token and applies two class-attention transformer layers, after which the final CLS representation is passed to a classifier head. No positional encodings are used, because jet constituents are treated as an unordered set.

BitParT does not quantize the entire network. The quantized components are the feed-forward networks within the transformer blocks and the dense classifier layers after the CLS token. By contrast, all multi-head self-attention computations, including the $Q/K/V$ projections, remain in full precision, as do the PairEmbed module, input MLPs, normalization parameters, bias parameters, and the CLS token itself. About 67% of model parameters are therefore binarized to 1 bit, with quantization concentrated in the most parameter-heavy submodules. This division is central to the design: physics-specific attention biases and the core attention mechanism are preserved in full precision, while the bulk FFN and classifier weights absorb the compression.

A recurrent misconception is to interpret BitParT as a fully binary Transformer. That description would be inaccurate. The implementation is explicitly a binary-weight architecture with selective quantization; the attention path and several auxiliary modules remain full precision.

3. Internal representation and BitLinear mechanics

The input representation follows the ParT formulation. Each jet is represented by a per-particle feature matrix

$X \in \mathbb{R}^{N \times d},$

with up to $N=200$ particles, together with a pairwise feature tensor

$U \in \mathbb{R}^{N \times N \times d'}.$

Per-particle features include kinematic quantities such as $\Delta \eta$, $\Delta \phi$, $\log p_T$, $\log E$, and $\Delta R$, along with PID information and track displacement parameters. Pairwise features are constructed from particle four-momenta through variables including $Q/K/V$0, $Q/K/V$1, $Q/K/V$2, and $Q/K/V$3, and the logarithms of these quantities are encoded by a shared 1D convolutional encoder to form the attention bias tensor (Rai et al., 10 Aug 2025).

Self-attention is modified by the physics-motivated pairwise bias:

$Q/K/V$4

This preserves the ParT mechanism in which pairwise physics information is injected directly into the attention scores.

The key quantized primitive is the BitLinear layer. For input $Q/K/V$5, BitLinear first performs per-row input centering and computes a scaling factor from the mean absolute value of the centered input. The weights are likewise mean-centered, binarized by a sign function, and associated with a weight scale and an optional learnable scaling factor. The reported output form is

$Q/K/V$6

This construction preserves average magnitude after binarization and is designed so that the binary matrix product can, in principle, be implemented through XNOR and popcount operations rather than floating-point MACs (Rai et al., 10 Aug 2025).

To stabilize optimization, BitLinear is embedded inside a BitLinLayer block comprising BitLinear, batch normalization, ReLU, dropout, a residual connection with learnable scaling, and a second batch normalization after residual merging. The quantized FFN in each transformer layer follows the pattern

$Q/K/V$7

with the FFN output scaled by a learnable factor before residual addition and normalization. Two analogous quantized CLS encoder layers are used in the class-attention stage, and the final classifier consists of one BitLinLayer followed by a final BitLinear mapping to the binary output classes.

4. Optimization procedure and benchmark definition

Training of the full-precision custom ParT and of BitParT is performed under identical conditions: AdamW, initial learning rate $Q/K/V$8, weight decay $Q/K/V$9, batch size $X \in \mathbb{R}^{N \times d},$0, and $X \in \mathbb{R}^{N \times d},$1 epochs of training. The learning-rate schedule is cosine annealing, decaying to $X \in \mathbb{R}^{N \times d},$2 of the initial learning rate over the last $X \in \mathbb{R}^{N \times d},$3 of epochs. Training uses quantization-aware training with a straight-through estimator for the non-differentiable sign function, so that during backpropagation the gradient is approximated as $X \in \mathbb{R}^{N \times d},$4 (Rai et al., 10 Aug 2025).

The stabilizing elements emphasized in the study are per-row centering and scaling of inputs, per-weight scaling and optional learnable scaling, residual blocks with learnable scaling, batch normalization inside BitLinLayer blocks, and cosine learning-rate decay. The paper explicitly notes that it does not introduce additional devices such as stochastic rounding or special initializations beyond this framework.

Evaluation is conducted on the public Top Quark Tagging Reference Dataset. The dataset contains $X \in \mathbb{R}^{N \times d},$5 simulated proton-proton events at $X \in \mathbb{R}^{N \times d},$6 TeV, generated with Pythia8 and passed through Delphes simulation using the ATLAS detector card. Jets are reconstructed with the anti-$X \in \mathbb{R}^{N \times d},$7 algorithm with $X \in \mathbb{R}^{N \times d},$8, and only the leading jet per event is kept. The kinematic selection requires $X \in \mathbb{R}^{N \times d},$9 GeV and $N=200$0. Signal jets are matched to parton-level tops with $N=200$1, and all top decay products must lie within the jet cone. Each jet is represented as an unordered set of up to $N=200$2 particle-flow objects, zero-padded when necessary and ordered by $N=200$3. The data split is $N=200$4 training, $N=200$5 validation, and $N=200$6 test.

The reported metrics are classification accuracy, area under the ROC curve, and background rejection at fixed signal efficiency. For a chosen signal efficiency $N=200$7, background rejection is defined as

$N=200$8

The working points reported are $N=200$9 and $U \in \mathbb{R}^{N \times N \times d'}.$0 signal efficiency.

5. Reported performance

The central empirical result is that BitParT preserves nearly all global discriminative power of the custom full-precision ParT while incurring a moderate loss in background rejection at fixed efficiency. The reported test-set comparison is as follows (Rai et al., 10 Aug 2025).

Metric	Custom ParT	BitParT
Accuracy	0.940	0.9399
AUC	0.9862	0.9856
Rejection at 50% signal efficiency	$U \in \mathbb{R}^{N \times N \times d'}.$1	$U \in \mathbb{R}^{N \times N \times d'}.$2
Rejection at 30% signal efficiency	$U \in \mathbb{R}^{N \times N \times d'}.$3	$U \in \mathbb{R}^{N \times N \times d'}.$4

The AUC loss is reported as less than $U \in \mathbb{R}^{N \times N \times d'}.$5 and less than $U \in \mathbb{R}^{N \times N \times d'}.$6 relative. The ROC curves are described as having very similar shapes, with BitParT slightly below ParT, mainly in the most background-rejecting regimes. At $U \in \mathbb{R}^{N \times N \times d'}.$7 signal efficiency, the reduction in background rejection is about $U \in \mathbb{R}^{N \times N \times d'}.$8. The same pattern persists at $U \in \mathbb{R}^{N \times N \times d'}.$9 signal efficiency, although the absolute rejection remains competitive.

The study also places BitParT against models reported elsewhere in the literature for the same benchmark. ParT-f.t. is listed with accuracy $\Delta \eta$0, AUC $\Delta \eta$1, $\Delta \eta$2, and $\Delta \eta$3, while ParT is listed with accuracy $\Delta \eta$4, AUC $\Delta \eta$5, $\Delta \eta$6, and $\Delta \eta$7. ParticleNet, PCT, LorentzNet, and related models are described as occupying similar AUC ranges around $\Delta \eta$8–$\Delta \eta$9, with rejection factors between approximately $\Delta \phi$0–$\Delta \phi$1 at $\Delta \phi$2 efficiency and approximately $\Delta \phi$3–$\Delta \phi$4 at $\Delta \phi$5 efficiency. Within that landscape, BitParT’s AUC of $\Delta \phi$6 is described as competitive with many state-of-the-art models, even though its rejection performance remains below the best fine-tuned ParT variant.

6. Deployment implications, limitations, and outlook

The paper’s deployment argument rests on parameter precision and arithmetic structure rather than on direct hardware measurements. Both models contain 1.23M parameters, but in BitParT more than $\Delta \phi$7 of the weights are binarized. For those layers, this implies roughly an order-of-magnitude or more reduction in model memory footprint, together with a corresponding reduction in memory-bandwidth requirements. Because BitLinear replaces real-valued matrix multiplications with binary operations, the design is aligned with hardware implementations based on XNOR and popcount, which are particularly attractive for FPGA and ASIC targets (Rai et al., 10 Aug 2025).

The scope of the evidence is nevertheless limited. No explicit latency, throughput, or energy benchmarks are reported for FPGA or ASIC deployment. Evaluation is confined to a single dataset and a single task, namely top tagging on the Top Quark Tagging Reference Dataset. The implementation focuses on binary weights rather than full weight-and-activation binarization, so it does not establish the feasibility of a fully binary network for the same physics task. The observed reduction in fixed-efficiency background rejection may also remain relevant for analyses operating in especially demanding rejection regimes.

These limitations define the stated future directions. The paper proposes hardware-aware implementation and co-design on FPGAs or ASICs; application of BitParT-style 1-bit Transformers to broader datasets such as JetClass; exploration of hybrid precision and structured pruning; and investigation of multi-bit quantization, activation quantization, and improved gradient estimators or regularization. Taken together, these directions suggest a broader research program in which selective quantization of parameter-heavy Transformer submodules is used to preserve physics-specific inductive biases while aggressively compressing the dominant memory and compute paths.

BitParT’s broader significance lies in demonstrating that a binary-weight Transformer can achieve near-parity in AUC and accuracy with a full-precision ParT on a demanding HEP benchmark while quantizing the majority of parameters. In that sense, it functions as a proof of concept for transferring ultra-low-bit Transformer techniques from LLM-oriented efficient inference research into collider data analysis, with explicit attention to real-time and resource-constrained deployment environments.