Bits-per-Byte (BPB) Metric

• BPB is a normalized metric that quantifies information density per byte across various digital systems.
• It is used to assess compression efficiency, execution trace redundancy, and encoding performance in applications from sensor readouts to deep network quantization.
• BPB provides actionable insights for optimizing protocols and system designs by benchmarking against theoretical entropy limits and practical encoding methods.

Bits-per-Byte (BPB) is a central metric in information theory and digital systems for quantifying the information density, compression efficiency, and fidelity of data representations across a range of domains—including program execution analysis, detector readout, neural network quantization, large-scale LLMing, coding theory, and high-sensitivity communications. While its interpretation varies by context (as entropy-rate, normalized compression ratio, encoding efficiency, or information per symbol/byte), BPB universally provides a normalized, unitless measure describing how many useful information bits are encoded or transmitted per nominal byte or byte-equivalent unit.

1. Definitions, Theoretical Foundations, and Interpretations

BPB is strongly rooted in information theory, with its fundamental expression typically derived from entropy-rate or compression limits. In program analysis, for input or execution traces $\{x_i\}$ of length $n$, BPB is instantiated as

$$\lambda_{\text{exe}} = \lim_{n \to \infty} \frac{\log S_{\text{exe}}(n)}{n}$$

where $S_{\text{exe}}(n)$ is the number of possible execution traces of length $n$ (Cui et al., 2013). This rate quantifies the average number of bits required to represent each symbol (instruction or byte) produced by the "source" (i.e., the running program).

Similarly, in binary detector readout, the minimal number of bits required (from an entropy perspective) is

$H = \log_2 \left( \binom{n}{k} \right)$

representing the configuration entropy for $k$ events over $n$ possible channels (Garcia-Sciveres et al., 2013). The actual BPB for a given encoding is then $H$ divided by the actual bits used.

In probabilistic modeling (e.g., LLMs, density estimation), BPB corresponds to the cross-entropy or negative log-likelihood per byte. In compression, it measures the actual number of output bits after a lossless encoding, normalized by the number of input bytes (Yu et al., 2023, Egli et al., 20 Feb 2025).

The table below summarizes core BPB-related formulas across different contexts:

Context	Core Formula for BPB or Equivalent	Reference
Execution trace entropy	$$\lambda_{\mathrm{exe}} = \lim_{n \to \infty} \frac{\log S_{\mathrm{exe}}(n)}{n}$$	(Cui et al., 2013)
Binary detector readout	$\varepsilon_0 = H/B$	(Garcia-Sciveres et al., 2013)
Probabilistic compression	BPB = $$-\frac{1}{N}\displaystyle\sum_{i=1}^N \log_2 p(x_i | x_{<i})$$	(Yu et al., 2023)
Photon information efficiency	PIE = $C/F_s$ (bits per photon)	(Dacha et al., 23 Jan 2025)

BPB is thus a broad and general abstraction for the "information density" per encoded or decoded unit, normalized with respect to a byte or related unit.

2. BPB in Program Execution Analysis and Software Testing

Viewing a program as a signal generator, BPB—implemented as the information rate per instruction or per byte—captures the local and global information density of execution traces (Cui et al., 2013). In practice, instantaneously measuring BPB involves:

• Recording assembly-level execution traces for a run.
• Compressing the trace with a universal algorithm (typically Lempel–Ziv).
• Computing instantaneous BPB as $x_r(i) = \dfrac{\text{bits to compress block$B_i$}}{\text{bytes (or instructions) in$B_i$}}$ for segments $B_i$.

The resulting BPB signal $x_r(i)$ may be analyzed for coverage or redundancy in software testing. Further, applying a discrete Fourier transform (DFT) to the mean-removed signal yields a BPB spectrum $|X(k)|$ that reveals the periodic structure or "information-rich bursts" in the execution trace.

For finite state transition systems—used to model programs formally—the information rate for the system or subcomponents is

$$\lambda_{\mathcal{M}} = \lim_{n \to \infty} \frac{\log S_\mathcal{M}(n)}{n}$$

with $S_\mathcal{M}(n)$ the number of execution paths of length $n$. Algorithms (e.g., iterative edge-deletion with information-rate checks) can be used to extract "information-rich components" (IRCs) with BPB rates above a threshold $\theta\,\lambda_{\mathcal{M}}$ within the control-flow graph.

This methodology enables precise test coverage metrics and prioritization based on information density, rather than only syntactic or structural program features.

3. BPB as an Efficiency Measure in Detector Readout and Digital Encoding

In digital sensor systems—strip or pixel detectors—BPB operationalizes the encoding efficiency of lossless readout (Garcia-Sciveres et al., 2013, Garcia-Sciveres et al., 2015). For sparse binary patterns, efficiency is defined as

$\epsilon_0 = H/B$

where $H$ is pattern entropy (the minimum necessary bits) and $B$ is the actual encoding length. Realistic efficiency measures further consider engineering overhead (DC-balance, framing bits):

$\epsilon_1 = \frac{H + E(H)}{B_1}$

with $E(H) \approx \frac{1}{2}(\log_2(\pi H) - 1)$ the overhead cost. The calculations extend to include context, cluster, or error-correcting bits, giving efficiency at various levels $\epsilon_2$.

Methods such as Pattern Overlay Compression (POC), which combines multiple low-occupancy patterns to approach optimal entropy conditions (near 1 bit per bit), directly improve BPB efficiency—lowering the number of bits per byte required for reliable transmission.

Pixel detector readout systems further decompose the entropy (and hence BPB) into subcomponents: address, cluster shape, total charge, and charge fractions. For each, entropy is directly estimated, and the sum gives the theoretical lower bound for BPB:

$H_{\text{hits}} = H_A + H_s + (H_{QT} + H_{QF})$

Observed implementations (e.g., the FE-I4 chip in the ATLAS experiment) typically use 35–37 bits per cluster while the entropy limit is $\approx$24.5 bits, indicating significant BPB inefficiency and thus room for improved encoding methods (Garcia-Sciveres et al., 2015).

4. BPB in Model Compression, Hashing, and Deep Representations

BPB directly impacts the storage and computational trade-offs in hashing, neural network quantization, and learned embedding systems.

For example, in vectorized integer compression (e.g., VByte), BPB quantifies the average bit cost per stored integer—ranging from 8–16 bits per integer depending on data distribution (Plaisance et al., 2015). Advances such as Masked VByte enable SIMD-based decoding that achieves higher decompression speed without sacrificing BPB, thereby decoupling storage efficiency from computational cost.

In hashing-based models (Pb-Hash (Li et al., 2023)), BPB efficiency is improved by partitioning a single $B$-bit hash into $m$ $b$-bit chunks and reusing them. This reduces model size from $2^B$ to $m \times 2^b$ while incurring only a controlled accuracy loss, as quantified by a variance multiplier

$$R_{m,b} = \frac{\operatorname{Var}(\hat{J}_m)}{J(1 - J)}$$

For deep learning, quantized models (e.g., Binary Neural Networks and their ensembles (Zhu et al., 2018)) leverage BPB by encoding parameters/activations as single bits. Ensemble techniques (BENN) convert "more bits per network" into "more networks per bit," regaining representational and predictive power while maintaining the low BPB that underpins hardware efficiency.

5. BPB in Tokenization, Byte-Level Modeling, and Compression Algorithms

BPB is central in byte-level and bit-level tokenization and compression frameworks for LLMing, multimodal modeling, and other sequential tasks. The theoretical underpinning in tokenization is formalized in the paper of Byte-Pair Encoding (BPE) (Kozma et al., 13 Nov 2024), where BPE approximates the optimal compression utility (and thus BPB) to within a constant factor ($0.333 \leq \rho \leq 0.625$)—even though finding the optimal pair encoding is APX-complete. This explains BPE's empirical success in reducing BPB for LLMing.

Recent advances extend BPE below the byte boundary, leveraging common prefix structures and frequency distributions in UTF-8 encoded data to losslessly re-encode sequences (Moon et al., 9 Jun 2025). For highly diverse languages (CJK, emoji-rich text), this bit-level fallback can reduce sequence length and improve BPB efficiency by collapsing redundant prefixes and using variable-length bit tokens.

Hierarchical architectures (e.g., MegaByte (Yu et al., 2023), Multiscale Byte LLMs (Egli et al., 20 Feb 2025)) patchify byte streams, enabling sub-quadratic attention and large context window modeling. BPB serves as the main model quality criterion: models with lower BPB demonstrate stronger predictive power, better compression, and increased cross-domain flexibility, such as joint modeling of text and serialized image bytes.

The table summarizes BPB-influencing design strategies in tokenization:

Technique	BPB-Impacting Mechanism	Representative Reference
Standard BPE	Greedy pair merges; approx. 0.33–0.63 optimal	(Kozma et al., 13 Nov 2024)
Bit-level BPE	Prefix sharing, lossless deduplication	(Moon et al., 9 Jun 2025)
Hierarchical byte-level patchifying	Patch decomposition, reduces effective BPB	(Yu et al., 2023, Egli et al., 20 Feb 2025)
Inference-time byte sampling	Precise conditioning, prompt alignment	(Hayase et al., 17 Jun 2025)

6. BPB as a Metric in Communication Systems and DNA Data Storage

In photon-starved communication systems (Dacha et al., 23 Jan 2025), BPB (equivalently photon information efficiency, PIE) quantifies the maximum information rate per received quantum:

$\mathrm{PIE} = \frac{C}{F_s}$

where $C$ is the channel capacity and $F_s$ is the photon arrival rate. The experiment achieved 14.5 bits per incident photon, a record in the optical domain, corresponding to extremely low energy per bit (only 0.069 photons per bit at 1550 nm). This implicates BPB/PIE as a central performance figure for long-range, energy-efficient communications, and for applications where each symbol (e.g., photon) is highly valuable.

In DNA data storage, encoding density is the analog of BPB: the number of useful bits per nucleotide (bits/nt) (Li et al., 2021). Hybrid variable-length and pattern-aware encoding systems (e.g., DP-DNA) adaptively select encoding schemes based on actual data patterns, achieving up to 1.98 bits/nt in payload—very close to the physical upper bound of 2 bits/nt. By integrating a variable-length regime and dynamic selection (via DPAC), these methods nearly double previous benchmarks for storage density, directly translating to superior BPB efficiency for DNA archival systems.

7. Future Directions and Open Questions

BPB will remain fundamental for both designing and evaluating information systems as data volumes, hardware diversity, and cross-modality continue to scale. Key future directions include:

• Tightening the theoretical gap between greedy and optimal compression/tokenization strategies for the lowest achievable BPB, especially in fixed-alphabet or resource-constrained regimes (Kozma et al., 13 Nov 2024).
• Extending bit- and byte-level modeling to omnimodal foundations, where low BPB across heterogeneous inputs (text, image, audio bytes) becomes a defining criterion for unified architectures (Egli et al., 20 Feb 2025).
• Advancing techniques for identifying and leveraging information-rich components in complex processes, to minimize BPB via targeted compression or adaptive modeling (Cui et al., 2013).
• Integrating BPB efficiency arguments into hashing, quantization, and embedding methods, balancing statistical accuracy against memory and computation (Li et al., 2023, Zhu et al., 2018).
• Exploiting BPB-aware design in communication and storage—engineering for physical limits (e.g., single-photon, DNA-nt regimes), and pursuing codes and protocols that push BPB toward the entropy bound (Dacha et al., 23 Jan 2025, Li et al., 2021).

A common misconception is that BPB minimization always correlates with accuracy or model quality. In practice, excessively aggressive compression can degrade interpretability or robustness, and the best efficiency/accuracy trade-off is highly application-dependent. Furthermore, the complexity of achieving BPB optimality often requires hybrid or algorithmically sophisticated approaches, especially as alphabet cardinality, context, and distributional complexity increase.

BPB remains a unifying lens for understanding, engineering, and benchmarking information density, whether in hardware, software, or biological substrates.